One doesn't have to imagine - there have been attempts to estimate this. Of course, the incarcerated would not all be unemployed if they were released, but it's likely that a higher proportion of them would be than of the general population.

I can't find any details right now, but I seem to remember that with normal incarceration levels, unemployment figures would be about 2% higher than they currently are, while employment figures would drop a bit (I think by less than 1%). Does anybody have reliable data for either of these figures?