Maybe, just maybe, the fact that there are 50 million unemployed Americans has nothing at all to do with foreign workers. There are a maximum of 65,000 H-1B visas issued per year, even if those all went away and every one of those jobs was filled by one of the 50 million unemployed you'd have a 0.13% reduction in the number of unemployed.
MAYBE the issue is more complicated than "the dirty foreigners are taking all the work!". MAYBE the issue is the geographic location of the unemployed being very different from the geographic location of any available jobs. MAYBE the 50 million unemployed don't have the education, skills, or experience needed to perform any of the jobs those H-1B visa holders are performing... or any of the other available jobs.
But reality never gets in the way of a good freakout, does it? It's so much easier to just blame the foreigner, and that's such a popular option in this country. You've got to love deeply ingrained xenophobia.