But isn't this the same argument people make against illegal immigration?
It is the same argument and there is nothing inherently wrong with it. On both ends, employers intentionally use immigrants status as leverage to drive wages down.
An American citizen with more options would demand at least $20 an hour to pick grapes in the 105 degree heat in August.
You will find that similarly tough jobs (oil field service comes to mind) that are worked exclusively by citizens pay significantly more than Ag jobs do.