caveat: I live in Canada, your mileage may vary.
When I see a person panhandling on the street, I know full well that there are social assistance programs, welfare, employment insurance, homeless shelters, retraining programs, health care programs, etc. In many cases, they have trained professionals who will probably do a much better job than me of helping people that need help. In a sense that means that *I have already helped them* by supporting a society and government that includes these programs, and by funding them with my tax dollars. In many cases the problem is just getting people into programs that will work for them.
That said, I still help people out sometimes if I think it makes sense. Most of the time though my charitable giving goes outside the country to places with less of a social safety net.