I guess drawing the line in instances like this is difficult, but here is my thoughts:
* If you are passing by a car accident or someone bleeding out in the street, action is required (often by law - depending on the country / state).
* If you're parking your car in a downhill street, you're required to engage the handbrake/emergency brake before leaving the car - action required of you to avoid hurting other people.
As the chain of actions (or people) between the (missing) action and the outcome grows, our perceived responsibility is diminished. The same is true if the distance (or time) from us to where (when) the impact is felt increases. As you pointed out, we need some kind of filter - or the consequences of all our inactions would crush our sanity.