Okay, so you're not next to the switch, you're on a bridge over the track. You still have no way to warn the people on the track. But there's a very fat man standing on the bridge next to you, and if you pushed him off to his death on the track below, it'd stop the trolley. Do you do it?
Most people say "no", and even most of those who say yes seem to struggle with it.
The reason people struggle with it is because the scenario doesn't make a ton of sense. Everyone has seen videos of trains smashing cars like the car isn't even there, it's hard to believe that a fat guy would be heavy enough to stop the train. What if I push him, the train hits him and then continues on to hit the people? And if the fat guy is heavy enough to stop the train, doesn't that mean he's going to be too fat for me to push? I'm a skinny guy, physics wouldn't be on my side here. What if I try to push him, fail, and then he turns around and wonders why the hell I was trying to push him into a train? And even if it works, people might think I pushed the guy because he was fat, rather than to save the people.
All of that compared to me flipping a switch to save 4 people at the cost of 1. If I flip the switch the outcome is pretty set.
It might be better to pose the fat guy variant of question as "there's some dynamite near the track and someone is sitting there eating their lunch on it, do you blow it up to stop the train, killing the fat guy in the process?"