A lot of different topics, I'll try my best to address most of them at least.
First, most countries in Europe are actually rather small. Only 8 of the 53 have more than 20 mil people, 14 have more than 10. But even in some of the big countries, like Germany, Spain, Italy, France, you have a rather large amount of parties and they are by far not so "ancient" that you could say they grew out of a time when there were no firmly entrenched parties. Europe has a long history of political plurality, and voters tend to shop around.
Israel is a very special case indeed, a showcase of how a small party can have an impact and also one how a small party can have an impact that it's by no means entitled to. That's the threat when you have a three party system, where you have two large parties that can simply not agree on anything and a tiny one that could cooperate with either of the big ones and that will "sell" its handful of votes to whatever side offers them the biggest share of the cake.
But it's by no means hard to get an idea to national level, provided you're not the only one who has that idea. A good example is the Green movement of the 1980s. You have Green parties in pretty much every state of Europe, and independent of the size of the country they are all fairly successful, of course to varying degree. Germany, where the Greens actually made it into the national government a while ago, is a counter example to your claim that such new parties can only succeed in small countries because you cannot rally enough people to your "cause".
The problem of the US, and why it is so hard to form a new party, is not so much the size or the organization structure required to get it off the ground. The problem is a psychological one that stems from the first-past-the-post system. You will notice a similar psychological barrier to voting for a certain party in Europe, too, if people don't think that the party will make it past the election threshold. In most countries, you have to get at least 3-6% (it differs from country to country) of the votes to actually win a seat in the parliament. The goal is to avoid what happened in Italy where you suddenly had a few dozen parties sitting in the parliament which made it near impossible to govern the whole shit. It also means, though, that people do here the same they do in the US, just at a smaller scale. There are usually some parties with rather interesting concepts and ideas in many EU countries that would have a following but that following does not trust them to get in the parliament and hence their vote being "lost". So instead they give their vote to a party that does not match their preferences so well but at least has a chance to make it in.
It's the same with the US. To take up your example, if you're one of the group you mentioned, it would make a lot of sense to vote for the party you propose, but as long as people don't think that this party would gain majority they will rather fall back onto D or R, depending on what they're rather leaning towards.