Even if we assume that for every active terrorist there is 100 people supporting them (a high estimate, but not outside the realm of possibility at all), we're still talking about only hundreds of people.
Well, official estimates say that some 1,200+ people have left France to join jihadist groups in Iraq and Syria so I'd say your estimates are quite low in how many support the terrorists ideals, 1,200 people have been so enraptured by the same ideals that they have left their homeland to take up arms in a distant country, how many more must there be that support them but haven't been so moved yet as to actually take physical action? I think the problem is much, much larger than your musing imply, and it's something that needs to be talked about and dealt with. Is every muslim in the west a terrorist? No, of course not. Have the majority of terrorist attacks against the west in the last 2 decades been perpetrated by extremist muslims, yes.