An infrastructure funded by the entire community, both those that sign up for high-speed Internet and those that don't. That sounds wonderful - free infrastructure for the providers, funded by taxpayers...
Who said that the taxpayers funded the network? It was funded by those that chose to connect. If not enough people connect, then no fibre for the area. In my case it was the local district heating company that also lay fibre when they dug up the streets anyway. Cheap and effective.
But, sure, we have (had) people who aren't "interested" in high-speed (aka "only speed") access. Like I'm sure was the case with electrification and city water and sewage treatment. They've mostly come crawling back with their tail between their legs when/if they get the chance to get connected. So I have no fundamental problem with taxpayers footing some of the bill. Much like I don't have a problem when I have to pay for railways I don't personally use, roads I don't travel on, schools I (no longer) send my kids to, etc. etc.
One reason we Swedes manage to stay competitive is undoubtedly our infrastructure, which internet connectivity is a recent part of.
Fun fact: The buildout of fibre connectivity is faster in rural areas, where many communities band together in co-ops to get fibre as it's seen as one way to get young people to stay, instead of moving to the city. So they're fighting hard not only for schools, and groceries these days, but mostly for fibre.