This idea that the government needs to be interested in you, personally, for surveillance to matter, is rooted in an outdated understanding of surveillance.
They no longer need to send someone to your house to tap a phone line. They no longer need to allocate man-hours and a vehicle to have someone follow you. It's no longer a manual process for someone to listen to your recorded conversations, summarize them, and transport that information to someone who can make a decision negatively impacting you. Even that "someone making a decision negatively impacting you" at the end of the chain is gone - it can be entirely automated.
The tools available to them now mean they don't have to take a personal interest in you.
The risk I'm concerned about is something like the Chinese social credit system.