If you're worried about compromised CPUs being used to compile executables that are used by others, then reproduceable builds are a great countermeasure. Just use reproduceable builds on many different CPUs, and compare them to ensure they are the same (for a given version of source and tools). The more variations, the less likely that there is a subversion. If what you're compiling is itself a compiler, then use diverse double-compiling (DDC) on many CPUs.
If you're worried that an INDIVIDUAL may end up with a compromised CPU, then yes, it's much harder to counter attack. On some systems, you can isolate the system (no network traffic, etc.). That said, an adversary has to send packets to subvert a specific system, then every time they do the subversion they risk being detected, so it's far less likely to be used for bulk surveillance... it would more likely be one well-resourced organization (e.g., a government) working against another well-resourced organization.