That's a cop-out, though. Yes, there is always an element of trust in whatever you do. That's unavoidable, though it's smart to minimize the amount of trust you must put in others. Taken to the extreme it's ludicrous, as you've pointed out. But, that doesn't mean that there's no merit in limiting the amount of trust you put in third parties. Just because you can't completely trust your OS or compiler, doesn't mean that you should throw the entire concept of limiting trust out the window. It's dishonest to suggest that the risk is the same between trusting (your compiler), (your compiler + your OS), and (your compiler + your OS + the CA system).
The CA system is truly an honor system by design. It requires you to put your complete trust in a large, and growing, list of opaque and unfamiliar third parties and the decision to trust them is made by others though an opaque and unaccountable process. It's putatively a "security system", but is insecure by design. It depends entirely on unaccountable, secretive, and self-selected "authorities" to determine who should trust who.
Look at your OS's list of trusted CAs sometime. Any of these organizations, or anyone delegated by any single one of them, are implicitly trusted by your system. Completely trusting Microsoft, Apple, or various Linux devs is naive, but completely trusting everyone in the root CA list is absolutely insane!