It's kind of the same issue with open source software, as far as the "most people don't care" aspect, but at an even greater disadvantage that open source software. I don't have a chip fab (at least I could compile open source software), and so even if I were capable of understanding the chip design, there's not much of a guarantee that the physical chip I purchase doesn't have some proprietary back door built into it.
Like most people, I'm even lazy about the open source software I use. While I try to download from trusted sources, there's no guarantee that what I actually install matches the current stable version in the repo. I'm taking a leap of faith.
In both cases (including the former where I indicated my ignorare about chip design), presumably I am counting on other experts to understand the chip or understand the source code for me, but only in the latter case could I actually assemble the product myself in order to guarantee matching the reviewed, stable code.