No matter how many times I read that, I can't seem to find the clause that says "Except when..."
The opposite is implied, that you have no right to refuse reasonable searches and seizures or warrants issued upon probable cause, supported by oath or affirmation, and particularly describing the place to be searched, and the persons or things to be seized. That it is possible for the police to "break into" your device is similar to how it's possible for the police to break into your home or for that matter, pull out a gun and shoot you dead. Whether or not that's legal depends on the circumstances and whether or not your device is encrypted or not won't change the legality. This is more like building a safe with built-in sensors that'll self-destruct the contents with thermite if the police wants to drill it, should "police-proof" systems be a consumer commodity? Yes, I know the government can be the bad guy but they're not the only bad guy, there are real criminals being caught through a constitutionally valid process. It doesn't say "The right of the people to be secure in their persons, houses, papers, and effects, against all searches and seizures, shall not be violated"
The real issue here is that you don't trust the government to play by the rules, which given the NSA affair and such is probably not a bad thing. A backdoor doesn't mean it has to be a covert backdoor though, it could be that the next time you enter your PIN you get a pop-up saying "Your phone has been searched by law enforcement. Here's a scan of a signed warrant authorizing the search. For further details, contact [police department] on [number]." Granted, it's not so safe as it being impossible, but getting your chance in court is what you get if they break down your door by mistake. And if they shoot you dead by a misunderstanding, you don't even get that. Besides, last I checked he didn't actually want to outlaw it which would create a whole lot of problems for open source and everything else without a back door. He's saying it would be bad for law enforcement, which would be good for criminals and ultimately worse for the general public. Right or wrong, he's certainly got the right to make the argument.