As far as I'm concerned, a hardware router...
There is no such thing. A device that moves data from one location to another, using some policies to examine and transform it, is not just a "hardware" device. It's also software. And if it interfaces with software, then it can be compromised. Or haven't you noticed the news about D-Link routers? A lot of these routers have 2MB or less of flash, which makes it difficult to find a useful exploit, but "difficult" doesn't mean "impossible."
It's pretty unlikely that anyone will come up with a useful attack on a device that's just doing port blocking, NAT, and basic routing. At worst, somebody might DOS it or turn it into a well-connected zombie to aid in DDOSing somebody's server, but neither of those is compromising your data.
With just a little paranoia, I can imagine someone finding a way to get those routers to copy your traffic, or at least the headers, to some hostile entity. It doesn't take full knowledge of your traffic to destroy your privacy.
A router is a type of computer. It's subject to all the same concerns about trustworthiness as any debate about proprietary and free software.