I prefer my TV's to be dumb displays
They should be limited to take video in, modify resolution/contrast/etc as per settings and display it on the screen, and provide a control interface
IF I want to play media on it, I will use a device for that
Modularity is better
I hear you, but this (the current) approach has some serious drawbacks, including cable mess and multiple remotes (or one poor universal), power extensions when you only have two sockets, etc.
The right approach would be for each TV to come with a hidden and swappable "smart" unit (or bought "naked" if you wish), controlled by the main TV's remote control, powered by the TV and with a standardised interface. This way, you'd have the best of both worlds, you'd be rid of the cable and remote control mess and it'd still be modular. People could then create Boxee boxes, Cable/Satellite set top boxes or fully functional PCs to adhere to the standardised smart unit interface. You may also want the WIFI or Ethernet connectivity in the TV, so that you could have multiple smart units all connected to a hub in the TV.
The problem is that nobody but the consumer is interested in standards. They would all create their own proprietary slot suitable only for their own smart units.