Replaceable - but not really swappable.
At the very least, future consoles should at least make the storage hot-swappable, encouraging that you can install a large library across multiple storage units.
It might be that you want to have some other fixed internal storage for game saves, to help support cloud backup facilities.
But to be honest, it's about time we went back to cartridges. Memory costs are low enough now, and game prices so high, that the cost of manufacture, even at Blu-Ray scale, isn't prohibitive.
Simply being able to unplug one game, plug another in, and start up instantly was a big part of the console experience.
Moving to CD was mildly problematic with loading screens, but well executed titles largely got around that problem.
But now, even if you buy a game on disc, you have to spend hours installing it (and then updating it). And it doesn't save you on space on the console, so you still have to juggle the games.
Owning a console just isn't any fun anymore.