Exactly. That's why a modular PCs were never created. There's no way you can get high performance when the user can pick their own RAM, CPU, motherboard, video card, hard drives, etc.
Oh, wait.
Size matters. Desktop PCs are easy to make modular (unless you want an iMac). Laptops are harder, and besides removable batteries, only a few had any modular components (like a DVD drive swappable for an extra battery). Phones are much more space-constrained. Every millimeter counts, and modularity takes up quite a bit of space at that scale, because each part needs to be enclosed, securely attach to the others, etc.
In short, a modular phone is possible, but the trade-offs will be severe, and you'll be able to pick one or two things (e.g. speed, battery life, extra features, small size, etc.) but not all at the same time. And the prices won't be good, because manufacturer(s) will lose economies of scale: it'll be hard to compete with Apple and Samsung making millions and tens of millions of identical units.