Fragmentation on a handset is different than fragmentation on a PC.
Some people's PCs have unsupported hardware, and you can either replace the hardware, try a different distro, see if you can find drivers, or replace the PC altogether. You have options.
Most people in the mobile space are locked into a handset for at least one year, probably two, and there's no way to change the hardware in it. Plus, many if not most of these come from a carrier that locks down sideloading and customization to some degree, so it's not within the average user's ability to do a clean install.
This leaves them with what effectively is a fragmented platform, and one which will be on a moldy version of android soon because updates only come as long as the phone company feels it's worth their money (i.e., while the handset is continuing to be sold). Once the handset is discontinued it's not worth time or money to test new versions, so eventually the OS and the Apps leave you behind in dependency hell.
The level of fragmentation is what the user feels it as, not what Android is in an ideal world where everyone has a Nexus One with no contract. (which only a few thousand people do)