but I as a developer sure do notice. The biggest issue I keep running into (developing backend software for my companies frontend software) is that testing on a mix of devices means learning the quirks for every single manufacturers user interface that they have bolted on top of Android. We've also had some weird issues based upon the Android version installed, across two devices with the same Android version number (4.0 for example) with the carrier/device manufacturers changes we have a bug on one but not the other.
This is highly annoying.
One issue that Android users hail as the greatest thing since sliced bread (alternate keyboards) actually meant having to write work-arounds because some keyboard implementations were simply broken, or actually caused issues with entering text in certain situations. An alternate keyboard shouldn't be able to have that sort of an effect!
Fragmentation is real, and it is an issue. Consumers don't notice because they only use a single device, developers and power users that may switch more often than the average user will notice and it is an issue.