I don't disagree at all... however, those binaries don't come from Google, but from the phone manufacturers (at least to my understanding, please correct me if I'm wrong). This isn't much different than the binary nvidia drivers I use in Linux for my video card... although yes, there are open source alternatives (and different hardware) in that case, but using the binaries doesn't make the Linux distribution itself any less open source.
Also, there were pure Android Gingerbread builds available for phones such as the Nexus One, straight from AOSP, before any binary drivers were actually released... I didn't run any of those builds, as I understand there was some reduced functionality (which makes sense if a driver was missing)... but Android itself was still built and running from its pure open source release. I believe the reduced functionality was worked around by people like Cyanogen creating shims that could make use of the Froyo binary drivers in Gingerbread, though.
I guess what I'm saying is that you're completely correct, but I don't feel a company separate from Google choosing not to release their drivers for their proprietary hardware as open source means that Android itself is not still open source. I believe other Linux variants, such as OpenWrt for routers, have done similar things with Broadcom chipsets as well.
Again, please correct me if I'm wrong on any of this... this is just my understanding from my following of ASOP, CyanogenMod, xda-developers forums, and other sources. I'm certainly not claiming to be any kind of authority on the subject.