Er, the whole point of each of those package-managers dragging in a whole stack of stuff is to avoid the whole "one manager/upgrader stomped another's files" problem that got this whole thread started. Fink and MacPorts each have their own (for example) libxml2 so that neither one's nor apple's feature-enable/disable, interface-compatibility, or other changes affects anything except its own packages, which presumably know how their own lib is built. If CPAN had behaved the same way, we wouldn't be here, because apple would know exactly how its versions of things were and its upgrades would be self-consistent, and user-installed versions *somewhere else* would not be touched and would remain self-consistent with themselves. That's really the only solution that's easy to make work well (or at all) as a few-parentlevels-up notes, and that's precisely what you criticize Fink/MacPorts for doing.
If you build from source yourself, you're using apple's versions of libs, which is fine as long as you trust apple not to change things like this. If you do so and keep your files in /usr/local like you're supposed to, at worst, you'll have to recompile when an OS lib changes. The problem here is when users *didn't* restrict locally-installed things to /usr/local, which is just silly.