Fink and MacPorts both have the same problems:
1) It's impossible to successfully compile anything complex with 64-bit compiler flags (i.e.: -arch x86_64 -m64) since all libraries will be built without said flags... So you have to manually figure out all the dependencies yourself or backtrack the 32-bit install piecemeal.
2) X11 is/was often broken and needed XFree86 compiled instead of Apple's X11. This might have been dealt with in all cases now.
3) They are hopelessly behind every major O/S release. It took forever to have a 10.5 compatible fink and/or macport that was officially released.
All in all, I had to build a 64-bit application chain for 10.5/Intel and it took me two days to get ~30 packages compiled and linked successfully and fink/macport couldn't be used. I often had to reverse engineer fink mac patches for the packages that it did have, it's silly that I couldn't use fink to build them in 64-bit!
The rationale for a 64-bit build was a 40% performance improvement.