Memory is fairly cheap (though if you really want lots of memory in addition to the cost of the memory itself you have to consider the cost of the platform to accomodate that memory) but cache, particulally the lower levels of cache that are closest to the CPU isn't so cheap, if you have a pointer-heavy workload (e.g. data structures that are mostly cross-references implemented using pointers) then you can fit a lot less of your workload in cache with .64-bit pointers.
For java (which is very pointer heavy) this was so bad that they implemented tricks in the VM so that if the java heap size is below a certain level then "ordinary object pointers" can be stored in 32-bits.
On linux there is something out there called x32 which aims to combine the advantages of the 64-bit processor mode with the advantages of 32-bit pointers but it's unclear if it will ever catch on.
you can fit 32Gb+ in a consumer grade desktop system.
If you had left off the + then you would have been more on the mark. The largest reasonablly available modules of desktop memory are 8GB and the most slots you will find in a mainstream desktop board is 4 (and many boards only have two). If you want to go over 32GB you have to move to the high end desktop platform (and if you want to go over 64GB you have to move to workstation/server platforms).