Actually he did give about 50% of his net worth to charity via a number of foundations. That's what moved him from being the richest person in the US to the 2nd richest behind gates. Over the long term he has pledged that 99% of his wealth will go to philanthropic causes.
When I used to for the John D and Cathrine T MacArthur foundation I found it surprisingly does take a lot of time, and man-power to give away that much money (assuming you limit it to programs that actually help people and not get wasted, stolen, mis-used). Usually the amount of research, financial tracking, charitable tax paperwork, legal department, IT department, international relations, government relations, etc people is very much like a 200-300 employee endeavor right there. And all the time your doing all this work to ensure that the money goes to people who really need it you are making more money.
On top of that Buffet is going to all his Billionaire buddies and saying "do you really need to have a net worth of 20 Billion? Why not give away 10 billion, cause you have the ability to make that back over a decade or two anyway".