That's not necessarily true. If you are taxing a charger to its limit, there will be a lot more waste heat. Using a charger that is running at half its max will generally be more efficient.
50% is more efficient than 100%, yes, but 80% is generally much more efficient than either. You're generally best off using a power supply that's 120% of the max power the device will use 80% of the time, as long as 'absolute maximum' is still within the supply's reach.