For some codes it makes a major difference (Score 1)

For most programs source level optimization does not make a major difference, but for some problems it is an absolute must. I have hand tweaked numerical codes and observed improvements of 500+ percent, just by interchanging loops, intelligent dereferencing, linear memory access and other minor tweaks . Yes, that's more than 5 times faster - and thats a lot. Think of reducing the running time from 1h to 10min.

For most applications however source level optimization is not necessary, either because processing speed is not the bottleneck, or because the code does not require a lot of time to run.