Well, I'll give you my rule zero for optimizing code
My direct experience says most of the people saying "don't optimize" are the ones who wrote the shittiest code in the first place because they simply assume all libraries are fast and efficient.
By the time you've made that shitty and slow code, it's probably too damned late to try to optimize it.
I cut my teeth writing on bare metal, and libraries which were called over and over.
If you don't start with some consideration of what is efficient, and you just do stupid things which rely too much on the library