I work on a code base that is, in parts, 40 years old. There is nothing more satisfying that digging into some bizarre lockup or slowness, discovering it's because of assumptions or constraints from the 80s that no longer hold, ripping out or rejigging a load of code and seeing the problem go away not just in one application but in all the applications. Yes of course you need to maintain code without breaking things, and the damage footprint is higher the more dependents the code has, but there are well-established techniques by which you mitigate that risk (i.e. automated testing).
Libraries can have their issues but they are also absolutely a part of the answer to improving performance. Any library you include will be used by more than just you, and is therefore statistically more likely to have encountered, had reported, and fixed any performance issues before your application encountered them. Plus it will have been written by someone who cares more about the problem than you (after all, they bothered to write the code whereas you wanted to focus on other things to the extent that you went looking for a library). That said, knowing when to use a library (don't try to write a faster strcmp() than libc's) and when not (you can do better than qsort() in many circumstance) is something that only comes with experience.