You are assuming too much too fast and without thinking before writing.
Let's give some practical examples, since you want so much, about some things that I decided to install from source:
Freetype: I wanted to install from the source in order to enable the use of subpixel hinting and other useful settings that are not enabled by default. Clean, clear documentation on what each option does, few dependencies and therefore easy to compile. Compiled and installed without problems.
Kernel: The first thing I set up in a new Linux installation. Tons of settings that you need to worry about but usually the documentation on them is reasonably good and you therefore do not have to "shoot in the dark" to learn about the important settings and avoid those that may cause problems.
Look, a new version of Gimp with those features I wanted to use! Oops, source only, maybe one day we'll provide the package for your distro. Ok, I have the means to install from the source so let's see ... First attempt fails because of a library I've never seen before, but that's okay, I can find it somewhere on the internet. Second attempt fails because my XYZ system library is not the latest possible (later you find out that it is still in beta). Ok, let's try, dependency resolved. Third attempt fails because of a really weird error, you search on forums and only find a half dozen references and no response other than "look up Google" (funny, you arrived at that forum exactly for having searched on the Google).
Fourth attempt, it now works! Oh oh, but now my sound player crashed because the library (in beta) I installed before was used by the sound player and nowhere mentioned that.
I decide that it is better to wait for the GIMP packaged properly for my distro and then try to use the package manager to roll back the library (in beta) to the previous version (note that the beta library installation was done via package management, I will not mention for you the times when I installed every single dependency from the source code)
Okay, package manager says it can return my beta library to the stable version, but in the process it insists on trying to remove half of the desktop-related packages (eg GNOME/KDE/etc). WTF??
Oh, there was also the time when I decided to be aggressive and I installed an application and all the packages it needed (and the dependencies of those packages) from the source for being a more unusual application (I do not remember which one now, sorry) and therefore with fewer users and therefore less documentation. I was able to install of course, but after that my desktop was never the same again and I had to reinstall everything from scratch to go back to normal.
So, you're still thinking I'm just making it up?
And note that any grammar errors are not intentional, English is not my native language and is pretty difficult to translate from my language into English without losing some meaning.