Unless your build environment is really broken (or you have a seriously a-typicall code base) compile time should not matter nearly as much as the resulting code. Don't forget that the resulting code has a big impact on the test phase of your cycle.
Normally during the edit phase you only touch part of a codebase, and proper dependency tracking should result in only a small part of it being rebuild and linked.
Proper dependency handling is not a job of the compiler.
LInking is also not the job of the compiler. (And untill lld is mature enough llvm and gcc use the same one anyway, and even then it still has to prove itself)