This is what modules do for e.g. PCI devices. You can compile them all, and only the ones that are detected will get actually loaded.
Yet the unused modules in a generic kernel are still occupying space on the boot drive. For a smaller flash-based system, this can become significant. Even on an HDD-based system, more modules loaded before the file system comes up means more time to load the initial RAM disk.