Absolutely. The situation is not sustainable.
Even worse, because every SOC is a haphazard pile of random and arbitrarily buggy peripherals, there is no deterministic way (at run time) to enumerate all of the peripherals, and thus which various driver variants (and even worse, binary blobs) are required to make them work.
So by definition, none of this can EVER go into the mainline. Every kernel fork is its own disconnected universe, dedicated to a single snapshot of a single SOC and its particular collection of peripherals.
But if you try to explain this to a PHB (or, say TI), you'll get nothing but blank stares. There is nobody home.