There are basically three x86 processor manufacturers. The two smaller players work hard to stay compatible because their livelihood depends on it. Most of the interface functionality is off-chip.
There are many well-known ARM processor licensees. They all strive to differentiate their product offerings. In the majority of cases all of the major peripherals (which are one of the primary opportunities for differentiation) are on-chip.
As such, where minimizing differences by processor was clean and relatively straightforward for x86, expecting it to continue to work well for ARM is nonsensical. I really think Linus is missing the forest on this one.