If licensed like DOS, it would have every bit as many compatibility problems.
Oh, not as bad, at least at first. The companies licensing MacOS would have had to make suitable hardware, and Apple could have held their feet to the fire to get compatibility and quality.
In those days, there was so much pent-up demand for Mac laptops that there were companies that would buy a Mac, crack it open and pull out the ROMs, build a laptop with the ROMs, and provide some sort of docking station so the original Mac would not be useless. This was about the most expensive way to make a laptop ever, but it was the only legal way to do it. Apple took forever to release a laptop product, and when they did, it was not what the customers wanted (heavy due to the lead-acid battery for one thing). Third-party Macs could have cost significantly more than generic "beige box" PCs and customers would have paid happily.
The thing is, Apple was charging crazy money for Macs. If Apple had adopted the Microsoft model, they would have had to accept lower margins on each Mac, and made it up on volume. Third-party Macs would have cost less than Apple official Macs but still would have sold a lot and buried the DOS-on-x86 PC. Apple was marking up Macs by about 100%... They were successfully getting a 50% margin on each Mac. Nobody else got away with that kind of markup, before or since.
It was great for Apple while it worked. But eventually Windows got to the point where it was kind of usable. And a Compaq running Windows would cost less than half what Apple was getting for a Mac. Hastings's Law: Adequate and cheaper tends to win against better but more expensive. Windows sales took off and Apple nearly died.
What saved Apple was the PowerBook, a laptop that really was what customers wanted. And a string of other successful products. And now Apple is doing very well. But IMHO, Apple could have had success like Microsoft in the 1990's had they adopted the Microsoft strategy of licensing to everyone and making a small profit on a huge volume; instead they nearly went out of business.
Even now, Apple isn't getting anything close to 50% margins on Macs. Those days are over.