These gentleman's agreements are bunk, making the very idea of sports competitions a joke. These are not the best of the best, they're the best of what they feel like allowing - for now.
Maybe they need to have both. For instance, in sailing, there are several types of competition. There is One Design racing, where the boats are all required to be pretty much identical. Different boats have different restrictions (some restrict costs by doing things like limiting how many sets of sails you can buy each year, others pretty much say no limits), but within a fleet, the boats are pretty much identical. At some events, boats are even provided and/or you rotate between boats. This really means that the best sailor wins since they can't do anything to significantly alter the boat.
Then there are classes where the boats are all similar, but not identical. They are built to a specific rule, e.g. "As long as the boat fits in X dimensions and has sail area less than Y and the ballast doesn't weight more than Z". The boats tend to be fairly even, since a fast design soon takes over the competition, so the best sailors still tend to win. But it also leads to innovation. For instance, the Moth class has evolved from a boat that looks like this, to a crazy high performance design that hydrofoils above the water like this. The old moth would never be able to compete with the new designs...but that's ok. That's the price of allowing people to innovate.
There are even events that really have no rules...but generally people don't care much about them since the competition completely turns into a money pit. Generally what happens there, is a standards body assigns a handicap to level the playing field. So the super rich dudes still compete for who can have the flat-out fastest boat, but everybody else (and the people who have the fastest boat from 10 years ago) just compete for the best corrected time according to handicap.