"Connecting directly to the destination network is typically free
Exactly. When the costs naturally split fairly evenly, there's no reason for anyone to pay anyone else. If both networks do their fair share, and we presume the traffic benefits both parties roughly equally, there's no reason for anyone to pay anyone else. Any argument that X should pay Y equally argues that Y should pay X. Senders pay to send, recipients pay to receive. This is a pretty typical case, and it's the justification for settlement-free peering.
So why don't you get access to their entire network? Because for some parts of the network, bringing the traffic to the destination network is *not* anywhere close to half the work. When the work doesn't naturally divide equally, settlement-based peering is used. This is how it's been for decades.
In the case of traffic from someone like YouTube or Netflix to a customer of an ISP like AT&T or Comcast, the costs don't naturally divide evenly. YouTube and Netflix use a small number of sources located wherever the cost is least. AT&T and Comcast have a large number of destinations located wherever they happen to be. This is a direct and inevitable result of the business model companies like Netflix and YouTube have chosen. We presume the traffic benefits both sides equally and so each side should pay half the cost.
In the vast majority of cases, the sender should bear roughly half the cost of delivering their traffic and the recipient should bear half. The result of this kind of imbalance has always been settlement-based peering.