Exactly. ICANN and IANA don't exist because they have a mandate from the US government, they exist because there is a consensus that they're doing a reasonable job. You don't own an IP address because IANA says so, you own an IP address because the people who configure the BGP routes for backbone networks agree to send packets for you to the place that you've asked. They currently do this because they perceive the assignments made by IANA (and then subsequently by national organisations) to be fair and equitable. If it looks like the USA is imposing too much control on IANA, then their authority goes away and there is likely to be a new consensus about whose assignments become the real ones (probably with a long interim process where bits of the Internet were broken or unreliable).
Ironically, a lawsuit like this is exactly the sort of thing that would push the consensus away from the USA.