That's the problem with exponential functions, the human brain is too easily tricked. Doubling the bit length of a key doesn't just make it twice as hard to break.
Over the past 40-ish years, we've transitioned from 8-bit computing to 16-bit, 32 and now 64 bit is common. We might need pointers bigger than 64-bits eventually, but we will never need a pointer bigger than 256-bits in length.
The same is true of encryption, for the same reasons. We measure the strength of a crypto system based on the number of keys we would need to attempt in a brute force search. Sometimes we find mathematical short-cuts that weaken a crypto system, reducing the number of keys we need to try. But if we can't do that, we need to test every value.
Counting through all possible values of a 128-bit number would use enough energy to raise the oceans to 100 decrees Celsius and then convert all of the water to steam. This is an amount of energy that we might be able to do harness one day, if we could be bothered. Counting through all values in a 256-bit number would require capturing all of the energy released by every star we can see.