I think you've drifted into confusion between the rental value and the sale value, also the unimproved value and the improved value. Property taxes are generally assessed on the sale value which includes improvements.
I am saying that a landowner should, in effect, rent the land from the community at its market rate as unimproved land. If he makes improvements that should not affect the 'ground rent'.
If he sells it with improvements, for more than he bought it for, he can keep the profit. He earned it.
If the community builds a city round it and rezones it as suitable for millionaires' mansions, that should affect the ground rent.
That may mean it is no longer viable as cattle pasture (or whatever) in which case the owner can either build mansions or sell it to a mansion builder.
If he sells, should he keep the profit? He didn't really earn it, it was created by the community. But the community is recovering the additional value through the increased ground rent, so perhaps we can consider the profit as compensation for having to move.