The most fundamental right, upon which all other rights are based, is the right to life.
And that right only exists because human beings have developed civilisation and society and, gasp, government and the rule of law.
A gazelle on the African plains doesn't have a right to life, at least in any meaningful sense. If a lion catches and eats it, there's nothing wrong with that .
From the right to life derives the right to voluntarily trade for the goods that make life worth living, and real estate is also among those goods.
The ability to "own" a piece of land rests entirely on an artificial concept with the force of the law behind it (as opposed to the simple ability of the individual to defend it). Again, lions don't "own" the land they hunt on, they're just able to fight off other lions who want to be there too.
That doesn't mean it's wrong, but it's no more a natural right than my right to own an iPhone.
Too often, "social contract" is a verbal fog that sneaks in hidden restrictions against the life of an insufficiently careful thinker.
The term "social contract" is used to emphasise that human society is something other than everyone doing what they want, that there is give and take, mutual responsibilities and so on. The term "social compact" is perhaps more accurate, since as you say, a contract is voluntary.
As in any agreement, yes there are restrictions involved. In a rational human society, I am not free to murder you. If we were lions, I would be entirely free to try to kill you, and if I succeeded there would be no repercussions.