That's a logical position, but I don't think it's entirely compatible with the accepted notion of human rights.
For example, the U.S. Constitution guarantees the right to a jury trial. This is a "positive" right that other citizens are forced to grant.
The Universal Declaration of Human Rights also includes several positive rights: Everyone has the right to a standard of living adequate for the health and well-being of himself and of his family, including food, clothing, housing and medical care and necessary social services, and the right to security in the event of unemployment, sickness, disability, widowhood, old age or other lack of livelihood in circumstances beyond his control.
Personally, I think we need to invent a new category of "reasonable expectations" that sits between "rights" and "privileges".