He was poor and from a poor country, but he did go to school and learn math there. He just happened to be fascinated by it and continued to work on it, neglecting everything else. He obviously also had a knack for math. That has nothing to do with poor or rich.
Math, being theoritical, does not require a lot of external resources (like laboratories etc.)