Well, I'd say it is possible (it surely is, as some people actually do it !), but is indeed extremely hard.
Attaining the level of a master degree in any field seems possible to anyone with the required intelligence and power of will. The real gap might lie somewhere between master degree and PhD. At this level, the resources are pretty rare and sparse, and there is no easy entry point. It's quite understandable, as there are much fewer PhDs than BSc's, so few books are written for this audience. Also, I believe the field is not always cleared enough to be explained in one, standardized way like undergraduate or even graduate-level science: the didactic tools, standard exercises etc. might not always have been built - but that's only my impression, I might be wrong. And at this level, it's usually expected that you'll be guided in the field by some older, more experienced scientist, who knows what's most important and what you have to study first to understand some concept.