Not sure if this was covered above...
I suppose it depends on what areas you will focus on. As in most fields, astrophysics and astronomy have many sub-fields that would require a more extensive background in that area. It also depends on which route you are taking (experimental or theoretical). With a math background, I would expect a theoretical route would be very appropriate.
Additionally, some schools have different notions of what to include in an Astro grad program. I know someone in one now in which the program assumes little or no previous Astro experience, while some assume extensive undergraduate coursework.
I completely encourage you to pursue this, but it may be a daunting task. If you made it through math, it is likely that you have the brain juice to do the physics. There is just a lot of physics out there to learn... so keep a pot of coffee on at all times... :)
For the basics (as most have suggested) check out the Feynman Lectures.
The Schaum's Outline series provides a good overview of many subjects.
Your instructors will probably tell you what to read for your particular astronomy and astrophysics courses.
Most basic and intro areas in astro or physics have a wide variety of books to choose from. Yes, some are better that others, but they all have the information you're looking for, so just grab a few and dive in!
(just a few thoughts from a fellow astrophysicist / college professor)