You've got a tough road.
Many graduate programs have core graduate physics courses, perhaps 4 to 6. The enduring standards over the last few decades are Classical Mechanics by Goldstein, Electromagnetics by Jackson, Statistical physics by Reif, QM (not sure if there are enduring classics there).
You've got to be able to work problems out of these books quite well.
I would reccomend that you read a survey course book in astronomy like Chaison's Astronomy today. It's a gentle introduction into the current astronomy suitable for even a freshman. A good intro to physics is a University physics book such as Sears and Zemansky (now written by Young and Freedman). This covers the calculus version of the basics - 1100 pages covered in two to three semesters.
Then, in order to get to the level of the graduate texts, there are typically one to three courses between the basics and the graduate level. This is in electromagnetics, quantum theory, statistical physics, classical mechanics, thermodynamics. Optics is another probable option.
The feynman lectures are great for explanations and concepts. Use them in concert with other texts for good results.
you also need to learn error analysis and basics of experimental methods taught in undergrad labs.
Remember, math has the principle tools of science but it is not the science.