Organic chemsitry is not a fascinating subject in its ownright. And even though it falls in the purview of physics -- like, uh, everything -- it is best understood apart from physics, as a unique lense. Just as biology is not best understood as complicated chemistry, but rather as a completely different perspective.
It demonstrates the raw power of abstraction. For example, ask an experienced organic chemist to propose a synthesis of any arbitrary molecule. A good one will normally be able to come up with something plausible in minutes, and refine it to something practical in hours. A physical chemist, let alone a physicist, even with the incredible computing resources for the complex quantum mechanical calculations required wouldn't be able to tell you how to make it if you gave her months! Guarenteed.
That is the power of organic chemistry. It teaches you how a handful of simplifications, fuzzy rules, and fictional symbols can give you incredibly unique and practical skills. This is not unlike treating the human body as a group of organs, cells, cellular machines, etc., rather than subatomic particles. Of course it's all physics, but viewing systems through appropriate paradigms can yield incredible results.
If people see "orgo" as just a test of rote memorization, their professors should be ashamed -- they've missed it point.