I think many of your (valid) complaints stem from weak and/or immature writers relying on "magical" things and/or super tech to achieve the desired story line. I dislike "magic" in most stories as it seems to be used mostly as a crutch for weak writing, or a writer unwilling to deal with unpleasant consequences in a story.
My understanding is that the transporter was originally used to workaround using a shuttle craft for all extra-vehicular excursions (for screen-time efficiencies) and later became a useful story device. But, you're right that this new trans-warp beaming-device is simply "plot magic". Kahn could have beamed to a near-by ship and flown to Klingon space. (Furthermore, why didn't an Enterprise retrieval party simply re-use the trans-warp transporter to beam themselves as did Kahn... saving themselves the trouble of the trip.
I've been watching Trek for like 30 years and didn't know that about TOS transporter. Thanks. I guess I need to read more Trekkie history.
The two J.J. Abrams ST films seem full of unnecessary / stupid things - like parking the Enterprise underwater to avoid being seen by natives when parking it in *orbit* would have accomplished the same thing. Granted, watching the ship rise from the ocean was a ST boner moment, but still technologically unnecessary. (JJ's signature move seems to be having the Enterprise rise up through clouds, water, etc...)
I wonder about orbit. They generally enter a moderate orbit, perhaps? A large starship streaking across the sky at regular intervals would be noticeable near sunrise/sunset. I'm nitpicking I suppose, but it reminds me of C.J. Cherryh's excellent Foreigner series, which had humans get really lost and end up finding a planet that happens to have an intelligent steam-age society. That light orbiting up there seriously got the natives thinking, "WTF?". (umm, I don't recall how much, it's been years since I read the first). But they were semi-advanced and had astronomers.
Oh, and in First Contact they *were* noticeable from orbit, and as we know at that point there were likely plenty of people around with telescopes and eyeballs and tech savvy. Nobody else noticed Enterprise up there? Well, probably I guess, but it didn't matter in that story. No more missiles to shoot it down, save Cochran's!
But I suppose the Prime Directive made an exception to being noticed as merely a light in the sky