Two things, "Even Ham radio operators?" When did they become the retards of the RF world - I thought that title belonged to CB'ers? Honestly, hams are not interested in your phone.
While, yes, technically anyone can communicate with your modem; anyone can communicate with your wifi card or your bluetooth adapter as well. And it would appear that the samsung radio interface IPC layer at least has a modicum less access to the entirety of your device than your wifi driver - which is in the kernel. People have, in the past, exploited mistakes in wifi drivers and wifi card firmware to remote exploit via wifi. (*: The specific instance I remember, was with an old intel 802.11b/g card and specially crafted management frames which could be trivially spoofed and didn't need to be encrypted to be accepted by the wireless card. The proof of concept was able to issue busmaster DMA read/writes which, ostensibly, would allow rewriting arbitrary kernel ram, etc.)
Across the scope of samsung phones I was able to check (ok, two of them), the radio interface, the android host side of this communications channel, runs as uid 1001 (radio). As far as my cursory inspection revealed, meant that the radio/modem can read/write the files in /efs and only read a number of other places, such as /sdcard. Granted, /sdcard contains a lot of your personal data. My point is that, in this case, a compromised modem is still less privileged than a compromised android service or, worse, compromised driver/kernel. Also, given that these IPC instructions are used for reading/writing modem "nvram" data such as the handset IMEI, to describe them as a "backdoor" is horribly inappropriate.
So, yeah, as you said, "huge technological challenge." Agreed. But, the idea that a data modem may be exploitable is by no means new.