In phones with removable batteries, the SD card is generally hidden next to the SIM and requires moving the battery to get to it. So you can't swap while the phone is usable at all.
Sorry, but that's just plain not true. I've had 3 different smartphones with removable battery and SD (holding off on getting another because those features are so hard to find now). In exactly 0 of them did you have to remove the battery or SIM to change the SD card. To change the SIM card, yes; to change the SD card, no. (On the last 2 you do have to remove the back cover -- snap-off -- but not the battery)