Oh, I do that too, but I haven't figured out a good, _simple_ way to access this with my Android phone media players. I do not want to have to install a special server, that's what sshd is! I do not want to have to root my phone (e.g., in order to use sshfs). Maybe it's out there, but I haven't found it yet.
I agree - it kinda sux that the sshfs implementation on android needs root. However, if you want to stream to an android media player from your home server, and you don't want to root your device, a combination of an openvpn vpn, and samba filesharing will do the job. A lot of routers support openvpn out of the box now - but failing that any linux server can be configured to provide vpn addresses.