For almost a year now, I've been using xt_geoip to block connections to ports on my internet server that require authentication from countries where no authorized user is located. Right now, I'm the only authorized user, so that's easy. The only ports open to the world are 25, 80, and 443. Port 25 has SMTP AUTH disabled, and I'm considering adding GeoIP filtering to certain virtual hosts running on the web server. I pull GeoIP databases weekly, to keep things up to date.
Sure, it's basically security by obscurity, due to proxies and such, but it's only one layer and it dramatically cuts down the amount of incoming attacks that the various servers and Fail2Ban then have to deal with. Sure, there's a chance I might screw up and lock myself out, but console access is still there as a backup.