So, my first /. journal entry. Momentous occasion, perhaps. Perhaps not. What prompted it was the large number of people visiting hostip.info from my (allegedly non-existant) journal page... So I thought I'd put something in here :-)
Hostip itself started off as a "wouldn't this be cool" idea, and a first version was born. The 'individual privacy' minded will have a field-day with this, but the inspiration actually came from watching 'Enemy of the State' on a '747 flight :-) I wanted to do (in a very limited way, of course) something similar using the web. As always in projects like this, it's the data that's the hard part of the equation, not the coding...
This first version allowed people to type in new cities, and it would auto-associate with their IP address. This was (as I should have forseen) a complete disaster. The number of Martians living here on Earth is truly amazing. We apparently even play host to a couple of Alpha Centaurites; to these fine beings I say 'Welcome to Earth' in "Will Smith" fashion. (Yes, I'm a fan...)
Once it was clear that if bad data was trivial to enter, it would indeed be entered, I raised the bar a little. Now you can only choose cities that already exist (and which I have latitude and longitude for), or email me with the details of a previously-unknown city, and I'll check it out before entering it into the DB. This has made the database more useful... Needless to say, cleansing all the bad data from the DB was a monumental task. It literally took weeks, and if I'd known at the start how long it would take, I'd not have started it!
It's still possible to lie to the machine of course (and I dare say lots do, on purpose, simply because it's their principle to do so). I have in my own way tried to get around that - the DB keeps a track history of assignments to a /24 netblock (that's the smallest unit it tracks), and since you can only reassign your own IP address, as soon as 2 others on your netblock tell the truth about where you are, it will switch to the real location... It's certainly not foolproof (hell I can think of a half-dozen ways around it!) but it raises the bar...
Up until this point, hostip was a purely text-based system. Next came the map data. I got in touch with the US National Geophysical Data Center in Boulder, and asked them for the highest-resolution data they had. That turned out to be 30 arc-second elevation data for the entire planet. Wow! So I spent some time writing tools to efficiently extract the correct data and colour it nicely/correctly for the small maps I needed - this took a week or so... Just loading the data into RAM took a lot of time (eventually I remembered mmap() and things went a *lot* smoother!).
The dataset consists of a 43000x21500 image, at approximately 1km/pixel, taking ~2.6Gbytes to store. Even things like ppmtogif can't handle that much data :-( The current database size (from du -sk on the mysql db directory) is 623Mbytes. All this needed to be correlated together before the applet started to look even vaguely reasonable. It still has lots of errors (mostly where I have the decimal point wrong in latitude or longitude figures :-( but it's useful now, and I tend to get told [grin] when something is wrong...
One of the reasons I wanted to do this (apart from the obvious coolness of the idea :-) is to give something back to the people who've given me so much 'free' software over the years. Those from this nameless multitude, I salute you - I hope you get as much out of hostip as I got from your various projects/programs.
I happen to think the applet is (even though I wrote it myself, [grin]) one of the coolest ones I've seen so far, although that may be down to knowing just how much hard work went into making it [big grin].
I have more plans for hostip, but perhaps I'll leave them for another journal entry...
Simon.