Slashdot Log In
Map the Internet... In One Day?
Posted by
jamie
on Fri Nov 14, 2003 03:09 PM
from the graph-paper dept.
from the graph-paper dept.
rjbrown99 writes "There have been numerous stories over the past few years on Bill Cheswick's Internet Mapping Project. The Lumeta folks even created a company out of it. Well, now there is a competitor. A single guy with a single computer is working to accomplish the same feat - within ONE DAY and using open-source tools to do it. The new project is called Opte and can be found at www.opte.org." He's made some progress and is looking for volunteers.
This discussion has been archived.
No new comments can be posted.
Map the Internet... In One Day?
|
Log In/Create an Account
| Top
| 263 comments
(Spill at 50!) | Index Only
| Search Discussion
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
This server will die ! (Score:5, Informative)
This project was started by me (Barrett Lyon) as a response to a conversation with my colleagues at Network Presence. Over a lunch we were discussing William Cheswick and Hal Burch's Internet Mapping Project. I was not very impressed with the results of their project, they produce beautiful maps but they don't seem to be very useful nor do they release their code freely. Their mapping also takes nearly six months to generate a single map. My comment was that, "I can write a program that can map the entire net in a single day." The comment was met with some hostility. Thus, this project was born.
What
The goal of this project is to use a single computer and single Internet connection to map the location of every single class C network on the Internet. It is obvious that the Internet is not routed as a bunch of class-c networks, but it is easy to see that by treating the Internet IP space as a bunch of class C networks, it will be possible to make a detailed map of the entire Internet. The global Internet address space currently offers 32 bits worth of unique host addresses, or a theoretical maximum of 2^32=4,294,967,296 hosts. In reality, the address space has been allocated in fairly large contiguous blocks, which renders strictly optimal utilization difficult. The smallest block that is logically routed via BGP or allocated by ARIN is a class C network (CIDR
At the rate of 194 traceroutes per-second it is possible to scan the entire theoretical 2^24 space within a single day. Thus about 16,777,216 class C networks could be processed by a single computer in a single day. Yet, there are huge portions of network blocks that are no longer used, many network blocks fall into the RFC 1918 standard and other blocks that are reserved by ARIN.
According to ARIN there are about 47 class A networks in the reserved status (search ARIN for OrgName "Internet Assigned Numbers Authority".) Doing the math results in a reduction of 3,080,192 class C blocks to be removed from the scan list, leaving us with a theoretical list of 13,697,024 blocks.
Applying some additional thought large portions of the 13.7 Million blocks may route to the same place. By testing about 20 routes at random within a class B and comparing the results, it is possible to see if there are multiple routes worth investigating or if the entire thing goes to the same place. By applying that logic it increases the speed of the scanning.
After some testing and beta code I proved that with enough bandwidth it is possible to scan the entier Internet with a single computer. The 1/5th of the Internet map only took about 2 hours to create, yet it generated nearly 200k/sec of traffic and put my machine at a load of 60+ while scanning. If you apply the math, the entire internet would take about 10 hours to scan and another hour or two for the visual map output.
I found a lot of value in the project, so after the proof of concept was completed I continued to program. I turned the entire system into a distributed client/server model. The clients request a chunk of random IP space from the server and when it is completed the IP space is registered with the server. This is done until all of the IP space has been scanned. I'm also working on a stats system so I can monitor the productivity of the different scanning nodes and users involved in the project.
By taking a more distributed approach the data will look more like the real Internet. It will show more of the backup routes, more of the smaller links in different countries, etc. When the first version of the code is done I should have about 5 to 10 different scanning nodes running on the Internet. If you would like to donate a computer and some bandwdith to this project, please contact me. I can give credit where credit is due!
When
The first scanning tests began in late October 2003 and I wish to have the project generate a new map every week.
Where
Currently the project is hosted in San Francisco on a multi-homed fiber ba
Re:This server will die ! (Score:5, Insightful)
Here ya' go... (Score:3, Funny)
(Last Journal: Monday December 08 2003, @09:32PM)
He needs more bandwidth (Score:5, Funny)
Slashdotting was never easier!
Re:He needs more bandwidth (Score:4, Funny)
(http://dukeytoo.blogspot.com/)
Turn left at the third router... (Score:5, Funny)
(http://www.redstream.org/)
Re:It has to be asked.... (Score:5, Informative)
(http://www.5vs1.com)
Mapping the Internet weekly will allow us to see major disasters in different parts of the world. The Internet is a huge disaster censor. If I had maps of pre-war Iraq and then compared them to today, one could see how badly Iraq was destoryed. The idea of a metaphysical representation of the real world is very interesting to me.
The project can show the Internet growth.
The project is art.
Re:It has to be asked.... (Score:5, Funny)
Ok heres my part... (Score:5, Funny)
(Last Journal: Monday October 11 2004, @09:43PM)
IP Address: 127.0.0.1
Computer: The one from Microsoft with the Start button in the bottom left hand corner.
Location: my bedroom.
His Map Is Wrong (Score:5, Funny)
(http://www.advantura.com/)
Been there, done that (Score:4, Informative)
(Last Journal: Wednesday July 28 2004, @08:06PM)
There's John Quarterman [mids.org] who's been doing it for years, and then the CAIDA [caida.org] visualization tools, and Cybergeography [cybergeography.org] and the Internet weather report [internetweather.com] and damn maps [mapblast.com] and more maps [mapquest.com].
Note to everyone: please stop mapping the internet.
Slashdotted ! (Score:5, Funny)
(http://technocrat.net/ | Last Journal: Monday March 29 2004, @12:53PM)
I can map that Internet in ... (Score:4, Funny)
Top that!
Creepy (Score:5, Interesting)
When I first saw the image on the right [opte.org] it looked like a human brain. It would be creepy if the Internet had a sort of fractal self-similarity to our physiology.
It's too easy... (Score:5, Funny)
(Last Journal: Wednesday July 28 2004, @08:06PM)
He's mapping the Internet. Why am I not surprised he's single?
rsync (Score:3, Interesting)
(http://www.p0wn3d.com/ | Last Journal: Tuesday March 09 2004, @09:43AM)
Internet Mapping Project does daily maps (Score:5, Informative)
[Internet Mapping Project's] mapping also takes nearly six months to generate a single map. My comment was that, "I can write a program that can map the entire net in a single day."
The Internet Mapping Project maps the Internet in under two hours (105 minutes for this morning's run). I'm not certain where the six months came from. The rate limitation is the packet rate limit we set (500 packet per second).
Map layout time is not included in that time, but that is not done on a daily basis. A map layout take about six hours, as I recall. It only took a couple weeks to produce all the layouts necessary for a movie of the Internet [lumeta.com] from Aug 1998 to Jan 2001 based on the daily runs.
CAIDA [caida.org] also creates daily maps of the Internet as part of their Skitter project. Their schedule varies between measurement points. In addition, other projects, such as the Mercator project and the RocketFuel projects, also map or did map the Internet.
Each project has slightly different goals. Skitter focuses on paths to major web and DNS servers. Mercator attempted to discover networks with limited pre-knowledge. RocketFuel wants a very accurate map of a particular ISP. The Internet Mapping Project is focused on the router connectivity within and between public backbones.
I have considered something similar (Score:5, Interesting)
(http://insecure.org/)
Then I came to my senses and decided to work on more practical and less controversial projects such as Nmap Version Detection [insecure.org]. But the subversive in me still hasn't given up entirely on Nmapster :).
-Fyodor
Brilliant! (Score:3, Funny)
(Last Journal: Friday May 02 2003, @12:35PM)
1 Setup a site saying you want to map the internet.
2 Get posted on slashdot.
3 Parse the referer logs.
4 ???
5 Profit!
bad for business (Score:3, Interesting)
(http://slashdot.org/ | Last Journal: Tuesday June 24 2003, @04:25PM)
I mean with this tool, I would look up where my new IP would land me and try to find a host closer to the main backbones. Is this already done now by most people?
(on another subject the maps remind me of the species origin stuff)
Hierarchy (Score:3, Insightful)
This means concentration of power. So, the real, failure-tolerant internet is gone, at least it seems to be.
Re:Hierarchy (Score:4, Interesting)
(Last Journal: Monday August 20 2001, @01:45PM)
No matter where you are on the net, your view is going to look like a tree with you at the center. Traceroute-type mapping will not capture the redundancies.
Let the mountain come to Mohamed (Score:3, Informative)
(http://slashdot.org/)
P.S.
Is there such a thing as trecart ?
The Internet according to Garp (Score:3, Insightful)
(http://slashdot.org/)
To illustrate, if I map routes from, say Chicago, I'm likely to miss the direct connection between Seattle and San Francisco, as there is no traffic I could generate that would take that path.
Re:Lets face it (Score:3, Informative)
(Last Journal: Wednesday October 20 2004, @01:41AM)
Don't feel too bad, the government here (USA) is on your side mainly. I would disagree with you as there is always good money to made here but you have to be creative. The idea is to push each other further to create new ideas and technologies where you can make money.
Dude. (Score:5, Funny)
Web pages are NOT internet hosts.
Web servers are relatively few compared with other types of hosts on the internet.
The World Wide Web is NOT the internet.
The World Wide Web is NOT the internet.
The World Wide Web is NOT the internet.
The World Wide Web is NOT the internet.