I don't see what would prevent you from screen scraping the archive, and bring our history with us to the new site.
url_pattern = "slashdot.org/archive.pl?op=bytime&keyword=&year=&page="
for year in (1998 to 2014):
thread_page = 1
while true:
url = url_pattern.replace(").with(year).replace("").with(thread_page)
threads = extract_threads( fetch(url) )
if no threads:
break
for thread in threads:
comment_page = 1
while true:
comments = extract_comments(thread + "?page=" + comment_page)
if no comments:
break
append_to_database(comments)
comment_page += 1
thread_page += 1
How long would it take?
The Slashdot archive spans 16 years.
Let's say there is an average of 35 pages worth of threads each year.
Each such page of thread links takes about 7 second to load.
In an hour you should be able to get all thread URLs.
Let's further assume it takes 4 seconds to load a comment thread and that there are 270 threads URLs per page.
That gives us a total of 16 x 35 x 270 ~ 150000 thread pages.
Getting all those threads so you can extract the comment related data and meta data would then take about a week (7 days).
Finally, lets assume each discussion thread has two pages, so we double that time. Still it's only two weeks of linear scraping to get the entire slashdot archive.
Now if you parallelize this, it'd naturally go much quicker. A few hours of scraping is all that's needed, or a bit longer if we don't want to put undue stress on the site (being good netizen).
Go for it I say, before the PHB DICE folk realize that they bought a platform for technical people by technical people and start removing useful features from the site (like the archive).
