Google's Academic TB Swap Project

eldavojohn writes "Google is transferring data the old fashioned way — by mailing hard drive arrays around to collect information and then sending copies to other institutions. All in the name of science & education. From the article, 'The program is currently informal and not open to the general public. Google either approaches bodies that it knows has large data sets or is contacted by scientists themselves. One of the largest data sets copied and distributed was data from the Hubble telescope — 120 terabytes of data. One terabyte is equivalent to 1,000 gigabytes. Mr. DiBona said he hoped that Google could one day make the data available to the public.'"

Should we be continuing this fallacy?

[garcia]

One terabyte is equivalent to 1,000 gigabytes.

Uhh, no it isn't. It's really 0.9765625 terabytes.

Large datasets

[BWJones]

This is absolutely the most cost effective way of transferring large amounts of data like this. If you do the calculations on terrabyte size files, sneakernet (of FedEx net) is actually faster and less expensive. We also went to one of Jim Grey's seminars when he was here giving an Organick Memorial Lecture and he made an incredibly compelling demonstration using a variety of data types. We ended up talking with him for some time after about new projects we are engaging in that will also be generating terrabytes of data and his suggestion was to pass applications rather than data which was interesting.

This is becoming more and more the norm in scientific research and Google's work is quite welcome.

Re:Should we be continuing this fallacy?

[wizzard2k]

From wikipedia:
(a contraction of tera binary byte) is a unit of information or computer storage, abbreviated TiB.

1 tebibyte [wikipedia.org] = 240 bytes = 1,099,511,627,776 bytes = 1,024 gibibytes

The tebibyte is closely related to the terabyte, which can either be an (inaccurate) synonym for tebibyte, or refer to 1012 bytes = 1,000,000,000,000 bytes, depending on context.

Re:Should we be continuing this fallacy?

[garcia]

Thanks for pointing out that I should have been hitting Preview instead of getting First Post :)

1000GB = 0.9765625 TB, not 1TB.

Mod parent up

[ari_j]

Here's what happened when I FedExed my RMA to Newegg, packed very carefully. Note the bent motherboard - I didn't even know you could do that. The good news is that FedEx paid part of my claim ... they paid $100 plus the $8.33 that the FedEx store charged me to fax in the claim forms. The bad news is that they did not refund my original shipping or pay more than $100 on the over $280 of damage that they did. It also took about 4 hours of phone calls to even convince FedEx that I was not the seller, and then they lost my claim in their e-mail system (and did not reply to my e-mails) and closed it out for inactivity after a month or so, until I called them and asked what happened.

• box front [theari.com] - doesn't look bad from this angle
• box rear [theari.com] - run over by a truck?
• formerly a CPU [theari.com] - this was packed inside the box with lots of noodles
• motherboard top [theari.com] - not terrible-looking, but note the USB/Ethernet header
• motherboard end view [theari.com] - here you can see the curvature of the motherboard
• another picture of the CPU and motherboard boxes [theari.com] - almost certainly run over by a truck

On a side note, don't bother with UPS insurance. I insured something when I sent it to myself once, and they broke it and the insurance remedy was to return it to the origination address and ask to see an original purchase receipt to award the insurance claim. If you happened to make something yourself or even received something as a gift, don't insure it when you ship it. And hire a private courier (unless someone has found a common carrier that doesn't suck).

Re:so..

[oneiros27]

So, if Google takes the raw data and does that color assignment itself, well, the result is theirs.

I'm not so sure that the result in theirs, necessarily. They'd need to properly attribute it. Many science archives have rules about how to properly attribute their work.

Don't get me wrong -- many of the scientists want people to use their data (eg, see The Astronomer's Data Manifesto [ivoa.net]), but they also want to know who's using it, because it's how they justify the value of their projects, and the costs incurred from distributing the data (especially for non-active projects).

The science community is also working on the Science Commons [sciencecommons.org] (an equivalent of the Creative Commons for marking scientific data) and various federated search engines (eg, night time (astronomy) virtual observatories [wikipedia.org], as well as other space and earth science discipline specific VOs. [nasa.gov]).

Re:Large datasets

[Agent Orange]

Yup. There was a paper a few years back entitled "terascale sneakernet", by jim gray and a couple of guys at MSFT research division on this. You can find it in the arxiv [arxiv.org].

This concept has also been applied to such things as the Sloan Digital Sky Survey [sdss.org]. Astronomers do tend to generate a lot of data with large surveys such as this.

Nope

[sheldon]

How you measure a terabyte depends on whether you are buying disk, or monitoring disk usage on your server.

The disk manufacturers define it as 1000 megabytes which is 1000 kilobytes which is 1000 bytes.

The OS measures it as 1024 megabytes, which is 1024 kilobytes, which is 1024 bytes

Why? Because when you're buying a drive, 750 Gigs sounds bigger than 698.5 gigs.

Re:1TB = 1024 GB

[91degrees]

Well, the IEC and IEEE as well as the CIPM and NIST all agree thatthere are 1000 bytes to a Kilobyte and 1024 bytes tothe kibibyte. So there:P

Re:Mod parent up

[MajinBlayze]

As a former UPS employee, (I worked as a package handler, the guy that beats the shit out of your boxes as he loads them on the truck) I will never ship anything of value without paying extra for the insurance. when you do that, a couple of things happen:

1. the item goes into a big bag (by itself, not mixed with other items) with red/white stripes, so employess know not to mess with it)
2. it gets hand-carted to the destination truck, and is the last thing to be loaded, and first unloaded
3. only seasoned workers ever touch your package, and generally care about the state that it's in
4. finally, they are good about paying up if the item arrives damaged.

did I forget to include ???? and Profit!

Re:Mod parent up

[evilviper]

Besides, insurance is meant to cover damage due to normal mishandling, such as dropping a box by mistake, not the kind of (at least nearly) intentional damage that must have been involved in my case. Or maybe you have a theory of how my box got squashed that badly in the normal course of FedEx's business.

I still don't know where you get that idea. Insurance is meant to handle any kind of damage, including being completely destroyed in plane crashes, car accidents, train derailments, theft, loss, and anything else that could possibly occur.

For your package, I imagine it was run over by one piece of equipment or another. Forklifts, tractor trailers, etc. Or it may have been some sort of freak accident with equipment in their automated package handling system. I certainly don't have any reason to believe it was intentional, unless you have some reason to believe you've seriously pissed-off your local fedex office employees beforehand...

Re:Should we be continuing this fallacy?

[jonbryce]

The other primary place where the prefixes are in use is RAM chips, and they do use 2^10 rather than 10^3.