Behind the Scenes At Google

An anonymous reader writes "University of Wahington TV Presents "behind the Scenes With Google." From the site: 'Search is one of the most important applications used on the internet and poses some of the most interesting challenges in computer science. Providing high-quality search requires understanding across a wide range of computer science disciplines. In this program, Jeff Dean of Google describes some of these challenges, discusses applications Google has developed, and highlights systems they've built, including GFS, a large-scale distributed file system, and MapReduce, a library for automatic parallelization and distribution of large-scale computation. He also shares some interesting observations derived from Google's web data.' "

Google's dirty secret revealed

Google is actually a giant super computer which has become self-aware. Every person it "hires" is actually one more person it saps knowledge from. In the not too distant future, it hopes to be able to network every human completely so that it can collect the remaining knowledge on Earth more easily.

Network everybody together, eh?

Can't wait for the "I'm Feeling Lucky" feature on that one!

Re:Google's dirty secret revealed

I think I saw this on "Star Trek" (and, also, "Futurama" - the "scooty-puff, junior" episode) one time. "Borgoogle : Resistance is results 1-10 of about 200."

What -- I Have To Watch TV Now?

Man, that's *so* twentieth century. I came to /. for the bleeding edge in information acquisition technology: realtime optical scanning blocks of glyphs encoding human language.

I can't absorb information I can't copy/paste.

UW mirror

Also hosted by CS at:

http://norfolk.cs.washington.edu/htbin-post/unrest ricted/colloq/details.cgi?id=274 [washington.edu]

Jeff Dean

Abstract Search is one of the most important applications used on the internet, but it also poses some of the most interesting challenges in computer science. Providing high-quality search requires understanding across a wide range of computer science disciplines, from lower-level systems issues like computer architecture and distributed systems to applied areas like information retrieval, machine learning, data mining, and user interface design. I'll describe some of the challenges in these areas, discuss some of the applications that Google has developed over the past few years. I'll also highlight some of the systems that we've built at Google, including GFS, a large-scale distributed file system, and MapReduce, a library for automatic parallelization and distribution of large-scale computation. Along the way, I'll share some interesting observations derived from Google's web data. Jeff Dean joined Google in 1999 and is currently a Distinguished Engineer in Google's Systems Lab. While at Google he has worked on Google's crawling, indexing, query serving, and advertising systems, implemented several search quality improvements, and built various pieces of Google's distributed computing infrastructure. Prior to joining Google, he was at DEC/Compaq's Western Research Laboratory. He received a Ph.D. from the University of Washington in 1996 working with Craig Chambers on compiler optimization techniques for object-oriented languages.

Few women in CS.

So, I'm always reading about how unfair the tech world is, because there are so few women joining it. But if you watch the video, the audience is surprisingly full of them.

50% female is the goal

When google was recuiting at Georiga Tech they stated that one of their founders had the 'vision' of having half of google female in the near future.

One of the thecnical female googerls mentioned how that was probably impossible, but by shooting for the impossible you acheive a lot more than you would have otherwise.

Google & Backup

I wonder how Google backups its data -- especially the Gmail data. Does the GFS support automatic replication?

WTFV?

Whoa, whoa.. it's hard enough for us to RTFA but now we've got to WTFV (an hour long one too)?

The average slashdotter has an attention span of 5 secon.. ooh look a birdie!

the director...

can anyone confirm that Leni Riefenstahl [wikipedia.org] was behind this film?

Behind the scenes?

Disclaimer: my opinions expressed herein are not necessarily those of Google, Inc.

That having been said, as a long time insider I have a pretty good idea about what really happens "behind the scenes" and let me tell you, both conspiracy theories crackpots and our slashdot fanboys are quite amusing, but the boring fact is that we are neither trying to take over the world, nor are we the best thing since the second coming of Jesus.

We used to be a very successful startup, yes, and now we are a fairly successful corporation. Yes, there are a lot of smart people working here, but don't fool yourself, "the most interesting challenges in computer science" are happening in academia, not in corporations. (Besides, anyone who knows Jeff is perfectly aware that he often tends to grossly exaggerate our importance, but to be honest that is a part of his job which he is doing really great.)

All in all, I love to work here, I thing there are a lot of very smart people here, but if you think that we are the only place on the planet where geniuses cluster lately, you are just not being reasonable. If you want to find real discoveries you have to look in places where people don't have shareholders telling them what to do. The point is that we haven't done anything new per se, only the scale of our implementations is unprecedented.

For example, in my 20% time (Google allows us to spend 20% of paid work time on personal projects) I am working with KeyKOS right now and let me tell you, this is what I call innovation. It was done in the '70s and no mainstream OS has implemented its ideas to this day so far. I'm sure that when after a decade or two a Big Corporation (be it Google, Microsoft, Apple, or IBM) reimplements KeyKOS, the Slashdot crowd will wet their pants screaming "wow, what an innovation!" completely forgetting that it was an innovation back in the '70s of the 20th century when Norm Hurdy et al. were working on it quitely with no buzz and fanfares. Please remember that "The Next Big Thing" is always an old idea but this time backed with $$$ and marketing. Please never forget it, or otherwise the people who are worth their salt will only consider you uneducated.

University Recruiting Talks

Google is constantly giving talks like this at universities. I saw one at Harvard back in the fall.
They aren't really news worth reporting on slashdot, since they all contain the same content.

Equal Time

Hey -- I love Google. Use it every day, and I think they're doing some really neat stuff. But this was an hour-long commercial for Google - -to me it looked designed to recruit from college campuses. While I think it's great that Google does this (it sure sounds like a great way to get cheap qualified labor) is it really new or interesting? Or even geeky? So we have redundant clustering, LISP-like patterns, and issues of dealing with BIG stuff. Hasn't the industry already done all of this, like dozens of times? You can't tell me VISA international doesn't handle this size data, or that General Motors doesn't have some of the same scaling issues. I read somewhere that Wal-Mart has one of the biggest computer systems in the world. To me the signal-to-noise ratio was out of whack to make it worth an hour of my time. Just my opinion folks.

Google innovates? It's news to me.

May be Google has done some nifty things with their file-system, but can't we forget about it already? Their search hasn't changed much http://www.google.com/ [archive.org]">in the past six years. Of course, the fanboys will salivate over Google calculator [google.com] and Google unit converter [google.com], but on the scale of Internet these "innovations" barely register.

Some of the other search engines are comparable in quality to Google (Teoma [teoma.com], Vivisimo [vivisimo.com]), and may be better, depending on how many points you take away from Google for spam-infested results, too many blogs, too many Wikipedia clones, too many commercial sites, etc. And some sites are so much further on the innovation scale (meet BrainBoost [brainboost.com], an artifically intelligent Internet reference desk answering any questions asked in natural English, with amazing quality and accuracy in a very friendly and usable interface) that they put Google to shame.

Re:I use Google at work

Now I have some pretty important lists which I need to keep tight control over. The information really ought not be distributed outside my office. However, because of the nature of my business, I must do frequent searches using various search engines to fill in my lists.

If you want to keep something private, don't put it on the publicly accessible internet. Including searches. Duh.

How am I assured that my searches remain anonymous and secure with Google?

You aren't. Did you sign a contract to that effect? No.

And frankly, if you can find things with google, it isn't too secret.

Re:I use Google at work

You happen to be only one of the millions of people searching for adult pictures online.

You are about as anonymous as it gets.

Re:I use Google at work

a) Don't use Google.
b) Use a different anonymizing proxy for _each_ single search, preferably using SSL.
c) Assume your searches AND non-encrypted web requests aren't anonymous and secure.

If I were running the NSA or some other spook agency, I'd tap the pipes leading to Google (and a few other sites too).

Same if I were a dubious org/agency.

Lots of finance institutions/orgs/ppl get the bulk of their info from just a few sources e.g. Bloomberg. So if Bloomberg gets/sends the bulk of their info down just a few pipes... ;)

Re:mediocre or no Linux support!

Like any tech company, they went with the biggest platform first. Gmail works on non-Windows browsers now. It just took them a while.

Re:GFS

At the time OpenMosix had their own distributed filesystem called MFS. But it's proved inadequate, which is why they are switching to GFS

I'm sorry, did I miss the point at which Google made an open source implementation of GFS? Last I knew, the only docs for GFS were the papers that Google published on the concept. And those papers (unfortunately) seemed to lack a few of the finer details of implementation.

Re:GFS

Ok, I looked it up. You're confusing Sistina's (now Red Hat) Global File System with the Google File System. The two ARE NOT THE SAME.

Here's Red Hat:

http://www.redhat.com/software/rha/gfs/ [redhat.com]

Here's Google:

http://www.cs.rochester.edu/sosp2003/papers/p125-g hemawat.pdf [rochester.edu] (PDF)
http://64.233.161.104/search?q=cache:m0TMQYgIlIoJ: www.cs.rochester.edu/sosp2003/papers/p125-ghemawat .pdf+Google+File+System&hl=en&client=safari [64.233.161.104] (HTML)

Re:GFS

I mean, they are both distributed filesystems with the same name. What are the odds? ;)

Considering that it's in vogue to name file systems with one letter in front of "FS"? About 1 in 26. The odds are even better if you discount commonly used file systems such as XFS, UFS, FFS, NFS, and JFS.

Re:Fsking video format.

$ man mplayer /dumpstream

Download the .asx File, look inside. This is your URL. Have fun.

Re:Fsking video format.

Please explain how these programs provide patent-free, Open Source, non-crappy video codecs.

Dirt? That more like modelling clay

Given the bias of the site if that's all the dirt they can dig up, Google must be a pretty good company, and/or the people at that site are just crap at digging up dirt.

Think about it, if someone really hated any of the Fortune 500 companies and bothered to dig up some dirt, there'd be tons more dirt.

I suppose Google is a young company. Give it a few more years and more parasites would have found their way into Google. Then you'd have a lot more dirt.

Re:G4/TechTV

Discovery channel is a shadow of its former self. They used to actually show science programs. Now all of their programming is merely Hotrod this or that.