Forgot your password?
typodupeerror

Google File System Evolves, Hadoop To Follow 53

Posted by ScuttleMonkey
from the i-wanna-be-like-mike dept.
Christophe Bisciglia, Google's former infrastructure guru and current member of the Cloudera start-up team, has commented on Google's latest iteration on their GFS file system and deemed its features well within the evolutionary capabilities of open-source competitor Hadoop. "Details on Google's GFS2 are slim. After all, it's Google. But based on what he's read, Bisciglia calls the update 'the next logical iteration' of the original GFS, and he sees Hadoop eventually following in the (rather sketchy) footsteps left by his former employer. 'A lot of the things Google is talking about are very logical directions for Hadoop to go,' Bisciglia tells The Reg. 'One of the things I've been very happy to see repeatedly demonstrated is that Hadoop has been able to implement [new Google GFS and MapReduce] features in approximately the same order. This shows that the fundamentals of Hadoop are solid, that the fundamentals are based on the same principles that allowed Google's systems to scale over the years.'"
This discussion has been archived. No new comments can be posted.

Google File System Evolves, Hadoop to Follow

Comments Filter:
  • Hadoop (Score:5, Funny)

    by Jurily (900488) <jurily@NOSPam.gmail.com> on Monday September 14, 2009 @06:15PM (#29419963)

    I wish they would stop taking names from Star Wars.

    • Re:Hadoop (Score:4, Funny)

      by Tablizer (95088) on Monday September 14, 2009 @06:18PM (#29419999) Homepage Journal

      I wish they would stop taking names from Star Wars.

      These are not the names you are looking for.

    • Re: (Score:3, Interesting)

      by Tablizer (95088)

      Speaking of names, check this out from TFA:

      this overhaul of the Google File System is already under test as part of the "Caffeine" infrastructure the company announced earlier this week.

      If they keep naming things with coffee references (including Java), what would happen if it's discovered that coffee causes cancer or shrunken balls or what not? It's already going to affect acceptance in Utah. This is why corporations find bland mean-nothing names like "Teamware" or "Altria" or "Inprise". I personally like

      • Re: (Score:3, Insightful)

        by Korin43 (881732)
        But would you really rather talk about companies with names like that? Google knows their audience. There's the normal people who will use anything that's set as the default, and the nerds who are the ones setting the defaults. Google can't convince normal people to switch (because telling someone to click on the search box and choose Google is "too complicated"), so it makes sense for them to target very specifically at nerds, who will then do their work for them.
      • Re:Hadoop (Score:4, Funny)

        by dakameleon (1126377) on Monday September 14, 2009 @07:21PM (#29420501)

        The day "caffeine" becomes a word that is objectionable to a non-trivial chunk of my customer base is the day I know the PC crazies have won.

      • Stuff 9? As in stuff 9 fingers? That's almost like fisting. Pervert!

      • If they keep naming things with coffee references (including Java), what would happen if it's discovered that coffee causes cancer or shrunken balls or what not?

        Don't have to wait - in the UK one of the more egregious papers regularly publishes a scare story about cancer. So much so that there are sites dedicated to Daily Mail Oncology Ontology.

        Curiously coffee falls into both the good and bad camps [tumblr.com].

        actually it's not that curious - never let consistency spoil a good rant

    • Re:Hadoop (Score:5, Informative)

      by e9th (652576) <e9th@@@tupodex...com> on Monday September 14, 2009 @07:06PM (#29420377)
      This NY Times article [nytimes.com] includes a photo of Doug Cutting, Hadoop's creator (and now Cloudera employee), holding his son's toy elephant, Hadoop.
    • by Jeian (409916)

      I'm actually hearing the Street Fighter 2 announcer yelling "HADOOPKEN! HADOOPKEN!" in my head.

  • by eldavojohn (898314) * <eldavojohn&gmail,com> on Monday September 14, 2009 @06:40PM (#29420189) Journal
    The quoted text seems to be coming from this register story entitled "Google File System II stalked by open-source elephant" [theregister.co.uk], not the one linked in the summary. Also, I can't follow the Firehose link below the story to see if this was changed from the original submission.
  • by XPeter (1429763) *

    Kill M$, take the fat.

  • I thought GFS was a file system only meant to be internally by Google. And if thats the case, then how is it competing with anything?
  • deemed its features well within the evolutionary capabilities of open-source competitor Hadoop.

    I didn't know that file systems were living beings that could evolve. I thought they were inanimate and were designed by humans? Should I be afraid? Is it sentient yet?

    • by FunPika (1551249)
      Not yet, but when it does become sentient...get as far underground as you can before it nukes us all.
  • It Rebelled.
    It Evolved.
    There are many Copies.
    And it has a Plan.

  • by mcrbids (148650) on Monday September 14, 2009 @09:23PM (#29421413) Journal

    I've been on the market for a distributed, clustered file system for some time. Unfortunately, Hardoop is not really what I'm looking for. What I'm looking for:

    1) Redundancy - no single point of failure.
    2) Suitable for standard-sized file I/O.
    3) Performance that doesn't completely suck ass.
    4) Graceful re-integration when bringing a cluster portion back online.
    5) Accessible through standard interfaces. (EG: Posix F/S)
    6) Doesn't require a PHD in the technology to administer.
    7) Doesn't require insane quantities of cash to build.
    8) Stable.

    There are clustered file systems that have some of these qualities. None that I've found so far have *all* of these qualities.

    Hardoop fails on #1, #2, and #6. It has a single nameserver commanding the cluster, so if it goes down, well... (shrug) It also does poorly for "normal" sized files, somehow having a 10 GB file is the norm for Google. And setting a multiple node cluster up is definitely non-trivial.

    Of all that I've reviewed, GlusterFS did the best [gluster.com] but even in that case, I ran into severe over-serialization that brought my 6-node cluster to its knees. I tried three times to roll it out, and had to roll back all three times. I fiddled with the brick setup and caches for days before finally throwing in the towel.

    Now I get by with rsyncing program files, and a homegrown data distribution setup using network sockets and xinetd. Not optimal to be sure, but so far it's scaled linearly and provides decent performance, at the price of a PHD in said technology. I guess you could compare our technology to MogileFS [danga.com], only our scheme

    A) uses DNS records to coordinate the cluster so that it scales up,
    B) has a richer "where is the file" schema than the simple flat keys used by Mogile, and
    C) has the ability to execute programs against files for performance. (EG: grep for searching text files, tar/gzip for compress/uncompress, virus scans, etc)
    D) has the ability to "hang open" for activities like logging.

    So far, this has held up well with about 500,000 file operations and millions of log entries per business day with an average file size of about 1-3 megabytes and every sign that growth can continue by simply stacking on more hardware. No, I'm not talking about massive throughput, but I *am* talking about the need for high availability systems that scale nicely without bottlenecks and exorbitant expense. Yes, it works pretty well, but we've had to invest significant programming time to do this.

    Guess it's like the old engineering saw: Convenient, Cheap, Quality: pick any two!

    • Re: (Score:3, Insightful)

      by Rakishi (759894)

      Hadoop is not really a file system or rather as you found out it doesn't make a good one. It's a framework for doing a certain type of parallel computing (map reduce) on very large amounts of data. There's a filesystem (hdfs) in there but it's pretty much designed for running such parallel jobs rather than being a clustered NAS. The filesystem is in some ways even irrelevant as there's actually support for various filesystems (Amazon S3, etc.).

    • by AlXtreme (223728)

      No, I'm not talking about massive throughput, but I *am* talking about the need for high availability systems that scale nicely without bottlenecks and exorbitant expense. Yes, it works pretty well, but we've had to invest significant programming time to do this.

      Is there any chance your project would be released?

      As you found out there are only a couple of Linux clustering filesystems, all with drawbacks. It would be interesting having a new one designed from the start around reliability.

  • ...it is being intelligently designed.

If I had only known, I would have been a locksmith. -- Albert Einstein

Working...