Follow Slashdot stories on Twitter

 


Forgot your password?
Close
typodupeerror
×

Amazon's Werner Vogels on Large Scale Systems 49

Posted by ScuttleMonkey from the horses-mouth dept.
ChelleChelle writes "When it comes to managing and deploying large scale systems and networks, discipline and focus matter more than specific technologies. In a conversation with ACM Queuecast host Mike Vizard, Amazon CTO Werner Vogels says the key to success is to have a 'relentless commitment to a modular computer architecture that makes it possible for the people who build the applications to also be responsible for running and deploying those systems within a common IT framework.'"
This discussion has been archived. No new comments can be posted.

Amazon's Werner Vogels on Large Scale Systems More

Amazon's Werner Vogels on Large Scale Systems

Comments Filter:

  • Good Computing (Score:3, Funny)

    by Anonymous Coward on Monday July 24, 2006 @11:35PM (#15773783)
    Best question in the interview (page 3):

    "MV: Given the size and scope of Amazon, there's a lot of talk in the industry about good computing. Most of the talk is around scientific applications. Do you see good computing playing a role at Amazon in the future?"

    Actually, no... Amazon likes bad computing.
     

  • scale? (Score:5, Informative)

    by lecithin ( 745575 ) on Monday July 24, 2006 @11:38PM (#15773790)

    "When it comes to managing and deploying large scale systems and networks, discipline and focus matter more than specific technologies.

    How about:

    When it comes to DOING ANYTHING, discipline and focus matter more than specific technologies.

    If you are at a 'small scale' environment and are limited to specific technologies, discipline and focus matter even more. Your choice is less with technologies and more with how you use them.

    "the key to success is to have a 'relentless commitment to a modular computer architecture that makes it possible for the people who build the applications to also be responsible for running and deploying those systems within a common IT framework.'"

    We have a BINGO!!!!!

    • scale? (Score:5, Informative)

      Oh dear, and here was me thinking it was merely insightful, but apparently most people here at slashdot didn't already know this.

      -Qyiet

      • Re:scale? (Score:2, Funny)

        by kfg ( 145172 ) *
        . . .apparently most people here at slashdot didn't already know this.

        They're all busy trying to integrate Java and XML.

        KFG

    • Re:scale? (Score:3, Interesting)

      by syousef ( 465911 )
      "the key to success is to have a 'relentless commitment to a modular computer architecture that makes it possible for the people who build the applications to also be responsible for running and deploying those systems within a common IT framework.'"

      This works well sometimes. The developer supporting their own application. For other things it makes more sense to divide the role. My experience is that the more complex and customised the software, and the more quickly it is changing, the more important you ha

  • his comments are only 1/2 true (Score:3, Interesting)

    by Anonymous Coward on Tuesday July 25, 2006 @12:21AM (#15773901)
    Werner's comments are only 1/2 true. While many of the things he deals with are website centric, there is a whole world behind the website. No one buys from Amazon because of the website necessarily, they buy because the fulfillment is (mostly) accurate and fast. These backend systems are not nearly as clean as werner indicates.

    Myopic vision on his behalf imo.
    • Myopic vision on his behalf imo.

      The thing I find more amazing than an anonymous coward claiming to know Amazon's backend better than Amazon's chief technologist does, and saying "omg your system isn't nearly as clean as you say it is," is that other people at Slashdot are actually moderating him up.

      I've actually seen Amazon's backend codebase, and it's remarkably cleanly modular. Whereas I suspect that cleanliness has been built progressively over the years, it becomes quite clear that Amazon believes deep

  • Share nothing architecture? (Score:4, Interesting)

    by QuantumFTL ( 197300 ) * on Tuesday July 25, 2006 @12:35AM (#15773938)
    I've been programming for many years now, but I'm new to web-app development. I've been learning Ruby on Rails (for various reasons) and one of the points the book I'm reading makes (Agile Development with Rails) is that good scalability is best achieved through the use of a "share nothing" architecture - basically reduction of chokepoints by reduction of shared content in a system.

    I'm studying this as I'm looking at scalability concerns in an app I'm putting together, and I did a google search on the topic, but the only thing of interest I could find was this article [zefhemel.com], which doesn't really go into the downsides of this approach. What does slashdot think about this?
    • If I were you I would not worry about scalibility too much. At this point ROR is in production taking 3 million hits a day in at least one production environment. Worrying about scalibility at this stage is premature optimization. You may find that once your application gets to be so popular that ROR can't cut it anymore. More likely you will never even come close to hitting that wall.

      Just code using the ROR conventions and you should be fine.
    • You'll be able to find a lot more about this technique if you search google for the (quoted) term "shared nothing"

    • Shared nothing is useful but overhyped (Score:4, Informative)

      by Stu Charlton ( 1311 ) on Tuesday July 25, 2006 @08:45AM (#15775313) Homepage
      The best resource (though getting dated) on the origins and meaning of shared nothing v. shared-something archticture is Greg Pfister's In Search of Clusters, 2nd ed. [amazon.com].

      There's "degenerate" shared nothing, which is what I find most people referring to today -- you have web server farm and you don't store session state, or if you do, you "pin" it to a particular server. Or you just rely on the database. It's degenerate because, sure, it's scalable (memory isn't as directly linked to concurrent users), but it really just shifts the burden to the database, which tends to be 1 big box.

      So the question becomes, how do you scale the database horizontally?

      In the database world, the term has become somewhat overloaded. Originally it meant physically shared disks and/memory vs. using network interconnectivity. But with the rise of I/O shipping technologies over networks (iSCSI, high speed NFS/NAS, SAN fibre-channel), this isn't really true anymore. So now, it comes down to how your data is partitioned and how you ship a read/write function to that node. Does a node "own" it's data (or a replica)? Or can any node touch any data? That's the debate.

      In short, it works well in some cases: read-mostly parallel queries and/or search, which is why Google's using it, or why you see it with data warehouses (Teradata, DB2 UDB). It works OK if you have mostly have transactional data updates within a well-defined partitionable set of data (such as the TPC-C benchmark). It works less well when dealing with transactional updates spread across the entire data set (assuming a normal distribution), as you'll need to update replicas with a two-phase commit. The load balancing of your data across nodes also requires care in picking the appropriate partitioning key: sometimes a hash works well, sometimes range-values work well. If you need to re-partition your data for whatever reason, it's going to be a big job.

      Commercially, Oracle 10g's Real Application Clusters is an example of a shared disk database, though they use an interconnect between nodes for cache coherency. Microsoft SQL Server, DB2, Teradata, MySQL, etc. are all "shared nothing".
      • Excellent, this is precisely the kind of answer I was hoping for. I've done some fairly interesting things with databases in the past (including a rather odd sort of partial replication - for security purposes), and while I'm up on much of the theory of these things, I don't have much practical experience. Do you have any books you can reccommend that would cover the issues of access to redundant databases, and this partitioning (which from what you said seems to be a method of load balancing, akin to how
        • I don't think "shared nothing" means "everything shares the same thing." If it is a buzzword used to sell databases, sorry, I didn't know. But the whole idea of "autonomous" systems, at least in the sense that Amazon (refusing to accept the label) does, is that even databases aren't shared. App 1 might collect user data and store it in a postgres database. App 2 might collect order data and store it in an oracle database. App 3 might keep track of referers in a flat file (perhaps apache's access.log) A
          • Amazon, interestingly enough, uses a LInux/Oracle RAC data warehouse, for data analysis. It is one of the larger deployments in the world, at over 15 terabytes. I found this article [adtmag.com], but there have been many over the years, some on Oracle's site.

            And, to re-iterate, Oracle RAC is shared-disk... ;-)
      • but it really just shifts the burden to the database, which tends to be 1 big box. // So the question becomes, how do you scale the database horizontally?

        There really aren't any major databases which don't cluster, and some architectures like Mnesia and KDb are built specifically to handle being the generic workhorse behind data requests. That said, what you're discussing isn't shared nothing - it isn't even degenerate shared nothing. If it was, then there wouldn't be any state to share. Moving the share
        • That said, what you're discussing isn't shared nothing - it isn't even degenerate shared nothing. If it was, then there wouldn't be any state to share.

          This was partially my point, though I was wrong in saying it was "degenerate". What I meant to say is that few actually implement shared nothing -- they say they do, but they really don't. And the reason is mostly due to the tradeoffs between fault tolerance and concurrency management that make it difficult to use for transactional data management.

          Microsof
        • Thanks for the pointer Mnesia, great reading.
    • Do yourself a favor and investigate other methodologies than RoR. There's a reason that the same people who called ASP.Net an unimportant flash in the pan, who called ColdFusion a dead technology and who laughed when Oracle said the Web would make them the new Microsoft are yawning in the faces of RoR, Python and Nails. Sure, it's possible that RoR is the next big thing, but if you look at the track record of the people being unimpressed, it seems likely that being familiar with traditional, proven method
      • For me right now, RoR is a great prototyping system... I'd probably use J2EE in real life, but it's also possible that RoR will pan out. If nothing else, it's great for internal webapps that need set up quickly and don't need to scale.
  • This is a dupe of a story run in May:

    http://slashdot.org/article.pl?sid=06/05/17/045320 8 [slashdot.org]

    I was RTFA and thought it looked mighty familiar - that or DeJaVu.

  • 'Duh' (Score:1, Insightful)

    by Dryanta ( 978861 )
    Even after RTFA, the whole thing seemed like the guy sucking himself off.

    WV: Yeah. So, I think there are two answers to your question there. The first part addresses autonomic computing, which is I think just like Web services or service-oriented architecture, one of those buzzwords, you know?

    One of those 'buzzwords, you know?' was your entire interview buddy. Imagine that? Scalability is achieved through many different technologies with many different engineers? I would never have thought that. I guess you

    • Vogels was a distinguished academic in distributed systems prior to Amazon. Read his blog [allthingsdistributed.com] some time. He is quite insightful, and this queuecast was a great one. Yours is the first comment I've seen in many forums over the past few months that seems to think it was tripe, so I find it curious.

      His point is that Amazon has found that a decentralied archtiecture that can work reliably but still respond to new demands with agility. That's a huge deal, considering the contortions, pain, and centralized bottl
    • Even after RTFA, the whole thing seemed like the guy sucking himself off.

      He didn't seem that way to me, though you rather do. Your post seems to be there essentially to make you feel smarter than him.

      Scalability is achieved through many different technologies with many different engineers? I would never have thought that.

      That's not actually what he said. What he really said was that the specific path to scalability during constant change which had worked well for Amazon was to maintain absolute modularity

  • Yep: you support what you wrote (Score:2, Informative)

    by Anonymous Coward
    Under normal circumstances, at Amazon you'll have to support what you wrote. That means if your code crashes all the time, you'll get paged in the middle of the night.

    Now, even if you get rid of some incompetent programmer (say by moving him to another team), the rest of the team will still get bogged down with supporting the code he wrote. And since engineers now have to do support for the other teams using their service, their productvity eventually grinds down to a halt and new development becomes extr
  • Just as Amazon was using SOA long before it was named, the same is true of DBMS2. [dbms2.com] Add that to SAP's adoption, [dbms2.com] and we're getting somewhere. :)

  • ...in 37 signals [37signals.com] (the guys who made Ruby on Rails [rubyonrails.org]), I wouldn't be suprised to see some more RoR happening there as some commenters here have suggested. Good times...
  • I think YOU missed the point Stu.

    Yours is the first comment I've seen in many forums over the past few months that seems to think it was tripe, so I find it curious.

    I'm not saying that his commentary is 'tripe' as you say, so much as bonehead obvious and not news-worthy. He managed to talk for three pages about how 'varied technologies and disciplines with a varied engineering staff makes Amazon worx lawlz!' I believe he is scholarly - otherwise he would not be able to ramble on about nothing for so lon

  • At Java One this year there was a pretty interesting talk by a guy from Ebay, talking about "creating an object model that spans the world". If you join the Sun Developer Network (for free) you can get the slides, and even audio/video I believe.

Slashdot Top Deals

A morsel of genuine history is a thing so rare as to be always valuable. -- Thomas Jefferson

Close