Become a fan of Slashdot on Facebook

 


Forgot your password?
Close
typodupeerror
×

The Amazon Technology Platform 109

Posted by ScuttleMonkey from the under-the-covers dept.
Don420 writes "Jim Gray has an interview with Amazon CTO Werner Vogels for ACM Queue. It is filled with a lot of details about the Amazon architecture that we have not seen before: 'If you hit the Amazon.com gateway page, the application calls more than 100 services to collect data and construct the page for you.' But also quite a strong statements about developing software at Amazon: 'Developers of our services can use any tools they see fit to build their services. [...] Whatever tools are necessary, we provide them, and then get the hell out of the way of the developers so that they can do their jobs. [...] Developers are like artists; they produce their best work if they have the freedom to do so, but they need good tools.'"
This discussion has been archived. No new comments can be posted.

The Amazon Technology Platform More

The Amazon Technology Platform

Comments Filter:

  • 100 Services ? (Score:4, Insightful)

    by jonv ( 2423 ) on Wednesday May 17, 2006 @04:45AM (#15349306)
    'If you hit the Amazon.com gateway page, the application calls more than 100 services to collect data and construct the page for you.'

    and this a good thing ?

    • Re:100 Services ? (Score:5, Informative)

      by shmlco ( 594907 ) on Wednesday May 17, 2006 @04:55AM (#15349332) Homepage
      Ever considered the amount of data that has to be churned through to build your average custom My Yahoo home page? Especially one with a ton of custom news items, stocks, local weather, local movie listings, and so on?

      Major web sites are just a "little" more complex than your typical iWeb home page...

      • Ever considered the amount of data that has to be churned through to build your average custom My Yahoo home page?

        Considered it? Heck, I coded one such system!

        I also paid to host it for awhile, which is exactly why I made sure to take full advantage of the viewer's cache.

        Amazon is huge! Redundant data wastes bandwidth. Bandwidth costs money.

        C'mon Jeff (Bezos)! You are a money guy! You know should know better...

      • Err , so what? (Score:1, Offtopic)

        by Viol8 ( 599362 )
        You seriously think a web page its the pinnacle of large data access?
        I work in a bank where we process terabytes of data a day, and no we
        don't use 100 services everytime we need to access a load of data. We
        use a few f*ck off big RDMBSs and some Big Iron.

        Don't make out that serving up a web page is a big deal , compared to
        real hard code data processing its mickey mouse.
      • Ideally the hundred of services need for the 'my amazon' type services should not be called from the home page - only when they are required.

        Perhaps amazon has looked at how users interact with the site and find they goto these options straight away - more likely the marketing department wants to present the user with lots of stuff they might like to buy as soon as possible.
      • > considered the amount of data that has to be churned

        :%s/data/crap/g[Enter]

    • Re:100 Services ? (Score:2, Insightful)

      by simonjp ( 970013 )
      It didnt say that it collected 100 pieces of personal data etc from your computer, more likely its all internal requests :) I mean it usually provides suggestions on your past orders, what pages you've recently viewed in the system (which isnt exactly personal data being collection IMO as its all "internal" to Amazon.

      But the question is does this make for a better shopping experience?

      When I've used Amazon it's always been helpful.

      However, Amazon UK seem to be a little less reliable in shipping recently.
    • Not 100 external services... 100 internal services, created and hosted on Amazon servers. Essentially they have compartmentalized their software development so that each piece of functionality is a black box, providing its content via a public interface so that changes to that component can be made without changing the home page at all (which gets thousands of hits per second).

      In other words, they're using good software design techniques to minimize interdependancies and maximize code reuse.

      Yes, it is a goo
    • This concept, of specializiation of tools, is championed on the UNIX command line and if it can be used efficiently is should be championed for web-development. Why not have a bunch of specialized services that you can quickly combine in any way you like instead of having a single monolithic web service?
    • The implied statement is that "the webpage calls 100 different services, and yet still renders in about the same amount of time that a plain ole HTML file would render." And that is a good thing, because it means the many service calls don't take long to complete.

  • Unlimited budget (Score:4, Funny)

    by DonCarlos ( 222830 ) on Wednesday May 17, 2006 @04:45AM (#15349307) Homepage
    Having unlimited development budget is definitely THE good thing I sometimes miss myself ;)

  • Sounds (almost) too good to be true... (Score:5, Interesting)

    by allanj ( 151784 ) on Wednesday May 17, 2006 @04:47AM (#15349312)
    I wonder if the actual developers/coders see it that way themselves. Sadly, CxO's often have a warped view of how things work "on the floor".
    • While I don't see myself as an "Artist" (not since my Subway Sandwich Artist days anyways), I do consider myself a professional that knows more about what I need then the average CxO or boss in general does. Which is why I'm glad I work for a company that gives me the freedom and benefit of giving me a task and then staying the fuck out of my hair. I know my job, I know what the boss wants, I know when the boss wants it, and as long as the boss don't screw me around then the boss will get what he wants.

      Ca

      • No offense, but the management side of me just toted up the total employee carry cost for 6 months at somewhere between 100k and 200k (depending on benefits and taxes and overhead) and immediately wanted to see a product evaluation for test automation tools in that price range.

        The maintenance costs on the licenses will probably be a wash compared to the in-house maintenance costs on a tool that takes half a year to build and will most likely be dwarfed by the cost of new test creation anyway

        Classic buy-vs-b
        • Keep one thing in mind. You've gotta put time into building the scripts if you want good automation testing, whether you buy or build. I was going to spend as much time using a pre-packaged tool to write scripts as I've so far spent writing scripts while building the tool. Money-wise, you're looking at $100,050 thousand as opposed to $100,899 spent on regression testing at this point, relatively speaking. Plus the knowledge I've gained of Perl scripting can be transfered to other areas of the company (a
          • That is a solid defense, and your reasoning sounds valid. Your original post just hit my "NIH" detector pretty hard and for some reason I felt like saying something.

            I had actually thought the same of web app test automation tools (mostly crap, expensive, brittle) until a colleague of mine started working with Selenium [openqa.org]

            It blows me away - he's able to code up tests extremely quickly and he has access to the Javascript DOM so the tests aren't that brittle. He said it takes around 1-2 hours per test. Granted, he
  • Of course, Amazon can't be around for much longer at the rate they're cutting it down.

  • Thanks (Score:1, Interesting)

    by Anonymous Coward
    "Developers are like artists" is exactly what these primadonnas needed to hear. No, developers are nothing like artists. Creating something that didn't exist before doesn't make you an artist. Builders create something new every day. Developers are more like inventors if you give them a free hand in what they develop, but most of the time developers are very much like builders: They create what you ask them to create.

    Or maybe I got you wrong and you meant that developers are like artists: Poor, starving, li
    • "only valued once they are no longer available."

      You've never heard of COBOL then I take it?
      • I've heard of KOBOL, its some ancient planet where mankind supposedly came from before their fall from grace with the gods, on KOBOL they lived with them but after their fall they decided to leave in 12 tribes from KOBOL for 12 planets which later became the "12 Colonies of Mankind". There was also supposed to be a 13th tribe which came here and settled on earth.

        Anyway, I'm going off the point there, KOBOL in the end was left deserted and the gods buggered off at some point as well. There is a series bein
    • Creating something that didn't exist before doesn't make you an artist. Builders create something new every day.

      Builders don't design what they are building. Most developers are like architects (the kind that design buildings). Sure "they create what you ask them to create", but so do architects. You give them specs and a budget, and they design your solution. The difference is that software developers ignore the 'budget' part. Note that the 'Most developers' statement excludes code monkeys who are g

    • Some developers are like artists, some are not. Ever seen a developer looking at his/her code with dreamy eyes and a stupid smile? Yes I say! So some are like artists. Sure they create what you want them to at times but there's many ways to do the same thing...including a 'beautiful/elegant' way!
    • Interesting how you put that AC.
      If it's not an art to invent, why does it seem so hard for people to be able to. The same basic elements of inspiration and an understanding of your media are necessary. Sure, a lot of compscis are USED like builders, but to develop something new isn't like the builders you mentioned, so much as the designer or architect creating the idea.
      Also, any true coder knows what an art it is to debug efficiently. On that level, I'm sure there are parallels to your builders.

    • Re:Thanks (Score:4, Insightful)

      by stlhawkeye ( 868951 ) on Wednesday May 17, 2006 @10:42AM (#15351338) Homepage Journal
      Or maybe I got you wrong and you meant that developers are like artists: Poor, starving, living for their work and only valued once they are no longer available.

      Move away from the coasts. I make $75,000/year working 40 hour weeks. I'm not on-call, have flex hours, get 3 weeks of vacation, and unlimited sick time. Quit working for IT sweat shops. Move somewhere where family time is valued and it's impossible to hire people unless you are willing to give them that flexibility. I've been through four employers in the St. Louis area and been able to land jobs with a deal akin to this one at all four. Developers are "poor"? No. Elementary school teachers are "poor." Starting salary for a developer in a low-watt market is close to $40K without a degree. That's not "poor." That's not starving, and that's not living for their work, unless by "living for their work" you mean that you're expected to show up on time and do your job.

    • Obviously you are not a developer so your opinion is incorrect and uninformed; hence the anonymity.

  • But they need good tools (Score:4, Informative)

    by wysiwia ( 932559 ) on Wednesday May 17, 2006 @05:13AM (#15349386) Homepage
    Sure anybody need good tools to produces something exceptional. But what can you do if the needed tools aren't available? What can developers do if they aren't happy with their tools or their environment?

    For users the answer is easy, they simply switch to something different, but for developer it's not. You usually first have to get a lot of knowledge which needs time. But one does never get more time!

    So developers have to think in advance sometimes several years. This means constantly be on the edge of the available knowledge. Tools can certainly help but nothing prevents you from getting the knowledge in advance.

    O. Wyss

  • Artists? (Score:5, Funny)

    by should_be_linear ( 779431 ) on Wednesday May 17, 2006 @06:12AM (#15349529)
    Well *Some* developers are like Artists, others are more like Naïve painters with unlimited budgets for colors and huge canvases.

  • Tools are all well and good... (Score:4, Interesting)

    by dtsazza ( 956120 ) on Wednesday May 17, 2006 @06:23AM (#15349569)
    ...but at the end of the day it's the developer's talent and experience that tell most of the picture. It sounds like Amazon do let their developers decide (to a large extent) how their products are going to work.

    The transition from the monolithic Obidos to the current SOA makes me wonder how exactly that part of the system works. Though it's not (that I could see) explictly stated, it certainly seems that adding scalability was a long and painful process. Planning for future developments like this is something that developers tend to be much better at than managers - so I wonder whether the developers didn't think about including scalability hooks in their initial efforts, whether they decided (back in the early days) that it wasn't worth it, or whether they wanted to but were told not to bother.

    All said, I do applaud the public stance that Vogels is taking in his attitude. If more CxOs shared it, we'd likely have beeter-designed systems all over the shop. You hire the developer because (s)he's good at developing - so let them go to it!

  • Good Tools? (Score:1, Insightful)

    by Anonymous Coward
    My father always said it was a poor craftsman who uses his tools as an excuse.
    • I guess you've never had to program in PL/SQL!

    • Re:Good Tools? (Score:3, Informative)

      by mahangu ( 576268 )
      There is an idiom in my native tongue of Sinhala for this - natanna bari minihata polova adai, which translated directly (albeit a little clunkily) reads the ground is always uneven for the person who can't dance.

      Having said this, I'm sure everyone agrees that a certain amount of tools are necessary to be productive. All in all though, I think this article [nidahas.com] sums up the value of tools pretty well.
  • application calls more than 100 services to collect data and construct the page for you.

    No wonder it takes so long to load
  • and then get the hell out of the way of the developers so that they can do their jobs. [...] Developers are like artists; they produce their best work if they have the freedom to do so, but they need good tools


    gee, i wish my boss(es) was/were like that
    just so we didn't have to end up doing
    half-assed jobs with half-assed resources
    ending up in half-assed products...
    (and maybe half-assed revenue?)

    oh well...

    • Is the other half of your ass posting on slashdot, then? ;)
      • Is the other half of your ass posting on slashdot, then? ;)

        yeah, most of the time ;)
        maybe thats why asses have two cheeks
        so we can choose to be half-assed on will? :)
        well, myself being a half-breed born in japan
        between a japanese mom and an american dad
        just might help my cause too ;)

        <beer>cheers!</beer>

  • My question is (Score:5, Interesting)

    by Aceticon ( 140883 ) on Wednesday May 17, 2006 @06:54AM (#15349689)
    Developers of our services can use any tools they see fit to build their services.

    I wonder how they avoid the maintenance nightmare which is having multiple application components done using various obscure technologies/tools and the person that did it leaving the company and somebody else having to maintain/extend those application components?

    Do they standardize their build tools, require good documentation on the service implementations or just overwork the poor sods that have to do maintenance to death?

    • Re:My question is (Score:1, Insightful)

      by Anonymous Coward
      I worked at Amazon. It was a nightmare. And I can say this: absolutely none of the above. Amazon tools are largely undocumented, not requiring any kind of standardization on platform/libraries/processes, are totally heterogeneous and are often scary bits of random code floating out in the ether.

      This attitude sounds like it's great, but something that this attitude absolutely encourages is non-standardization and a huge lack of planning for the future. They encourage you to go 'just do it' - and then 10 diff

    • Re:My question is (Score:1, Insightful)

      by Anonymous Coward
      I currently work at Amazon, and typically most projects aren't done using obscure technologies, they're usually pretty standard technologies. However, there are a *lot* of standard technologies, and every project really does use a different one. So if you're on one team and you're using Java/Spring/Hibernate, you may move to another team and be using Perl, or be using Java/Struts/SomethingElse.

      Good documentation, however, is something that is definitely *not* required and something that I hope will change i

      • Re:My question is (Score:1, Insightful)

        by Anonymous Coward
        I started at AMZN as a dev - hoped to do development of the kinds I never did. And I'm not saying I didn't. However, in retrospect, what I think fucked me up the most is my inability to estimate the time needed between being on a pager 24 hrs/day, 6-7 days out of every month and the time needed to maintain/fix old legacy (perl-based mostly) code - which left me with little time to do any project work. I suppose I've got no one to blame but myself for not knowing how to predict time required for projects, h
  • Wow their CTO is a lot more in touch with development than our CTO... :(
  • 'nuf said.
  • I'll preface my comments by stating that I regularly order through Amazon.com and have never had a customer service-related problem.

    This interview, while I'm sure sincere on the part of the CTO, comes across as a recruiting pitch. Obvious fallacies:

    "Developers themselves know best which tools make them most productive and which tools are right for the job."

    This sort of development mishmash depends on the developers never leaving (which most do after 2-3 years). Maintenance is, at best, nightmarish and lead

    • Re:Recruiting interview? (Score:4, Interesting)

      by AuMatar ( 183847 ) on Wednesday May 17, 2006 @12:22PM (#15352167)
      Wow, troll much? Working at Amazon currently, I'll comment on a few of these things:

      1. Low pay - it's a great place for budding "artists" fresh out of school to build "experience" that has to be unlearned at a more organized shop.

      Not at all. I'm making 15% more here than I did at HP last year. I make more than devs with equal experience at MS. The pay is pretty good.

       
            2. Dot-Com mentality - Lots of excuses for your desk being a door on two filing cabinets as well as the lack of organization.

      THe door desk is aq culture thing. THere's a whole story behind it. Think of it like a tradition a sports team has. In the end, it works well- its not pretty, but its sturdy and does its job.


            3. Turnover rate - as soon as people get experience, they leave for better paying jobs; Amazon is *always* hiring.


      Amazon is also growing- we have more developers than the year before. Yeah, there's turnover. Welcome to the IT industry- its rare for anyone to work at 1 place for more than 3-4 years. A lot of it is people staying just long enough to vest their stock grant (thats right, our sign on bonus is a grant, not an option) and then leave for another bonus elsewhere. Moneywise its the smart thing to do at any company.

      • Nah, I haven't even started trolling and didn't even intend it as a troll. If I was trolling there would be no doubt on anyone's part. :-)

        I notice that the software engineering points weren't disputed, so I'll offer the following observations.

        1) I didn't realize that HP paid that poorly. Thanks for tipping me off. :-)
        2) I've heard the story. However, at some point cultures need to mature and move on (avoid the Peter Pan syndrome at all costs).
        3) IT is a volatile industry, but I'm merely repeating the commen
    • My buddy from my current job just left for amazon. He has a degree and about 2 1/2 years experience. They offered him 85,000, plus gave him $20000 in cash and stocks as a signing bonus. They are also paying for all his travel expenses and putting him up in a apartment for two months. Sounds like they don't pay bad too me.. Jake
    • In my experience, most developers are nothing like artists and more closely resemble petulant, undisciplined children. Often they ignore the most basic principles of good software development (like version control, automated build and test suites, documentation, etc.) because "those are boring".

      I see that you don't understand most corporate development shops. First, most development teams uses version control. If you're talking about one person working on a pet project, well, maybe you have a point, but mo
      • On the contrary, I've worked in corporate IT shops for decades and have seen very few shops that consistently do the right things. In fact, I'm often the one who insists on deploying version control, requirements management, change control, etc. to try to minimize the wasted cycles. Invariably the changes are met with skepticism and resistance until the first project is completed successfully because:

        1) It's too much extra work
        2) There's not enough time
        3) It's to
      • "It's that clients don't care about or understand those basic principles, and managers often don't see the ROI for such activities. Documentation and test suites are going to take significantly more time for initial development of the product. Yes, if the project is even remotely complicated, these efforts will reduce long-term development time of the product through fewer downstream bugs, easier integration of new features or other products, and less ramp-up time for new team members."

        Oh, one other note. Y
    • In my experience, most developers are nothing like artists and more closely resemble petulant, undisciplined children.

      Er.... so they're just like artists then? ;)

      FWIW, this is meant in the spirit of a friendly jest, so there's no need for flamebait mods. On the other hand, as someone in a band hence many musician friends, and also lots of thespian friends, there's an element of seriousness to it as well ;)

  • The next wave: Service Oriented Architectures (Score:3, Informative)

    by NovaX ( 37364 ) on Wednesday May 17, 2006 @01:24PM (#15352714)
    What Amazon is describing is their SOA and their efforts to make it a generic, base platform for a large multitude of services. The idea of having a service grid, where services are easily developed, deployed, and work seemlessly together, has been gaining a lot of momentium in the last few years. A number of companies are posed to shift the playing field from an ASP model to a network of service.

    The article is impressive in hearing how Amazon successfully migrated from their legacy platform to a SOA. They may become a real contender in this emerging market, considering that they already have the user base and are quickly maturing a powerful platform. The other major contenders are Rearden Commerce and Salesforce.

    Rearden Commerce, the company I work for, has developed a very pure SOA. They are currently targetting enterprise customers in order to gain the critical mass and user adoption necessary to succeed, which can be very difficult for a startup working in the consumer market. Their goal is to provide a web-based personal assistent that you can use to book plane tickets, dinning, etc. and all coordinated with your peers and working with your calendar and notification preferences (email, SMS, voice). It looks as if Amazon is on a similar path, so it will be interesting to see what happens in the next few years.

    After the critical mass and the base platform are available the next big issue is getting 3rd party developers on the platform. That's something that everyone seems to be working on, which is why we're seeing so many AJAX and other toolkits emerging from companies like Yahoo, Google, and Zimbra. Imagine another company's product integrating just as neatly with Gmail as Google Calendar, yet staying very decoupled. That's part of the promise, and is the next big hurdle for the SOA leaders even though their platforms are still quite fresh and new.
  • All of Amazon's technology is great, but it seems to be the one engine that does not work with Squid proxies in some cases. We use Astaro as our gateway/proxy appliance, and it uses Squid as its proxy, and Amazon (as well as Amazon-powered sites, such as Target, Toys R Us, Barnes & Noble, etc) does not work.

    We have worked with Astaro support, and they have narrowed down the problem and sent the information to Amazon, but there has been no response yet.

    In researching the issues, I have found posts a

  • Developers are like artists; they produce their best work if they have the freedom to do so, but they need good tools.

    if only all IT managers knew this.
  • "Are self-describing XML documents flowing around inside your service-oriented architecture?"

Slashdot Top Deals

"There is no statute of limitations on stupidity." -- Randomly produced by a computer program called Markov3.

Close