Want to read Slashdot from your mobile device? Point it at m.slashdot.org and keep reading!

 


Forgot your password?
Close
typodupeerror
×

ILM's Datacenter 156

Posted by CmdrTaco from the something-to-read dept.
kylegordon writes "CGW has inside scoop on Industrial Light and Magic's facilities after they moved from San Rafeal to San Franciscos Presidio. With 3000 disks, it can shift 170Tb to 5000 rendernodes over 10GbE and 1GbE network links. It's an impressive system, for impressive films."
This discussion has been archived. No new comments can be posted.

ILM's Datacenter More

ILM's Datacenter

Comments Filter:

  • Hurray for Movie Technology! (Score:5, Funny)

    by eldavojohn ( 898314 ) * <eldavojohn@noSpAM.gmail.com> on Thursday March 30, 2006 @12:17PM (#15026594) Journal
    With 3000 disks, it can shift 170Tb to 5000 rendernodes over 10GbE and 1GbE network links. It's an impressive system, for impressive films.
    Unfortunately, all that storage can't provide decent acting, quality humor or plot lines without holes for their movies.

    I bet I could make a graph that represents how the quality of movies is characteristically inversely proportionate to the amount of CGI effects in them. Oftentimes, eye candy is used to shroud the plot and mask the bad acting/directing. American audiences especially just go looking for explosion sequences and CGI in the annual summer action flick hunt. We often fear a movie that might prove to be too cerebral and that pretty much disgusts me. Way to reinforce bad movies that are only good for one viewing with volume set to 'loud' and TV set to 'huge.'

    ILM is responsible for making movies like The Mask [imdb.com] (of which there are seven films) and characters like Jar-Jar Binks [wikipedia.org] possible. Be sure to thank them for that.
    • So you're going to hate the technical achievements here due to the lackluster plotting of directors and writers? Blame hollywood, not the engineers, technicians, and artists whom the article is trying to show in a bright light. Imagine you're a worker at ILM and get to do your work on these machines. That's what we're focused on acclaiming here and not the results of Hollywood's money-grubbing attitude.
    • Is it really ILM's fault? My understanding is that they are just there to supply the studios with special effects. I mean, shouldn't we be placing the blame on studios and the general public for creating and paying for movies that are crap? I mean, ILM just seems like a bunch of folks that make shit look cool when requested to... ...I do agree with you on the lack of quality...but come on, if I could make a living rendering fun stuff I would too, and it seems that is all ILM is doing. Meh, but, I could
      • You are absolutely correct, ILM just prvides VFX work. They are just a contractor, like the guys that do the catering on the set. They have absolutely no control over the story or directing of the movies they work on (except for a few exceptions).
    • There is absolutely nothing wrong with "The Mask". It's a great film. One of the few Jim Carey films I can bare to watch.
    • ILM is responsible for making movies like The Mask [imdb.com] (of which there are seven films)

      There were only two "The Mask" films. And only the first one counts if you don't like stuff that sucks.
    • American audiences especially just go looking for explosion sequences and CGI in the annual summer action flick hunt.

      You say that like it's a bad thing.

      Okay, CGI of explosions is bad, though. Stuff should get really blown up. Big stuff, not just models. It's the American way!

    • CGI is not the enemy (Score:3, Insightful)

      by Kombat ( 93720 )
      I bet I could make a graph that represents how the quality of movies is characteristically inversely proportionate to the amount of CGI effects in them. Oftentimes, eye candy is used to shroud the plot and mask the bad acting/directing. American audiences especially just go looking for explosion sequences and CGI in the annual summer action flick hunt. We often fear a movie that might prove to be too cerebral and that pretty much disgusts me.

      OK, here's the thing. Movies that are "cerebral" and thought-prov
      • Given high ticket prices and other factors, movies which truly deserve to be seen on the big screen are the ones that get my theater-watching priority.
      • Look at "Air Force One." Decent movie, spoiled by absolutely terrible special effects at the end.

        You mean the final splashdown? It was fine. What stretched things just a bit was a middle-aged politician who could beat the shit out of a whole squad of elite Spetsnaz; or trying to work out the motivation of the Secret Service traitor; or.... the plot was a much bigger obstacle to enjoyment than the SFX, for me. Nevertheless, Gary Oldman, William Macy, and even Ford made it watchable.

        • The theatrical edit showed painted matte plates (obvious - huh? kind of error) between the CG composites when the plane started going down. It was *Very* noticable and distracting. I believe the made for DVD and TV edits fixed it.
    • Seven Mask films? I suppose that the other 5 haven't been released yet? (or even made yet). Also, ILM wasn't responsible for Jar-Jar....you can only thank Lucas for that.

      Also, ILM didn't "make" those movies...and hasn't actually "made" any movies. They're an FX house. They provide a service to someone that pays them. Like any other service industry really.

      But as the other posters said above, don't blame ILM for movies sucking. They did some effects for "Schindler's List" and "Saving Private Ryan"...so shoul
    • CGI doesn't make the bad movie good, but it can also be used to contribute to making a good movie better. ILM is also responsible for the effects in countless other movies like Back to the Future that are seasoned classics. However, in the greater sense of movies "these days" I do agree that visual stunnery is being used for more than is needed to convey the plot line -- making some movies that could have been a 30 minute tv show into a 3 hour movie. -Jeff
    • Unfortunately, all that storage can't provide decent acting, quality humor or plot lines without holes for their movies.

      This statement is more true than you think. One of my high-school friends who went to work for ILM lamented that, as the most expensive special effects house in the business, they attract particulary the films that have nothing going for them but a high budget. No engaging plot, no spectacular acting, just a dumptruck full of money.

      What they end up with, and why he was so upset, is t

    • ILM is responsible for making movies like The Mask (of which there are seven films) and characters like Jar-Jar Binks possible. Be sure to thank them for that.

      ILM is a CGI studio. It may be responsible for a lot of what you actually *see* on the screen, but there's whole a lot more behind making a movie then just that (even the really bad ones). On a technical level, almost everything they've done has been top-notch. Just like I wouldn't blame them for The Mask movies, I wouldn't say they're the reason S

    • I suspect your inverse relationship theory will be pushed to the limit, as they push this new datacenter to its limit in Stealth II.

      quality ~ 1/cgi

      Anybody that saw the first installment of Stealth should understand where I'm coming from.

      Skycaptain is another one in this category...

      Way to go ILM! I want my money back!!!

      Comic Book Guy's Voice: "Worst. Movies. Ever."

      -@
    • The Mask is one of my favourite movies. The acting and the plot may be poor (though I like the good/bad girl twist), but I thought it was very funny. I like funny movies. I didn't like the new Star Wars stuff at all. Maybe ILM is best at slapstick humour movies.
    • I bet I could make a graph that represents how the quality of movies is characteristically inversely proportionate to the amount of CGI effects in them.

      Will your graph account for Pixar?

    • From TFA:
      "Data access over fiber between San Rafael and San Francisco was very fast, but when you're shooting packets to Singapore and introducing millisecond delays, the computers start bogging down," says Thompson. "It's not the throughput; it's the round-trip time. We're looking at Network Appliance, Hewlett-Packard, and a lot of start-up companies that deal with these WAN issues for a solution."
      A solution for round trip time to the far side of the planet? So he's looking for packets that travel faster t
      • True, there's no escaping the speed of light. The biggest issue, however, isn't latency itself; it's the fact that TCP scales badly across high-latency connections - the transmitting host will send packets up to the maximum window size, and then spend so much time waiting for the ACK to come back such that big pipes don't get efficiently utilized. There are solutions available, both hardware-based [orbitaldata.com] and service-based [internap.com], however, which proxy the TCP transmission for maximum throughput no matter what the latency
      • The solution is more local storage caches like Network Appliance and others provide. The latency between you and a local server is pretty small. Then all you have to worry about is your cache being up to date with the real data, which is where probably most of the storage providers R&D time is being spent now.
    • Correlation is not causation ... e.g. exhibit one:

      "The Incredibles"

  • Nice network (Score:1, Interesting)

    by liliafan ( 454080 ) *
    They seem to have a really nice set up there, I would be curious about how their 'hybrid' NAS/SAN benchmarks, and see some comparisions against some of the big boy equipment like IBM sharks.
    • The real question is:

      If 3000 disks are spun up in the forest, and no-one is there to listen, will there still be a nationwide blackout?

    • Re:Nice network (Score:5, Interesting)

      by flaming-opus ( 8186 ) on Thursday March 30, 2006 @12:55PM (#15026961)
      That's a funny question because I used to work at ILM's (San rafael, much less shiny) lab, benchmarking raids, including the first version of the IBM shark. At that time we came to the conclussion that the IBM raid was reliable, and reasonably fast, but the price was so far out of line, that it wasn't a real contender.

      The shark, and many of the high-end raids, are really designed around transaction oriented applications (databases). ILM's application are classic video codes, which work better on a classic raid5, than they do on the data-sprinkler style raids like the shark, eva, clariion, etc. Netapp makes pretty decent storage boxes, and they're highly configurable, so I'm sure they have them fine tuned to the apps' preffered i/o size.

      Furthermore, the nas/san has more to do with the spinaker software than the raid of choice. Back when I worked there, ILM was testing cluster sollutions, but the renderfarm was a bunch of sgi origins. The storage was hung off of a couple of 8-way irix boxes, and pushed around with NFS. Since then they've upped their compute capacity by a factor of 30, there's no way they'd be able to do all that I/O with NFS to a couple of big servers. The san setup lets them distribute the NFS load to a large number of servers, all sharing access to the storage on a san. A lot of other cluster filesystems allow this too.

      From the benchmarking I've done of these types of storage clusters, you don't get the same single stream performance as you do from a big-iron server setup, but the aggregate across a large number of nodes is pretty good. Managing the mess, and reliability can be problematic. I've never used spinaker, but I've used almost all the other products in this space, and they're all in the "pretty good" category. My current favorite is apple's xsan, because it is really inexpensive, and so is the hardware.
      • Have you head of BlueArk? What do you think about their SAN's ?

        • Re:Nice network (Score:3, Informative)

          by flaming-opus ( 8186 )
          I've tried bluearc. It works alright, though not as well as the whitepapers say. This is true of most everything, though. What really pissed me off about bluearc is the pre-sales engineers who seemed to have drunk a whole hell of a lot of the company coolaid. The whole story is that the filesystem is "implemented all in hardware", so it's really fast, and that should solve all your problems.

          Well, I've been around the block enough times to know that no filesystem is actually implemented in hardware. They may
      • Bah, XSan is *not* enterprise ready. When I was deploying it for a customer we discovered that the version of XSan we were using was totally incompatible with ACLs. The setup the customer wanted really required ACLs (regular unix permissions weren't flexible enough) and so we ended up switching off the XSan and using them as regular file systems.

        Although without ACLs, yes, XSan is pretty nice. And it's true, XServe RAID with XSan is very nicely priced compared to setups from IBM and Dell, etc.
  • Plus, it can hold a decent percentage of my pr0n collection... well, the JPGs anyway.

  • Way to go Spinnaker! (Score:3, Insightful)

    by drewzhrodague ( 606182 ) <drew@nOsPaM.zhrodague.net> on Thursday March 30, 2006 @12:23PM (#15026654) Homepage Journal
    Way to go, guys! Who would have known that a small startup from Pittsburgh with some killer engineers, could make it into ILM's datacenter. Hi, Gus!
    • Pittsburgh has CMU, which has a top-tier CS program, a top-tier ECE program, and the ETC (Entertainment Technology Center); what boggles the mind is why there aren't _more_ success stories like spinnaker comming out of Pittsburgh.
      • I think that those people fucking leave almost instantly. There are a couple of small companies and startups, but I think it is more difficult to make it as a tech business here in Pittsburgh, than in places like San Francisco, or Boston -- even though there's a whole hell of a lot more competition. There are a few standouts, and it is easier to stand out here, but people are generally less receptive to new ideas.

        I met lots of CMU people in Boston, though.
        • While I'm by no means and engineer and my evidence is strictly anecdotal, I know of at least a few hundred bright people from that area (I hail from about 40 miles east) that headed to either coast in search of better/more abundant jobs.

          More specifically, the Johnstown area, has suffered greatly from brain drain as there just isn't anything there (aside from family, which, thanks to technology, is much easier to keep in touch with these days) to keep young people from getting out at the first chance th

  • By my calculation ... (Score:3, Interesting)

    by LaughingCoder ( 914424 ) on Thursday March 30, 2006 @12:25PM (#15026668)
    According to Mapquest a trip from San Rafeal to San Francisco would take about 35 minutes (Est. Distance: 21.06 miles). Therefore, if I loaded up all 170TB on a truck my effective bandwidth would be about 3.06e28 bps (or roughly 3e16 Tbps). Once again for huge data repositories there is no substitute for shipping physical media.
    • How long would it take you to copy the data for transit (you're not suggesting you transport your master copy, surely), and then restore it?
      • It's probably already backed up, so you move the backups first, see how they fare, then move the masters. You'll get only half the transfer rate though.
        • > It's probably already backed up

          After what happened to Wallace and Grommit, it had better be.

          More interesting than their historic real estate, would be exactly *how* they backup this amazing amount of data.
    • Wow, you must drive really fast dude! 21 miles in 45 femtosecond! What's your secret?
    • What I love about the 'station wagon full of backup tapes hurtling down the highway' analogy is that it never seems to account for the integrity, security, or transfer time of the data to and from the medium itself.
    • > According to Mapquest a trip from San Rafeal to San Francisco would take about 35 minutes

      You've never tried to drive through SF proper into the Presidio, and you've definitely never had to find a place to park.

      Very pretty place for an office building. You'd never run out of stuff to do within walking distance on your breaks. (Real nice work if you can get it.)
    • You obviously don't know what traffic is like in the SF Bay Area. During rush hour, you'd be working at 801.11b speeds.
    • Try .65 Tbps...

      170TB*8=1360Tb

      1360Tb/(35*60)s = .65 (rounded)

      As another poster said, your travel time would have to be VERY VERY fast to get 3e16 Tbps...

    • Therefore, if I loaded up all 170TB on a truck my effective bandwidth would be about 3.06e28 bps (or roughly 3e16 Tbps). Once again for huge data repositories there is no substitute for shipping physical media.

      Bandwidth is not the issue here - latency is...

      A 70-minute Ping can really ruin your killing spree in Quake ;-)

  • What?!?! (Score:2, Interesting)

    by Machina Fortuno ( 963320 )
    Okay... not everything they do is shit. That, and CG doesn't make movies worse... only if it sucks. You can go watch claymation if you would like.

    http://www.ilm.com/ilm_services.html [ilm.com]

    Look at all they have done. While some of the stuff on there may have sucked... there is some really fucking good stuff on there.

    Also, if I remember correctly, they were some of the first to experiment with particle renders for CG (they used it in the Mask to create some of the storm/tornado transformations). Anyways...
    • It isn't that fucking good yet, so get off your high horse.

      • Liar (Score:5, Insightful)

        by Kombat ( 93720 ) <kevin@swanweddingphotography.com> on Thursday March 30, 2006 @12:50PM (#15026895)
        I ALWAYS notice CGI.

        No you don't. You think you do, but you don't. When you do notice it, you point it out and say to yourself, "that was so obvious, CGI sucks." But when you don't notice it, you don't realize that what you're looking at is CGI. You think it's real. You think the man really has had his legs amputated ("Forrest Gump") or Arnie really did jump his motorcycle off a 15 foot ramp ("Terminator 2"). CGI is used all over the place in movies now, not just for the big explosions that still may not look 100% convincing (however, it's much better than stop-motion animation).
        • you don't realize that what you're looking at is CGI. You think it's real.

          Stop, man, you're tripping me out... This sounds like Total Recall or something. It's heavy.

        • Re:Liar (Score:2, Funny)

          by slagell ( 959298 )
          however, it's much better than stop-motion animation What are you talking about. Gumby kicks ass! In fact, I didn't realize it was fake until last year. Alas, I will never get Gumby's autograph.
        • Someone jumped that bike off a high drop in the T2 scene. That stunt was performed by a trained professional on a real bike.
          • Someone jumped that bike off a high drop in the T2 scene. That stunt was performed by a trained professional on a real bike.

            With a crane and steel wires that were digitally removed. That was my point.
          • IIRC the jump was real and so was the stuntman, but the bike was being guided/supported by wires that were painted out later on.
        • Umm...the motorcycle jump in Terminator 2 wasn't CGI.

          It wasn't Arnie on the bike, but it wasn't CGI.
        • CGI is used all over the place in movies now

          Yeah, I was surprised to learn how much of Brokeback Mountain was shot on flat sandy backlots with mountains composited in later. You'd think it would have been cheaper just to film in Wyoming. Ah, well, I don't try to understand Ang Lee, or why people seem to think he's so great.

    • Re:What?!?! (Score:4, Insightful)

      by east coast ( 590680 ) on Thursday March 30, 2006 @12:56PM (#15026972)
      CG doesn't make movies worse...

      IMHO, you are wrong. CG can make a movie suck. Once Hollywood understands this maybe good films will not be as uncommon as they are today. Good CG and/or a good story can both do something for the fantasy aspect of things but when you put too much CG in to make up for a lame storyline than CG does suck (*cough* matrix 2 *cough*).

      It really doesn't bother me to watch an episode of (the old) Dr. Who, ST:TOS or Twilight Zone and notice that rocks are made of foam rubber or that a costume is little more than a pie plate and a grocery bag painted green on someone's head when the story is good and these series put out consistantly good stories. No amount of CG can make up for poor acting/writing.
      • My point was never to completely remove blame from CG, but to point out that not all CG is bad. And not that movies without CG are worthless. The point being, that CG can really make a movie rock, especially from an imaginative standpoint.

        You may have noticed in my previous post that I recognized that yes, some of these movies sucked.

        And here is the fun part... you don't notice good CG. Bad CG makes movies suck when you notice it.

        "WELCOME TO PLATO'S CAVE BITCH" - Someone I Can't Remeber
        • Bad CG makes movies suck when you notice it.

          Not to me, but that's just me. I can accept having a budget that can't make everything spiffy or that effects just aren't that important if the writers and actors dedicate themselves to the overall concept.

    • Also, if I remember correctly, they were some of the first to experiment with particle renders for CG (they used it in the Mask to create some of the storm/tornado transformations).

      ILM introduced the concept of particle systems for film. It was first used for the Genesis Sequence in Star Trek 2. William Reeves then presented a paper at SIGGRAPH 83. He was also awarded an Academy SciTech award for it:

      Particle Systems -- a Technique for Modeling a Class of Fuzzy Objects [acm.org]
      Particle Systems [siggraph.org]
      Particle Sys [oscars.org]

  • The new ILM DataCenter [jedinet.com] looks alot like the one at Trinity College in Dublin, Ireland.
  • Nice setup you have there ILM. Its a shame if something should happen to it ;)
    • "...and ten petabytes of fiber channel disk."

      "Fiber Channel, Dino."

      "Be a shame if someone was to set fire to them."

      "Set fire to them?!?"

      "Things burn, Colonel."
  • Jar Jar Binks wassa born here, meesa know!
  • It's an impressive system, for impressive films.

    If it's for impressive films, why are they using it for soulless dreck? Some sort of beta testing period maybe?
  • It's an impressive system, for impressive films.

    Thank you, I will decide what's impressive. This is like when vendors tell me that their product is "really cool" or "great".... tell me what it DOES, SHOW ME, then I will decide if its "cool" or of value to me.
  • I've got to echo a lot of the comments here, that sadly ILM and hearing details about ILM, has lost its luster a bit since the old Star Wars days. Not that the acting or the plot was that great in the early Star Wars films either, but there was just a rebelliousness to it, a certain type of moxy, plus a lot of us Gen Xers were very young back then, so maybe it has to do with that.

    Still, there was Jurassic Park, which had that wow effect, but only in a suburban, sterilized kind of way. Maybe it was just t

    • ILM doesn't make movies. They only do special effects shots. Movie studios hire ILM to add effects shots into the movies. What they do with them, and how they cast/write/act/script the movies is really out of ILM's hands.

      ILM used to have a digital movie group. Steve Jobs bought it, and it's now called pixar, err... Disney.
  • am i the only person who thinks its funny that there is quite a few machinima (game engine rendered movies) that are of better film quality than anything ILM makes on their billion dollar systems?
  • Nice to see that ILM had the sense to use Ethernet rather than InfiniBand. IB has some great features in hardware. Too bad the software to drive it is less than wonderful.
    • I put in 4 years working on IB for SilverStorm so I have been doing IB since before there were any ASICs or even a 1.0 spec. Can you be more specific? What kind of problems do you have with IB software and which IB software are you talking about? OpenIB? VendorX's proprietary stuff? Which HCA (Mellanox, Pathscale, IBM, etc)? Which switching (Voltaire, SilverStorm, Cisco, etc.)? What kind of setup were you doing (big HPC cluster or little database)? I not looking for flame war, I am genuinely curious
  • Massively parallelized Markov chains!

  • Virtualisation (Score:3, Funny)

    by ettlz ( 639203 ) on Thursday March 30, 2006 @01:24PM (#15027234) Journal
    Why have 5000 render nodes when you could virtualise with 250 physical processors running 20 apiece?
    • Perhaps because 5000 dedicated render nodes can render a s**tload more frames than 250 nodes each running 20 VMs!!!!
    • Because raw processing power is what's important. Lets say you have 1000 SMP machines, each processor running a rendering process (2000 render nodes). Does adding another process to each CPU (4000 rendering nodes) double your processing capabilities? NO! Because each processor is mostly utilized, with only a small percentage of idle cycles. You have a finite number of CPU resources, dividing them up into small pieces does not improve anything (in this case). It would make sense if rendering CG was more
  • now Lucas can rape my childhood memories faster than ever.
  • That's San Rafael, not San Rafeal. A sidenote on place names: Skywalker Ranch was located in Lucas Valley, but the valley had its name long before George located there.

    • > Skywalker Ranch was located in Lucas Valley, but the valley had its name long before George located there.

      Skywalker Ranch IS located in Lucas Valley. It's the compound where George lives -- and it's not going anywhere. The only thing that moved is the ILM crew (I think the game people were already somewhere else). Skywalker Sound is still out there.

      From the other comments, it's interesting to see how many people seem to miss the point of what ILM does. ILM is NOT LucasFilm. LucasFilm is George's

      • Skywalker Ranch IS located in Lucas Valley. It's the compound where George lives -- and it's not going anywhere. The only thing that moved is the ILM crew (I think the game people were already somewhere else). Skywalker Sound is still out there.

        Small calrification. Skywalker Ranch is where Lucas works. He lives elsewhere (though close). Skywalker Sound and the Lucasfilm production offices are at the Ranch. ILM and LucasArts, which used to be in separate locations, are now at the Presidio location, tho

  • Imagine a Beowulf cluster of these things.
  • I wonder if they are using NICs that have offloading capabilities in the compute nodes. InfiniBand, Myrinet, and iWarp NICs are all designed to get rid of TCP/IP processing. One would think with relativly large data sets, TCP could be a big CPU consumer. Also, standard NICs using TCP have horrible latency compared to InfinBand, Myrinet, etc. That latency really eats up cluster performance when the nodes all wait for something (like new data, results, etc.) In lots of high performance computing applicat
    • a low latency interconnect is only needed if you're doing message passing between nodes, which ILM is not. There's a batch scheduling node that passes each render node a list of tasks it's gonna do, and then the traffic is mostly pulling texture maps from the fileserver (into a local cache) and outputting the rendered frame. No need for infinaband. multiple gigE rails might help though.

      TCP offload sounds like a good idea, but I've seen it introduce a lot of bugs. It's also not terribly well supported on lin
      • I agree the cost for enough FC ports to handle the entire cluster would be quite a bit. I am just curious if it would be faster or not.
        Another option would be to use lots of local storage and use a clustered file system over a second 1 GigE lan. The cost may balance out in that case. 5000 nodes can provide a lot of shared storage to themselves and if each node had a reasonable SATA drive they could all keep up with the GigE demands.
        Of course spinnaker uses multiple smart gigE links per box and multiple b
        • Yes, fibre channel is almost always faster than gigE. Even 1gig fibre provides higher bandwidth than 1gig ethernet. The payload for each packet is a lot bigger than ethernet, even with jumbo frames. GigE, on the other hand, is cheap.

          There's a reason noone offers local-node exported cluster filesystems in the commercial space: What happens when a node fails? How do you get at your data? You can do raid across the different nodes, well, then how do you reconstruct the raid? How does it behave in degraded mode
  • I can't imagine a Beowulf cluster of ILM's facilities, sadly.

Slashdot Top Deals

To do nothing is to be nothing.

Close