Slashdot is powered by your submissions, so send in your scoop

 


Forgot your password?
Close
typodupeerror
×
IBM

IBM Sets SPECweb Record 88

Posted by justin++ from the doesn't-run-linux-:( dept.
the frogspotter writes "IBM has an article on their site talking about their new RS/6000 that can supposedly do 40,161 http ops./sec. as measured by SPECweb96. Jeez...that'd be 3469910400 http ops./day. " And it only took 12 (out of 24) processors (on an unreleased server) and a hacked kernel! It seems as if modifying the kernel for benchmarking is becoming quite a popular tactic these days...
This discussion has been archived. No new comments can be posted.

IBM Sets SPECweb Record More

IBM Sets SPECweb Record

Comments Filter:
  • Good point, why in the hell is hacking the kernel and getting direct support from the engineers who made the software considered fair in benchmarking??? When I first saw the Mindcraft results, I about laughed my ass off after reading that MS had direct help in the tuning of the server.
    When I can get personal hacked/tuned kernel from MS, and have their engineers come out and work on my servers, I might consider running NT on the server side.
    Come on IBM. Take it out of the box, do some Real Guy Tunning(TM) and then test it over serveral weeks. Then get back to us.
  • It's also not that good. (think of the recent NT vs Linux benchmarks where the webserver only servers static pages...)

    However, the "new and improved (tm)" SPECweb99 is almost ready: See the SPECwww99 home page [spec.org]

  • Web servers are frequently asked to send users static Web pages.

    With all these web benchmarks coming out using static web pages, I'm wondering if there are some statistics available somewhere that can either confirm or deny this.

    I know /. is heavily dynamic (are there any pages that are static?). I know my homepages are netscape and yahoo are dynamic. All search engines serve dynamic content. Certainly web sites designed for E-Commerce, the market for IBM's web servers, are dynamic (searching for products, filling out order forms, etc).

    For some reason I don't think the statement in IBM's release is totally accurate. Why not create a new benchmark or modify an existing one that had a mix of dynamic content that matched certain typical uses for web servers. A search engine server, a portal server, an e-commerce server, etc. This would give more accurate real-world results, IMHO.
  • Are there any projects out there aimed at modifying Linux so that it is THE leading web server platform? Obviously you can't compete with IBM and 24 processors, but what about a distro that is aimed at serving web pages, static and dynamic, at such a high level that it can't be matched in price/performance?

    I know Zeus is fscking fast, but what about an operating system that is designed from the ground up to be a web server? All OS bottle-necks removed.

    Is this a crazy idea? Is it possible? Seems to me like Cobalt's systems are a lame attempt at doing this (no optimizations, just easy setup).

    Stew
  • Well, I guess this answers my questions in comment #3!

    But, how accurate is this going to be? How often will the mix of dynamic/static content be updated to reflect current statistics? Are the ratios changeable within the benchmark and will testers use the changeable parameters to show their server/OS/hardware is "better" than the competition.

    Besides, who wants to pay $800 for a friggin benchmark test?
  • Vive l'AIX!! The single most weirdest, scariest flavour of Unix! Has anyone seen those 21" RS/6000 monitors that are perfect squares? Even weirder.

  • It has always been like this (Score:3)

    by QZS4 ( 7063 ) on Saturday July 03, 1999 @02:40AM (#1819741) Homepage

    Special tuning has always been a part of benchmarking. There are many compilers out there with special optimizations designed only to improve the performance of a specific benchmark (detection of certain matrix operations and other snippets of code which almost exclusively occur in benchmark programs). Many compilers, for example, recognize this piece of code:

    x = sqrt(exp(log(x)/t1))
    which gets transformed to
    x = exp(log(x)/(2*t1))
    thereby skipping the sqrt calculation. Since this code almost only occurs in the Whetstone benchmark, in the general case it will never get used. (Hennessy & Patterson, CA:AQA, chap. 1.8)

    Some of the compiler optimizations out there will even generate faulty code if they are switched on when not compiling a specific benchmark program. Might be worth thinking about.

    And don't think that only software can be optimized. Many of the processors out there have some special hardware hacks designed only to improve some benchmark program, and which almost never is used in "real world" programs.

    Bottom line is, don't be surprised that they use some special tuning, it has always been this way and always will.

  • Is there any free web benchmarking tool?

    Webbench from ZDNet only runs on Windows (".EXE for all platforms", right.)

    SPECWeb96 costs $800!

    Any suggestions?
    --
    http://www.wholepop.com/ [wholepop.com]
    Whole Pop Magazine Online - Pop Culture
  • I know Zeus is fscking fast, but what about an operating system that is designed from the ground up to be a web server? All OS bottle-necks removed.

    It's Not Linux (INL), but MIT's Exokernel [mit.edu] would likely be appropriate for this. They post numbers that seem to be as meeningless as IBM/Mindcraft/etc's.

  • Looks much better. Tests some of the things that are difficult about web serving, espeially the question of having thousands of low speed connections open at the same time. Optimising for this benchmark might even benefit real world web serving performance. Lets hope they also specify a dataset that is much larger than RAM so you can't just use a RAMDISK to get good scores. That may be a good idea if your dataset is small enough, but most web servers have much bigger data sets than RAM so it's not too realistic.

    The tuning that has gone on after Mindcraft has probably benefited performance for someone, and that's great, but I doubt it has made any difference to real web servers anywhere.

  • Is there any hope that vendors will stop citing tests on a platform that is never going to be commercially available unless the results are specifically labeled as R&D?

    I have not gone looking yet at the SpecWeb99, but I think some of the TPC benchmarks now require the platform to be commercially available within six months.

    At least IBM was honest enough to tell us that there was special tuning and new hardware involved. That gives me some hope.

  • The c't [heise.de] benchmark used a Linux-based benchmarking tool that is available from them [heise.de].

  • Even a site that serves only dynamic pages still serves static files. Even if nothing else, the images are all static. So if you've got a busy site, it's important to have a server that's fast at serving static files. Thus, you might run an image server on a different server than your pages and dynamic content.

    -Imperator
  • We separate integer benchmarks from floating point benchmarks because they reflec different situations. Why don't we separate static and dynamic web pages in different benchmarks?
  • Why is everybody so negative ?
    Let them do what they want, if they can get Linux above microshit, I'm happy :)



    =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
    Belgium HyperBanner
    http://belgium.hyperbanner.net
  • Isn't it the idea of Linux that the source
    of Linux is freely available, and that it's
    purpose is to be adapted by others ?!

    I say, IBM did a fine job. If they release
    that pathches it would even be better...

    This might lead to new and faster implementations...

  • Benchmarks... (Score:3)

    by Signal 11 ( 7608 ) on Saturday July 03, 1999 @04:49AM (#1819759)
    should be left on the bench. The only use is to justify your expenditures to management. Personal experience is the best measure of what works, and what doesn't. Let the marketing departments drool over them. Personally, I'd file them in the circular.

    Benchmarks only test a very narrow spectrum of performance, and is rarely representative of real world performance. Witness the mindcraft benchmarks. Specint is a "standard" now in benchmarking, hence it's to be expected that companies will be adding special optimizations that wouldn't otherwise be done.

    --
  • Instead of all this stuff with httpd in software, why not just build it into the hardware? That would fry the bacon, so to speak. Imagine a ATM card with a 1 gb RAM buffer caching your web pages ready to saturate any goddam bandwidth that you can find.

    Seems to me to be the obvious way to do it. They you can apply the general purpose CPU stuff to creating dynamic content where needed.
  • Yes, that's true - but if you actually read the article you'll see that it was running AIX, which is not GPL'd. They can release all the patches they want but unless you have teh AIX source code it's not goung to to a damn bit of good...

  • and what is /. server configuration ?
  • Posted by Ungrounded Lightning Rod:

    Seems to me like Cobalt's systems are a lame
    attempt at [modifying Linux so that it is THE
    leading web server platform] (no optimizations,
    just easy setup.

    Wasn't it a Cobalt guy who modified the
    TCP code to make it much more able to handle
    large numbers of short connections?

    I recall catching the tail end of his
    presentation at SVLUG about a year ago.
    A big piece of it was shrinking the data
    structure during the disconnect timeout,
    when the bulk of it was unused, so tens
    of thousands of dying connections didn't
    eat the RAM. I think there were a few
    other changes, such as reorganizing the
    data structure so it could be searched
    quickly when it was large.

    If it hasn't been folded into the distribution
    yet it should still be available (with the rest
    of the cobalt source code) on their web site.
  • Posted by Ungrounded Lightning Rod:

    of the previous poster's comment. It DOESN'T
    seem to ME that Cobalt was a "lame attempt".
  • Did you even read the article?

    How we did it
    Web servers are frequently asked to send users static Web pages. AIX takes each request for an existing, unchanged page and stores it in a cache controlled by its kernel. "What's different here is having the cache under the direct control of the kernel--the heart of the operating system," noted Jim Beesley, manager of AIX Marketing. "This reduces overhead and improves performance."

    The next time this Web page is requested, the AIX kernel cache is accessed first, and if the page is there, retrieves it without initiating the Web-server process. This saves time by avoiding a path that goes into the server application and back out again.

    A related feature is the large size of the cache that can be created on 64-bit RS/6000 servers. AIX used two gigabytes of cache for Web pages while running SPECweb96

    Ok, so it's not quite all hardware, but being able to store up to 2 GB of content in cache is gonna speed up just about any web server.
  • The correct pronunciation of AIX (IMO) is
    "aches"...
  • Surely IBM can write a parellel version of httpd and blow away any numbers out there? When are they going to drop these silly standalones and market the machines with some real power?

    Now if they could port PSSP to Linux, that would rule!
  • Posted by 2B||!2B:

    Good answer!

    Not that targetting benchmarks is necessarily bad. What we need is to keep making up new benchmarks in a way which deliberately targets flaws (flaws that actually make a difference, that is) in existing systems. It's much easier to make improvements if you have a simple way to measure the results on any changes.

    To be truly productive, we would have to target flaws in both Linux and in NT, to give a real incentive for both sides to improve. As much as I dislike NT, I would still love to come up with ways to make them improve it. If you can't beat it, fix it.
  • That's an excellent example. Hennessy & Patterson is an excellent book for Architecture (and it spends a great deal of time discussing benchmarks as well as excellent, clear explainations of Architecture, of course).

    For people who are interested in benchmarks, mistakes/ways people fudge numbers and how to measure, analyze and model system performance accurately, another gem to add to your library is: The Art of Computer Systems Performance Analysis by Raj Jain. It even includes a refresher on the stats course that you slept through (although a perfect knowledge of stats is not required to understand and profit from the information in this volume). It will give you an view into real performance analysis, modeling and measurement. Including all of the tricks that everyone seems to call "unfair".

    In many cases, they are wrong or unjustified. But it is not a bad idea to use tweaked hardware to perform a benchmark, although publishing the results may be in poor taste. Many times, management makes signficant decisions on what hardware to sell and what changes need to be made to future hardware, based upon these benchmarks. For instance, if a 10-by machine performs more poorly as a 3-by machine because of bad filesystem code (which is not an unreasonable possibility if the operations in the benchmark are IO-bound or depends heavily on IO performance), the management might decide to put pressure on for cleaner fs code. The only time this problem code is isolated and fixed may be during the benchmarks with tweaked code.

  • Grand Pa IBM steps up and says "Boys this is how its done if you want a good benchmark, build your own hardware, write your own software, do the test in-house, and write your own press release."

  • No doubt! AIX rules. JFS, LVM, SMIT, ODM, 64bit, solid as a Mac truck. I used to work in AIX defect support in Austin, was my 1st UNIX to play with. Jaded form the get go? Yes, but I've never seen a better UNIX in my years of playing with Solaris, DGUX, Linux, FreeBSD, SCO....

    See 'ya,

    Jim
  • And what's its keyrate? :)

    -Chris

  • IBM HTTP Server (Score:3)

    by Zombie ( 8332 ) on Saturday July 03, 1999 @08:36AM (#1819780) Homepage
    Sure /. servers up static content. All the bitmaps! There's a lot of static content out there!

    By the way, IBM HTTP Server [ibm.com], A.K.A. "IBM HTTP Server Powered by Apache" is Apache, now IBM's preferred webserver. So any performance improvements IBM makes to "their" webserver will go back to the community.

  • Sure, AIX packs some neat features, but c'mon, "solid as a (sic) Mac truck"? Yeah, the NFS implemenatation that shipped with the 4.3 release was solid as a Mac - like pudding. It made unfsd under Linux look good.

    I've worked with AIX off and on since 1991. I remember the 3.1.5 debacle. Believe me, AIX can't point fingers at anyone without having three pointing back at itself. I remember the joys of ODM corruption, I remember the text config files (there for backward compatibility) that did not reflect the true (odm) config of the system, I remember when they took away the ability for an admin to edit the ODM database directly so that you could fix the screwups caused by some of the install and system config utilities, I remember creeping JFS corruption, LVM management utils that would allow you to shoot yourself in the foot...

    Geez, AIX - why did you get me started? The only thing that would make me rant more are my years with the ultimate "Crappix" - SCO EunuchsWare. I will *never* look back. The only thing that could compete with it on the Crappix scale was HP/UX 8.x (and you thought 9.x was bad... boy, have I got some stories for you).

    Christ, somebody get me to alt.sysadmin.recovery before I go nuts and piss on the mains...

  • Do they mean disk cache or RAM cache? If it's ram that's a hell of a lot of cache.

    But the idea of storing frequently accessed static pages in a cache contolled by the kernel sounds like a cool idea. Wonder if there is any plan to implement this in the Linux kernel? Maybe as a module?

    Steve

  • Keeping in mind that ftp.cdrom.com has 4GB of RAM to handle its load...

  • There exists a kernel webserver! (Score:1)

    by Anonymous Coward
    There is a in-kernel webserver (although not quite mature)!
    It is called kHTTPd

    http://www.fenrus.demon.nl
  • It would be far more valuble to debunk bad banchmarks and make the world realize how much fraud (or at least close to it) go on in them.

    OSes are all about compromise, there are always ways to make things a little faster if you're willing to compromise reliabilty or stability. (I think MacOSX got bit by that bug with there little web bench problem not so long ago)

    How many record setting benchmark configurations are good for real world usage? Do we really want development effort going in to that? I'd rather have a benchmark loser that is rock solid all the time. There are certainly cases where it makes sense though, I think mindcraft revealed some very real problems that need to be addressed but I don't think it should be our norm.

  • I understand what he means... a disk cache in main memory. Sorta like SmartDrive from DOS. Still a cool idea that I hope at least gets looked at for the Linux kernel.

    Steve

  • Benchmarks can be relativley valuable.

    Or if not benchmarks, some form of standardized teating. I don't much care for "single number" benchmarks, that claim to sum it all up, but it's nice to see how different machines handle the same load. You do need a lot of them though. computer speeds are very complicated things, and you need to look at a lot of variables before you can conclude whether a system is "right" for you.

    And is pretty impractical to try all the different machines in order to decide what works.

    Benchmarks are a tool, not an answer.

  • AIX is a royal pain in the arse. My current position is admining a whole bunch of RS6000s and while it's rock solid and quick, it's _DIFFERENT_ to everything else.
    My last job was working on Data General AViiON systems running DG/UX, they are a real pleasure to use and administer, leaves AIX and Solaris in the dust.
    It's a pity Data General's management now have their tongue stuck so far Microsoft's arse.
  • DG/UX can do this with ease, but is a damn site smoother with it.

    I work with AIX, but wish I worked with DG/UX. :(
  • The same is also true in the world of HDD's. A typical SCSI drive can recognize the access patterns of a benchmark and adjust its caching algorithm accordingly.

    The thing is, I don't see this as being bad on the part of the drive (or microchip or compiler or whatever). Benchmarks are supposed to measure real world performance. If tuning your product to do well in a becnchmark doesn't make it perform better in the real world, I'd say that you need to get a new benchmark.
  • While the Exokernel stuff is interesting, I think that VIA-like optimizations (http://www.viarch.org as previously noted on /., owned by Compaq/Intel/MS) are a better way to go. The idea was originally from a thesis paper and implemented for Unix. (Author not handy due to a recently deceased HD.)

    VIA stands for Virtual Interface Architecture. The idea, generally, is to build device drivers that can directly interact with processes or threads with little or no interaction with the kernel. The idea is generally that the driver presents a virtual device interface to each process. The viarch.org guys appear to be concentrating on applying the idea to SAN I/O, but I think that the network/OS/Webserver (SMB, etc. also) is even more interesting given the smaller payloads and higher overhead.

    What I would like is to have a process or group of threads that would interface to the NIC and receive a stream of packets and transmit a stream of packets. The exchange could be completely async with large queues and no context switches or other OS interaction (for network traffic). The service processes would contain an integral TCP/IP stack and HTTP server. The device driver would send packets for a particular IP address only to the VIA processes while handling other IP addresses normally.

    This would allow optimizations just like those for the Exokernel. People could experiment with various zero copy schemes and other TCP/IP stack optimizations without affecting the operation of the system they were running on.

    We need to do this now!

    Is anyone working on something like this? Email me! I might even be able to provide some funding.

    sdw
  • Moving static requests back up into the kernel is a neat trick, to be sure. But I'd rather have web serving where it belongs: user space. The idea of being able to serve a page to every user on the planet five times over on a given day from my web server sounds nice on paper, but not very useful in real life.

    Having a bit of up close and personal AIX experience, I feel they should work on making their kernel a bit more stable, anyway. This is simply posturing for a benchmark, probably not real useful in a real-world scenario, and to be frank, I think IBM knows better.
  • All this complaining that the OS & software used to set this record was tweaked!!

    I don't see the problem. These people had access to source for AIX and they used it! Just the sort of thing that anyone with the skills can do to Open Source software.
  • >personal experience generally turns int personal
    >prejudice.
    >Example: how many slashdot
    >users have used/developed on more than one OS?
    >And the *nix world counts, essentially, as 1 OS,
    >btw.

    Some of us have. I used Macintosh computers for 4 years, then I primarily used windows for four years(95 for three years, 98 for about one), and I recently switched to linux, as I had gotten sick of windows crashing multiple times an hour. I also occasionally have used DOS, although that would probably be considered windoze by your definitions. Personal experience is not necessarily synonomous with prejudice-I consider linux the best operating system I have tried so far, but still intend to try other operating systems, and I have used windows, but I am not claiming that it is a superior operating system.

    irrelevant note-I call myself newbie, because I've been using linux since last sunday/monday.

  • Where's the overhead? (Score:1)

    by Anonymous Coward
    while dynamic vs. static is an important distinction, i think we're looking at it wrong.

    true, the majority of the data passed is static (images are almost all static), where's the big load show up? is it serving the images or generating the pages?

    as a side note, i generally agree that benchmarks are mostly worthless, showing performance in only a very narrow range of situations. perhaps it's time to start an open benchmark initiative (or, more likely, i missed it being announced ;^).

    /jbm, another moron who lost his password.
    time to get it back.
    jbm@intertek.net
  • The web-server IBM used was Apache (with their own name stamped on it). I believe IBM is releasing their improvements back to the community (but Apache's not GPL'd so they don't have to if they don't want to).

    --Ben
  • This came from the U-Net project at Cornell - http://www.cs.cornell.edu/U-Net/
  • AFAIK web polygraph is free (see http://polygraph.ircache.net).
  • People seem to be confusing several different performance-oriented activities, acting as though they're all the same. Here are some examples of things I've seen done:
    1. Changing already-existing parameters to suit a particular workload. It's really hard to argue that this isn't legit.
    2. Adding options/features that are only good for a particular workload. This is generally OK, especially if the machine is only sold for that purpose, or if the extra options/features can be turned off if the machine is used differently.
    3. Adding code that only helps the benchmark but nothing in the real world. For example, there was at least one case where a vendor shipped a compiler that would use pattern-matching to detect whether it was compiling Dhrystone and, if so, do a bunch of optimizations that would never be applied to any other code. A less extreme case might involve optimizing how a particular "weird" behavior or edge case just because a certain benchmark does things that way. This practice is generally frowned upon.
    4. The worst is when vendors make changes for the sake of a benchmark that actually sacrifice correctness. One example I had to fight against was failure to flush data to stable storage even when explicitly required to do so. Even the people who perpetrate this kind of thing don't consider it justifiable; they just hope they don't get caught.

    On a slightly different point, I just have to laugh at Linux advocates complaining about "hacking the kernel" to get better performance. Does anyone really think the very design of Linux's major subsystems hasn't been heavily influenced by a desire to improve performance in particular situations (notably web service)? In my experience there are about a hundred times as many Linux hackers worrying about performance than about correctness, which is why we allow abominations like a non-journaling filesystem with delayed metadata writes (because if you try to force synchrony you get undetected data corruption, and the authors admitted as much in a public paper). This is the very worst sort of benchmarketing, case 4 above. Linux advocates must tread very lightly indeed when they presume to criticize other OS vendors of benchmarketing sins.
  • Wouldn't dynamic web server performance be governed by other system standard benchmarks? It seems tome that generting dynamic content is a function of processor+disk+bus performance and so we already have good measures for these things.

Slashdot Top Deals

The most difficult thing in the world is to know how to do a thing and to watch someone else doing it wrong, without commenting. -- T.H. White

Close