My 4.5 modules worked with 4.6 with no changes. My 4.6 modules worked with 4.7 with no changes. My 4.7 modules worked with 5 with very minor changes. My 5 modules worked with 6 with very small changes. I haven't tried porting any 6 modules to 7 yet, but looking at the list of modules that already support 7, I'm not quivering with fright at the prospect.

The API is not guaranteed not to change, but a lot of people work hard to make sure that the change is evolutionary and the thousands of automated tests help to make sure that

I seriously object to my comments being marked as troll! This was not trolling. It's a serious point. It's hardly reasonable to say that there's one rule if it happens in the US, and another for everywhere else. Yes, I know that there were/are wars in both places, but there _have_ been deaths that were murder rather than battlefield deaths, and US citizens have not been held to account in courts of the countries where the actions took place.

http://www.huffingtonpost.com/2010/04/05/wikileaks-exposes-video-o_n_525569.html
http://edition.cnn.com/2009/WORLD/meast/05/12/iraq.soldiers.killed/
http://www.nytimes.com/2010/01/08/world/asia/08blackwater.html

Which is to say that there is no absolute rule that a person should be tried in the jurisdiction of the place under which the action had effect.

US citizens have been guilty of murder in Iraq and Afghanistan, and yet we do not see them on trial there. Why not?

The lesson I take from this is that it's not a rule that a crime is tried in the place in which it was committed.

You are absolutely right.

Except that there was a recent report that terahertz radiation was resonating DNA and partially unzipping it, creating 'bubbles' of unzipped regions. I haven't seen any corroboration of this, so it may not be proven, but it's certainly worth a pause because it's not an effect that had been predicted. Now mm waves are not sub-millimeter waves, but they are adjacent in the spectrum, and it's this sort of thing that we should worry about.

Probably there's no cause for concern, but again - when the potential impact is very high, though the risk is low, we should be taking great care.

The health effects of millimeter wave scanning are what we should be worried about - there's an unknown risk but a high possible impact: imagine if in 10 years time millions of people start developing melanomas as a result of being scanned.

The x-ray backscatter machines are much less worrying; we've had 100(?) years of experience with X-rays and we understand what x-rays can do to DNA.

But we have very little experience of mm level radiation.

What I've seen in the press is cheerleading. 'Experts say there is little cause to worry' with unknown experts talking in vague generalities. I've seen articles saying that the energy involved is less than the energy emitted by a cell-phone. That may well be the case, but it's not in the cell-phone spectrum, and even a little energy in the wrong place can do a lot of damage. Just see what a match can do to a pile of paper*.

Of course it's impossible to completely prove something is safe. But I don't think we don't have empirical evidence that it's safe. Or at least - I've not seen it.

I absolutely have not made an exhaustive search for literature on the health effects of mm radiation, and I'm not an expert. In the brief searches I have made I have seen that there's a lot of scare-noise based on what seems like only a few sources which imo are not applicable. What I don't see is a long list of studies. And even more striking is that that non-existent list of studies is not full of papers saying 'we found no observable effects'**.

But I have found that it's possible to cook bacteria with mm waves! Maybe this is a hint that there's a potential problem. And in the diagrams of atmospheric absorption of radiation that I've looked at, for example, it looks like mm waves are mostly absorbed, which suggests that we'd have little evolved defense to mm wave damage.

What I'd really like to see is a series of mm wave experts saying things like 'we've studied the health effects of mm wave scanners for 10 years now, and I'd have no qualms about subjecting my three month old baby to a scan because I'm confident that there's no health risk associated.'

* Pedants: yes, I know the energies involved are different orders of magnitude - it's a metaphor.

** It's quite possible I was looking in the wrong place. I'd be very pleased to see a detailed response with a link to this list.

Didn't American's fight a war of independence because of this? Maybe American politicians have forgotten, Someone seems to think that it's ok to make law without reference to the people it affect.

For the people who are helped by music when programming:

I listen to a pretty wide range of music, from classical through to current pop, with a lot of stuff inbetween. I find that some music that I like helps me to program, but other music, which I also like hinders me.

I work better with rhythmically strong music - a heavy beat without deep complexity. Music which encourages me to pay attention to lyrics is bad, but that might just be that familiarity is a requirement so as not to get sidetracked into _listening_ to the music.

The best music I know of for taking me to the zone is Talking Heads. And their best programming album of all for me is Stop Making Sense, which I must have listened to thousands of times.

It's starting to get repetitive. I would like some new programming music: what music helps you most?

Mail it to Linus, Alan Cox and the maintainers of subsystems which it abuses. Include clear notes of how it works, and what can be done to protect the systems. If you can't trust these people with it, then you should not trust Linux with your data at all. Even better, since you understand the tricks it uses, if you can write some patches, and submit them, together with your proof of exploit.

On a personal note - I also want to say thank you for doing this work. I use Linux both on servers, and as my normal desktop, and I'm immensely pleased that people are looking at making it safer: thank you.

If you are reading this, then you are probably under 45 years old.

I think the average life expectancy is rising - currently 75 years old or so in most Western countries - for men.

But more importantly medicine is going through an exponential increase in its abilities. It hasn't yet reached microprocessor rates of doubling every 18 months, but we certainly have learned more in about the last 20 or 30 years about how the human body works than we learned in all of history before that.

This strongly suggests that the average life expectancy will start rising exponentially too, with a lag to account for the development of applications from the basic discoveries.

So it is not unreasonable to expect some of you reading this to live to 150, barring any nasty accidents. And that means that climate change will have a direct impact on your life. You may have to live through mass starvation, wars fought over resources like water, extinction of once common species and the rise of diseases once only found in the tropics.

So, whether or not it's a totally proven phenomenon is irrelevant - _you_ personally can't afford to act as if it is unproven: you need to take personal action now, in the hope that it will help to mitigate the consequences.

We as a species need to learn the lessons of all the previous civilisations that have managed to extinguish themselves through resource starvation; don't spend time arguing that it may not be happening - act as if it is while hoping that it's not. Definitely carrying on with research at the same time. Maybe we will show that it's not happening, and that would be fantastic, but we can't wait for that.

I can't answer most of your questions because I haven't read the book, but I can tell you that a node is the smallest unit of addressable content that Drupal deals in. What's that in English, you ask? By addressable, I mean it has a URL, and can be displayed provided you have the rights to view it. By content I mean that it's something that someone entered into Drupal, and it's stored in the database. By smallest I mean that as a reader of the site, while you often view a node by itself, nodes can also be combined to produce a page - for instance by the Views module or the Panels module. Usually this combination is a node itself.

Nodes can also be differentiated from one another by their Content Type, for instance one node may be a Story or Article node, while another might be a House Listing node. This helps in making lists of nodes, and in searching them.

From a programming point of view, a node is an interface abstraction - if you write a module for dealing with a new sort of content, and want all the benefits of the framework (for example, making any data held in your new sort of content searchable), then you implement a few well-described interfaces, and voila!, Drupal is able to manipulate your new sort of content. This interface is in the form of hooks that you implement for your new sort of content.

You are also free to ignore these hooks, putting your data outside of the range of what Drupal considers to be content. An example of a module that does this is Webform - which is used to collect forms that site users fill in. This data does not appear in the Drupal search index, or benefit from any of the automatic rights that nodes enjoy.

Sorry about the state of the documentation: it's hard to find the right level for everyone. Perhaps it needs more signposting.

Contrariwise: it's very well structured. Each module has a defined structure. The core is modular too, with functionality being in predictable places. Functions are given predictable names, thanks to the hook mechanism.

If you are going to contend that it could have been laid out better, give firm examples.

Isolation. PHP4 does not have namespacing. PHP4 does not support interfaces. Until recently PHP4 was supported by Drupal. The point of isolation is to enhance security and attempt to limit the effect of bugs. Given that the language makes few attempts to support programming-in-the-large, it's a bit much to expect software developed using it to do so.

I suspect that by 'interfaces are pretty much defined by giving modules access to the entire core', you meant that you would like to sprinkle getters and setters liberally, then you are right. Drupal eschews a layer of make-work that prettifies without adding any value. PHP is not an especially speedy language, and adding them would have little benefit in return for a unhealthy slowdown. If you mean that the there is an agreed structure to key variables, and they are passed between procedures without limiting access to subparts, then yes, you are right. You might expect this to cause problems, but in my experience, it doesn't. Partly this is because of the test layer. Partly because the system depends upon modules obeying implicit rules.

I see no cop-out at all. Modules that respect the rules of interaction work fine together. Where modules are known not to work together, this is documented.

'If the module API isolated internals...' This isn't possible in PHP without using objects. PHP4 objects did not support the hook mechanism. A smart a object-based hook mechanism for PHP5, that didn't require registration of each hook, only became possible with magic methods. Only supported since PHP5. And last time I looked the performance of magic methods was abysmal.

'The code is garbage all the way through.' Another sweeping statement. Clearly all the tens of thousands of lines of code are rubbish.

Globals. The point of reducing the use of globals is to minimise the impact of a change in one place impacting unexpectedly in another. Drupal6 has 45 globals. Most of these are sensible as globals: for instance $language - an object containing information for the active language. Of these 45 only 1 is in wide use: $user which represents the current user, storing their preferences.

View, logic & database access. Database - Drupal contains a database independent data layer and has since Drupal5. All access is through the database layer. Views layer: Drupal has a strong theming layer, with more than 1 theming engine to choose from. All presentation is through themes. The data passed to themes is a limited structured subset of the data held by the core. Hooking into output requires you to work through the theme layer. Logic is separate from the theme layer and is usually event driven. Ownership of these tasks is implicit in the names of the functions - for instance - mymodule_block_theme is the callback for theming the block output by mymodule.

Methods are long. Yes, sometimes. Sometimes not. It depends on what is sensible. The same chunks of code: I'm sure there are repeats. I've not seen whatever it is that you mean - perhaps some examples.

I don't contend that Drupal is perfect. It's not. But what I see in your list is a bunch of accusations, with no proof.

Yes, each version is an evolutionary jump on the previous version. Not surprisingly, D6 has got better caching than D5.

Boost: It replaces pages with static html. This static cache is updated on a schedule. But if you have someone who is logged in, because it completely bypasses PHP, you have to replace all the page elements that are different for logged in people with Javascript driven widgets which load the block data dynamically. For a site with lots of elements this is a lot of work. Plus you are depending on Javascript being switched on. So if you have lots of logged in users, and lots of blocks just for logged in users, and you don't have much control over their environment the site is viewed in, then this isn't really a starter.

Global cache clear whacks everything, obviously, but it's quite surprising to me that item/page cache clear does this. I've never had a close look at caching on D5, but knowing that there's a potential problem here is useful. Thanks.

I'm guessing that you already track the groups.drupal.org that deal with high volume sites and the distributions intended for your situation like PressFlow, Mercury, OpenPublish, and http://groups.drupal.org/high-performance. Pressflow in particular has spent lots of effort on optimisation, and there's a D5 version. Pressflow also only supports MySQL, which answers one of your concerns. Maybe you've already trawled through it, looking at SQL optimisations they've done.

InnoDB: I can't comment, but I'm pretty sure that someone in the high-performance group has experience to offer. I think most high-performance sites are using InnoDB, and I'm pretty sure it's the Pressflow default. I know there's an issue with node counts on InnoDB, and I think that Pressflow changes the way that this is handled.

BTW - check out http://groups.drupal.org/node/25617 not so much for the comparison of performance stats on D6, because it's concerned with non-logged in users, but because the discussion below the article will probably interest you.

Once more, thanks for your very good posts; it's always useful to learn other people's experiences.

Thanks for the detailed reply. Sorry, I wasn't meaning to teach you to suck eggs. That part of my reply was for readers who don't know much about Drupal beyond it having a strange name.

Interesting: I didn't know about the table lock on cache! Are you using Drupal 6? I've just done a search, but didn't find anything. I did find a status report I didn't know about! At admin/reports/status/sql which shows lock hits amongst other things. Maybe you are on Drupal 5? The cache updates I found where all straight inserts/updates or deletes. But whatever - I'd say that you were right in moving caching to memcached if you have to support millions of pageviews.

Yes - I agree the cache layer needs PHP execution, and that's not so great. Actually, IMO it's not the PHP, but that it's a hit against the database, even though it is a single query. A hit against the file system would avoid the whole DB connection and query. Drupal Boost caching module wouldn't bring huge benefits for you because you have lots of logged in users, but it uses static files and doesn't hit the database.

Block cache can be efficient at cutting down database hits and code execution. And if you are using Views2, it will also use block caching for HTML and queries. It has options for lots of different caching strategies - from one for everyone through to per user per block. Block caching not well documented though.

One thing that I'm not sure of is why one website would interfere with the cache of another if you are not sharing content. [Readers: you can configure Drupal to use a single database for all websites, or multiple databases. If you have a single database, then you can configure each installation to use a different set of tables, through setting a prefix to the table name. It's very unusual to share any of the cache tables across sites except in the shared-content scenario, where the same content is being presented on more than one website.] Maybe I've misunderstood. It might be that the sites are running on a cluster against a db shared across the cluster. In this scenario, I can imagine one machine locking others out, though I'm not sure where in the code a table lock is.

Multisite: Good points and bad points. The management of update is the bad point, when you put in a new module with database upgrades. You have global downtime unless you do progressive updates by putting the module first in the specific directory for each of the particular sites and running update for that site. But it does lower the admin cost.

SQL rewriting works in some circumstances. See hook_db_rewrite_sql. [Readers: The base problem is that it's difficult with modularity to avoid explosion of queries. This is true of all modular systems with database interactivity. You can avoid it if one module is intimately aware of another, and then the two can co-operate, though this is context dependent. For instance - if the first module loads the user details (name, user account, email, etc), and a second, optional, module loads a user profile, then you could cut down on the queries by including the user profile columns as part of the user table, and loading the two at the same time. Modularity, however, is not your friend here - to make the management of a profile module easy, it's much better to have it in its own table. But now you are doing table joins or loading one and then the other. If you have a heavily customised system, then you can make the joined table by hand, and modify the queries. You can do this either by changing the core code, or by rewriting the SQL on-the-fly, by writing a new module. With the second option you can leave the core alone, and your changes are overlayed, at the cost of a small drop in performance of the PHP (since there's an extra function to run). The PHP drop is almost certainly negligible compared to the database speed-up, except where the database is already in memory.]

I absolutely agree - look at the source code - one of the reasons that we standardised on Drupal was that the code is well structured and well thought out. It's based on a call-back system with hooks triggered by naming convention. It is well documented, and has a wide series of tests that are automatically run against it, for regression tests of changes. The API is documented at api.drupal.org, and includes examples of how you can extend the system.

All the above applies to the core code, and the widely used modules just outside core - like Views, which is a module to help non-programmers build structured queries and present the results.

I agree that amongst the third-tier modules, the quality varies hugely, as you see in much open-source software. Some of them are very well written, but others are not.

Each module has an issue queue, which is publicly visible, and usage statistics tell you how widely it's used, which allows you to make some decisions even if you aren't a coder.

The only criticism I have with the core code is a technical one that doesn't matter to end-users; most of it is not object-orientated. This is not to say that it's unstructured: it is very modular. And there's a good reason it's not object oriented: it's because PHP4 made it very difficult to build a name triggered callback system with objects. However, I expect the core will become more and more object oriented over time since PHP5 objects are much more robust with reasonable performance characteristics.

What is large-ish in your context? I'm not sure that readers will realise that you are coming from a particular point of view. I maintain a Drupal installation with 1.5m nodes, which I wouldn't call large, because it has a limited purpose and does not use hundreds of extra modules. We have not had to patch or customise. And it runs on a VM on commodity hardware.

I assume that you have a site with a great deal of functionality. Not every reader will realise that a CMS that is modular is naturally going to result in an explosion of queries; every module controls its own queries, each with its own tables. There are several ways of combating this: you can cache content and results, you can lazy-load daa, or you can reduce modularity, combining tables.

Drupal already uses the first two strategies, and occasionally prticular modules are designed to work together and implement the third. For anonymous users, pages are delivered from cache. For logged in users pages can be put together from cached blocks, if your site is set up to do this. (There are some restrictions on this; it's not available in every circumstance.) It sounds like you may have shared content which is an unusual installation. Under normal installations, one site of a multisite install does not ineract with another site. The caching is not designed with shared content multisites in mind; I've also had problems with this, but I think that this is an edge case. It would be great if your organisation would contribute your experience, by posting your rewritten cache engine.

I don't however agree with your suggestion of a focus on a single database. Yes, specialising to a single database will allow the use of features available on that database only, but at the cost of a portion of the audience. Similarly, we could say only a particular version of a particular database was supported, to allow us to use the optimisations available for that engine, but again this would cut the audience dramatically. Instead what is needed is a structured way to optimise. To a small extent this is already available through the SQL rewite hook, but this isn't especially safe. Drupal 7 deals with this by abstracting the SQL further, but the SQL rewrite is much safer, and more suited to rewrites for optimising.

I'd be really interested in where you've had full table locking problems. Do you have an issue reference on drupal.org?

Regarding "... if you're looking for something to build a high-traffic site with plenty of customization and member-only data, you're better off building your own framework from scratch..." With any CMS there's a trade-off: you can build your own, bearing the full cost yourself, or you can use one that's built already, whether closed or open source, amortizing the cost of the build over many organisations and individuals. By my estimation, Drupal core represents many millions of dollars of work. It's possible that your organisation can afford to invest that sort of amount in your own custom CMS, but the experience of companies that have built their own is that the cost of maintenance is too high, and many of them are migrating to open source software, which does more than they could hope to do, at a cheaper price, but which comes with its own problems. Organisations can use the money saved on the build to either make larger sites, more sites, or specialise the system, which is what your organisation appears to be doing.

It would be fantastic if others could learn from your DBA experience. Would you consider posting a case study to drupal.org?