The things you choose to engage on is kind of odd.

That observation was very much of a tangential to my argument. I was speaking more generally there as McKinsey has published several high level analyses of healthcare for public consumption and you seemed to be making a broader statement (RE: "I'll rely on McKinsey"). I know a a good number of current and former McKinsey people and stand by what I said.

Jean Drouin is in his mid40s

We're about the same age and surely know several people in common between Princeton, Stanford, and McKinsey. I would not call him a kid now, but at the time the particular analysis you cited was published (2008) he would have only been a few years out of school. Further, there's a good chance much of the (percieved) scutwork has performed by people even more junior than he was at the time.

The point is not that I have any particular reason to believe any of them are stupid or particularly naive, but that I don't put much weight on so-called expertise generally (especially not when I have better information to go on) and even less on this particular "mile wide inch deep" approach to the world. Getting to the bottom of topics like this tends to require a lot close independent study of the details and the McKinseys of the world basically represent the antithesis of this approach (not usually a lot of value add).

And if you think what you're linking to is "science", you're indulging in more of the same

It is the approach the makes it science whereas appealing to expert opinion is pretty much the opposite of science.

I think you may be confusing me with someone who gives a shit about arguing economics with you.

In other words, "I don't have any substantive objections and my sole contribution here is to blithely paste links to widely cited documents that just about everyone has already seen". Why even bother?

I'll rely on McKinsey and Stiglitz, thanks.

I'll take science over expertise any day. I am especially wary when the so-called expert analysis is a little more than the rudimentary analysis of a bunch of kids fresh out of school with only a skin-deep exposure to these topics (I have known many such people, including peers of these authors, and while most of them are reasonably intelligent people, their bluster far exceeds their subject matter expertise).

As for Stiglitz, he has not published anything on healthcare and shared any sort of rigorous analysis, so far as I can tell, so I see no reason to defer to his professed (left wing) preferences. He has, however, published stuff closer to his knitting with Amartya Sen, Jean-Paul Fitouss, and others that pertains to the discussion. Indeed, they recommend AIC and AHDI over and above GDP as an indicator of material living conditions.

Here are just a few instructive quotes:

"GDP mainly measures market production.... However, it has often been treated as if it were a measure of economic well-being. Conflating the two can lead to misleading indications about how well-off people are...Material living standards are more closely associated with measures of net national income, real household income and consumption – production can expand while income decreases or vice versa when account is taken of depreciation, income flows into and out of a country, and differences between the prices of output and the prices of consumer products.
[snip]
In a world of globalization, there may be large differences between the income of a country’s citizens and measures of domestic production, but the former is clearly more relevant for measuring the well-being of citizens...the household sector is particularly relevant for our considerations, and for households the income perspective is much more appropriate than measures of production.
[snip]
While it is informative to track the performance of economies as a whole, trends in citizens’ material living standards are better followed through measures of household income and consumption. Indeed, the available national accounts data shows that in a number of OECD countries real household income has grown quite differently from real GDP per capita, and typically at a lower rate. The household perspective entails taking account of payments between sectors, such as taxes going to government, social benefits coming from government, and interest payments on household loans going to financial corporations.
[snip]
Properly defined, household income and consumption should also reflect in-kind services provided by government, such as subsidized health care and educational services. A major effort of statistical reconciliation will also be required to understand why certain measures such as household income can move differently depending on the underlying statistical source"

Also, you mention Legatum as a data source

As a source? No, I don't think you don't understand. These services, of which Legatum is just one, have helpfully aggregated a wide variety of indicators from various 3rd parties that measure the sorts of lives that people live (health, happiness, material conditions, etc) and these indicators are overwhelmingly better predicted by household measures like AIC, the same measures that Stiglitz et al recommend, than GDP.

For instance:

Social Progress Index: r-squared and OLS.

World Happiness Survey: r-squared and OLS

Legatum: r-squared and OLS

The point, to be clear, is not that (self-reported) happiness or what have you causes more health spending, but that AIC and AHDI actually measure the household perspective and that these measures are theoretically and empirically more robust predictors of the behavior of a society than domestic production levels (GDP). Overwhelmingly, GDP only predicts "good" outcomes to the extent it predicts the household perspective, but once we have robust household measures (e.g., AIC, AHDI, etc) GDP tends to be insignificant to even a net negative on the margins (i.e., conditional on AIC, countries with more net exports, CFC, etc don't enjoy significantly better outcomes and often do worse in the great majority of cases)

In other words, it is not just luck that AIC or AHDI fully mediate GDP with respect to HCE, as we find the same patterns with just about anything else of significance, and these same measures quite well explain why the US spends as much as it does. Whether or not high US spending is "worth it" is another question entirely, but the evidence suggests this spending is not substantially a result of idiosyncratic features of the US health spending and that other countries behave very similarly with respect to the household perspective (despite essentially no relationship between spending and life expectancy amongst the OECD)

That's not much of an argument.

A few points:

1) Adjusted Household Disposable Income and Actual Individual Consumption are widely acknowledged by people that have studied this to be superior indicators of material living conditions. GDP is a measure of domestic production, full stop. It is not and was never intended to be a measure of resources actually available to households and it is the household perspective that matters here. GDP is often used as a proxy for these types of measures in lieu of better data, but they're not the same and they can and do deviate quite significantly for a number of reasons.

2) When it comes to predicting national health expenditures and other health system characteristics, these measures, AIC and AHDI, fully mediate GDP in multiple regression and in multiple specifications (much the same if one subtracts HCE from these household measures).

3) CMS uses a close analog to AHDI, disposable personal income, as the dominant exogenous variable in their long-term projections because their research and theory suggest it's much superior as a predictor.

4) These measures are also much stronger predictors of essentially all other measures of living conditions (e.g., life expectancy, life satisfaction, satisfaction with financial conditions, poverty rates, access to clean water, etc). Indeed, if one sets about systematically comparing indicators from organizations like Social Progress Index, Gallup/WVS, Legatum, and others, at least 90% of these indicators have markedly higher r-squared with AIC than GDP. Likewise, in OLS, AIC clearly mediates GDP or, if one disaggregates GDP, the remaining components of GDP (e.g.,net exports, CFC, etc).

5) There is also tremendous consistency in consumption patterns if one looks at disaggregated SNA data by function (COICOP and the like). The patterns the US exhibits in consumption across individual categories/functions is highly consistent with its aggregate AIC, adjusted household disposable income, and so on. The US consumes much more of almost everything in real terms (especially those goods and services that are clearly elastic at a national level). It certainly consumes more in most categories than the handful of significant countries with higher GDP, so it can hardly be surprising that the US would also consume much more healthcare (one of the most elastic categories with respect to national income).

Note: These results are also consistent with what we find if we look at other sorts of more granular measures (e.g., number of rooms per capita, household possessions, private car ownership rates, frequency eating out, etc).

What you are talking about in terms of running a small business is risky. It is risky because when Best Buy's Geek Squad, or any local large operator sees you as a threat, it won't be long before you are eliminated one way or another. Sometimes it's possible to operate "under the radar" but the fact that anyone would need to do so is clear indication that one already knows that there are risks of being destroyed by larger, predatory businesses.

I beg to differ. No doubt being an entrepreneur is risky and requires a lot of hard work (in most cases), but the overwhelming majority that do fail fail for reasons that have little to do with any kind of direct response from the competition (let alone anything unethical), e.g., insufficient capitalization, poor financial management, il-concieved products/services, bad execution/implementation, etc.

Big companies have a lot more capital and resources to throw at problems, but the reality is that are often slow to respond to anything and, when they do "respond", their response is based on group-think and a fundamental lack of understanding for the marketplaces in which they operate. These big companies remain successful because they have inertia and a lot of capital to acquire younger companies, not because they are effective at killing the competition or finding new areas for growth.

I have seen this time and time again, first hand as an entrepreneur and coming from a family of entrepreneurs, many of whom that have gone head to head against some of the largest companies in this country and come to be the dominate player in their chosen the market or at least carve out a strong niche for themselves, allowing for the many stakeholders to profit handsomely (including employees), without any of the sorts of shady behavior you imply.

I have known at least a few instances where the founders have actually cheered when their primary competition was acquired by multi-billion dollar corporation since they have a strong conviction that these big companies, even though being known to try to play a little dirty at times and have more capital to spend on things like marketing, actually lack the discipline to be as an effective competitor as the relatively smaller prior-organization.

To re-iterate, most companies fail due the reasons of their own making. To the extent competition is an issue at all, it's more that their product or service fails to provide a sufficiently compelling reason from customers to switch from the existing products/services or have something enough different to attract wholly new customers.

Finally, I'll point out that it's not a zero-sum game. Many markets have a lot of potential room for growth. Sometimes new competition is the best thing that can happen for all (or at least most) market plaeyrs because brings in some ideas/vigour and creates vital competition where before little existed -- which spurs all the companies to invest in R&D -- which leads to growth for all as the products become substantially more attractive for all.

One huge problem with these sorts of debates is that people seem to think that actionable policy descends automatically from scientific conclusions as if it is simply a matter of logic. This is simply not the case because personal values, risk preference, economic principles, and more play a necessary and important role in decision making.

For instance, although the science is fairly certain that the earth has warmed over the last 150 years and that CO2 acts as a greenhouse gas in and of itself, it does not automatically follow that we should abandon anything that produces CO2 or other GHG. Even if we were absolutely certain that X tons of additional CO2 today result in actual damages Y tomorrow (which we are nowhere close to), rational people can still choose other courses of action because they believe that the costs of said action are less than the proposed solutions today. Unfortunately many scientists have unnecessarily politicized this area by conflating their scientific findings with their personal preferences on policy. They are NOT the same thing.

Both science and policy outcomes are likely to be better when society appreciates the difference and proceeds with their debate accordingly.

To be very clear about this: I have argued from the start that most small to medium sized businesses would be better off outsourcing typically generic services like email because outsource companies enjoy systematic advantages (scale, specialization, expertise, etc). I was also specifically challenging your apparently sweeping arguments that they should be categorically rejected " simply because of liability and privacy" or because they are "not tightly bound by laws regarding retention and usage". You may have intended this to refer only to Google Mail/Apps specifically, but I think it was fair to interpret them as a broader attack on outsourcing IT services generally. Regardless, that is the point I took issue with.

however, all of this is academic and is entirely dependent upon process, policy,hardware, and software. It's like stating that it's less risky to let someone else drive than it is to drive your own car. In some cases yes, in my case, I would argue no, it all depends on the details - and again, I wasn't making generalizations, I was stating my opinion about why I don't use it and the opinion of those in similar positions to myself whose opinion I am aware of.

I agree there can be unique circumstances that make outsourcing a poor choice, but I think you overstate your case. A typical outsourced email operation enjoys substantial structural advantages that an in-house operation rarely has given more standard requirements. A better analogy would be asking whether you're better off flying in a single-engine plane with a private weekend pilot or flying commercial airlines in the US on a wide-body jet. OK, that's perhaps an overstatement with respect to comparative risks, but nevertheless....

But why would a 50 person company need 24-7 support for office applications and e-mail?

Some medical devices business, for instance, require this kind of support (my last company) since they are supporting patients in a clinical capacity nationwide 24-7. Likewise, some of my current clients require support at 10PM or later since they're sending multi-million dollar proposals at the last minute (a lot of money at stake). In any event, my point is that a 24-7 operation has an easier time doing maintenance (since they can schedule it after hours easily) and are more likely to be able to attack a problem as soon as it is detected (which may well be earlier too since their operations are often more professional/proactive). Perhaps you can have your admins stay till 5AM to fix a problem or do a routine upgrade, but they're probably going to make more mistakes because they're tired and will be of little use the following morning when/if something blows the following morning.

I'm sorry but we're an ISV and we have some pretty sharp people here, but none of our 'non techies' would easily be able to fill the role of a squashed Google Apps admin or manage mail difficulties, or be able to convey those issues intelligently to whatever support mechanism Google has in place.

I bet they can handle most of the routine stuff (e.g., add/drop/change accounts) long enough to comfortably locate a replacement (which, btw, probably is NOT a full-time IT person)-- certainly far better than they are to be able to re-build a RAID array or deal with a complex AD replication issue.

What downtime? Are you envisioning some scenario where there's a private company with an understaffed, overworked IT group and someone burns down a file server someplace right before a big sales presentation? While certainly what I would term 'uncommon' it is a very possible scenario; however, it is simple to argue that a small company could at least remedy the situation themselves whereas Google losing your mail (as has happened), or Google Apps not being reachable (happened several times that I'm aware of for long periods of time) is something you can do absolutely nothing about. That's ignoring the obvious problems with "Hey Bob, where's that spreadsheet you had ready for the board meeting today?" "Uh, well, I can't show it from my laptop because when I went to synch it this morning, we were having ISP difficulties, so I couldn't get it off the cloud..." In all instances I can think of, the worst case scenarios are all better on the 'no Google Apps' side of the river.

(Again, I do NOT think Google Docs is a replacement for MS Office generally speaking). Where I think you go wrong is in confusing the availability of Google Mail, practically the lowest possible cost option for an actual company, with the significantly higher cost hosted Exchange options using more stable software and established technology. I would challenge you to try to offer a better emails/contacts/calendar experience for all of your 50 users for a mere $2500 a year (including server acquisition, IT overhead, data center, bandwidth, licensing, etc) which is what Google is charging. I bet you spend at least twice this much in IT manpower alone to deal with backup/maintenance/issues/extra time necessary to handle it as-is (without more exotic/expensive systems to ensure higher levels of availability)

Why? Adding 50 people to your staff is much more a problem relating to hardware, imaging, OS infrastructure than it is to e-mail, file server space, or office productivity software. Adding 50 people to your staff and giving them an approach to office collaboration they've likely never used, seen, or heard of, would be a much bigger problem. Again, I can imagine scenarios where what you're saying is absolutely correct, but I certainly don't think it applies in the general case and certainly not in my case(s)

(Again, not talking about Google Docs!) Consider the costs to step-up server, licensing, IT staff, data center size, bandwidth, etc as you grow or add services. If you buy a server hoping it will last for 2 years as rapidly growing company, you will probably be forced to buy a lot more capacity than you really need if you think you will reach, say, 100 users in the next year and/or as mailbox sizes expand (if you allow this). What about your data center when your company needs to move to a new office space? It's kind of hard to hire 0.2 sysadmins. Ok, maybe you can find a decent part-timer where you are but you lose something in the process (imho). These are just a few examples of stepping issues.

I don't understand why using Google Apps or Google Mail would keep a company, who needed data centers to accomplish their goals, from needing data centers anyhow? Are you saying that these data centers exist simply to support something like Microsoft Office and Exchange?

Actually for this $60M (250+ FTE and rapidly growing) client of mine I've kept them out of a needing to invest in a data center entirely with judicious use of outsource services and I am running their entire IT operation with around 5-10 hours a week from my company (a bit pricey on an hourly basis, but they get a lot more value too). I don't need to be on-site to deal with any critical servers. With the servers that I do directly manage at Rackspace, I can be confident that if a hard-drive or power supply fails, it will be replaced in less than an hour--even if it happens at 2AM and I'm under some bus tire :-). This frees me up to focus on the tasks that really offer value and means that I can bring on a relatively junior full-time IT guy to manage the day-to-day issues when I go off-premises in a few months, after I've delivered a major piece of software, and that it will be more than enough even with increased demand. Likewise, I know when this client outgrows their current suite in 10-12 months I will be able to manage the move painlessly and with minimal investment of time.

I think it would be very simple to argue the exact opposite. Google is quite obviously a giant, virtually irresistible target to hackers. Given the hacking of their e-mail systems most recently, I would think this would actually be an argument against using Google mail for things you want to be private. The more and more people who use Google Apps, the more likely Google Apps will be penetrated (I know, that's obvious.)

My point here was that if you are a tenant in some kind of shared facility and you're a relatively obscure 50 person company in a relatively unsexy area, your data is not going to be of much interest to most would-be hackers or snooping employees. Google itself is obviously a huge name target for hackers (on the other hand, Exchange as an application is at least as large of a target, with highly exposed code (albeit in machine code), and it's difficult for Microsoft/Customers to test and deploy patches across the many different configurations and millions of installations.

Having also been acquired by large companies (both Microsoft and then at another company by SIEMENS) and 'integrated' (love that term which should be worded 'ugh, you change now, do things same way we do'),

Agreed :-) The acquirer of my last company sent in a "SWAT" team from IBM, directed by their usual middling mid-level yes-man people, (they treat them like dirt) and some other consultants to do the remaining integration that I resisted after I gave notice (they didn't understand wtf they were talking about) and it was epic fail by all accounts. Sigh. They're still trying to shoehorn their ERP system, several years later, at the cost of several million in customization (and that's just the estimate!) and it's going to massive clusterf*ck if they pull the trigger. I am half tempted to just deliver a clean functional (fully buzzword compliant) re-write and offer to sell it to them for half the cost (a mere $2M), but they'd never go for it if it doesn't come from SAP or one of their other established business partners. Then again, I think I'd probably commit seppuku if I had to deal with their C-level operating company people again (or, worse, wanna-bes) for any extended period of time.

I am simply making a general case for the pros and cons of outsourcing email and related services (albeit a bit more pro in the general sense). I am not trying to argue for Google Apps specifically and, in fact, would generally not recommend the product today to most businesses unless their needs were very minimal and/or cost was the overriding concern.

I don't believe that I'm giving short shrift to any "more likely risks."

I only meant this insofar as your actual argument on /. goes, not that you are making an inappropriate decision for your environment.

You may consider the failure rate of industry standard backup hardware to be riskier than storing all of your contracts, designs, patent documents, and corporate e-mail on a publicly traded U.S. company's servers, but I do not.

In my experience and in the experience of many others, there is a high failure rate when actual restores are required -- particularly in smaller organizations that do not systematically test their backups on a regular basis -- and it has less to do with hardware failure than with misconfiguration, small backup windows, occasional media failure, poorly understand expectations/training (e.g., time, granularity, etc), improper storage of tapes, etc. Furthermore, the shortcomings of doing it in-house extend beyond just the ability to do a successful timely restores. In the organizations I have managed, for instance, a few hours of downtime can easily cost them a million dollars in lost sales, not to mention potential hassles/penalty from regulatory agencies, customer relations issues, etc.

I do not see what baring the outsource company's publicly traded status has to do with this debate. Regardless, it is difficult for me to imagine a situation in the various reputable service providers today wherein senior management would make a decision to systematically rifle through their clients files--particularly not if they are contractually obligated not to do things like this and will likely lose customers (at the very least) if they are ever caught. However, I am willing to acknowledge it as a potential risk (however remote). I might be a bit more sympathetic to this argument if said company is a potential competitor of yours or might be likely to want to acquire you at some point. The more realistic concern, imho, is whether or not they have good security procedures in place and whether they actually execute on them to prevent their own employees from misdeeds or some outside hacker (likewise for in-house operations!).

Furthermore, if you really have this much truly sensitive material online that could threaten the very survival of your organization if leaked/stolen (i.e., you have a competitors that could actually substantially steal your IP and get away with it), I would argue that putting it all online in a readily accessible manner is probably a mistake no matter where you host it and email is generally not treated this way. If you're not behind secure firewall/VPN, with mandatory two-factor authentication, w/ minimal granular access levels, auditing, segregation of duties, etc in place.... the concern of company management stealing secrets seems a comparatively remote risk.

How quickly can we respond to a problem after hours? What does this have to do with Google Apps? Or is this now about all 3rd party services?

My point here is that most good service providers make strong commitments to high availability and have the resources in place to actually deliver on it well. Most 50 person companies simply cannot afford to staff even one highly qualified sysadmin 24-7, never mind all the other resources necessary to deliver. Without this kind of staffing, it's harder to do necessary maintenance and respond quickly to problems as they happen, i.e., before they impact primary business hours.

What happens if the admin gets hit by a bus (I use this analogy at the office a lot as well ;)) ? The same thing that happens if the person administering our usage of Google Apps gets hit by a bus.

A company can easily share credentials with a number of reasonably intelligent people to allow them to fill this role quite easily because it's doesn't require much technical know-how or knowledge of the environment. The situation gets very different when you get an even moderately complex in-house IT environment because it requires a lot more skill to operate reliably not to mention specific knowledge of the environment as it is configured.

If you need me to make a generalization I would be happy to state that I find it difficult (but not impossible) to imagine scenarios where private companies should use Google Apps.

I would argue pretty much exactly the opposite (albeit not specific to Google Apps).

A private company likely has:

1) fewer resources to survive lost sales or opportunity b/c of downtime.
2) less scale to make high availability cost effective
3) far more cash flow sensitive (survival)
4) larger stepping problems with rapid growth/movement/strategic reposition
5) can generally use extra cash flow far more productively elsewhere in the organization, i.e., better to hire an engineer or good salesman than buying IT hardware or hiring an additional IT employee.
6) more likely to need to re-locate and thus re-invest in their data centers (been in that situation several times)
7) probably a bit less likely to be a target of some hacker.

Having managed IT very much in-house from 10 to 800+ employees (very IT intensive), from private to publicly traded, to being subsequently acquired and having to integrate into a large (Fortune 50) company, and having consulted for a variety of SMBs (especially start-ups) outside of this... I can say with confidence that most would be better advised to outsource their email and other similar services unless they have very clear reasons for not doing so or already have a large investment in IT for whatever reason that cannot be more efficiently utilized elsewhere.

I am not claiming that all organizations that reject it do not have valid reasons. There are valid reasons for companies not to use it today -- even for some smaller companies. However, many IT organizations make these sorts of decision in an ignorant and/or self-serving fashion.

I have known many people in IT that think they will promote their own careers by maximizing their budget, head count under them, getting experience with latest/sexiest technology, etc without really considering the company's needs. I have known other that simply lack the ability to think critically about the issues (e.g., use an arbitrary requirement to rule out alternatives without considering the cost v benefit). For instance, they look at the per mailbox cost but fail to really take into consideration just how much overhead managing said technology in-house incurs or the cost of the risk they take by using shortcuts. Others are simply very conservative by nature (in the sense of personal career risk) and until outsourcing becomes the norm in their line of business or management essentially insists, they will not go with the program.

I do not know your organization. However, I wonder whether your preoccupation with trust in your IT organization is informed and rational. Yes, it is possible that the outsource company might do something in more of a top-down fashion that is not aligned with your company's interests. However, those businesses that actually profit substantially directly from their users (e.g., not gmail) have a strong incentive not to do something that might cause their client to lose confidence in them. A company that abuses their clients will ultimately hurt themselves.

In the mean time, I think you are giving short shrift to other more likely risks that correlate strongly with how these services are actually provisioned and managed. In other words, data loss, extended downtime, poor security, malfeasance of IT/operations folk, etc. An outsource company that serves one thousand times as many users with similar needs is far more likely to be able to do that job better and more cost effectively because they specialize in it and have the scale to do is cost effectively (though, as I said, the business is only came into maturity in the past few years and I believe it will take more time to be able to accommodate a wider range of requirements).

Consider:

How often do you test your backups?
How quickly can you respond to problems after hours?
What happens if you or key admin(s) gets hit by the proverbial bus?
How much expertise do your admins really have with your mail servers?
How many people have admin level access?
How many vendors or non-IT people have access to your data center(s) for various reasons?
Do you have offsite DR -- how good is it really?
What controls do you have against internal malfeasance?
How good are your data centers really?

My only point here is that you can't just look at "trust" in an outsource company in isolation of the other risks that hinge on decisions you make in this context. Your environment might be tighter and more cost effective given the stuff you really need but I haven't seen you articulate why this might be the case. Just some food for thought.

As a former CIO, I disagree with your diagnosis of the issues. Many companies, both large and small, outsource services to companies with access to all manner of sensitive materials (e.g., documentation destruction, electronic reading rooms, business continuity services, AR, etc). The difference is how those services are implemented and the trust in the organizations, not so much the laws that specifically regulate their offerings or even the ability to sue them.

In my opinion, the problem with Google Apps is that they:

1) don't make many important explicit commitments (e.g., availability, security, retention policies, restoration times, etc)
2) provide very little visibility into their implementation
3) their low cost service model provides little room for day-to-day customer service (e.g., mailbox restore) and the confidence to know that you can rapidly escalate a problem should one arise (not to mention offline backup)

I say this because this implies the issue is not inherent to outsourcing email in principle. The outsource service model is the future for generally commoditized services like email. There are several offerings today that I believe are generally superior to in-house for most SMBs that want Exchange functionality and need good availability. I have recommended Rackspace's Hosted Exchange to a $60M (revenues) client of mine and a few others. I am generally quite pleased with it, though there are a few shortcomings that will prevent others from adopting it today (especially larger organizations).

The biggest issues with the various Hosted Exchange offerings (those I'm familiar with at least):

#1: Authentication cannot be readily shared with other services, i.e., the employees need to juggle yet one more set of credentials.
#2: Limited ability to use 3rd party software (e.g., VM, Fax, two-factor authentication systems, etc) unless it exclusively uses exposed interfaces (RPC/HTTP, IMAP, etc).
#3: Won't scale well with large companies (with multiple subsidiaries/operating companies) that need/want to use more advanced AD features.

That said, these companies will figure most of this stuff out gradually until all but the most conservative big companies concede that they are better off outsourcing it, i.e., that an outside company has the scale and expertise to do a better job at less cost and in a more capital friendly way. When real customization is required then in-house makes sense, but the reality is that many of these issues are fairly widely felt and can be addressed with more generalized solutions.

If I can do anything for the cause of science, it's to repeat this: Scientists get famous by ripping the shit out of other scientists' work. The famous scientists you've heard of got famous by demolishing the work of others. As scientists, we know that. And we're always looking for some schmuck to use as a stepping stone. I know if I do bad science, I'll be a stepping stone. I know if I find bad science, I can use IT as a stepping stone. That keeps most scientists pretty damn honest.

These CRU emails pretty much prove that you put way too much stock in this. There were several emails in which these so-called scientists expressed major reservations about the quality of research being conducted by Mann, Briffa, and other researchers and yet none of this made its way into the literature and these same researchers continued to support each others work in public.

What you fail to acknowledge is:

1) The conclusions are not binary. Relative warming can be exaggerated in a climate reconstruction, while another substantially different conclusion will generally not be acknowledged as falsifying it.

2) The incestuous nature of the research community combined with the fact that they are rely on fundamentally similar and weak evidence means that it is not in their interest to try to directly discredit their peers research.

3) The vast majority of this research is NOT reproducible because complete data, code, and methods are generally NOT shared even within the so-called community. This makes it extremely difficult to prove bad methods were used or otherwise falsify the conclusions. When material is shared it is often incomplete and only shared with friendly peers which creates an incentive not to criticize for fear of getting cut off in the future and a feeling of debt.

4) There are probably no silver bullets that can easily overturn these models or reconstructions empirically. Unless someone has something like this that undeniably falsifies all prior recent work in the field all of their incentives tell them to go with the party line because they are unlikely to get published and will have scorn heaped upon them.

What possible motivation would the climate scientists have to do so? What do they gain from over hyping the possible scenarios? To promote renewable energy? Again, what do they gain from this?

Here are just a few reasons:

1) Further their own careers. Big (positive) claims about AGW are important if you want to get published in the high impact journals.

2) To get grant Money to stay publish and stay employed.

3) Face time with the media

4) Genuine-belief in AGW--even if not well supported by the actual evidence.

5) Insider politics -- why criticize a peer's research that largely agrees with your own? The incentives are reversed.

6) Other environmental motives, e.g., "even if AGW is wrong, reducing pollution, sprawl, cars, oil dependency, etc is good" (I have heard this argument a lot)

7) (Mistaken) belief in the precautionary principle, i.e., AGW is a risk and refusal to see it in cost vs benefit terms.

Scientists have always been egotistical, with their own pet theories and human idiosyncrasies. The saving grace of science has never been the scientists, but the method in which science is conducted. Peer review, vigorous debate, and cat-fights. What we believe scientists should be and what scientists are are two very different things. The problem here is the outside influences. You and me.

I think you have managed to miss the point and (sort of) contradict yourself. We will never remove the human element -- inside OR outside -- these negative influences have always existed in science and will never be banished.

The method is what matters most in ultimately arriving at the best possible science. Although we can never systematically remove all bias, bad faith, bad incentives, or what have you, we can sure as hell demand integrity and openness in science. The more politicized and the more important science is the more important it is that science stick to its core principle of openness. All of this empirical research MUST be fully replicable. In other words, all data, methods, and code should be reasonably obtainable by any qualified individual. Whether research is replicable by qualified individual is truly an objective fact that can be agreed upon by reasonable people on any side of an issue.

There is no excuse in this day and age with the internet and cheap storage why nearly all of this cannot be archived as a condition for publication. We certainly should not be making multi-trillion dollar decisions based on science without this critical step. Without replicability it's extremely difficult to falsify research and very easy to publish false or bad research when the prominent journals are all managed by like-minded individuals who have little incentive to find flaws in your research and a lot of incentive to keep you friendly.

What the CRU emails have done is demonstrate, amongst other things, a conspiracy to prevent reasonable requests (for essential information necessary for replication) for its own sake despite: the long tradition of this in science, the policy of several journals (albeit poorly enforced), the charter of these government agencies, and FOI requests. This behavior fundamentally abhorrent to good science.

These CRU emails have given some people a view into the naked politics (against science), bias, attempts to manipulate publication, suppressing legitimate internal criticism (for political reasons), etc... Taken together, they make compelling case for the corruptibility of science and the need for outside scrutiny. You do not need to be a climate scientist to point out fundamental statistical errors in so-called climate reconstructions or to observe that data is being misrepresented on graphs ("hide the decline")... The knowledge that outsiders can actually check research for errors at any point would do a lot to keep research honest -- before and after it is published. We may never be able to ensure that good skeptical research receives a fair chance to be published, but by preventing bad research from entering or remaining in the literature science can correct itself far more quickly.

Yes, clearly this is something that is better handled by lawyers, business people and their assorted paid off goons.

I said nothing of the sort. I merely suggest that the academic process is wholly inadequate given the cost of the measures that are being proposed today and the alleged risks. We should not be making multi-trillion dollar decisions based on the output of a handful of academics following the usual insular peer review process which was never designed to do this sort of thing.

We can fund and construct a far more open and rigorous process if the science is truly "settled".

Want more reliable proxy records? Gather a team of experts to determine the best proxies to examine, clearly state the assumptions and reasons for choices, then fund this research directly so that it can be used freely by all. This data should be as complete as possible, i.e., do not do what Briffa did with his Yamal paper by just including 13 trees to represent the last 100 years while using many many more in prior years.

Want to produce reconstructions that will convince a lot more people? Surely these scientists can collectively decide on the best most defensible way to produce a limited number of reconstructions, then they can document their exact methods and explain their rationale for outside review and commentary. Demand thorough independent analysis by one or more 3rd parties, don't just wave your hand at it and say it roughly agrees with my beliefs about AGW and looks OK on paper. Allow people with expertise in relevant fields to produce relevant criticisms, e.g., statisticians, dendros, etc. Do not allow artificial hurdles to be constructed like those which exist in academic journals (e.g., commentary can only be made X days after publication, can only be Y words long, etc).

Want to produce climate model that will attempt to predict future temperatures? Share the f'n code. Document it well. Explicitly state your assumptions and defend them. Allow people to see how it fairs against the instrumental record in detail, both past and future. It's not as if the only meaningful output is what happens with average temperature. These models make all kinds of assumptions about various phenomenon, particularly with respect to positive and negative feedback effects (esp. cloud cover)... well the outputs can be documented to so how well they hold up. If CO2 output deviates substantially from projections, fine, then run it with the actuals and see how it changes...

All of this data can be archived in one place for all to see at the click of a mouse. The Whitehouse spent $18,000,000 to re-design recovery.gov. Certainly we can spend money to have one definitive source for OPEN research on the settled science of AGW. I would gladly see my tax dollars be used for the goverment to set aside, say, 10 billion dollars to assemble a panel of leading scientists from a variety of areas of expertise to manage this process, direct research funds, etc. If the science is truly settled, it shouldn't take very long to make a compelling case for AGW and dispel mainstream concerns of bias and manipulation.

That is BS. If you bothered to read the refutations, the divergences are themselves a subject of many publications, and this has been out in the open forever.

My point is that this study and several others have deliberately used this 'trick' in their presentations and that this behavior is dishonest and anti-scientific. There is no excuse for it, period. Your excuse in the prior post that the actual temperature record reflects an increase is misleading and amounts to sophistry.

I do agree that access to the raw data could be better, and even that some of the statistical methods etc have been applied poorly (or even incorrectly). You might even find, somewhere in the stack of tens of thousands of climate science publications, some that misrepresent the data, perhaps even deliberately. Not all scientists are as expert as they should be in statistics, and scientists are human and have human frailties (although that doesn't excuse anything). But this does not appear to be one of those cases. You are reading far too much into one email, and you clearly are not aware of the context.

There are only a handful of published reconstructions that show 'hockey stick' shapes stretching over ~2K years and most of these rely on a small set of similar raw data, similar statistical methods, and are authored by a small circle of scientists at even fewer institutions. Many of these studies have already been largely discredited. Furthermore, expecting that every study should be immediately shot down if it is bad is unrealistic given the biases of the climate science community and their complete lack of openness on critical issues.

You need to have a great deal of faith in their particular peer review process to believe this issue is anywhere near settled given how reluctant they have been to allow any sort of dissent by barring access to data, methods, black-balling people from journals, etc. Many reasonable people look at the facts and believe that there is a need for skepticism.

Given:

1) Emails like this (clear conspiracy to mislead, hide embarrassing facts, hide & delete data, etc).

2) The small size and incentives within the climate science community

3) Numerous fundamental documented errors in many of these studies (bad statistics, poor quality data collection, etc)

I simply have little faith in the quality of the peer review going on.

If the climate science community wants to convince more people of the need to act, to spend many many trillions of dollars in the near term, then they MUST be far more open and honest at the very least. There is no excuse in this day and age not to share data and methods freely given the fact that almost all of these people work for publicly funded institutions and are collectively asking society to turn itself upside down to fix the alleged problem.

If all of the science of global climate change depended on a single set of proxy data, then you would have a point. But it doesn't, and you don't.

I never claimed that the entire science of global climate change relies on a single set of proxy data. Even if the science is correct and the model forecasts are 100% accurate, this behavior is still bad and inexcusable, i.e., my point still stands. That said, even though I acknowledge that the earth has warmed over the last century and that CO2 does act as a greenhouse gas, there is: a great deal of unsettled science in exactly how unprecedented today's temperatures are; exactly how sensitive the climate is to CO2; the behavior and extent of feedback effects; what will happen in the future given certain CO2 levels; not to mention the 2nd, 3rd, 4th, and further order impacts of warming (if it happens as predicted).

This issue is too politicized and too important to leave to the typical academic process.