ShakaUVM's Journal: Problems of and Solutions for the Scientific Method

Ugh, I wrote a long response and lost it. Essentially, as I said before, I agree the scientific method is very useful, but I disagree when you say that "I believe that science (in its ideal form) is not only the best method we've found so far, but the best method there could possibly be."

My criticism of the scientific method are:
1) Unique events can and do all the time (it's a consequence of the probabilistic nature of the universe). A unique event is simply one that we haven't observed before or since. Science can't deal with unique events, which are often the most interesting things to us. What is the standard deviation of the heights of Martians, if we have only met one Martian? Science can't answer that question -- but you could certainly ask the Martian to gain this knowledge.

2) The scientific method's reaction to inexplicable events is to reject the event, and the person reporting it. When Roentgen was working with these crazy new X-Ray things, he didn't publish his observations until he had established the cause and effect, so as not to risk his professional reputation. A method which can take an observation, find no explanation, and as a result reject the *observation* has a fundamental flaw.

3) Science should eliminate bias and politics. Studies performed by people who have a financial stake in the result must always be suspect, as stats can usually be massaged to show whatever it is you want them to show. A better model would be a sort of escrowing process where Intel or Merck or GM hands money to an escrow dealer (possibly the government, possibly a private entity) who then presents an unbiased question to a scientist in the field: "Are Intel CPUs faster at 3DMARK07 than AMD's?" "Does Vioxx reduce or induce heart attacks?" "What is the 0 to 60 time on the following cars...?" With the proviso that the results will be published regardless of if they are favorable to the original sponsor of the study. This would go a HUGE way to fixing the problems of the science of today.

4) You said: "The level of certainty science can provide is sufficient." Hormone Replacement Therapy was "scientifically" shown to reduce breast cancer risk. As a result, some 10,000 women have died of breast cancer from HRT. All scientific studies are uncertain to different degrees; as you stated, studies in physics are probably pretty reliable. But studies in medicine are overturned constantly. The level of certainty in medicine is really quite low.

5) This is also the problem of trust in science. The problem is not malice or fraud (though the case of the South Korean cloning guy shows this can happen) but that whereas you or I can understand that studies inherently have uncertainty in them, people go out there and make life or death decisions based on studies, thinking that something which is scientifically shown to be true means the same thing as something mathematically shown to be true.

6) The combination of (1-out-of-20 (p-value less than 0.05) or 1-out-of-100 (p-value less than 0.01) studies being due to random chance) x (a large number of studies per year) results in a huge number of conclusions being published *due solely to chance*. This should be caught (eventually) by reproduction of results, but...

7) Reproducing results (whether building on previous research as you say, or simply doing a new study on the same topic) is done haphazardly, and the result of a follow-up study that contradicts a previous one does *not* actually overturn it... if you have one study showing that eyeblink therapy stops PTSD, then a follow-up shows it doesn't, the literature will conclude "Studies are conflicted". Only topics which interest someone at NSF get the kind of treatment needed to conclusively establish something as true. Many interesting questions linger in the "studies conflict" category for years without any systematic approach to resolving the conflict. Reproduction of results is the cornerstone of the scientific method, but studies are expensive, so follow-up studies are often neglected.

8) People from all times and places always think that whatever we-know-now is truth, whatever people-used-to-know is helplessly ignorant, and there will probably not be a lot new in whatever people-will-know in the future. We're hopelessly arrogant in our present-ness. Hence the resistance to science's heretics. The Kuhnian Crisis model for science is not a very efficient one for finding truth.

In conclusion: As a result of all this, especially in fields like medicine where it is very difficult to isolate cause and effect (was the heart attack due to his diabetes or because of his Vioxx?) you will have hundreds of wrong results added to the literature every year. A press and public that thinks that whatever we know is the perfect truth that will jump on whatever study has captured their attention, combined with a very haphazard follow-up process which doesn't efficiently weed out the results published due to random chance means that the scientific method could indeed be much better than it is today.

Science seeks not to collect random facts, but to discover the general underlying principles of reality (which you refer to as "the natural world," as if to imply there is another).
If there is another, science can't tell us about it, which is precisely my point. =)