You have no idea what you're talking about. The abstract clearly illustrates their type I probability as at most
.05 which is pretty standard.
Using other figures from the abstract I approximate their type II probability using the following
approximate critical t value (from the t-table in Wackerly's Mathematical Statistics) 1.96 for 46 df
t confidence interval
4.2 = 2.435 + 1.96*se
implies se =
since se = std.dev./sqrt(n) (recall n=47)
std. dev. = 6.1736 (approximately)
The effect size (d) is approximately
.3934 under equal variance
This leads to the following approximate power
For a One-Tailed (Directional) Hypothesis
Observed Power: 0.778
For a Two-Tailed (Non-Directional) Hypothesis
Observed Power: 0.671
They had a directional hypothesis (mu phone on greater than mu phone off) and since I'm only using approximate values, numbers from their abstract, and I assumed by their use of ANOVA that equal variance was satisfied (so I used the same value for both) that that
.778 is pretty close to the standard
.8. So I'm willing to bet their experimental design was such that it their power is actually at least
.8 and that the difference is from my rounding and approximation and not actually using their recorded standard deviations (since I didn't see them in the abstract and I don't have access to the JAMA article itself).
If any methodological challenge can be had from just reading an abstract (which seems unlikely since all of their methods are not expressed in it) its that they don't explain which version of ANOVA they used (it would seem repeated measures is most appropriate and a quick review of other studies suggests they likely used this, but since they don't say it explicitly its possible they used another ANOVA). One might also question why a regression model with mixed effects wasn't used given the repeated measures and potential for significant mixed effects in longitudinal study, but given the relationships between ANOVA and regression it isn't really a fatal flaw.
Now posting that ridiculous (see *) Science News article could only have one of a few purposes: (1) You're a Bayesian and you're rejecting a frequentist approach to this study. (2) You're against the use of all statistics based on their uncertainty versus deterministic/certain models from mathematics. (3) You think compounding errors effects this study. If I'm missing you're real point please feel free to elaborate it.
If (1) I'm not going to dive into the pro/con's of Bayesian vs Frequentist, but I will say Bayesian models are built off of Frequentist its not like they were independently developed in a vacuum. As such while Bayesian methods act generally under different assumptions they do inherit and are confined by some of the same restrictions as frequentist models. A real problem is that Bayesian models work well when you can effectively incorporate prior knowledge (as in domain specific knowledge) which a Statistician isn't likely to have, and how realistic do you really think it is to teach scientists Bayesian methods when they all require some understanding of frequentist models that they already have shown they don't fully understand?
If (2) well if you have a neat proof for PvsNP that P=NP tucked away you might want to get on with submitting it and claiming your million dollar prize and possible fields medal if not other accolades or if you have many previously unpublished exact methods please publish them we're at a point where computing power could actually do the necessary calculation and that too could possibly net you a fields medal and more. Since most methods used are based on maximum likelihood, most powerful test, or some other "at least this good" type method you don't need to fear the uncertainty and without a way to map probabilistic methods to deterministic ones meaningfully and with the infeasibility of conducting a census to actually have population parameters for most studies I believe these are the best tools available for the job. If that's not good enough try developing your own methods and see how well they measure up maybe you're onto something or maybe you have no appreciation for how sophisticated these methods actually are.
If (3) well there is a difference between sensitivity and specificity. With that in mind once you've collected the data for the sample no one is stopping you from implementing a quality control method to verify the validity of each entry. Also many statistical procedures (including most implementations of ANOVA -- because really who is calculating this by hand?) automatically adjust for changes in errors and minimizing error effects. Mistakes and errors can and do happen, but no one said there aren't ways to deal with such things. Statisticians, quality control engineers, and even artificial intelligence / machine learning researchers have been happily using such methods to deal with this problem with confidence.
So either you gravely misunderstand statistics or you're trying to fool other people for some reason. Either way you need a better statistics background badly!
(*) e.g. "Statistical tests are supposed to guide scientists in judging whether an experimental result reflects some real effect or is merely a random fluke, but the standard methods mix mutually inconsistent philosophies and offer no meaningful basis for making such decisions." -- No they aren't inconsistent or mutually inconsistent and the author
[who as a journalist and at best an undergraduate level chemist/physicist is hardly qualified to formally judge the merits of Statistical methods or their application in the sciences] shows his own misunderstanding of statistics. Stupidity such as "Statistical problems also afflict the “gold standard” for medical research, the randomized, controlled clinical trials" as if there were clinical trials before
statistics. Something akin to "damn statistics muddled my flawless clinical trial design" uhm... no
you'd have no clinical trial designs without statistics. Statistics can be misused by someone who doesn't understand the underlying model assumptions in such a ways as the article describes. The only "truth" (if such a thing even exists) in that article is that many people (who aren't statisticians) use statistics without understanding model assumptions, frequently misunderstand what type I and type II errors even are, statistical versus practical significance is not fully understood by these people, and as a result results are frequently misinterpreted. So I'd take that all to mean all scientists would benefit from having handy access to actual statisticians more than making them take a single semester course in statistical methods that leaves them hopelessly lost and unprepared to do statistical analysis. Which I can agree with, but doesn't mean the problem is with statistics, its with its use by non-statisticians.