Being a physicist myself I am very happy that this topic makes it into the news. But it is important to keep cool and skeptical. The statement that a statistical fluke has a probability of 0.05% implies that it is bound to happen if you let 2000 students do data analyses on independent data sets. There are indeed literally thousands of PhD students doing such analyses LHC data, trying to address hundreds of specific research questions that each require different data selections. So it is very likely that some of them will find a result several standard deviations away from the expectation. Actually 3.5 sigma deviations happen very often, because of all sorts of mistakes and inaccuracies in the analyses, but most of the time these mistakes are scrutinzed away before loud public announcements are made. After all scrutiny a few genuine statistical flukes should still remain, and recognized as such.
(For the xkcd inclined: green jellybeans linked to acne.)
- On slide 14 and 15 you see a summary of the estimated systematic errors and the final result: the deviation of the observed value from the expected value is 0.82 ± 0.21(stat.) ± 0.11(sys.) %. Estimating and combining systematic errors is almost by definition dark magic. It looks like the "3.5 sigma" was obtained by adding the statistical and systematic error in quadrature, which yields a total error of 0.237, and 0.82/0.237=3.5.
- The statement that the probability of this 3.5 sigma deviation is 0.05% is based on the assumption that if you repeat this analysis several times on more data with exactly the same experimental setup, the deviations from expectation are distributed like a Gaussian (bell curve) with a sigma equal to the total error mentioned in the previous bullet point. That is a major idealization, it could be distributed in many other ways, and then the relation between the deviation (in units of sigma, which is also defined for non-Gaussian distributions) and "the fraction of events with such a deviations or larger" can be quite different. Furthermore, when repeating the identical experiment the systematical errors do *not* fluctuate (that is one of the aspects in which they differ from statistical errors), so aforementioned idealized Gaussian would have an arbitrary offset with a magnitude of the order of the estimated systematic error (0.11), in either direction, and a width of the actual statistical error, 0.21. Depending on what this systematic error really is, the true statistical significance is much larger or much smaller than the quoted 3.5 sigma.
So this is a very interesting result, but more study is needed and in my experience such flukes almost always evaporate in the light of more data and scrutiny. Still, it's not completely excluded that this was indeed the first hint of a real discovery (otherwise no researcher would ever do all that work).
OK, enough for now. Sorry for misinterpretations and other errors I might have made.