Again, you have clearly no idea what you're talking about. I'd advise you to stop blustering and get more than a Masters level education in statistics.

Nowhere does the article claim that calibration is "twisting the data" or "changing the data". It quite clearly says that calibration is changing the values of variables used in the model: "Calibrating a complex model for which parameters can't be directly measured usually involves taking historical data, and, enlisting various computational techniques, adjusting the parameters so that the model would have 'predicted' that historical data."

What the article is describing is fitting a model: finding parameter values that cause it to fit the data.

"Calibration" is a commonly used statistics term in some sub-disciplines. (Others call it "fitting", "tuning", or "parameter estimation".) It literally means "fitting the model", or in a Bayesian context, computing a posterior distribution over the model parameters (e.g., this discussion).

It is simply false that a model which validates well on out-of-sample data will necessarily predict well. The article is in fact about circumstances under which this assumption does NOT hold, such as in statistically non-identifiable models.

One way in which this can happen is if your likelihood surface (or, more generally, objective function or error metric) is multimodal. It can easily happen that both your training data and validation data identify the same mode, but the true value ends up being a different mode, and you can only find that out farther into the future.

To understand better what the article is talking about, you may want to read this paper, which I suspect is by the same guy cited in TFA.

And yes, I do build statistical models for a living. Pretty much everything I do on a daily basis is model calibration. I am not a card-carrying statistician myself (i.e. my Ph.D. is not in statistics), but I'm trained in the field, all of my research is in statistics, I collaborate regularly with statisticians, and publish research in statistics journals.

The problem can show up with an "uninformative" prior as well. Usually this happens when the "true" values of the prior are on the edges of the prior range, e.g., you have a uniform prior on [x1,x2] and the true value happens to be near x1. If the range [x1,x2] is very wide, the prior mean (x1+x2)/2 will be far from x1, and it will drag the posterior there. Sometimes this is ok, if the predictive distribution near (x1+x2)/2 is similar to the predictive distribution near x1. But if x1 and (x1+x2)/2 have very different predictions, it's a problem.

A lot of people are holding forth on why economic models are wrong, but few comments are related to the actual subject of the article. (By the way, unless I missed something, the article itself is very vague on what work is done. I think it may be referring to this Jonathan Carter, and the research findings may be related to this 2005 paper.

The article is about the following situation: you have a model (statistical model, computer simulation, etc.) that you want to use for prediction. It has some "knobs" (parameters) that you can twiddle to change its output; this is necessary because the settings of these knobs are often unknown a-priori. So people "tune" or "fit" or "calibrate" the model to observed data to determine the parameter settings in order to make predictions.

A problem occurs if there are many different "knob settings" that cause the model to behave similarly on past observed data. Statisticians call this an "identifiability problem" (since you can't hope to identify the true value of the parameters from the observed data. Ecologists call it "equifinality", since there are many equally good ways to reach the same final outcome. And engineers call it "multimodality", where the fit of the model has many local minima. (Or you could get a whole "ridge" in parameter space that is equally good everywhere along the ridge crest.)

In such circumstances, you can't determine the true values of the parameters very well, even if the model is perfect. This isn't about imperfections in the numerical model, or in the mathematical theory. It's an inherent consequence of the relationship some models have with the data.

This also is not a consequence of imprecise data. There is always some uncertainty about model parameters given noisy data, so you'll never determine the true value of parameters exactly. But this isn't what it means to be non-identifiable.

An example of the real problem of non-identifiability: suppose your model is y = (A+B) * x + error. It's pretty clear that if you measure y and x, all you can hope to determine is the linear combination A+B, and not A or B individually, even if you have perfect data. (That is, unless you have some additional source of information to constrain their values other than y and x.)

The above is a case of perfect non-identifiability. Other models are just "nearly" non-identifiable (e.g., they have "almost flat" ridges in parameter space). Then you can identify the parameters eventually, but only with unusually good data, or multiple data constraints. As an example of the latter, you could observe one quantity that constrains the parameters to a ridge in parameter space, and another quantity that constrains the parameters to a perpendicular ridge, and the intersection of the ridges is well constrained. (Think of an "X" shape, or something like this figure, except the ellipses are stretched into ridges extending across the whole parameter space).

Non-identifiability is sometimes a problem for prediction, and sometimes not. The issue is that different parameter values can be consistent with the same data. If this relationship also holds into the future, then it may not matter: you might not know what the true value of a parameter is, but if all the allowed parameter settings lead to the same predictions, maybe you don't care if you get the parameters themselves wrong.

However, the relationship may not hold into the future: parameter settings that give similar predictions for historical data may lead to very different predictions for the future. This is the real problem, and it can't necessarily be solved with better data if the model is truly non-identifiable. Then you have to simply prepare for the wide range of possible outcomes.

What the article doesn't make clear is that not all models have this problem.

You clearly don't know what you're talking about. The article is talking about making predictions with non-identifiable models. (What it doesn't make clear is that many models are, in fact, identifiable.)

Any time you fit a model, you're doing "calibration". It doesn't matter whether your fit validates well in an out-of-sample test. If your model has identifiability problems, validation is no guarantee of future predictive skill, even if the model is perfect. That's the guy's point.

Well, Carter's argument is sometimes wrong. I do Bayesian calibration of computer models, and with some models the maximum a posteriori estimate, or the posterior mean, is consistently very different from the "true" parameter values (in a perfect model simulation study). This is basically a combination of non-identifiability in the model combined with insufficiently informative priors. It's hard to do anything about this, and it's a problem if the estimated and "true" parameter values lead to very different predictions. (Sometimes they don't, and if you only care about predictions, it may not matter that your valid predictions are based on "wrong" parameter estimates.)

The linked article has nothing to do with numerical error due to floating point precision. It has to do with the fact that many models produce similar outputs for very different inputs.

Saturday Morning Breakfast Cereal comic.

Blog post here.

They'll make up for it in volume.

Depends on how you define "relativistic mass". You can define it as E/c^2, and relate it to "rest mass" by a Lorentzian gamma factor. But this doesn't work if you try to plug relativistic mass into Newton's law; it turns out that you need a whole "relativistic mass matrix" instead of a scalar. This matrix can be decomposed into "longitudinal" and "transverse" relativistic mass. See Wikipedia for more.

"Invariant mass" or just "mass" is a better term than "rest mass", since it applies to photons. (Photons are never at rest and therefore it is meaningless to ask what their mass "at rest" would be.) For a photon, (invariant/"rest") mass is zero, but its relativistic mass (via the E/c^2 definition) is not.

The gravitational effect of a photon depends on its relativistic mass-energy (E, or E/c^2, depending on how you look at it), not its invariant mass (=0).

Basically, we don't know.

We don't know what happens when a black hole evaporates. That requires a theory of quantum gravity, which we don't have. Hawking radiation can be worked out in a semiclassical theory of gravity, so we know a black hole will shrink, but when you get down to "the last photon", we can't say what ultimately happens to the black hole.

Similarly, we don't know what would happen to a photon if you gave it Planck energy. That too requires a quantum theory of gravity. Below the Planck energy, it wouldn't form a black hole.

The point is that gravitons are not tachyons. In all quantum theories of gravitation they're massless spin-2 particles, which travel at the speed of light.

Light has energy and energy creates a gravitational field just like mass does. Interestingly, momentum also creates a gravitational field (as does pressure). These are all aspects of stress-energy.

It turns out that two parallel light beams in vacuum neither attract nor repel each other, because (in this special case) the gravitational attraction due to their energy is cancelled by a gravitational repulsion due to their momentum.

In general, light and matter attract each other.

As another poster pointed out, photons possess energy and therefore gravitate. This is a corollary of relativistic mass-energy equivalence (E=mc^2).

Yeah, well, it was probably intended to be an apparatus that an uneducated 4th grader could make in a classroom. See my other post.