The science is not, in fact, that easy, for a number of reasons. Off the top of my head:
- "Energy Out" calculations can't be made in a vacuum; the amount of energy your body burns doing any exercise (even at rest) is largely determined by your current body composition, ie. more muscle mass takes more energy to maintain
- All calories are not utilized the same; protein, for example, is used to build/repair structures and is not burned as "fuel". Counting protein calories is essentially useless, though tracking the amount consumed can be helpful (approximately one gram per pound of lean mass per day is considered a good baseline for an adult male of average activity level)
- Certain foods trigger certain hormonal responses in the body that can effect metabolism. Insulin is a storage hormone. Anything that triggers a strong insulin response (sugars/grains) will predispose you to storing more of your "energy in". Likewise, staying too low on the "energy in" side of the equation will make your body think it's starving which will adversely effect metabolism. There are numerous studies showing that starvation diets don't work beyond short-term.
These are just the easy ones I can come up with without doing any real research. For a great example of how you can lose a lot of body fat (not just "weight") while not counting calories in/out, see "Four Hour Body" by Tim Ferriss.