I was using this graph : http://earlywarn.blogspot.fr/2011/09/peak-oil-per-capita.html , but it's only oil (and you see a stagnation rather than a decline, but since the EROEI goes down, the available energy for society from oil also goes down).
Increased efficiency plays a role, but if I'm not mistaken it's about 0.5% per year. I'm not sure if a better metric than energy per capita exists : you would need to somehow sum up the transformations that this use of energy allowed. Maybe computing exergy would be better?
Then, I guess my hypothesis comes more from the feeling that we're "scraping the bottom of the barrel" with the hydraulic fracturing and tar sands, while many of western countries show signs of political rot.