Why Energy Forecasting Goes Wildly Wrong

Wednesday, 23 October 2013 00:00 Steve Yetiv and Lowell Field
AddThis Social Bookmark Button

Energy lies at the heart of the global economy.  Disruptions in energy supplies and serious energy price increases can cause economic and political dislocation, even recessions. Indeed, most recessions in the past forty years have been linked to supply disruptions in the Middle East. It should not come as a surprise then that when energy forecasters talk about future energy production and prices, people listen, especially if the modelers come from or represent vaunted organizations such as the International Energy Agency or the US Energy Information Administration. These are learned, serious people who try to excel at their jobs.  For these reasons people listen to them, make plans based on their assessments both as individuals, companies and nations, and we largely tend to lend credence to their forecasts.

So what’s the problem?    

Relying on such forecasts, particularly long-term projections going out 10 or 20 years, is largely a mistake because they are almost always wrong and sometimes wildly mistaken.  Importantly, it is important to understand why such forecasts tend to be mistaken, and the implications of society’s reactive over-reliance on such forecasts.
US Energy Information Administration (EIA) 

The US Energy Information Administration (EIA) is no better or worse  in its forecasts than other organizations, but it serves as a prime example  of the dubiousness of energy forecasting.   A focus on the EIA’s 2005 long-term energy market for The EIA's forecasts for year 2013 in its Annual Energy Outlook of 2005 were off by staggering margins in many cases. The forecasts are for every five years (2010, 2015, 2020, 2025, et al)  but it's easy enough to interpolate for an intermediary year, such as 2013. The International Energy Agency and other forecasting governmental institutions  and companies faced similar problems (e.g., nobody was predicting the oil price spikes of the 1970s, the oil glut and price collapse of 1985-1986, or the price spike again in 2008) suggesting general problems with forecasting. 

First, the EIA forecast in its "reference case" that oil prices would be around $25-$30 per barrel in 2013 with gasoline prices of around $1.50/gallon.   The actual prices?  More like $100 per barrel for oil $3.50/gallon for gasoline in the US market. That's off by a factor of 3-4 fold. The EIA also got the trend wrong in projecting that, from anticipated high levels throughout 2005, oil prices would decline gradually to $31 per barrel in 2010 then only to rise by about 0.8 percent per year to $35 in 2025.  That's not even close even when considering the EIA's ‘high oil price case’ of $48/barrel by 2025.   Of course, as the EIA put it at the time, "While the three cases [low, medium, and high price scenarios]  vary widely, they do not span the full range of possible scenarios."   That comment was eerily true given that oil prices increased to more than $100 per barrel and stayed extremely high for years. That wasn't even considered as a possibility by EIA in 2005 - in any of its projections. 

Forecasts of World Oil Prices 2010-2025

(2003 Dollars per Barrel)







AEO 2004 (reference case)










AEO 2005 (reference case) 










High A World Oil Price 





High B World Oil Price 





October Oil Futures 





Low World Oil Price





Source: EIA 2005 Annual Energy Outlook 

Second, the EIA forecast US crude oil production of about 5.7 million barrels per day (bbl/d) in 2013 with Lower-US-48 production declining steadily after 2009. Actual US production so far in 2013 has been about 7.3 million barrels per day (bpd) and rising fast.  Likewise,  the EIA forecast US natural gas production of about 20.6 trillion cubic feet (Tcf) in 2013, while actual production was far higher - 24.2 Tcf.

Third,  the EIA also was completely wrong about US oil imports. The EIA forecast that "Total US gross petroleum imports are projected to increase in the reference case from 12.3 million bpd in 2003 to 20.2 million in 2025."  What's actually happened so far is that US gross oil imports have fallen sharply from 12.3 million bpd in 2003 to just 9.7 million bpd in 2013. US net oil imports have fallen even more sharply, from 11.2 million bpd in 2003 to 6.7 million bpd in 2013. The EIA predicted the exact opposite.   

Fourth,  the EIA also missed the price of natural gas in the US by a huge margin: a forecast price for 2012 of $3.80/million cubic feet (mcf) compared to the actual price of $10.66/mcf. It also erred in  its 2005 US coal consumption forecast predicting that "domestic coal demand is projected to increase by 413 million tons (38%) in the reference case forecast from 1.095 million tons in 2003 to 1.508 million tons in 2025."  Instead, US coal consumption has fallen sharply, from 1.095 million tons in 2003 to 890 million tons in 2012.
Why forecasting is so faulty

A few reasons may well underscore why energy forecasts are so often wildly wrong. As an example,  what went wrong with the EIA's oil price forecast when it didn’t even consider that oil prices would increase to more than $100 per barrel and stay extremely high for years? The answer is a lot of things,  but among them was an inability to predict the end of cheap conventional oil. Consider this flawed reasoning articulated back in 2005: "Although OPEC’s share of world oil supply is projected to increase significantly over the next two decades, competitive forces are expected to remain strong enough to forestall efforts to escalate real oil prices significantly."  In fact, those ‘competitive forces’ have most certainly not prevented a massive escalation in real oil prices.  One reason is that higher oil output has come increasingly from more expensive and technologically challenging sources (e.g., ultra-deep waters, tar sands, ‘tight’ oils).  These unconventional oils are expensive to produce (on the order of $70 per barrel or so).  Meanwhile, OPEC's ability or willingness to increase its production has been extremely limited.   In fact, OPEC crude oil output has not only failed to "increase significantly," as EIA confidently asserted in 2005, but it actually declined slightly from 30.83 million bpd in 2005 to 30.18 million bpd this year.  That was wrong again, and this guesstimate undermined the EIA's forecast.

Second, forecasts are driven heavily by what has happened previously, by what current technology can accomplish, which often prove inaccurate.   EIA forecasts were way off on production forecasts because the EIA completely missed the boom in natural gas fracking.   A search in the Annual Energy Outlook reveals not a single mention of the words fracturing or fracking. This was in 2005 just before the enormous fracking boom took off.

Interestingly, the EIA is in fact part of the US Department of Energy (DoE) and the DoE itself had helped to fund what has become known as the fracking revolution.  In short, in its reporting the EIA completely missed reporting on and considering the impact in the introduction of advanced new technologies for recovering oil from tight formations and shale gas.

Having completely missed the fracking revolution for natural gas, EIA's entire analysis went wildly wrong.  For instance, "In 2015...North America is projected to consume 5.7 Tcf more [natural gas] than it produces, and in 2025 the gap between North America’s natural gas production and consumption is projected to be 8 Tcf illustrating the region’s growing dependence on imports."   In fact, US natural gas imports have been plummeting while US natural gas exports have been soaring.  North America is now roughly self sufficient in natural gas. So much for "the gap between North America’s natural gas production and consumption is projected to be 8.0 Tcf illustrating the region’s growing dependence on imports."

Third, it’s hard to predict public pressure as well as changes in government policy. Thus, the EIA missed the decline in coal use partly by wrongly estimating stronger, new emissions standards for US coal-fired power plants (and also the fact that renewables and natural gas has grown faster than the EIA had forecast).

Fourth, another possible reason for the chasm between forecasts and reality is that analysts tend to run in packs, as you can see in the Table from the 2005 Annual Energy Outlook by EIA. Note that almost every oil analyst was forecasting 2010 oil prices in the $20s or possibly $30s per barrel.   The actual world oil price in 2010? Closer to $80 per barrel. Note that we're currently over $100 per barrel, just 1.5 years from 2015, when not a single forecaster listed in the EIA comparison table had oil prices higher than $41 per barrel (the EIA's "High B" world oil price case), and where most forecasts clustered around $25-$30 per barrel. 

Running in packs

Analysts tend to stick together because they are sensitive to being criticized. In addition, if they make such a prediction and they're wrong, they could risk their careers. In contrast, if they're wrong, but everyone else is too, they can all stay safe within the herd.

The pressure to conform is strong especially if one seeks to forecast significantly higher oil prices.   Consider the case of Arjun Murti back in 2005. As a Goldman Sachs energy analyst, he predicted that oil prices would not just rise but “Super Spike” to $105 per barrel or more, within a few years.   At the time, world oil prices were averaging around $50 per barrel, and the vast majority of Murti's fellow energy analysts were predicting that they would probably level off or decline, and highly criticized Murti.   Reflecting broader views, Kevin Kerr of Kerr Trading International blasted his forecast as "irresponsible" and as "clearly an attempt to talk up the market on nothing more than hot air." Other analysts piled on.  Murti was right on this one, though quite wrong in other cases, and basically all the other top oil market analysts were wrong.

Fifth, it may be politically uncomfortable to forecast price spikes. Who wants to be the head of a government forecasting agency, and have to tell their political bosses, regularly facing re-election, that oil prices are going to double or triple and in the process likely tanking the economy and their superiors’ re-election hopes?  The issue is who's going to be the one to deliver to the Secretary of Energy or the White House unpleasant analytical findings?

Sixth, black swan events in the geopolitical arena are hard to predict and yet can have huge impact.   For instance, the 1973 Yom Kippur Arab-Israeli war and Arab oil embargo was one tipping point nobody had really foreseen, as were the 1978-1979 Iranian Revolution, the 1980 outbreak of the Iran-Iraq War, the 1985-6 collapse in oil prices, the Asian economic crisis of 1997, the 9/11 attacks,  the war with Iraq that began in the spring of 2003, the massive increase in Asian oil demand over the past decade, the Great Recession starting in 2007, the fracking- revolution in the US, the boom in Canadian "tar sands" production.  The list goes on and on and it will continue to do so.   

These events and technological advances have all had enormous impact on world oil markets and national economies, yet  none of them were predicted certainly not with any accuracy a priori.  This is not because forecasters are idiots.  It's because it's simply not possible to predict  this stuff.  And without being able to do that, one certainly can't forecast energy production, consumption, prices  or even the direction of movement in any of those areas with any degree of accuracy. 

Seventh, speculation can also be a factor although it is certainly not sufficient in itself because we also observe that forecasts have also been off-base for natural gas and coal which traditionally have not been sold on global markets. Still, the oil forecasts are the most problematic and this may have to do with global markets and the unpredictability that speculation generates.

Markets go up and down, sometimes based on much more than economics such as speculation and psychology in buying and selling bubbles (remember the dot.com boom and bust on the NASDAQ around year 2000?). For example, speculation appeared to be in play in the period from 2007 to 2008 when oil prices rose from around $57 a barrel to $147. In that type of market, traders buy oil futures to make a quick sale rather than to actually possess oil. They speculate trying to anticipate a cumulative effect of all those complex factors on oil prices so to speak. Such speculation arguably has periodically driven prices higher.

Most economists, market enthusiasts, and energy forecasters will tell you that speculation is not a major factor if at all.   And even if some of them concede that it is in play, they will say that it is just a normal part of how markets work and that if prices are driven artificially high markets will self-correct.  Economists see markets as operating under rational principles, unless they are behavioral economists who believe that markets can be irrational, even if predictably irrational. But we do have suggestive evidence that speculation is in play in oil markets contrary to what many economists would argue.

The rise of oil markets such as the NYMEX in 1983 has been revolutionary chiefly because it has allowed for non-fundamentals to more easily affect oil prices.  This explains, in part, why oil prices have shifted so radically over time from a historic low of $10 per barrel in 1998 to highs near $147 in summer of 2008 or even more dramatically from $50 in February 2007 to over $147 per barrel in July 2008.  Changes in demand/supply fundamentals may explain part but certainly not all of these shifts. It’s hard to ignore that oil prices were four to five times higher in the last decade than in the mid-1980s and through the 1990s. This indicates higher speculation in oil markets, not any shift in ‘fundamentals’ as energy analysts love to point out (even though it's basically just meaningless hand-waving, a way for them to look competent, despite the fact that they have no clue what's really going on). 

Consider that when seeking re-election, President Obama suggested that his energy plan might temper oil prices, but instead they rose from around $36 dollars per barrel in December 2008 to $110 dollars in 2010. Data released in March 2011 by Bart Chilton, a member of the US Commodity Futures Trading Commission, suggest that speculators increased their positions in energy markets by 64% between June 2008 and January 2011. That’s the highest level on record.

Importantly, oil prices rose despite mixed news on oil supply and demand, a diminishing chance of Saudi oil disruptions, and Japan’s human and economic catastrophe. Sometimes, they rose in short periods of time when nothing of note had changed on the supply or demand side or had even changed in favor of lower prices.  Tens of billions of dollars have been placed in US energy commodity markets in the past few years. That money is earmarked to buy oil futures contracts and that drives the price of oil higher.   Institutional and hedge funds are investing increasingly in oil, which even prompted President Obama, Bart Chilton and others to call for curbs on oil speculation.


Why does any of this matter, and why should any of us care if energy forecasts are off base most of the time? To the extent that policymakers believe erroneous forecasts, they can make wildly incorrect policy choices. For instance, if they believe that oil prices will remain far lower in the future than is the case, their forecasts will undermine efforts to conserve or to switch to alternatives. Why would nations, businesses, entrepreneurs, and individual consumers take such steps if oil prices are predicted to remain low? At a minimum, this will be one factor working against conservation and movement away from fossil fuels. 

But if we are right that forecasts are often mistaken and of questionable use, then what do they remain so critical? Maybe as human beings we like to design and want to believe in our predictive capacities.   Perhaps we forget when people make the wrong predictions and remember when they correctly predicted two of the last twenty recessions or major events. Maybe no one has a good incentive to point out wrong predictions. Then of course, forecasters may want to rationalize their failed predictions or confuse the matter by talking about sophisticated statistical language that 99% of normal people don't understand. 

Whatever the case the question arises: what should we do? How should we make national, global, and other investment and policy decisions?  One idea is that we would be better off without long-term forecasting or at least without such a great reliance on it.   If an actor believes that the price of a commodity is going to be relatively inexpensive for the next 20 years, and makes investment/policy decisions accordingly, and if instead prices triple or quadruple (as we've seen with oil prices), then that actor will be far worse off than in an alternative scenario where the actors simply ignored the forecasts. 

It’s important to stress that this paper analyzes one major case (the EIA) which we believe indicates a problem with longer term forecasting.  Still, it is just one case and we can only infer so much from it about longer term forecasting in general. In addition, we are not questioning short-term forecasting, albeit our hunch is that it also presents some problems. Nor are we impugning the work of the EIA at the Department of Energy which conducts excellent analyses on a range of subjects.

Nonetheless, the arguments put forward here have merit.  First, if long term forecasting is dubious, hedging and diversification would seem to be smart ideas just as they are for individual or institutional investors. Second, at the national and global level, we shouldn’t depend that much on longer term forecasts in making energy policy. This is because we already have a good sense of what needs to be done: namely, transition away from fossil fuels as rapidly as reasonable, for a variety of strategic and environmental reasons. Third, insofar as we seek to engage in long-term forecasting, we should give greater weight to qualitative analyses than to predictive models. 

Contributor Steve Yetiv is  the Louis I. Jaffe Professor of International Relations at Old Dominion University and Lowell Field is a former Senior Energy Analyst with the US Department of Energy.