On Forecasts

Overview

I was meeting with a client recently, and since it was the end of the year and they were in their budgeting process we were discussing forecasting methodologies. They were asking about their marketing function specifically, but the conversation applied more or less to all areas of the business which set targets and/or develop a budget. So basically, the whole business.

It’s important to note that in many companies budgeting and forecasting are the same thing, so this is as much a budgeting topic as a forecasting one. The reason that a lot of departments are asked to do forecasts in the first place is simply so that the forecast can be added as a line item in a budget.

These forecasts and/or budget line items are also often directly placed into compensation or bonus schemes either directly for a given employee or indirectly by so-called performance-based compensation packages for executives. In other words, forecasts end up touching almost every part of the business and every person in the business, though some stand to benefit more than others.

The problem is that in many businesses, budgeting and forecasting are totally broken. There is often a months-long budgeting process wherein thousands of hours are used in order to produce forecasts for inclusion into a budget. Often the honest estimates of employees are negotiated up (or down) in a series of meetings, the end result being a budget which may not be realistically achievable at all. The budget might address the dreams of an executive team or investors, but it may be impossible to achieve in reality.

In a perfect world, there would be no yearly budgets and no five-year plans, and we would all operate with rolling forecasts. This is explained in detail in a great book called Beyond Budgeting which is one of the two books I recommend to all of my clients (the other being The Art of Action.

I’ve had numerous discussions with companies about this topic over the years, but that experience has not been documented. The goal of this post is to begin to give an overview of those discussions, answer some frequently asked questions, resolve some common misunderstandings, and provide reasonable expectations.

Some preliminaries

First things first, is your organization mature enough in age and measurement capabilities to actually have historical data you can use for forecasting? For many startups, the answer is a clear NO. If your business has yearly seasonality, and you have only been operating for a year or two, then chances are that your growth rate will dominate seasonal effects. Equivalently, if you want to forecast sales for a brand new office or new subsidiary in another country then you simply will not be able to do that. You might be able to make an educated guess, possibly informed by deep domain knowledge, but you will not be able to make a forecast in the proper sense.

In these cases, the best bet is for you to make a guess and be explicit about the fact that it is a guess. Calling it a forecast only serves as an attempt to add legitimacy where there is none. It’s always better to acknowledge and manage uncertainty than it is to pretend it doesn’t exist.

In the case where you do have sufficient historical data (called a time series) on which to base your forecasts, there are many options on how to use it. So when we talk about forecasting sales for next quarter, or per-capita sick days per department, we’re well into the domain of time series forecasting.

Time Series Basics

A time series is simply a sequence of values which are ordered in time. The interval between each value is typically the same. Examples would be yearly sales for a company, the monthly average cost of fuel for a car, the number of calories you eat every day, and so on. This is typically the thing which you want to forecast.

The frequency of the time series is the interval at which the time series values are measured. This could be daily, weekly, or 1000 times per second.

Observed time series data, like monthly revenue, can be decomposed into three main components: trend, seasonality, and residuals. Time series decomposition can be done in other ways, and with methods requiring varying degrees of mathematical sophistication, but that discussion is beyond the scope of this post.

The trend is the general direction of the time series data. Is it going up or down over the interval? In many forecast problems for startups, the trend will almost completely explain the data and dominate any future forecasts on certain timescales, so your forecast will effectively be just the trend.

The seasonality is the variability which repeatedly occurs in the data. A common example would be a consumer products company which will see higher sales during the Christmas holidays every year, or restaurants which often see lower revenue on certain days of the week or months of the year. This is a consistent pattern related to the business operations.

The residuals is what is leftover after the trend and seasonal components of the time series have been extracted. This is also sometimes called the noise.

You can see what this decomposition looks like in this Stack Overflow post.

A hierarchical time series is one in which the time series data is aggregated into levels. In the figure below, from Forecasting: Principles and Practice you can see a 2-level hierarchy.

This is typically the situation in a company when they want to, for example, forecast their revenue for the next accounting period. They will typically settle for a top-down forecast or a bottom-up forecast, or some combination thereof.

In a top-down forecast, what typically happens is the leadership will set a target at the top and then ask the lower levels how they plan to get there. Note that this isn’t necessarily what a top-down forecast should be, but it’s often what occurs in practice.

In a bottom-up forecast the leadership will ask the lowest levels to produce forecasts, which then get rolled into higher level forecasts, which continues all the way up until there is a single forecast which has aggregated information from the entire organization.

A note on budgets.

For the purposes of budgeting, forecasts are often hierarchical. This adds a bit of complexity because you must answer the question of whether you will use top-down, bottom-up, a mixture of the two, and how you will combine the forecasts.

Of additional importance is the fact that the variance in the forecasts may combine in unexpected ways. It may be the case that your bottom-up forecast, by the time it gets to the top, has such a wide variance that it is almost useless. THIS IS NOT A BAD THING! Knowing the truth about the predictability in your organization is something to manage and minimize, but not something to ignore. If your forecast variance is so wide as to be useless, then it is better to spend your efforts on ways to manage uncertainty in your organization.

Organizational maturity

We can consider the maturity of an organization in a few different ways for our purposes.

1. Quality of data. Is it reliable and usable?

2. Historical data available. Is there enough data? Are forecasts even possible?

3. Quality of data infrastructure. Can we do the desired computations in the desired time?

4. What is the confidence about whether or not the target can actually be achieved? Are there additional factors which are not accounted for in the data but which materially impact the accuracy of the forecasts? Did one of your manufacturing facilities get destroyed for example, and you now have 20% lower production capacity compared to last year?

To the second point, some textbooks will mention that for ARIMA models you need at least 50 to 60 data points to start doing accurate forecasts. In reality, any time series can be modeled, it’s just a matter of signal to noise ratio and whether or not the forecast variance is acceptable. If I tell you that your revenue for next year should be either down 80% or up 400%, that might not be useful, but it is still a valid model.

All models are wrong but some are useful.

– George Box

It’s also important to note that a forecast is a guideline for the operation of the business. Until our AI overlords take over, humans still have the most complete context and should serve as the sanity check for any forecasts. Just because a computer says you might have a 200% revenue increase next year, doesn’t mean it is correct.

Top-down versus bottom-up

This is one of the main questions which comes up, and the answer is that it depends. For hierarchical time series forecasting, a combination of the two methods is often most useful and straightforward.

However, even simpler still is just doing a bottom-up forecast. Trust the people on the ground. Trust the data. They know more than you do and just because you don’t like what they’re telling you doesn’t make it false. If they came up with incorrect forecasts due to now having complete context and information from leadership, then leadership should consider how they can improve communication of pertinent information, which will also confer enormous benefits to many other areas of the business.

I constantly work with companies who are either concerned about hitting their targets because they didn’t accept the estimates and forecasts of the people actually responsible for delivering. Just because you want a forecast to be higher doesn’t mean you should turn a forecast into a stretch goal / big hairy audacious goal / SMART goal / whatever. A forecast is not an objective.

Variance in forecasts

One thing I almost always see is companies not putting variance or some measure of uncertainty around their forecasts. This usually happens because the senior management doesn’t want to see it, doesn’t want to acknowledge it, or would say the forecast was invalid or useless with that particular level of variance. None of that actually serves to decrease the variance however, and they will still require employees to produce forecasts, so everyone chooses to ignore the variance. The question then becomes, what to do if the variance is so high that the forecast is not deemed useful? That’s a separate business question, but the short answer is that it pays to focus more on business changes which can allow the organization to cope with the uncertainty and adapt to the rapid pace of change. Simply wishing for forecast variance to be smaller does not make it so. Channeling the spirit of Neil deGrasse Tyson for a moment, the nice thing about facts is that they are true whether or not you believe them.

Forecast intervals

Many of these forecasts are prepared for the next 12 months, because they are going to form the basis for the budget for the next operating year. In some contexts this can make sense, especially when operational costs are relatively fixed. If you have a multi-year lease for warehouse space at a fixed cost then you can be pretty certain about those costs over the next 12 months.

However, it’s almost always a horrible decision to incorporate forecasts with wide variance into a budget since that basically converts them from forecasts with forecast error to targets in a budget.

Let’s say you do create a 12 month forecast. Intuitively, your uncertainty about month 11 is far greater than about month 1. So why not just update the forecast as you gain more information? Then you can use these new forecasts to project out 12 months, with the understanding that the further you project out, the more uncertain you will be. This is the whole point of rolling forecasts. You can update the forecasts as you have new information, and you are not stuck with massive under or overestimation on a 12 month forecast because you aren’t held to those numbers. If you can’t accurately estimate a number for the next month, and there is plenty of historical data and context, then that is another problem entirely, but in a startup which has many product changes launching in parallel and many additional business development activities, 12 months is simply too long of a time horizon to make any sort of reasonable forecast for some things.

Conclusion

There’s no question that forecasting and budgeting are here to stay, at least for now. There’s a craving for certainty in business as in life, and a concrete desire for investors to truly understand the risks which might befall the companies in their portfolios. While this might make sense in larger companies which are very stable, the companies with the greatest promise for growth are also precisely those which are not amenable to accurate forecasting, and for precisely that reason. The growth potential is high, but the variability in possible outcomes is equally high. Especially in these cases, when the difference between forecast accuracy at 3 months and 12 months is so massive, it only makes sense to focus on rolling forecasts at short intervals instead of making budget commitments at longer ones.

This is not to say that plans should not be made. In fact, quite the opposite. Plans should be drawn up and contingency plans made as well. General Eisenhower said it best.

Plans are nothing; planning is everything.

– General Dwight D. Eisenhower

So yes, make plans, but do not expect everything to go as planned. Additionally, if you require your people to make a forecast or budget, and they will be penalized for not meeting the forecast, then they will always try to play things safe and intentionally provide low numbers and behave conservatively. They will focus on avoiding risks instead of pursuing rewards. In some businesses conservative behavior might be desired, though hopefully not encouraged via arcane budgeting exercises, while in others there is an insatiable craving for innovation. If you want innovation, you can’t require someone to build a business case with 5 years of forecasts for something that doesn’t even exist yet. If those numbers are ever achieved then it will be purely by chance.

The better strategy is to create an organization which can adapt to the changes it encounters and which is robust to changes in market conditions, revenue, and other factors. There are simply some things which cannot be predicted to a satisfactory level of certainty. If you are including the variance along with your forecasts, as you should be, then it may be the case that the variance is too high for the forecasts to be useful. At that point you must either confront the truth that the future is less certain than you want it to be, and plan accordingly, or you can live in denial and instead treat forecasts like concrete predictions of the future.