What It Costs to Keep a Forecast Honest

The number people ask about first is the accuracy figure: plus or minus 4 percent, across the engagements where I've built this practice. What they usually mean by the question is 'what's the model', as if forecast accuracy were a property of an algorithm. It isn't. I have used simple moving averages that landed within a couple of points in a stable environment, and I have watched a more sophisticated regression drift 15 points off in an environment with lumpy commitment renewals. The model matters far less than whether anyone is checking it against what actually happened, on a fixed cadence, and adjusting it when it's wrong. That check is the entire practice. Everything else is bookkeeping.

Forecasts degrade for a small number of recurring reasons, and none of them are exotic. Seasonality is the obvious one: a retail workload in November does not look like the same workload in February, and a forecast trained on a flat baseline will miss the spike every time until someone tells it to expect one. One-off spend is the quieter problem: a data migration, a load test left running over a weekend, a one-time licensing true-up. If you don't tag and exclude these from the training data, the model learns that irregular spend is normal, and it starts predicting noise as if it were signal. And then there are changes to the commercial terms underneath everything: a new savings plan, a reserved instance renewal, a committed-use discount that resets on a different date than the last one. None of these are hard to explain after the fact. They are only hard to catch before the fact, and the only way to catch them before they wreck a forecast is to have already built the habit of looking.

That habit is the forecasting feedback loop, and it is less glamorous than it sounds: at the end of each month, pull the forecast that was generated at the start of it, put it next to what was actually spent, and look at the gap. Not just the aggregate gap: the gap by service, by account, by the categories that actually move independently of each other. Most of the time the gap is small and boring, and that's fine, that's what a working forecast looks like. The times it matters are when the gap has a pattern: the same service overshooting three months running, or a business unit whose actuals keep coming in under forecast because someone quietly rightsized a fleet and never told finance. Those patterns are where you adjust the model, tighten a seasonal adjustment, add a variable, exclude a cost center that behaves differently from the rest. Skip this step and the forecast doesn't fail loudly. It just slowly stops being trusted, and the finance team goes back to spreadsheets and gut checks, which is the actual failure mode you're trying to avoid.

The automation side has its own version of this problem, and it took me longer to see it clearly. Running around 1,200 automated cost-governance actions a month sounds like a lot, and it is, but volume was never the hard part. Writing a rule that shuts down an idle dev environment at 9pm or resizes an over-provisioned instance based on a week of utilization data is not difficult. What's difficult is trusting that the rule is still doing what you think it's doing three months later, after the workload it was written for has changed shape. A rightsizing policy tuned against last quarter's traffic pattern can start downsizing something that's actually grown, and if you're not watching for it, the first sign is an incident, not a report. Automation at that volume is not free once it's running. It just moves the cost from manual review of individual actions to periodic audit of the rules themselves, and if you don't pay that second cost, you're carrying risk you can't see.

So the discipline that actually protects both halves of this, the forecast and the automation, is the same discipline: sampling your own output and comparing it to reality on a schedule you don't skip. For forecasting, that means the actual-versus-predicted review every month, not at renewal time or when something looks off. For automation, it means pulling a sample of actions taken by each policy and asking whether a human would have made the same call, not assuming that because an action fired without an alert, it fired correctly. Tagging coverage at 98 percent is what makes both of these checks possible at all. You cannot audit a rightsizing policy's decisions by team if half the instances it touched aren't attributed to a team. Forecast accuracy and safe automation volume are not two separate wins. They're the same feedback loop, applied to two different outputs, and the loop is the part nobody puts on a slide.