Measuring Forecast Accuracy

A forecasting tool that doesn't report on its own accuracy is asking for blind trust. TatvaAI instead shows a backtest error metric alongside every forecast, so users can judge how much confidence to place in the numbers.

What Backtest Error Means

Backtesting means checking how well past forecasts, generated using only the data available at the time, would have matched what actually happened. The resulting error metric shows how far off those past predictions typically were.

What WAPE Is

WAPE, or Weighted Absolute Percentage Error, is a common metric for measuring forecast accuracy across many series at once. It weighs errors by the actual size of what's being forecast, so a large series being slightly off contributes more to the overall error than a tiny series being slightly off — giving a more business-relevant accuracy picture than a simple average error.

Reading the Median WAPE Figure

A "median WAPE" figure summarizes typical forecast accuracy across all the series being tracked. A lower median WAPE indicates the forecasting model has historically tracked actuals closely; a higher figure suggests more caution is warranted when acting on the current forecast.

Using Backtest Error Wisely

Backtest error shouldn't be treated as a single pass/fail signal. It's most useful as context — a forecast with a higher backtest error deserves more scrutiny and cross-checking against other information (such as the order book or known upcoming events) before being used for firm commitments.

Practical Use Case

A finance team preparing quarterly guidance checks the median WAPE alongside the forecast itself. Seeing a higher-than-usual backtest error, they choose to lean more heavily on the P10 conservative scenario for external guidance, rather than the P50 median, to build in extra caution.

FAQ

Q1: What is backtesting?

Checking how accurate past forecasts would have been against what actually happened.

Q2: What does WAPE stand for?

Weighted Absolute Percentage Error.

Q3: Why is WAPE weighted?

So larger series count more toward the overall accuracy measure than smaller ones.

Q4: What does a high median WAPE suggest?

More caution is warranted before acting on the current forecast.

Q5: How should backtest error be used?

As context for how much scrutiny a forecast deserves, not as a simple pass/fail signal.