Small test set, big precautions
With a few hundred rows, a two-point gap means nothing. You have to think in intervals.
The situation is common among our clients: three years of sales, consumption or stock history, with missing months, late entries and a software change halfway through. They want to know whether a forecasting model will do better than the current method. And they have, at best, a few hundred usable observations.
Why a single number lies
On a small test set, a model's score depends as much on the luck of the split as on the quality of the model. Change the rows used for testing, and accuracy can move by several points without anything else having changed.
Announcing "model A reaches 82% and model B 80%" then amounts to comparing two coin tosses. The only honest question is: within what range does each model's performance really fall, and do those ranges overlap?
Three techniques for thinking in intervals
- Cross-validation. Rather than a single split between training and test, you make five or ten, and look at how the scores are spread. An average together with its spread says far more than an isolated number.
- Bootstrap. You resample the test set hundreds of times, with replacement, and recompute the score each time. This directly gives a confidence interval, without complicated assumptions.
- Paired comparison. To compare two models, you evaluate them on exactly the same cases and look, case by case, at which one does better. It is far more sensitive than comparing two averages.
For time series, one extra precaution: validation must respect the order of time. You train on the past, test on the period that follows, then move forward one step. Mixing up the dates lets the model read the future, and the resulting score is worthless.
Choosing the right error measure
The choice of metric matters as much as the method. A mean percentage error, very widespread, becomes absurd as soon as some actual values are close to zero: a product sold once instead of twice shows a 50% error. On stock or sales data with many small volumes, a mean absolute error, expressed in the business unit, is often more meaningful.
Above all, the metric must reflect the real cost of errors. Forecasting too low and running out of stock does not have the same consequences as forecasting too high and tying up stock. If the two errors do not cost the same, the measure should not treat them the same way.
Gaps in the data are not neutral
A missing month is almost never missing by chance. A stockout, a strike, a data-entry system failure, a reorganization: the missing data often tells a story. Filling it with an average erases precisely the event the model should have learned to handle.
Before any modeling, we therefore spend time with the business teams to qualify each gap: data-entry error, genuine absence of activity, or exceptional event. The treatment differs in each case, and this step often improves results more than the choice of model.
Always beat a simple baseline
A machine learning model is only worthwhile if it does better than a naive method: taking the value from the same month last year, or the average of the last three months. These baselines are free, understandable by everyone, and surprisingly hard to beat on short, noisy data.
If the model's performance interval largely overlaps that of the baseline, the conclusion is clear: the data does not yet allow you to do better. That is a result, not a failure. It avoids investing in a complex system that adds nothing.
Deciding despite uncertainty
The goal is not to obtain a certainty that the data cannot provide. It is to make an informed decision. So we present the results this way: the likely range of gains, the cost of an error in each direction, and what an extra year of data would bring.
With little data, rigor is not about finding the best model, but about knowing how to say when you cannot yet pick one.
And often, the most useful recommendation is not a model: it is to improve data collection right now, so that next year's evaluation is finally conclusive.