When Clever Gets in the Way

There is a particular sentence that has become one of the best compliments I can imagine receiving about something I have built.

“Yeah, that's about right.”

It sounds almost dismissive. In operations it is gold. It means the thing on the screen has passed the fastest and most unforgiving test available, it resembles the world the person is standing in.

I learned the opposite lesson while building an early forecasting engine on my own. I became obsessed with accuracy. More signals. More refinement. More effort to shave another piece off the error. At one stage the model was running at around 11% MAPE, a measure of forecast error where lower is better. Put roughly, the errors averaged eleven percent of the actual values across the observations being measured. Technically, it was becoming an impressive piece of work.

I showcased it to the CEO of a major workforce technology company and his Head of Engineering, a man with around twenty-five years of engineering experience and a team behind him. Their reaction was close to awe. I would be lying if I said that did not feel rather good, particularly because I had built the engine alone.

Then I showed it to a regional manager and a store user. Their reaction was closer to complete confusion. They were, how shall I put this, refreshingly blunt. I had optimised the engine and forgotten the driver.

That was one of the more useful failures I have had because the technology had not failed at all. I had. I had mistaken model quality for product quality. The forecast could be mathematically respectable and operationally useless at exactly the same time.

Since then I have become much more interested in whether someone can understand what a tool is telling them, whether it fits the decision they actually need to make and whether it earns trust quickly enough to be used. Accuracy matters of course, but accuracy nobody can use is an expensive form of cleverness.

If your system contradicts an experienced manager, you need to be very sure why before you assume the clever bit is the software.

Credibility has to be earned in the right order. Show people something they recognise as true. Let them understand why. Then ask them to trust you with something they could not easily see themselves.

The lesson was not “make the model less accurate”. It was much more humbling, never let the sophistication of the thing you built become more important than the person who has to use it.

I have also become more careful about what accuracy means. An eleven percent MAPE can be a useful measure of a forecasting model, but it cannot tell you whether a manager understands why tomorrow looks different, whether they know what action to take, or whether they trust the number enough to change a rota. Those questions are messier. They are also much closer to value.

The rebuild therefore started with a plain piece of paper and the answers the users had already given me. I went back to the beginning and rebuilt it from the ground up, putting their journey at the front of the app and layering the engineering underneath afterwards. Previously I had been asking how much more cleverness I could add to the engine. This time I asked what the person looking at it needed to understand and do next.

I stopped chasing fractions of a percentage point and aimed for around 85 to 88% accuracy, putting more of my effort into a journey someone could understand and use. The engineering still mattered, but it had to support that experience instead of becoming the main attraction.

That sounds obvious written down. It was not obvious while I was enjoying myself making the model better. There is a reason I keep returning to the people doing the work. They are remarkably effective at puncturing your cleverest assumptions, usually in fewer words than you would prefer.