Your data scientists can figure out which variables improve a forecast. They can build the model, test it, monitor it, and explain why it occasionally does something that makes everyone stare at the screen in silence. What they can’t do is decide what should change in your business when the model works. That part belongs to you.
I know this because I once got paid to make the mistake.
In the mid-aughts, after I founded the analytics department at Publicis, our team sold Hewlett-Packard’s imaging and printing division a marketing mix modeling project. This was early enough in the history of MMM that most meetings began with explaining what one was. I knew modeling, and I also knew change management mattered. I said so in the pitch.
Then I spent months learning how HP allocated marketing budget: who submitted requests, who approved them, which departments controlled which dollars, and how the whole thing moved through the organization. We built a model that fit that process.
What I never built, and never asked anyone to build, was a picture of what the process should look like after the model existed. We mapped the as-is, but never designed the to-be.
That wasn’t an analytics deliverable. It was a management deliverable. It never occurred to me that anyone had to create it.
The mistake taught me something: analytics isn’t primarily a technology process. It’s a management process. If you want analytics to change results, there are four decisions you can’t outsource.
The business decision your model is supposed to improve is only the first of them. Three more are still sitting on your desk.
1. The Target: What Decision Are You Trying to Improve?
When someone brings you an analytics proposal, resist starting with the technology. “We need AI,” “We need a forecasting model,” and “We need a customer data platform” may all be reasonable ambitions, but none of them describes a business problem. Those are just purchases for Accounts Payable to process.
Your first decision is what business decision the analytics is supposed to improve.
And the test is simple: What will someone do differently if it works?
A retailer is not really trying to improve demand forecasting. Someone is deciding how much inventory to put in each store. A sales organization does not need a lead-scoring model for its own sake. Someone is deciding which opportunities deserve attention. A hospital does not need predictive analytics in the abstract. Someone is deciding which discharged patients need additional follow-up.
You should be able to describe the project in one sentence: We are helping [someone] decide [something] in order to improve [an outcome]. If you cannot fill in those three blanks, you are not ready to discuss algorithms.
You may not even have an analytics problem. Sometimes the real problem is unclear authority, bad incentives, or a process everyone knows is broken and nobody owns. A model can’t improve a decision nobody is empowered to make.
2. The Evidence: What Would Change Your Mind?
Once the target is clear, the next question is usually, “What data do we have?” It sounds practical. It’s backwards. That’s ordering dinner from whatever happens to be left in the kitchen.
Ask instead: What would we need to know to make this decision better?
Available data has a habit of defining the project. You end up solving the problem your database happens to contain rather than the one your business needs solved.
Back at HP, one of the most important variables in the project never appeared in a spreadsheet. We modeled media effectiveness with great care, but nobody asked how tightly each department’s identity and influence were tied to its budget. That turned out to matter.
The model couldn’t measure it because the organization never had.
Good analytics means asking what evidence would cause you to act differently. What would raise your confidence? What would make you reverse a decision? What important factor is missing because nobody currently measures it? And most insidious of all, which assumptions have fossilized into “facts”?
Data is evidence, not truth. Historical data contains old strategies, old incentives, old customer behavior, and occasionally old mistakes preserved with remarkable numerical precision. Your analytics team can tell you what the data says. You must decide what it means.
3. The Adoption: Who Has to Lose Something?
Our marketing mix model for HP recommended reallocating spending across departments. We knew organizations dislike sudden budget shifts, so we had constrained the model. We capped how much spending could shift because we were trying to reduce risk.
About a third of the departments were asked to give up four or five percent of their budgets, mostly from traditional above-the-line advertising, with the money moving toward online and merchandising. Four or five percent was hardly a revolution.
The resistance was fierce.
That was the lesson. The risk we were managing wasn’t the risk that mattered most.
We were worried that aggressive changes in media spending might hurt business results. The organization was worried about something else entirely: whose budget was getting smaller.
The size of the transfer was almost beside the point. Money was crossing organizational boundaries. Some groups were losing resources and influence, and maybe a piece of the future, to disciplines that were becoming more important. A modest recommendation on a spreadsheet feels different when it says your group should have less and someone else’s should have more.
This is why “start small” isn’t always enough. One percent taken from my budget and handed to your department can create more resistance than a ten percent across-the-board cut. Direction matters more than magnitude. So does who is taking from whom.
Your model doesn’t really recommend percentages. Inside the company, it recommends winners and losers.
The same thing happens everywhere. A churn model can identify customers likely to leave, but someone still has to decide who contacts them, what they’re authorized to offer, and how success gets measured. Without those operational decisions, you don’t have a retention system. You just have a weekly list of unhappy customers.
Someone has to behave differently, and that person may have to surrender authority, budget, time, or a way of working they like. You must find that person before launch.
At HP, we had executive sponsorship. That became important later, because the postmortem concluded we needed earlier buy-in. It sounded right, which is probably why nobody challenged it.
But buy-in from whom?
We had support from the center from the beginning. What we’d underestimated was the resistance in the departments where the recommendations actually hit. Sponsorship was central. Resistance was local.
And I had never asked whether anyone had enough cross-divisional budget authority to make the model’s recommendations stick.
4. The Owner: Who Is Still Responsible Next Year?
A model doesn’t stay valuable because it once worked. Customer behavior changes, markets move, data definitions drift, and people invent workarounds. Yesterday’s clever model becomes tomorrow’s mysterious spreadsheet nobody wants to delete because someone vaguely remembers it being important.
That’s roughly what happened to ours.
We’d built the HP model as a first implementation for one country, with other countries to follow. The expansion was shelved. The model survived for a while, but not in the role it was designed for.
Instead of helping determine budget allocations, it was used to predict the likely results of allocations that had already been decided without it.
That detail captures the whole problem. We had built a model to help make a decision, and the organization turned it into a tool for describing a decision made somewhere else. It went from deciding to reporting.
Eventually, without periodic updating, the model degraded and was put out of its misery.
This is why every important analytical system needs a business owner, not just a technical owner. The data science team can own the model. IT can own the application. Finance can validate the savings. But one person has to be accountable for whether the decision itself is getting better.
Otherwise, everyone owns a component and nobody owns the result.
The Question to Ask Monday Morning
For years, the standard prescription for failed analytics projects has been some combination of better data, better models, more executive sponsorship, and better change management. All useful. None sufficient.
We built a good model. We constrained the model to make its recommendations safer. We understood the current processes. The project still failed to change the decision it had been built to improve.
We had never worked out what would happen when the algorithm produced an answer the organization didn’t want to hear.
So the next time someone brings you a model, dashboard, AI application, forecasting system, or optimization project, do not begin by asking whether the analytics will work. Make four decisions first.
- What decision are we trying to improve?
- What evidence would change our minds (and can we get it)?
- Who has to give something up (and do they know yet)?
- Who owns the result a year from now?
Your data scientists can’t answer for you. These are management decisions.
While your data scientists can build the model, only you can build the organization that will act on it.
Click here for more columns from Michael Bagalman’s Data Science for Decision Makers series.
