Is This a Prediction Problem? How to Decide Before Building a Model
A prediction problem is a business decision where past examples can help estimate an unknown future or current outcome before someone needs to act.
Many business problems do not meet that test. A model cannot repair a vague process, create a useful outcome label after the fact, or make a decision valuable when nobody will change their action. The first job is to frame the problem, not choose an algorithm.
Answer in brief
A prediction model is worth considering when there is a specific decision, a repeatable outcome to estimate, information available at the moment of prediction, enough relevant history, and a clear action that is better with the prediction than without it.
Scope
This guide helps teams decide whether to explore predictive modeling. It does not promise that a model will be accurate, useful, fair or ready for deployment without testing, review and ongoing monitoring.
What Will Someone Do Differently?
A model output only matters when it changes a real choice. Begin with the person who owns the decision, the time they need it, the action they can take, and the consequence of being wrong in each direction.
State the decision: For example: prioritize a support case, offer a retention intervention, schedule capacity, inspect a transaction or route a sales lead.
Name the decision maker: Identify the team or person who receives the prediction and can act on it.
Set the timing: Define when the prediction is needed and how quickly the decision must be made.
Describe the action: Write what changes for a high, low or uncertain score.
Compare the error costs: Explain the practical cost or risk of false positives, false negatives, delayed action and no action.
If every outcome leads to the same action, prediction may add little value. A dashboard, rule, clearer policy, process redesign or research may be the better next step.
A Model Needs a Defined Target
A predictive model learns from examples where an outcome is known. The target can be a category, number, ranking, probability or time-to-event, but it must represent the business question closely enough to guide the intended action.
Can the outcome be defined consistently, not only by one person's judgment?
Is it recorded after enough comparable cases to learn from?
Does the recorded outcome reflect what the business actually cares about?
Is the outcome known soon enough to learn and improve?
Could incentives or existing constraints distort the recorded label?
Example
“Will this customer leave in the next 90 days?” can be a prediction target if departure, the time window and the eligible customer population are defined. “Which customers matter most?” is a strategic question that needs a business definition before it becomes a modeling target.
Information From the Future Makes a Model Look Better Than It Is
For each proposed input, ask whether it is reliably available before the decision. A field completed after a case closes, a later customer action or a manually corrected status cannot be used to make an earlier real-world prediction.
This is data leakage: information that would not be available at prediction time slips into training or evaluation. It can produce impressive test results that disappear in operation. Keep the prediction timestamp, feature availability and outcome timestamp explicit.
List each candidate input and when it becomes available.
Check whether the field is created by the decision or process the model is meant to influence.
Use the same preprocessing and definitions in evaluation and live operation.
Treat missingness as a business signal to investigate, not merely a technical inconvenience.
The Right Answer May Not Be a Model
| Need | Better first option when | When prediction may help |
|---|---|---|
| Apply a stable policy | The rule is clear, consistent and can be written down | Cases vary enough that a single rule produces poor prioritization |
| Understand why something happens | The team needs causes, evidence or qualitative context | A timely estimate of a defined outcome changes a specific action |
| Find unknown groups or themes | The categories are not settled | The target label is already defined and repeated cases exist |
| Forecast total demand | The decision concerns future volume over time | Use forecasting rather than independent case-by-case prediction |
| Improve a workflow | The bottleneck is approval, handoff, data quality or ownership | A reliable estimate can help route or prioritize work inside the improved process |
Prediction is usually a component of a wider system. It should be compared with the current process and a simple baseline, not with an imagined manual process that has never existed.
Useful Data Is More Than a Large Spreadsheet
Teams need representative historical cases, trustworthy definitions and a practical way to produce the same inputs in operation. More rows do not fix a target that is inconsistent, biased by past decisions or unavailable at the needed time.
Coverage: the history represents the users, products, locations and conditions where the system will be used.
Quality: inputs and outcomes have stable definitions, known gaps and traceable sources.
Recency: historical patterns still resemble the operating context enough to be informative.
Governance: access, retention, privacy, security and sector requirements are understood before use.
Human oversight: people know what a score means, when to question it and how to correct or override it.
Proportionality: the expected benefit justifies the work, maintenance and risk for this decision.
For high-impact decisions involving people, eligibility, finance, health, employment, housing, safety or similar areas, involve appropriate domain, legal, privacy and compliance owners before deployment. A technically plausible model is not enough.
Test the Decision, Not Only the Metric
A useful evaluation compares a proposed model with a baseline using data and conditions that resemble the real decision. It should include the users who will act on the output and document the model's limits, error patterns and fallback path.
Choose a baseline: Use the existing process, a simple rule or a simple statistical method as the comparison.
Define the test period: Use a holdout period or a time-aware evaluation that reflects how decisions occur.
Set decision metrics: Measure outcomes that matter to the action alongside model metrics, such as workload, review quality, service level or loss avoided.
Inspect error cases: Look at false positives, false negatives, missing data and groups or settings where performance changes.
Decide the operating boundary: State when the output is advisory, when a human review is required and when the system should not be used.
Go/no-go question
Would the proposed system improve the named decision enough, under realistic conditions, to justify its ongoing data, review and governance requirements?
Illustration: Prioritizing Account Reviews
Illustrative example only, not a Fossilite client result: an account team cannot review every account before renewal. It asks whether a prediction could prioritize which accounts receive a human review first.
The team defines the decision as review priority, not automatic intervention. It defines a renewal-risk outcome and checks that product use, support history and contract information are available before the review window. It compares a simple rule-based queue with a small predictive prototype, then has account managers inspect cases where the two approaches disagree.
If the model cannot improve the review queue over the rule without creating unmanageable exceptions, the team keeps the rule and focuses on collecting better evidence. That is a valid result.
Common Mistakes
Starting with a preferred model or dataset before defining the decision.
Using a target that is easy to measure but does not match the business outcome.
Including information that appears only after the point where a prediction is needed.
Calling a forecasting, segmentation, research or process-design question a prediction problem.
Comparing a model only with a weak or undocumented manual process.
Measuring overall accuracy while ignoring the cost and distribution of errors.
Automating an action before people understand the output, limits and escalation path.
Assuming a model that worked on historical data will remain suitable after the process changes.
Prediction-Problem Checklist
A named decision maker can act differently because of the output.
The decision timing and required prediction horizon are defined.
The outcome label is specific, repeatable and connected to the business question.
Candidate inputs are available before the prediction is made.
A simpler rule, research, workflow or forecasting alternative has been considered.
Historical data is relevant to the intended operating context.
Error costs, affected users and risks are documented.
The model will be compared with a credible baseline under realistic conditions.
Human review, overrides, monitoring and a fallback path are defined.
Frequently Asked Questions
What is a prediction problem in machine learning?
It is a task where historical examples can help estimate an unknown outcome before a person or system needs to take an action. The outcome may be a category, number, probability, ranking or time-to-event.
When should a business not use machine learning?
Do not start with machine learning when there is no clear decision, no defined outcome, inputs are unavailable at decision time, a stable rule solves the need, or the expected value does not justify the maintenance and risk.
What is the difference between prediction and forecasting?
Prediction often estimates an outcome for individual cases. Forecasting estimates values over time, such as total demand or call volume, and must account for trends, seasonality and planning horizons.
How much data is needed for a prediction model?
There is no universal minimum. The answer depends on outcome frequency, data quality, complexity, variation in the operating context and the decision stakes. Start by testing whether the available history supports a stable and useful baseline.
Can a model make decisions automatically?
Only after the action, error costs, controls, affected people, operating context and accountability have been assessed. Many useful systems keep the model advisory and route uncertain or high-impact cases to people.
What is data leakage?
Data leakage occurs when training or evaluation uses information that would not be available when the prediction is made. It creates overly optimistic results and can cause failure in real operation.