Using Scrum to Build AI Products: A Practical Guide

Building products with AI features, such as recommendations, classification, search or generative assistants, brings a new kind of uncertainty. A team can plan a normal feature with reasonable confidence; it often cannot know in advance whether an AI model will perform well enough to be useful. Scrum was designed for exactly this kind of complex work. This guide explains what makes AI products different, how to adapt the Product Backlog, the Definition of Done and the Scrum events for them, and how to build AI features responsibly.

Key takeaways

• AI products carry extra uncertainty: whether a model will work, what data it needs and how it will behave with real users.
• Scrum's empiricism suits this uncertainty; Sprint Goals can focus on learning as well as delivery.
• The Product Backlog should include data work, experiments, evaluation and safety work alongside features.
• A Definition of Done for AI features usually adds evaluation, documented limitations and monitoring.
• Cross-functional teams that include data and machine learning skills avoid slow handoffs.

What makes AI products different

• Uncertain feasibility: you may not know whether a model can reach useful quality until you try.
• Data dependency: quality depends heavily on the data available, which may need collecting, cleaning or labelling.
• Probabilistic behaviour: AI outputs are not always the same, and some will be wrong. The product must handle that.
• Change over time: a model that works today may perform worse as real-world data changes.
• Extra risks: fairness, privacy, safety and transparency need deliberate attention.

Why Scrum fits

The Scrum Guide describes Scrum as founded on empiricism: knowledge comes from experience, and decisions are based on what is observed. With AI products, many important questions can only be answered by trying. Short Sprints, a usable Increment every Sprint and regular inspection with stakeholders help teams learn quickly whether an approach works, and change direction early if it does not. For the basics, see the five Scrum events explained.

Product Goals for AI features

A Product Goal should describe an outcome for users, not a technology. Compare:

• Weak: "Build a machine learning model for support tickets."
• Stronger: "Customers get the right answer to common support questions within minutes, without waiting for an agent."

The stronger goal leaves room to discover whether AI is the best way to get there. See Product Goal and Sprint Goal explained.

The Product Backlog for AI products

AI products need kinds of work that ordinary features may not. Make them visible in the Product Backlog so they can be ordered and discussed:

• Data work: collecting, cleaning, labelling and understanding data
• Experiments: testing whether an approach can reach useful quality
• Evaluation: agreeing how quality will be measured, and measuring it
• User experience for AI: explaining results, handling uncertainty and offering a fallback when the AI is wrong
• Safety and fairness work: checking for bias, misuse and privacy risks
• Monitoring: tracking behaviour after release

Our guide to managing a Product Backlog explains how to order learning-focused items early.

Writing experiment items

Experiments are most useful when they are specific. A simple structure:

• Hypothesis: "We believe a model can correctly route most common support tickets to the right team."
• Measure: "Accuracy on a sample of recent, representative tickets, checked by support agents."
• Threshold: "If it is right often enough to save agents time, we continue; if not, we try a different approach or stop."

A negative result is still valuable: it saves the team from investing heavily in something that would not work.

A Definition of Done for AI features

The Definition of Done is the commitment for the Increment, according to the Scrum Guide. For AI features, teams often add items such as these, as an example:

• Evaluated against agreed quality measures on representative data
• Performance checked across relevant groups of users, to spot unfair results
• Known limitations documented
• Behaviour when the model is uncertain or wrong has been designed and tested
• Privacy and security reviewed
• Monitoring and alerts in place after release
• Reviewed by a team member

See Scrum artifacts explained for more on the Definition of Done.

Sprint Goals and Increments

Sprint Goals for AI work can focus on learning as well as delivery, for example: "Find out whether automatic ticket routing is accurate enough to trial with one support team." Every Sprint should still produce a usable Increment that meets the Definition of Done. For AI features, that might be a working feature released to a small group behind a feature flag, an internal tool used by staff, or an improvement to data or evaluation that the team can build on.

The Sprint Review for AI features

Stakeholders need to see how the AI actually behaves, not just a polished demonstration. Show typical cases, difficult cases and failures. Invite stakeholders to try their own examples. Discuss quality measures and limitations openly. This builds realistic expectations and produces far more useful feedback.

Releasing gradually

AI features often benefit from careful release: to internal users first, then a small percentage of customers, then more, with monitoring at each step. Feature flags make it possible to switch a feature off quickly if problems appear. This fits Scrum's idea that an Increment can be delivered to stakeholders before the end of the Sprint, and that learning from real use is essential.

Team composition

AI products need data science and machine learning skills alongside product, design, engineering and domain expertise. When data scientists sit in a separate team that receives requests and hands back models, delays and misunderstandings are common, much like the dependency problems of component teams. Cross-functional teams that include these skills can learn and deliver end to end. See how to lead multiple Scrum Teams for more on feature teams.

Measuring success

• Offline measures: quality on test data, such as accuracy, useful for development but not enough on their own.
• Product outcomes: whether users achieve their goals faster or more easily with the feature.
• Trust and satisfaction: whether users rely on and are comfortable with the feature.
• Safety and fairness: whether problems are reported, and how different groups experience the feature.
• Flow: how quickly the team learns and delivers; see Kanban metrics explained.

Engineering practices that help

Teams building AI features often adopt practices sometimes called MLOps: versioning data and models as well as code, automating training and evaluation, deploying models through the same pipelines as other software, and monitoring behaviour in production. These practices make it possible to release improvements frequently and safely, which is exactly what Scrum's short cycles need.

A worked example: a support-ticket classifier

As an illustration, a team wants to route support tickets automatically:

• Sprint 1: the Sprint Goal is to learn whether routing is feasible. The team gathers a sample of anonymised past tickets, agrees with support agents what "correct" means and runs a quick experiment. Results are promising for common ticket types, poor for rare ones.
• Sprint 2: the team builds an internal tool that suggests a route for common tickets, which agents can accept or change. It meets a Definition of Done including evaluation, documented limitations and monitoring.
• Sprint 3: agents use the tool. The Sprint Review shows where it helps and where it errs. Feedback reveals that some suggestions are confusing, so the team improves how suggestions are explained.
• Sprint 4: with evidence that agents trust the suggestions, the team automatically routes the most reliable ticket types and keeps human review for the rest.

At each step, the team inspected real results and adapted.

Handling model updates in Sprints

AI features often need ongoing improvement: retraining on new data, adjusting thresholds or replacing models. Treat these as Product Backlog items like any other, ordered by value and risk. Each update should go through the same evaluation and Definition of Done as the original release, and be monitored after release, because an update that looks better on test data can behave differently with real users.

Estimating AI work

AI work is harder to estimate than most software work, because some of it is research. Rather than precise estimates, many teams use time-boxed experiments ("we will spend up to one Sprint finding out whether this is feasible") and forecast delivery work once feasibility is known. Flow data, such as how long similar items took before, becomes more useful as the team gains experience.

Managing stakeholder expectations

Stakeholders sometimes expect AI features to be perfect, or to be delivered as quickly as simple features. Be clear from the start that AI outputs will sometimes be wrong, that feasibility is uncertain until tested, and that the team will share evidence at every Sprint Review. Showing real examples, including failures, is the most effective way to set realistic expectations.

Common mistakes

• Starting with the technology, not the user problem.
• Hiding data and evaluation work, so the backlog looks simpler than reality.
• Demonstrating only the best cases at the Sprint Review.
• No plan for when the AI is wrong.
• Releasing and forgetting, without monitoring.
• Separating data scientists from the product team, creating handoffs.

Learn more

For risks and guardrails, see responsible AI use in agile teams, and for AI in everyday Scrum work, our complete guide to AI in Scrum. Product Owner AI Plus covers using AI in product work and Scrum Master AI Plus covers using AI in Scrum Master work; both include self-paced training and an online exam on ExamVault by CertExpert with three attempts included, and certificates and digital badges are valid for two years.