Agile teams are adopting AI quickly, both as a tool in their own work and as a feature in the products they build. Moving fast is part of agility, but so is inspecting and adapting when things go wrong. Responsible AI use means getting the benefits while managing the real risks: inaccurate outputs, leaked data, unfair results and misplaced trust. This guide explains the main risks, the frameworks organisations refer to, and practical guardrails Scrum Teams can put in place.
Key takeaways
- Responsible AI means using and building AI in ways that are accurate, safe, fair, transparent and accountable.
- Widely referenced frameworks include the OECD AI Principles, the NIST AI Risk Management Framework, the EU AI Act and ISO/IEC 42001.
- For Scrum Teams, the Definition of Done and a team working agreement are practical places to build in safeguards.
- People remain accountable for decisions and outcomes, whatever role AI played.
- This guide gives general information, not legal advice; follow your organisation's policies and applicable law.
Why responsible AI matters for agile teams
Agile teams touch AI in two ways. First, they use AI tools to help with their own work, such as drafting, summarising and coding. Second, many build AI features into their products. In both cases, mistakes can affect customers, colleagues and the organisation. Scrum's emphasis on transparency, inspection and adaptation makes it well suited to handling AI responsibly, as long as teams deliberately include AI risks in what they inspect. For the broader picture, see our complete guide to AI in Scrum.
Frameworks organisations refer to
Several frameworks and laws shape how organisations approach AI. Each has its own official documentation, and organisations should consult them directly and seek appropriate advice. In brief:
OECD AI Principles
Adopted by the Organisation for Economic Co-operation and Development in 2019 and updated in 2024, these principles call for AI that benefits people and the planet, respects human rights and democratic values including fairness and privacy, is transparent and explainable, is robust, secure and safe, and has clear accountability.
NIST AI Risk Management Framework
Published by the US National Institute of Standards and Technology in January 2023, the AI RMF is a voluntary framework organised around four functions: Govern, Map, Measure and Manage. It helps organisations identify and manage AI risks throughout a system's life.
The EU AI Act
The European Union's AI Act entered into force in August 2024, with obligations applying in stages over the following years. It takes a risk-based approach, with stricter requirements for AI systems considered high-risk and prohibitions on certain uses. Organisations offering or using AI in the EU should check how it applies to them.
ISO/IEC 42001
Published in 2023, ISO/IEC 42001 is an international standard for AI management systems, describing how organisations can manage AI responsibly in a structured, auditable way.
Risks when teams use AI tools
- Accuracy: generative AI can produce fluent but incorrect content, including invented facts and references.
- Privacy and confidentiality: information entered into tools may be stored or processed outside the organisation.
- Security: AI-generated code can contain vulnerabilities; AI systems that read external content can be manipulated by hidden instructions, a risk known as prompt injection.
- Intellectual property: questions about the ownership and licensing of AI-generated content are still developing; follow your organisation's guidance.
- Bias: outputs can reflect biases in training data, which matters especially when content affects people.
- Over-reliance: teams may lose skills or stop thinking critically if they trust outputs too readily.
Guardrails for Scrum Teams
A team working agreement
Agree, as a team, which tools you use, what data is off limits, how outputs are reviewed and how AI use is disclosed. Revisit the agreement at Retrospectives.
The Definition of Done
The Definition of Done is the Scrum commitment for quality. Teams can add items such as:
- AI-assisted code has been reviewed by a team member and passed all tests
- AI-drafted content has been checked for accuracy
- No personal or confidential data was shared with unapproved tools
See Scrum artifacts explained.
Data rules
Many organisations classify data, for example public, internal, confidential and restricted, and specify which classes may be used with which tools. Make these rules easy to find and easy to follow.
Human in the loop
Keep a person responsible for reviewing outputs and making decisions, especially where they affect customers or colleagues.
Transparency
Be open, internally and with stakeholders, about where AI helped. It builds trust and invites scrutiny.
For teams building AI features
For the full delivery approach, see using Scrum to build AI products.
Teams that build AI into products carry additional responsibility. Practical steps include:
- Include risk items in the Product Backlog: evaluation, bias checks, safeguards against misuse and monitoring are real work that deserves ordering alongside features.
- Extend the Definition of Done: for example, the feature has been evaluated on representative data, known limitations are documented and monitoring is in place.
- Show limitations at the Sprint Review: stakeholders should see where the feature works poorly, not only where it shines.
- Monitor after release: AI behaviour can change as real-world data changes, so keep inspecting.
- Plan for mistakes: make it easy for users to report problems and for the team to respond.
Who is responsible for what
| Who | Role in responsible AI |
|---|---|
| Product Owner | Weighs value against risk when ordering the Product Backlog; ensures risk-related work is visible |
| Developers | Build and review work to meet the Definition of Done, including AI-related quality checks |
| Scrum Master | Helps the team create and follow guardrails; raises gaps in organisational policy as impediments |
| Leaders | Set policy, approve tools, provide training and model responsible use |
A responsible AI checklist
- Is the tool approved for this purpose and this data?
- Have we removed personal and confidential information?
- Has a person reviewed the output?
- Could the output be unfair to anyone?
- Are we being transparent about AI's role?
- If we are building AI, have we evaluated it, documented limitations and planned monitoring?
- Do we know what to do if something goes wrong?
When something goes wrong
Mistakes will happen: an AI summary misleads a stakeholder, confidential data is pasted into the wrong tool, or an AI feature behaves unexpectedly. Handle it the way good agile teams handle any incident. Contain the problem, inform the right people according to your organisation's policies, then hold a blameless review focused on how the system allowed it and what to change. See psychological safety in agile teams for why blame makes problems harder to surface.
A worked example: a data slip
As an illustration, a Developer pastes a log file into a public AI assistant to help debug an error, then realises the log contained customer email addresses. What a responsible team does next:
- Stop and report: the Developer tells the Scrum Master and follows the organisation's data incident process straight away. Reporting quickly matters more than anything else.
- Contain: the right people check what can be done with the tool provider and whether anyone needs to be informed, according to policy and law.
- Review without blame: at a short review, the team asks how the system made this easy. It finds there was no approved tool for debugging help and no reminder about logs containing personal data.
- Improve: the organisation provides an approved tool, the team adds a working-agreement rule about sanitising logs, and a simple check is added before logs are shared.
Because the Developer felt safe to report immediately, the problem was contained and the system improved.
Training the team
Guardrails only work if people understand them. Short, practical training helps: what the organisation's policy says, which tools are approved, examples of data that must not be shared, how to check outputs and what to do if something goes wrong. Refresh it when tools or policies change.
Questions to ask AI vendors
- How is the data we enter stored, for how long and where?
- Is our data used to train or improve models, and can we opt out?
- What security certifications or assessments does the service have?
- What administrative controls do we have over users and settings?
- How are incidents reported to customers?
Where is your team? A simple maturity view
As an illustration, teams often move through stages like these:
| Stage | What it looks like | Next step |
|---|---|---|
| Ad hoc | People use various tools individually, with no shared rules | Check organisational policy and agree basic data rules |
| Agreed | The team has a working agreement and approved tools | Add review of AI-assisted work to the Definition of Done |
| Inspected | AI use is discussed regularly in Retrospectives | Measure whether AI actually helps, and adapt |
| Embedded | Guardrails are routine, and risk work appears in the Product Backlog | Share practices with other teams and keep reviewing as tools change |
Responsible AI in regulated industries
Teams in sectors such as banking, insurance, healthcare and the public sector often face extra requirements: formal approval of tools, stricter data handling, documentation of how automated decisions are made, and audit trails. In these settings, it helps to involve risk, compliance or legal colleagues early, treat their requirements as part of the Definition of Done, and make approval steps visible in the workflow rather than surprises at the end. Scrum's transparency makes it easier to show regulators and auditors how work was done.
Common mistakes
- No policy or working agreement, so everyone uses AI differently.
- Treating responsible AI as a one-off approval rather than something inspected regularly.
- Leaving risk work off the Product Backlog, so it never gets done.
- Assuming vendors have handled all risks. Teams still need to understand how tools behave and what data they receive.
- Blaming individuals when AI-related mistakes happen, which discourages reporting.
Learn more
For practical AI use by role, see how Scrum Masters can use AI and how Product Owners can use AI, and for writing safe prompts, writing AI prompts for agile work. Scrum Master AI Plus and Product Owner AI Plus cover using AI in everyday Scrum and product work; both include self-paced training and an online exam on ExamVault by CertExpert with three attempts included, and certificates and digital badges are valid for two years.
Frequently asked questions
What is responsible AI?
Using and building AI in ways that are accurate, safe, fair, transparent and accountable.
What frameworks guide responsible AI?
Widely referenced ones include the OECD AI Principles, the NIST AI Risk Management Framework, the EU AI Act and ISO/IEC 42001.
Does the EU AI Act apply to my team?
It depends on what your organisation does and where. Check with your organisation's legal or compliance team; this guide is not legal advice.
How can Scrum Teams build in AI safeguards?
Through a team working agreement, additions to the Definition of Done, clear data rules, human review and transparency.
What is prompt injection?
A security risk where hidden instructions in content an AI system reads cause it to behave in unintended ways.
Who is accountable for AI mistakes?
People, not tools. Teams and organisations remain accountable for decisions and outcomes, whatever role AI played.
Should risk work go on the Product Backlog?
Yes. Evaluation, bias checks and monitoring are real work and should be visible and ordered.
What should a team do after an AI-related mistake?
Contain it, inform the right people, and hold a blameless review to improve the system.