AI Governance Needs an Incident Response Plan

Most organisations have a plan for when their systems are attacked. Far fewer have a plan for when their AI systems behave in ways they did not anticipate.

Over the past few years, organisations have made significant progress in thinking about AI governance.

Policies are being written. Risk assessments are being introduced. AI inventories are being developed. Governance committees are being established. Organisations are beginning to define acceptable use, assign accountability and evaluate AI systems before deployment.

All of this is necessary.

But much of AI governance still focuses on one question:

How do we prevent AI systems from causing harm?

There is another question that deserves just as much attention:

What happens when prevention fails?

Because eventually, it will.

An AI system may expose information it should not disclose.

An AI agent may perform an action beyond its intended authority.

A model may begin producing systematically unreliable outputs after an update.

A retrieval-augmented system may surface sensitive internal information.

An automated decision system may generate outcomes that create operational, regulatory or reputational consequences.

Or an attacker may manipulate an AI system through techniques such as prompt injection, data poisoning or abuse of connected tools.

At that point, an organisation no longer has an AI policy problem.

It has an AI incident.

And that requires a very different capability.

---

Governance Before Deployment Is Not Enough

Traditional AI governance tends to be strongest before deployment.

Organisations ask whether the system should be used, whether the data is appropriate, whether risks have been assessed, whether human oversight exists and whether the use case complies with internal policies.

These are important controls.

But deployed AI systems do not operate in static environments.

Models change.

Data changes.

Users behave in unexpected ways.

Attackers adapt.

Third-party services are updated.

Business processes evolve.

And increasingly, AI systems are being connected to tools that allow them not merely to generate information, but to take action.

This changes the nature of the risk.

The question is no longer simply whether an AI model produces an incorrect answer.

The question may be whether an AI agent can send an email, modify a record, approve a transaction, execute code, access confidential information or trigger another system before anyone realises something has gone wrong.

The distance between an AI error and an organisational consequence is becoming shorter.

Governance therefore cannot end when an AI system is approved for deployment.

It must continue into operations.

---

AI Incidents Are Not Always Cyber Incidents

One temptation is to treat AI incidents as another category of cybersecurity incident.

Sometimes they are.

If an attacker compromises an AI system, steals model credentials or exploits an AI application to access protected data, the organisation's existing cybersecurity incident response capability should play a central role.

But not every AI incident begins with an attacker.

Imagine an AI system recommending decisions that appear reasonable individually but produce systematic discrimination over time.

Or an internal assistant repeatedly generating inaccurate regulatory guidance that employees begin relying upon.

Or an autonomous agent performing technically authorised actions that collectively create an unintended business outcome.

Nothing may have been "hacked".

The network may be functioning normally.

Authentication may have succeeded.

Every API call may be legitimate.

Yet something has clearly gone wrong.

This is where AI incident response becomes distinct.

Traditional cybersecurity incident response often asks:

Has confidentiality, integrity or availability been compromised?

AI incident response may need to ask additional questions:

• Has the system behaved outside its intended purpose?
• Can its outputs still be trusted?
• Has its level of autonomy exceeded acceptable risk?
• Are people being affected by its decisions?
• Should the system still be allowed to operate?

Those are not purely technical questions.

They are governance questions occurring at operational speed.

---

The 2:00 a.m. Governance Problem

Consider a simple scenario.

At 2:00 a.m., monitoring detects unusual behaviour from an AI agent used in an important business process.

The security operations team sees the alert.

The agent is not clearly compromised, but its actions are inconsistent with its normal behaviour.

Stopping it could interrupt a critical process.

Allowing it to continue could create greater consequences.

Who makes the decision?

The SOC analyst?

The CISO?

The system owner?

The Chief AI Officer?

The business executive responsible for the process?

The model provider?

The AI governance committee?

And perhaps more importantly:

Does anyone actually have the authority to stop it?

This is where many governance models may struggle.

A committee that meets monthly cannot govern an incident unfolding in minutes.

A policy requiring human oversight is insufficient if nobody knows which human has decision authority during an emergency.

A risk classification completed six months earlier does not tell an analyst whether an autonomous system should be disconnected now.

AI governance therefore needs something cybersecurity learned decades ago:

Governance must define decision rights before the incident occurs.

---

From AI Governance to AI Incident Response

Organisations do not necessarily need an entirely separate incident management organisation for AI.

But they do need to extend existing incident response capabilities to recognise the characteristics of AI systems.

A practical AI incident lifecycle could contain six stages:

1. Detect

Identify behaviour that may indicate an AI incident.

Detection cannot rely only on conventional security alerts.

Organisations may need to monitor model behaviour, unusual tool usage, unexpected access patterns, abnormal volumes of autonomous actions, policy violations, significant performance degradation and user-reported outcomes.

The challenge is that an AI system can fail while the underlying infrastructure remains completely healthy.

2. Assess

Determine what is happening and how serious it is.

This requires more than asking whether the system has been compromised.

Teams may need to understand:

• what the AI system did;
• which users, systems or data were affected;
• whether the behaviour is reproducible;
• whether autonomous actions are still occurring;
• whether the issue is security-related, model-related, data-related or operational;
• what could happen if the system continues operating.

AI incidents may therefore require collaboration between cybersecurity, data science, legal, risk, privacy and business teams.

3. Contain

Limit the system's ability to create further harm.

This is where organisations need controls that are designed before deployment.

Containment might mean restricting access to particular tools, reducing permissions, disabling autonomous actions, isolating a model, reverting to human approval or temporarily suspending the system.

For highly autonomous systems, containment may need to occur within seconds.

That makes pre-defined authority essential.

4. Investigate

Understand why the incident occurred.

This may require evidence that traditional systems do not always capture:

• prompts;
• model outputs;
• retrieval context;
• tool calls;
• agent reasoning traces, where available and appropriate;
• model versions;
• configuration changes;
• policy decisions;
• human overrides;
• external API interactions.

Without sufficient observability, organisations may know that an AI incident occurred without being able to reconstruct how it happened.

AI governance therefore has a direct dependency on AI observability.

You cannot govern what you cannot reconstruct.

5. Recover

Return the affected process to a trusted state.

Recovery should not simply mean switching the AI system back on.

The organisation must determine whether the conditions that caused the incident have been addressed and whether confidence in the system has been restored.

In some cases, the safest recovery strategy may be operating temporarily without AI.

That possibility should exist before an incident occurs.

6. Learn

Every significant AI incident should feed back into governance.

Risk assessments may need updating.

Permissions may need changing.

Monitoring rules may need improvement.

Models may require additional evaluation.

Policies may need revision.

Human oversight mechanisms may need redesign.

This creates a feedback loop:

Govern → Deploy → Monitor → Respond → Learn → Govern again.

AI governance should not be a gate. It should be a control loop.

---

The Kill Switch Is a Governance Mechanism

The idea of a "kill switch" for AI systems is often discussed as a technical safeguard.

But the difficult part is rarely the button.

The difficult part is deciding:

Who is allowed to press it?

And:

Under what conditions?

If an organisation deploys an autonomous AI system, it should define explicit criteria for reducing or removing that autonomy.

For example:

• If confidence falls below a defined threshold, require human approval.
• If the system attempts an unauthorised class of action, revoke tool access.
• If abnormal behaviour exceeds a risk threshold, move the system into restricted mode.
• If significant harm is suspected, suspend the system entirely.

This creates something more useful than a binary kill switch.

It creates graduated autonomy.

Instead of treating an AI system as either fully operational or completely disabled, organisations could design several operating states:

Autonomous → Supervised → Restricted → Isolated → Disabled

The appropriate state could depend on risk, confidence and observed behaviour.

That would allow governance to become dynamic rather than static.

---

Incident Response Should Begin During AI Design

The worst time to decide how to stop an AI system is after it has already failed.

Incident response requirements should therefore be considered during system design and procurement.

Before deploying an AI system, organisations should be able to answer questions such as:

• Who owns the system?
• Who monitors it?
• What constitutes an AI incident?
• What telemetry is retained?
• Can individual capabilities be disabled?
• Can autonomous actions be suspended without shutting down the entire business process?
• Can the system operate in a degraded or human-controlled mode?
• Who has emergency authority?
• When must legal, privacy, cybersecurity or senior management become involved?
• How will affected users be informed?
• How will evidence be preserved?
• What happens if the AI provider itself experiences an incident?

These are architecture questions.

But they are also governance questions.

And increasingly, the distinction between the two is disappearing.

---

From Human-in-the-Loop to Human-in-Command

Much of responsible AI has focused on keeping humans "in the loop".

That remains useful, but it may not be enough for increasingly autonomous systems.

A human approving every individual AI action could eliminate much of the value of automation.

At the same time, removing meaningful human control creates obvious governance problems.

The more useful objective may therefore be human-in-command.

Humans do not necessarily need to approve every action.

But humans must retain meaningful authority over the system.

They should be able to:

• observe it;
• challenge it;
• restrict it;
• override it;
• and, when necessary, stop it.

This becomes particularly important as AI systems evolve from tools that recommend actions to agents that execute them.

The governance challenge is no longer simply oversight of AI outputs.

It is governance of machine authority.

---

The Next Phase of AI Governance

AI governance has spent considerable effort defining what organisations should do before deploying AI.

The next phase must address what organisations do after something goes wrong.

Policies matter.

Risk assessments matter.

Ethical principles matter.

Human oversight matters.

But mature governance is demonstrated under pressure.

When an AI system behaves unexpectedly, organisations need more than principles.

They need telemetry.

Escalation paths.

Decision rights.

Containment mechanisms.

Recovery procedures.

And people who already know what authority they have.

The defining question for AI governance may therefore become surprisingly simple:

When your AI system fails at 2:00 a.m., who has the authority to stop it?

If the answer is unclear, the organisation may have an AI governance framework.

But it does not yet have operational AI governance.