How to run periodic AI system reviews

The record of the review is the evidence, not the meeting.

10 min read
In short: An AI system review confirms that a system's owner, purpose, providers, and controls are still right, and records that the check happened. Review every 180 days by default, and sooner on purpose drift, a new provider, violations, or a missing owner. In Arcjet, Mark reviewed sets the date, and the audit log records the change.

What is an AI system review?

An AI system review confirms that a system's owner, purpose, providers, and controls are still right, and records that the check happened. Every 180 days is a sensible default. Review sooner when the system drifts from its declared purpose or starts using a new provider. The record of the review is the evidence, not the meeting.

An AI system is an application or agent together with the model providers and tools it uses. Each one in your AI inventory carries an owner, a purpose, provider decisions, and a last-reviewed date. A review is how those fields stay true after the day someone first filled them in. For how reviews fit into a governance program, see what is AI governance.

NIST AI RMF 1.0 (the National Institute of Standards and Technology AI Risk Management Framework) asks in GOVERN 1.5 that ongoing monitoring and periodic review of the risk management process and its outcomes are planned, with defined roles and a chosen frequency. A review date on each system, and a list of the ones that are overdue, is evidence for that subcategory.

How often should you review an AI system?

Review each AI system at least every 180 days, and sooner when something about it changes. Arcjet uses 180 days as its single review interval: when you mark a system reviewed, the next review falls due 180 days later, and the system appears under Review overdue after that.

NIST doesn't set an interval; GOVERN 1.5 leaves the frequency to you. Annual reviews are common in other risk programs, and some teams pick them for that reason. A shorter cycle fits AI systems better for two reasons. Providers change models and terms more often than most suppliers change anything, and the tools an agent calls change whenever someone edits its configuration. Six months is long enough that a review isn't busywork, and short enough that a system can't drift for a year unnoticed.

Use your risk tier to decide how much each review covers rather than how often it happens. A low-tier internal assistant may need ten minutes. A high-tier system that makes decisions about people needs the owner, an engineer, and someone from security or legal. For how tiers work, see AI risk tiers.

What should trigger an early AI system review?

Review a system early when an observation shows its record may no longer be true. The following table lists the triggers that Arcjet surfaces and what each one suggests.

TriggerHow you see itWhat it suggests
Purpose driftA drift notice to the owner

The system handles a data class or calls a tool its purpose doesn't declare

A new provider or MCP serverAn unreviewed provider or server on the system

The system's providers or tools have changed since the last review

ViolationsViolations in the owner digest and Discovery lists

People still use a prohibited provider, so the decision or the approved alternative may be wrong

No owner

The system appears under the No owner filter

The owner left the team, and the system needs a new one
Frequent held actionsHold requests to the owner

The agent keeps trying actions that need approval, so its policy or purpose needs a closer look

Purpose drift is the most useful trigger. Arcjet compares the data classes and tools a purpose declares with sensitive-information findings and captured tool calls. When a support agent declared to handle order numbers starts receiving payment card numbers, the owner gets a notice. For how the comparison works, see AI purpose drift.

What does an AI system review check?

A review checks each field of the record against what Arcjet observes, and changes the field or the system when they disagree. The following table lists what to check, where to look, and what to change.

CheckWhere to lookIf it's wrong
The owner is still the right personThe owner fieldAsk an administrator to reassign it
The purpose matches what the system does

Declared data classes and tools against sensitive-information findings and captured tool calls

Change the purpose, or change the system or its policies to match it

Every provider has a decisionThe providers and clients on the systemApprove or prohibit the undecided ones
Decisions holdViolations since the last review

Follow up with the people involved, or revisit the decision and the approved alternative

Runtime policies fit the purpose

Guard policies on the system, and whether they're live or in dry run

Ask engineering to add or tighten a policy, and test it in dry run

The risk tier and EU AI Act role are rightThe risk tier and role fieldsChange the tier, which can change how deep later reviews go
The system still runsRecent activity for its actor

Retire it. Deleting an actor revokes its keys and keeps the record

A review that changes nothing is a valid result. Marking it reviewed records that someone looked on that date and found the record true.

Who should take part in an AI system review?

The owner runs the review, and brings in other people only for the checks that need them. A review that requires a meeting of five people for every system doesn't happen every 180 days, so keep the default small and widen it by risk tier.

The following list shows who joins and why:

  • The owner, for every review. The owner answers whether the purpose is still right and whether the system is still needed.
  • An engineer who works on the system, when the purpose or the providers changed, or when a runtime policy needs to change. The engineer knows which tools and data the system actually reaches.
  • Someone from security, when the system has violations, held actions, or a drift notice involving secrets or personal data.
  • Legal or privacy, for high-tier systems, or when the EU AI Act role or the data classes change.
  • A team administrator, to record changes to the owner, purpose, or provider decisions, and to mark the review. The owner can be the administrator.

Record who took part in your own ticket or notes if your auditors ask for it. The review date in Arcjet shows that the review happened and when. It doesn't list participants.

How do you run an AI system review, step by step?

The following procedure is one review of one system. Run it for each system the overdue list or the owner digest names.

  1. Open the overdue list or the owner digest. Filter the inventory by Review overdue, or start from the systems listed in your weekly owner digest. Add any system with a drift notice or a new provider, even if it isn't due.
  2. Check the purpose against observed tools and data. Compare the declared data classes and tools with the sensitive-information findings and captured tool calls since the last review. Read each drift notice.
  3. Check provider decisions and violations. Confirm that every provider and client the system uses has a decision, and read violations since the last review.
  4. Confirm or change the record. Update the purpose, owner, provider decisions, risk tier, or runtime policies where they no longer match. A team administrator edits the owner, purpose, and decisions. Retire the system if it no longer runs.
  5. Mark the system reviewed. A team administrator uses Mark reviewed to record the date. The next review falls due 180 days later.

Batch reviews by owner. An owner with six systems reviews them together, because the same person answers the same questions for each one.

What evidence does an AI system review leave?

A review leaves a dated record on the system and an entry in the WorkOS audit log. Mark reviewed sets the last-reviewed date and the next due date, and the audit log records that the review field changed, when, and who changed it. Changes to the owner, purpose, or provider decisions made during the review are logged the same way, with the fields that changed and the new owner. The purpose text itself isn't written to the log.

On the evidence page, the review.current signal is the share of systems reviewed within 180 days, and it supplies evidence for NIST AI RMF GOVERN 1.5. It shows that systems are reviewed. It doesn't show that your risk management process itself is reviewed, which GOVERN 1.5 also covers and which stays with you.

Export the evidence after each review cycle. Arcjet doesn't recompute the evidence page for a past date, so the export you keep is the record of what was true on the day. For how to export it, see how to export AI governance evidence.

Where Arcjet fits

Arcjet keeps a last-reviewed date and a due date on every AI system, actor, and provider in the AI inventory, and lists the overdue ones. The weekly owner digest tells each owner what's overdue and which providers have violations, and drift notices tell them where behavior moved away from the purpose, so reviews start from evidence rather than from memory. Drift notices and new providers prompt a review before the 180 days are up.

A review in Arcjet confirms the record against what Arcjet observes. It doesn't cover systems outside the inventory, the competence of the owner, or the review of your governance process as a whole.

After reviews, the last how-to in this series is getting the evidence to your auditors. See how to export AI governance evidence, and for the full governance picture, see what is AI governance.

Frequently asked questions

Is 180 days the right review interval for every AI system?

It's a sensible default, not a rule. NIST AI RMF sets no interval, and annual reviews are common in other risk programs. Arcjet uses 180 days for every system. Use the risk tier to decide how deep each review goes rather than how often it happens.

What is the Arcjet owner digest?

A weekly email to each owner that lists violations on their providers and their records that are overdue for review. It sends nothing when nothing needs attention, so an owner who receives one has something to act on.

Does a review have to change something?

No. A review that finds the record still true is a valid result. Marking it reviewed records that someone checked on that date, which is the evidence NIST AI RMF GOVERN 1.5 asks for.

What should happen when a review finds the system no longer runs?

Retire it. Deleting an actor in Arcjet revokes its keys and keeps the record, so the evidence shows when the system was retired and who retired it. That supplies evidence for NIST AI RMF GOVERN 1.7.

Can you prove a review happened at a past date?

Yes, from the WorkOS audit log, which records the change to the review field, when, and who made it. The evidence page shows only the current state, so keep your evidence exports from each review cycle as well.

AI runtime security in your code

Protect your AI agent workflows with Arcjet

Get allow, deny, and redact on agent actions before the side effect.