Governed AI Agents
Can AI Agents Be Used Safely in Regulated Industries?
Safe use is a design decision, not a product feature. The controls are known, the regulators are specific about the principles, and the work is mapping them to your workflows. Kriv AI helps regulated teams do that.
Yes, AI agents can be used safely in regulated industries, but only with controls around what they are allowed to do. That means least-privilege tool access, human approval before high-impact actions, logging of every agent step, and ongoing monitoring, mapped to the rules of your sector. Without those controls, an agent's reach exceeds its oversight.
context
What Makes an Agent Different From a Model
A model produces text. An agent acts. It can call tools, read and write to other systems, and chain its own outputs into the next step. In a regulated setting that changes the question from whether the answer is accurate to what this system can do, and who is accountable when it does it.
Where the risk actually sits
OWASP describes Excessive Agency as "the vulnerability that enables damaging actions to be performed in response to unexpected, ambiguous or manipulated outputs from an LLM, regardless of what is causing the LLM to malfunction." It names the root cause as "excessive functionality; excessive permissions; excessive autonomy."
That framing is useful for regulated teams because it moves attention from the model to the design around it. A model that sometimes errs is manageable if the agent around it can only do a small, reviewed set of things.
What the controls look like
OWASP's mitigation list for agent systems is concrete: limit the tools an agent can call to the minimum necessary, give those tools only the permissions they need, and run actions in the context of the specific user. It also says to "utilise human-in-the-loop control to require a human to approve high-impact actions before they are taken," and to "implement authorization in downstream systems rather than relying on an LLM to decide if an action is allowed or not."
OWASP adds that logging and monitoring can limit the level of damage caused, even though they do not prevent Excessive Agency on their own. In a regulated business, that second category is what auditors ask to see.
capabilities
How Regulators Frame It
Banking: model risk management
The Federal Reserve's SR 26-2, dated April 17, 2026, is titled "Revised Guidance on Model Risk Management" and "supersedes and replaces SR letter 11-7." It says the agencies updated the guidance "to emphasize a risk-based approach to model risk management that is tailored to a banking organization's model risk profile and the size and complexity of its operations." For an agent, that means the depth of validation and oversight should scale with what the agent can affect.
Life sciences: credibility for the context of use
FDA's January 2025 draft guidance, "Considerations for the Use of Artificial Intelligence To Support Regulatory Decision-Making for Drug and Biological Products," is labeled a draft that "Contains non-binding recommendations" and is "Not for implementation." It is a signal of direction, not a final rule. Our page on the AI model credibility framework for FDA drug submissions covers how to read it.
Cross-sector: the NIST AI Risk Management Framework
NIST describes the AI RMF as "intended for voluntary use" to improve "the ability to incorporate trustworthiness considerations into the design, development, use, and evaluation of AI products, services, and systems." NIST also released a Generative AI Profile, NIST-AI-600-1, on July 26, 2024. Many regulated firms use the framework as a common vocabulary for governance even where no regulator requires it.
Healthcare and insurance
Health and insurance rules, including HIPAA and state insurance regulators, each add their own obligations. We do not restate them here. See our pages on HIPAA requirements for AI systems and NAIC AI model governance for the detail, and confirm obligations with counsel.
ai gap
A Practical Safety Checklist for an Agent
Treat the following as the minimum design review before an agent touches a regulated workflow. First, list every tool and permission the agent has and remove any it does not need. Second, identify the actions that are high-impact or hard to reverse and put a named human approval step in front of each. Third, make sure authorization is enforced by the downstream system, not by the agent's own judgment.
Fourth, log each step the agent takes, what it was given, and what it did, in a form a reviewer can follow later. Fifth, set up monitoring and rate limits so a misbehaving agent is noticed before it does significant damage. Sixth, assess any vendor behind the agent before deployment, covering where data goes and what the vendor can change without telling you.
None of this is exotic. The difficulty is applying it consistently across many agents and teams, which is a governance problem more than a technical one.
differentiation
Where This Page Fits Among Our Other Pages
This page answers the broad question of whether agents can be used safely. For the security threat view, see our page on AI agent security risk in the enterprise. For why deployments fail reviews, see why AI agent deployments fail governance audits. For decision records in a clinical setting, see our guide to auditing an AI agent's decisions, and for the banking angle, see AI model risk management for banks.
engagement
How an Engagement Works
A scoped engagement usually starts by inventorying the agents and automations in use or planned, and the systems and data each can reach. We then design the permission boundaries, approval points, audit logging, and monitoring, test them against your own policies and the sector guidance above, and hand over controls your compliance and technology teams can run. We do not give legal advice and do not resell any vendor's product.
tiers
What You Get at Each Tier
1. Enterprise / regulated (banks, insurers, health systems, life sciences)
Agent inventory, permission and approval design, audit logging, monitoring, and a vendor risk review for the workflows in scope.
2. Fractional CTO / AI governance lead
Ongoing ownership of the agent governance program as new agents are proposed, and review of each new use case against it.
3. Specialized advisory
A single session or second opinion on whether a specific agent design has adequate controls before it goes live.
rate card
Kriv AI's Rates for This Work
These are Kriv AI's own published rate floors, not an industry average.
| Track | Kriv hourly rate | Typical engagement model | Minimum engagement |
|---|---|---|---|
| Enterprise / regulated (banks, broker-dealers, payment processors) | From $200/hr | Fixed-scope project or retainer | $8,000 |
| Fractional CTO / AI governance lead | $300 to $400/hr | Part-time, ongoing (monthly) | $8,000 |
| Specialized advisory (vendor evaluation, second opinion) | $400 to $700/hr | Hourly, per-session | Varies by engagement |
| Small business | $150/hr | Referred to Kriv AI's partner network | n/a |
get a quote
How to Get a Real Quote
The rates above are floors, not a quote. Actual price depends on which workflows the agent touches, what systems and data it can reach, and how much logging and human review already exists. Book a discovery call and we will scope it honestly.
Sources
Cited sources
- OWASP Gen AI Security Project, LLM06:2025 Excessive Agency
- Federal Reserve, SR 26-2: Revised Guidance on Model Risk Management
- NIST, AI Risk Management Framework
- FDA, Considerations for the Use of Artificial Intelligence To Support Regulatory Decision-Making for Drug and Biological Products (draft guidance)
Related resources
Continue exploring
Straight answers
Frequently asked questions about Can AI Agents Be Used Safely in Regulated Industries?
Can AI agents be used safely in regulated industries?
Yes, with controls around what the agent can do: least-privilege tool access, human approval for high-impact actions, step-level logging, and monitoring, mapped to your sector's rules.
What is the biggest risk with AI agents?
OWASP names excessive functionality, excessive permissions, and excessive autonomy as the root causes of damaging agent actions.
Do regulators ban AI agents?
Not as a category. Guidance such as the Federal Reserve's SR 26-2 emphasizes a risk-based approach tailored to the institution's risk profile and complexity.
Is the NIST AI RMF mandatory?
No. NIST describes it as intended for voluntary use.
Does a human have to approve every agent action?
Not every action. OWASP's guidance is to require human approval for high-impact actions, and to enforce authorization in downstream systems.
What does this cost with Kriv AI?
Kriv AI's rates start at a $200/hr floor for enterprise and regulated work, with an $8,000 minimum engagement. Specialized advisory runs $400 to $700/hr.
Talk to the team that would do the work
Bring your requirements to a working session with the person who'll actually deliver.
Book a Discovery Call