What HIPAA Requires for AI Systems in Clinical Settings

The Security Rule, at 45 CFR Part 164 Subpart C, breaks down into three safeguard categories, and an AI system has to satisfy all three the same way a billing system or an EHR does. Administrative safeguards cover risk analysis and workforce training. Physical safeguards cover facility and device access. Technical safeguards cover access control, audit controls, integrity controls, and transmission security, and that last category is where most AI deployments actually fail, because nobody mapped where PHI flows once it hits a model API.

The Privacy Rule's minimum-necessary standard, at 45 CFR Part 164 Subpart E, does not get suspended because a model is doing the reading instead of a person. If an AI agent pulls a full chart to answer one narrow clinical question, that is the same over-disclosure problem as a person pulling more records than the task requires, and it needs the same access controls and use limitations a manual process would need.

The Breach Notification Rule, at 45 CFR Part 164 Subpart D, does not care whether unsecured PHI ended up in the wrong place because of a person or a process. PHI captured in a vendor's model logs, retained in a training dataset, or sent to a subprocessor with no data-handling terms covering it is a security incident under the same rule that governs a lost laptop.

The piece most healthcare organizations miss: an AI vendor that creates, receives, maintains, or transmits PHI on your behalf is a business associate under 45 CFR 160.103, whether the vendor's own marketing calls itself a processor, a platform, or an AI assistant. No signed business associate agreement naming that specific use case means no HIPAA-aligned basis for the AI system to touch PHI at all, regardless of how good its technical controls are.

The audit-controls standard at 164.312(b) does not distinguish between a decision a person made and one an algorithm made. The covered entity is accountable either way, so the audit trail has to exist at the same granularity you would want for a clinician's chart note, not a summary log of API calls.

Agentic systems make this harder than a single-prediction model, because one user request can trigger a chain of tool calls, data lookups, and intermediate reasoning steps before a final output appears. A defensible audit trail captures each step in that chain, not just the final answer, because an examiner, or a malpractice attorney, will ask what the agent looked at along the way, not only what it concluded.

The control that actually protects a hospital is the human review checkpoint: a documented record of who reviewed the agent's output before it reached a patient or a chart, and what they based their acceptance or override on. Without that record, the AI's output effectively became the clinical decision with no accountable human in the loop, which is the exact scenario a risk assessment is built to catch.

HIPAA's text does not use the word explainability. In practice, examiners and internal compliance teams increasingly reach for the NIST AI Risk Management Framework's Manage function to fill that gap, treating ongoing monitoring, incident response, and human-oversight documentation as the operational evidence HIPAA implies but never spells out for AI specifically.

A system can satisfy every Security Rule technical safeguard, have a signed BAA, and pass a HIPAA audit, while still producing an unreliable clinical recommendation. HIPAA was written to protect data, not to validate model performance, so compliance with it is necessary and not remotely sufficient for clinical AI safety on its own.

For AI embedded in a regulated device or diagnostic workflow, FDA software oversight and 21 CFR Part 11's electronic records and signatures requirements can layer on top of HIPAA, particularly in life sciences and laboratory settings where GAMP 5 governs the validation methodology itself.

The NIST AI Risk Management Framework is not a legal requirement, but it is the framework most compliance teams and examiners reach for when a question falls into the space HIPAA's text is silent on, like drift monitoring, model change control, or ongoing human oversight of an agentic system. Treat it as the operational layer that answers the how, where HIPAA only answers the what.