AI governance in healthcare: making human oversight work
By ChatGPT Health. Editorial revision with AI assistance, September 28, 2026. Named clinical review: pending. This article offers educational recommendations, not a clinical protocol or legal advice.
Assign responsibility before deployment
A healthcare AI project needs a named owner before it affects patient care. Define the intended use, the decisions a qualified person must review and the evidence required to proceed. Documentation, patient education and clinical decision support carry different consequences when the output is wrong.
Human oversight needs time, relevant information and authority to stop an unsafe action. A required approval click does little if the reviewer cannot inspect the evidence or change the decision.
Define what an AI agent may do
An agent can use connected tools to carry out a sequence of tasks. Its permissions determine whether it can retrieve information, draft a note or change a record. Before enabling those actions, document the permitted steps, the information the agent can access and the point at which a person must approve.
Test the proposed workflow with failure cases. Keep records of the actions taken and provide a usable fallback when the agent or its connected services fail.
Keep consequential decisions with qualified clinicians
Our editorial recommendation is to retain qualified clinical responsibility for resuscitation, prescribing that carries substantial risk, final diagnosis and difficult conversations with patients and families. A team should specify who reviews AI output, what they must check and how they escalate disagreement. These recommendations do not establish a universal legal rule for every product or country.
Review changes to the model and workflow
Models may remain fixed after deployment or change through planned updates. A change to the model, data source or surrounding workflow can affect performance, so the organization should assess each proposed change before release.
In the United States, FDA guidance describes predetermined change control plans for certain AI enabled devices. A plan describes proposed modifications, methods for developing and validating them, and their expected impact. FDA reviews the plan within a marketing submission. This mechanism has a defined scope; it is not permission for unrestricted model changes. See the FDA guidance on predetermined change control plans.
Examine unequal performance and access
Review the population used to develop and evaluate the system. Examine performance across the groups it will serve and investigate missing data or proxies that could distort a decision. Explain what is known about the output and its limitations. An explanation alone does not establish that an output is correct or safe.
The WHO guidance on ethics and governance of AI for health places ethics, human rights and accountability within the design, deployment and use of health AI. Its recommendations provide a starting point for evaluating responsibilities to patients, communities and care teams.
Control access to health information
Document what information enters the system, who can access it and how long it is retained. Evaluate consent, security controls and the possibility of identifying a person from data described as anonymous. Have qualified privacy and security staff assess the actual service and contract.
Use the rules that apply to the service
Requirements differ across jurisdictions and product functions. Identify the applicable rules with qualified local experts, and distinguish guidance from binding obligations. International frameworks can help organize questions, but do not replace that assessment.
Make oversight part of daily work
Train the people who use and review the system. Monitor errors, workload and patient consequences after release, with clear thresholds for investigation or a pause. Keep an owner and next review date for each use.
Use the governance lifecycle, plan a local evaluation or define when a system should be paused.