When Does Healthcare AI Need a Patient Conversation?

Health services are deploying AI in places patients may barely notice: drafting clinical notes, sorting messages, supporting image interpretation, identifying people for follow-up and shaping the information that reaches a clinician.

That does not mean every use demands a lengthy consent process. But neither does it make a notice buried on a website an adequate answer whenever an algorithm influences someone’s experience of care.

A qualitative study published this week in *PLOS Digital Health* offers a more useful way through the false choice between blanket consent and near-invisibility. Its central finding is that patients want transparency and agency calibrated to the role AI plays in their care. Low-impact uses may need clear disclosure. Tools that affect communication, records or care pathways may warrant a more direct conversation. Where AI has material consequences for treatment, consent and a genuine ability to refuse become more important.

The study is small and not a measure of public opinion: it draws on interviews with 20 members of patient and family advisory councils in one US health network. Its participants were also more educated and less diverse than a representative population. Yet the practical distinction it makes is valuable. It turns “patient trust” from a vague adoption objective into a service-design question.

Patients do not need a lecture on every algorithm; they need to know when technology changes their care, choices or information.

A notice is not the same as meaningful information

The study proposes three levels of communication: disclosure, assent and informed consent. The labels matter less than the operating principle.

Disclosure fits routine, low-risk uses where a patient is unlikely to experience a meaningful change in their choices or care. A health service might explain, in plain language, that AI is used to help prioritise messages or assist clinicians with administrative work, while making clear who remains responsible.

Assent is more appropriate when a tool touches the patient directly or changes the texture of the encounter. Consider an ambient scribe recording a consultation to draft the clinical note; a system generating a patient message; or an algorithm shaping the order in which appointments are offered. Here, a timely explanation and an opportunity to raise concerns are part of the service itself. A generic privacy policy does not meet that need.

Informed consent belongs with high-stakes or more autonomous applications, particularly where a person’s treatment choices, access to services or exposure of sensitive information could be materially affected. The patient should understand the role of the system, the realistic alternatives, the clinician’s role and what happens if they decline.

This is not a call for consent theatre. Asking patients to approve a tool when there is no practical alternative, or when refusing would quietly result in slower or worse care, creates the appearance of control without its substance. Good communication says plainly where choice exists, where it does not, and why.

The real unit of design is the care moment

Healthcare organisations often treat AI information as a policy or communications task: publish an FAQ, add a privacy notice and prepare staff with a script. Those are necessary foundations, but they can miss the moment a patient actually needs an answer.

A person receiving a text reminder does not need the same information as someone whose consultation is being recorded, or a patient reading an AI-assisted explanation of test results before they can speak to a clinician. The relevant question is not simply, “Does this service use AI?” It is: what is the patient trying to do when this tool becomes visible?

For each use case, teams should map four practical questions:

• What changes for the patient: speed, access, the information they receive, the person they speak to, or the route into care?

• What data is being used, created or retained?

• Who can correct an error or challenge an output?

• Is there a clear human route when the patient needs explanation, reassurance or a different outcome?

That map should be tested with patients who are likely to encounter the service under pressure, not only with digitally confident panel members. People managing several conditions, those using interpreters, carers, people with low confidence online and those who have previously struggled to be heard are especially likely to identify friction that a polished demonstration conceals.

The question is commercial as well as clinical. A patient-facing tool that creates uncertainty can increase avoidable contacts, complaints and abandonment. One that makes the next step clearer may reduce effort for patients and staff alike. Adoption is shaped by whether the tool fits the everyday work of being a patient: understanding a letter, preparing for a consultation, reporting a concern or deciding what to do next.

Clinician oversight must be visible, not assumed

Patients in the new study were not rejecting AI outright. They were more receptive when it strengthened the capacity of a clinician rather than displaced clinical judgement. That distinction should be made legible.

NHS guidance already states that AI can support a health professional’s decision-making and that professionals provide advice and allow patients to make final decisions about their care and treatment. The challenge is translating that principle into language patients encounter at the point of use.

“Your clinician reviews this before it is sent” is more useful than “AI-enabled communications”. “This tool helps us identify who may benefit from a review; it does not diagnose you” is more useful than a technical description of a predictive model. These are not cosmetic changes. They define expectations, help patients recognise when something appears wrong and protect the clinician-patient relationship from avoidable ambiguity.

Teams should also be wary of assuming that human oversight automatically solves every concern. A clinician who receives a flood of AI-generated prompts, summaries or recommendations may have limited capacity to interrogate them. The patient experience of oversight depends on whether staff can explain the tool’s role, identify an error and act on the concern without adding a new administrative loop.

Patient involvement should shape the proposition earlier

The MHRA’s new Patient and Public Involvement and Engagement Plan, published on 5 November, reinforces a wider shift: lived experience should inform how medical technologies, including AI, are assessed and explained. For product teams and providers, the useful implication is not simply to recruit a patient representative near launch.

Bring patients into the work while the service proposition is still flexible. Test the language used to explain the tool. Ask what evidence of benefit would feel meaningful. Identify what people believe they should be able to see, correct or decline. Then return after launch to examine whether the service has delivered on those expectations.

The strongest health AI propositions will not promise an abstract blend of innovation and trust. They will show patients where AI is helping, where human judgement remains, and how people can retain a meaningful role in decisions about their own care.