Experts advise healthcare organizations to ask: “Who will be held accountable for decisions influenced by artificial intelligence – the doctor, the department, the provider, or the hospital? And what will be required to defend AI-influenced decisions?”
Two of the leading companies in the generative AI race, Anthropic and OpenAI, gained prominence this month with impactful announcements of their respective offerings aimed at the healthcare sector, reports HealthcareITNews.
The adoption of new AI tools dedicated to healthcare by healthcare systems and integrated with electronic health records and other healthcare IT tools, providers – and patients – has increasing associated risks.
AI may solve some of healthcare’s biggest challenges, but experts say there are also many reasons for concern.
For example, the potential for AI to provide inaccurate medical information – the infamous “hallucinations” – combined with the false sense of diagnostic certainty of some models, remains a major challenge.
Furthermore, there is often a lack of clinical accountability for these tools. This creates considerable legal and ethical gaps regarding who is held responsible when an algorithm-influenced recommendation results in patient harm.
OpenAI and Anthropic have announced a dedicated AI aimed at improving the use of medical records by healthcare organizations, including physicians and hospitals.
According to OpenAI, the launch of the product suite for Health allows various healthcare systems to utilize ChatGPT for Health to support patient care and reduce administrative burden.
OpenAI gave several examples of organizations whose clinical teams can use ChatGPT for Health to integrate medical evidence into their patient care work, including healthcare systems such as AdventHealth, Boston Children’s, Cedars-Sinai, HCA, Memorial Sloan Kettering, and Stanford Medicine.
Elation Health, which offers an electronic health records platform, announced this week the integration of Anthropic’s Claude AI into its clinical insights program. Doctors use the tool to create instant summaries of complete patient medical records.
According to Elation Health, healthcare professionals using the software integrated into the Claude AI have been getting answers to their questions 61% faster.
Michael Abrams, of Numerof & Associates, a consulting firm specializing in the Life Science sector, states that AI solutions are aligned with the Trump Administration’s action plans to leverage technology and empower patients to make more informed decisions about their health.
In this sense, patients can upload their medical records directly to a separate product called ChatGPT Health or synchronize data from their wellness apps and request insights from the AI. They can also connect with Claude as subscribers to the HealthEx app and ask questions about their medical history across multiple providers.
“First, we need to acknowledge the math: there simply aren’t enough doctors and healthcare professionals to meet global demand,” said Adam de la Zerda, CEO and founder of Visby Medical, a medical diagnostics company.
“Access to healthcare is a crisis, and AI is the only tool capable of expanding that expertise, so this is a fantastic step in the right direction,” added de la Zerda.
On the other hand, there are also plans to use language models on a large scale “to automate registration and form completion, develop process improvement plans, improve operational efficiency, and much more,” said Abrams.
Michael Abrams emphasizes, however, that “responsible boards and executives must recognize the risks associated with these new tools and take steps to protect the institution and the staff involved.”
The use of LLMs in healthcare requires industry leaders to be realistic and disciplined regarding governance and accountability.
“While data privacy is paramount, the real governance challenge is the decoupling between certainty and responsibility,” said de la Zerda.
Chase Feiger, CEO and co-founder of the AI company Ostro, warns that “Success will not come solely from the quality of the model,” clarifying that “It comes from the ability of organizations to adopt real discipline regarding governance, accountability, and the practical functioning of medicine.”
“Everyone involved understands that the greatest interest in these tools will be for diagnosis, by consumers, patients, and doctors,” said Abrams. “They also need to understand that, for diagnostic purposes, these tools, despite the tests performed, represent risks for use.”
“For the providers and the institutions to which they belong.”
A more critical concern may be that a machine learning model offers answers with a tone of diagnostic certainty, “but without any clinical accountability,” noted de la Zerda. “This creates a dangerous gap, where a patient may rely on an AI summary with an authoritative tone, but lacking professional nuance,” he said.
According to la Zerda, “Governance must ensure that we do not replace human judgment with a confident, but ultimately irresponsible, algorithm.”
Ali Diab, CEO and co-founder of the health benefits platform Collective Health, states that “Natural language processing capabilities allow AI to go beyond simply sharing information.” For Diab, “This raises new and interesting questions about who is responsible if that analysis or recommendation is wrong.”
What to do first
There is a set of issues that must be considered to begin with. It is necessary to recognize the significant challenges of the current system before demanding more from AI; it is necessary to create procedures to deal with situations that… Errors will likely occur, such as when the AI tool issues a demonstrably wrong diagnosis. Furthermore, it is essential to define who will be held accountable for decisions influenced by the tool – the doctor, the department, the provider, or the hospital. Similarly, it is crucial to define what will be required to defend AI-influenced decisions in medical malpractice or peer review processes, and how to ensure compliance with these requirements, but also to define if there are areas where the use of the tools should be restricted until their capabilities and accuracy are better understood.
Machine learning tools (LLMs), such as GPT and Claude, should be used before and after care, “not autonomously,” argues the CEO of OSTRO. “The important thing is that OpenAI is finally treating healthcare as a category with different rules, less tolerance for errors, and real consequences when something goes wrong,” he adds.
When an AI solution influences a clinical action, it is not the same as when it is incorporated into workflows.
Feiger expresses concern not about the model itself, “but about the ease with which tools like this can generate overconfidence,” concluding that “They seem confident, even when they are wrong, and in medicine, that’s dangerous if the limits aren’t explicit,” he warns, adding that “responsibility cannot be vague” and that “consumer use and business use cannot be confused.”
Diab cites China as an instructive example of the risks inherent in the use of long-term learning models (LLMs) in healthcare by consumers. These risks can be managed “with the appropriate level of disclosure and, potentially, regulation.” It is at this point that the Chinese AI regulatory framework classifies some AI applications as Class III medical devices, which require an approval process equivalent to that of the US FDA (Food and Drug Administration).
“If consumers start relying too heavily on these AI-based tools for self-diagnosis, and if this leads consumers to potentially make deductions or decisions about their health that turn out to be correct, causing harm to themselves or a loved one, I think we will face a reckoning with these tools that the enthusiasm surrounding them may now…” “It’s becoming obscured,” Diab concluded.
