How to adopt AI responsibly in your clinical practice

In 2018, Elaine Herzberg became the first recorded case of a pedestrian killed by a self-driving autonomous car while pushing her bicycle across a road in Tempe USA. But who was responsible for the accident? Herzberg herself? The ‘safety driver’? The car? Uber? The safety managers at Uber? Engineers at Uber? Arizona government officials who allowed Uber to test their cars in Tempe? All of them? None of them?1 In 2023 the safety driver who was allocated to the car pleaded guilty to a reduced charge of endangerment and was sentenced three years of supervised probation.

AI applications are becoming increasingly widespread in healthcare. As an administrative tool, AI scribe technology is supporting clinicians to keep accurate patient records in real time, and AI-driven x-ray and screening tools are supporting early diagnoses. The technology has the potential to be transformative to patient outcomes and almost every day in the news around the world we are hearing how AI is detecting diseases like cancer earlier from scans, predicting patient deterioration in hospitals, accelerating the discovery of new drugs, and enabling more personalised treatment plans for patients. However, as with any technology, it is not infallible. 

Who is responsible when AI goes wrong? 

Who is liable in AI-related healthcare malpractice claims is still murky territory. Is it the AI software developer for defective design, or false representation of accuracy or inadequate warnings could face a product liability claim? The clinician or hospital could be found liable for incorrect implementation, training or oversight of AI systems? Third-party vendors which have products integrated with the original AI system? Furthermore, courts are also not clear on how to view AI - as a medical device, a service or simply as a clinical tool?

What recent cases and studies tell us about AI risk 

It is interesting to look at how some AI-related legal cases and studies in the healthcare arena have played out in other jurisdictions, for example: 

• In September 2024, Pieces technology sold a clinician-assisted AI product which worked within a USA hospital’s electronic healthcare record programme to generate progress notes, discharge summaries and multidisciplinary care plans with the aim of "returning time to medicine" by reducing the administrative and documentation burden of doctors, nurses, and hospital administrative staff. They claimed the AI system hallucination rate (the rate by which AI systems generate false or misleading information and present it as factual) was less than 0.001%. The Texas State Regulators found this to be an underestimation and proceeded with a successful enforcement action in court.

• In an external validation study, the Epic Sepsis Model, which was integrated into patient electronic healthcare records in hundreds of hospitals in the USA to predict the early onset of sepsis, was able to assign higher risk scores to patients who developed sepsis only about 63% of the time, and only identified 7% of patients whose sepsis diagnosis was missed by a clinician. Meanwhile, the company claimed an accuracy rate of 76-83% 

• A systematic review of 86 studies evaluating commercial radiology AI algorithms found that 81% of the algorithms showed decreased performance during external validation, with 24% showing substantial declines.

• A review into AI bias from the British Journal of Radiology found various areas in the development of clinical AI tools which may be prone to bias. It also found that from data collection to data preparation and annotation, and from model development and deployment to evaluation, the earlier the bias is introduced into the data, or the more types of bias are involved, the worse the final model will perform with external data validation. 

Understanding the core risks of AI in clinical practice 

When assessing the potential legal risks of AI and how they may impact a clinician using AI, there are many important factors to consider. A key consideration, ‘The black box’ problem, results where deep learning models are unexplainable as they are either hidden or incomprehensible to humans (making them unverifiable and open to significant bias).

The ‘black box’ problem and bias in AI systems 

Additionally, the developmental data set used to programme the AI tool may not include all relevant patient populations or characteristics, creating an inherent and perpetuating bias specifically towards minority, under-resourced or disadvantaged groups. Both themes leave the clinician exposed to significant risk. Furthermore, HCPs should be aware of aforementioned issues like errors and hallucinations, and the uncertainty around liability. Alongside this, they must be wary of their under or over-reliance on the technology and how use of AI may erode their skills over time.

Steps to consider to help reduce medicolegal risk 

• When choosing an AI product, ensure that it is appropriate for the specific task you are using it for and check for external validation tests of the product. 

• Treat AI as a decision-support tool, not as a replacement for your clinical judgement, in line with HPSCA guidance. Always consider how the AI correlates with your clinical findings and question generated results in the same way that you would any investigation result. 

• Where AI systems are used within an organisation, ensure that appropriate initial and ongoing training is provided for all users. 

•  Review whether metrics such as false positives/negatives and hallucination rates are available and double check or audit some of the results. Alternatively, ensure that the organisation is doing so and you are kept up to date with any updates, changes or concerns raised.

Consent, transparency and patient choice 

• Importantly, your patients must be informed if AI was used in their management and give their voluntary consent which should be recorded. To obtain informed consent also involves clinicians being able to explain to patients how the AI reaches its outputs and how data is stored. 

• Keep in mind that a patient has the right to refuse AI-assisted care without the refusal impacting their care.

Tools and resources from Medical Protection 

AI represents an incredible opportunity to advance medical care and make significant leaps in improving patient care. At Medical Protection, we want to help ensure practitioners and their patients reap the potential benefits of AI and we provide advice and support which helps practitioners mitigate any emergent medicolegal risks.

We have developed the AI Safer Practice Framework to help practitioners to safely and responsibly integrate AI into their practice. The Framework is divided into two parts - INFORMED and RECORDS - and is structured around these acronyms to ensure it is practical and memorable.

Medical Protection members can also access the ‘Managing the risks of AI in medicine’ course on The Learning Hub, as a benefit of membership. The course introduces the AI Safer Practice Framework to help healthcare practitioners integrate AI into practice safely, ethically, and effectively. The interactive course can be accessed any time: https://prism.medicalprotection.org/course/view.php?id=1834