AI only delivers on its promise if clinicians trust it
At the same time, these expectations exist alongside widespread use of general-purpose AI tools such as ChatGPT and Google Gemini, which operate outside the control of the health system and may themselves include advertising or sponsored content. This creates a fundamental disconnect: patients are comfortable exploring information in less regulated environments but are demanding significantly stricter guardrails when that information is used inside the clinical environment.
As a result, clinicians are increasingly tasked with interpreting and validating inputs generated from tools that do not follow the same standards of transparency or oversight required within the health system. This expands the role of the clinician beyond traditional responsibilities, adding a layer of evaluation and contextualization that requires both time and training.
When digital tools are no longer confined to what’s been deployed within the health system, it means organizations are not just implementing AI. They are responding to it, interpreting it, and, in some cases, correcting it in real-time.
Many organizations are not yet equipped to support this shift. Resource constraints and ongoing staffing challenges continue to impact AI implementation plans, with only around 60% of provider organizations expressing confidence in their ability to train and upskill staff and/or effectively leverage AI in patient care.
Taken together, these dynamics suggest that the success of AI in healthcare will depend on the ability of organizations to define and enforce clear boundaries around their use. Without alignment between how patients engage with AI independently and how clinicians are expected to use it within the system, even well-intentioned implementations risk introducing new friction rather than reducing it.
