Trusted medication intelligence to support high-stakes clinical decisions

Effective medication management requires a trustworthy intelligence layer

The role of the LLM should be to interpret, surface, and support the underlying clinical content—not to change, override, or independently generate medication recommendations. Some applications of AI are superior choices for clinical solutions—especially in medication dosing when margins of error are small and patient safety is a priority.

Deterministic clinical logic is rules-based and operates on fixed clinical rules and workflows. This contrasts with probabilistic, or GenAI, which works from predictions and likelihood, creating a dynamic where—if not carefully validated—outputs can vary even with identical prompts or even hallucinate, which may impact patient safety standards.

In medication management, AI solutions need this rules-based logic and to be built on content generated and guided by clinical expertise. A 2025 study found that a generic LLM solution scored only 54% on basic drug interaction checks. A 2024 study from the National Institutes of Health linked adverse drug events to 5% to 28% of acute geriatric medical admissions—some of the most severe consequences of inappropriate medication use for older adults. Globally, the World Health Organization attributes more than half of all medical harm to medications.

Medication management cannot tolerate unnecessary variability—making verified clinical logic, governance, traceability, and expert-curated content critical components of trustworthy clinical AI. Strong governance at the final stage of the workflow is essential to preserve the integrity of the original expert-driven content.

DIY medication data collides with clinician needs

Many organizations are experimenting with internally assembled medication management workflows using commercially available probabilistic tools. This approach fails to infuse clinician insights and expert content into the entire medication workflow—and often lacks the consistency required to help inform high-risk decisions.

This is a problem because medication information is connected to highly-engaged clinical content and workflows that clinicians rely on every day. Over the last five years, clinicians using UpToDate® sought dosing guidance in 32% of safety-related protocols. This comes second only to infection prevention searches, which were 40% of all patient safety-related lookups.

High levels of clinician engagement underscore how central medication decisions are to daily clinical workflows—and why gaps in intelligence layers can introduce outsized risk.

Medication management workflows are becoming more complex

The intersection of AI and medication decision-making is becoming higher risk as drug development expands and high-alert medications are increasingly in play.

Between 2010 and 2019, the number of new drugs approved for sale by the US Food and Drug Administration increased by 60%. From 2023 to 2025, there was a continual flow of drug approvals with 151 novel drug approvals and 61 new biological treatments.

The risk implications are high when it comes to medication management. If AI hallucinates or guesses, the smallest dosing math error can impact patient safety. The most frequently searched high-alert drug topics in UpToDate from 2021-2025 are:

• DOAC dosing – 1.61 M
• Heparin & LMWH – 1.43 M
• Warfarin/VKAs – 773,000
• Vancomycin - 673,000
• Digoxin toxicity/dosing – 567,000
• Aminoglycosides – 430,000

In these scenarios, AI supported by governance and built on expert-generated clinical content is critical to support reliable care decision-making. Grounding AI governance in these standards can also reduce the need for high-risk trial-and-error with solutions not purpose-built for healthcare enterprise needs.

A trustworthy AI intelligence layer starts with expert input

As medication safety gatekeepers, pharmacies and clinicians need reliable AI, with experts-in-the-loop, built on content created by clinicians, and supported by enterprise-level governance that flows to the moment of care.

Traceability builds confidence

Clinical trust is impossible without a source of truth. When a clinician or pharmacist receives an AI-generated recommendation, they should be able to follow it back to the source—clearly answering, "Where did the AI agent go to retrieve this information?” This traceability standard extends to how the system was created—from prompting to verification to governance.

Technology providers should meet the same standard. A health system with an AI governance committee should be able to trust their vendors to have equivalent and aligned internal structures. The committee should be able to review proof of an audit of the entire chain—from data curation to prompt development, through to verification.

Structured inputs enable trustworthiness

AI that touches clinical functions should include an instruction layer within a governed solution—recruiting clinicians as guides in building content, defining structure, and determining governance. For medication management, this means:

• A drug interaction checker
• Allergy checker
• Dosing

These clinical instructions created by human expertise should precede any LLM interaction and help define the response structure. Clinicians should augment engineers as experts-in-the-loop at every level of governance, so AI experiences remain grounded in clinical logic and expert content while augmenting, not replacing, clinical judgment.

Expert-guided medication intelligence is the infrastructure of trust

Across the ecosystem, AI is already embedded in medication decisions—formally and informally.

The risk of AI adoption in high-stakes points of care is greatest when governance is reactive and not formalized. This gap invites unapproved AI applications not designed for the exacting needs of medication management.

Deterministic logic can help strengthen generative AI for medication management

Governing medication intelligence involves intentionally pairing deterministic clinical logic with generative AI capabilities, allowing each to contribute where it is most effective.

Deterministic logic is designed to help improve the chances that a drug database produces a consistent response to the same clinical question every time. Its governing principle is built and tested by experts before deployment. This is a fundamental distinction from models that operate on probabilistic patterns, such as hyper-scaler LLMs. Key characteristics of deterministic clinical logic include:

• Outputs that are verified against expert-curated data.
• An instruction layer built by clinicians and embedded at the integration level.
• Requirements that outputs pass threshold-based testing before use.
• Audit trails that make every output traceable.
• Built-in system controls of what data can be used and how.

GenAI can be useful for summarizing and augmenting outputs, especially when it references authoritative deterministic medication logic. When GenAI is grounded in expert-created medication intelligence and verified clinical logic, it becomes a trustworthy infrastructure and extends governance to the moment of care.

Weaving trusted AI into the connected healthcare ecosystem

With more clinicians turning to AI to augment their work, patient safety goals depend on supporting clinicians with trustworthy tools to help them make informed decisions.

As one of the highest risk applications of clinical AI, medication management must be a governance priority. By leveraging expert-validated content as the foundation of medication intelligence, leadership moves past rudimentary AI adoption—operationalizing trust, consistency, and governance directly within clinical workflows.

Learn more about medication management as the core of clinical decision-making in the UpToDate Point of Care report, “Patient safety in the AI era.”