Safe and transparent AI in highly regulated environments

First and foremost: we don't compromise on trust! We have 189 years of history going back to JB Lippincott within our nursing business. And our customers trust us to be the gold standard with evidence based tools and support. We bring technology innovation to the market with a focus on safety and efficacy.

Things obviously have been changing very fast, particularly with the introduction of GenAI. I've been doing this for a long time and AI presents a lot of unique challenges. You have to develop this intuition. You have to be able to adapt and pivot quickly. We're talking about non determinism... It's not obvious that you have a clear path to success – that you can get a good problem solution fit.

In the clinical setting, you're building AI to support professionals and that’s really demanding. You're looking at entirely new methods of evaluating clinical relevance and whether or not the solution is going to add the right value. It's demanding in new ways.

When you think about some of the changes that have been occurring with GenAI, I think about the overall day in a life. So whereas we used to be heavily focused on architectural design, UX and wireframes, that's now evolving.

Dealing with agentic AI, we want to own the outcome. We want to deliver the outcome, not the little breadcrumb trails, the transactional moments, all the clicking, those interfaces that tend to be challenging to use.

And that traditional focus on containers, microservices, API's, I'll be honest with you, we're talking a lot more about P value, confidence interval and sample size. That's a huge shift. We're moving towards experimental with rigorous validation.

And of course, traditional IT still plays a really important role. But this is a big change for technologists. They just haven't had this type of experience in the past. Some data scientists may be familiar with some of these things, but in particular in the healthcare setting that's where we're really fortunate to have access to on staff, to licensed physicians. These are folks that are involved in care delivery and they understand evidence based practice.

They understand this is based on research and experimentation. So we developed a strategy that we call Expert AI and it's really AI solutions by experts, for experts. This is a focus on expert in the loop at every stage of design, validation and testing.

It's rigorous, it's ensuring safety, efficacy and clinical relevance. And that has really helped us align, bringing together technologists and what they do best, and clinicians and what they do best.

And that has really been a key part of the long term guiding principles that are shaping our current early work and early focus around GenAI and agentic and these new technologies. Certainly it's going to be part of our long term vision. So Expert AI: by experts, for experts…

The experts really help us break down complex problems. ‘Owning the outcome’ means you need to really understand, what the intent is. What is it you are trying to achieve. What the purpose is. What the risks are.

We feel like it doesn't make a lot of sense to have AI engineers develop complex solutions in the clinical setting for which they really know nothing about. And then to rely on simple benchmarks and statistical proxies as their guiding principles…

That really doesn’t reflect the complexities of the real world. That expert in the loop, establishes the guardrails, the firm foundation and makes sure that – as we innovate and really speed up those innovation cycles – that we're doing it in a safe and trustworthy manner and that we can provide that explainability and transparency along the way.

We have technologists going through this transformations this journey. They are shifting from all these details around building enterprise technology, cloud APIs, microservices, and getting a more of an experimental understanding of how we add value.

Going back to basic statistical principles and these are not familiar themes. Experimental design, measurement error, uncertainty, risk of bias... These are not typically the types of themes that you would expect a technologist to command and that extends all the way up to the executive stakeholders that are learning and are beginning understand that the path to AI value is just not linear.

It's not structured in the way that your traditional enterprise SaaS offerings have progressed historically, with specific planning, basically breaking down features and using your agile SAFe. Those are good, solid techniques… we tend to decouple from the we call the Agile Release training.

We're doing really experimental work and in order to refine that right value – to make sure that we do it safely and effectively – it requires new forms of governance and as well as an increased focus on go to market. There are new challenges out there.

First of all, when we look at AI, and our experience over the past couple of years with GenAI is: it's really not as effective as a bolt on solution. There is the entire ecosystem to consider. And that tends to be true across industries and it takes time to develop that right value. The impact in the environments you're deploying it into, needs to be considered.

Also, there may be new training requirements for people using solutions and you may be involved in piloting and testing and increased governance. This may be very different from the way that you were building your traditional SaaS products in the past.

Most importantly: AI is tying together stakeholders more closely. Your technologists, your product experts, your going to market people, our expert clinicians, our customers… are working a lot more closely. That takes time and you can't just release AI into the wild.

And I think we're learning that as we go. We are maturing. You're not only experimenting to see if you have a good problem solution set using the core technology. We've got great success at doing that so far. You're also experimenting to get that right product market set, the commercials, the adoption, the willingness to pay, the overall governance..

With AI, a particular GenAI and agentic, it's all still very new. We're figuring that out as we go. And this is the part that I really like to emphasize: as a CTO you may find you feel more like the chief explanation officer; explaining what happened with these experiments and our assumptions. How do we pivot? What are we going to do here?

Are these investments going to bear fruit? Do we have the right strategy? And I've been doing this for 25 years and this is the biggest change I've seen and I like to relate it. Having a PhD helps a lot when you're the chief explanation officer. If you find yourself in an executive meeting that feels like a thesis defense, that's probably a good sign that things are working and moving past the early challenges of AI adoption and really get into that real value.

A few months ago, we released our UpToDate Expert AI and that is: Expert AI is built on decades long commitment to curating the gold standard and evidence based clinical decision support that's used by approximately three million clinicians around the world.

We continue that tradition by grounding GenAI and our trusted and verified content, having experts in the loop. This prevents hallucinations but it's also designing the solution, making sure that it is explainable and transparent.

So that's providing on demand guidance and recommendations at the point of care, safely and effectively. It's saving clinicians valuable time and it's allowing them to focus on patient care – another area that we're finding really good value.

Three years ago, we were talking about AR, VR and the metaverse... This is ‘pre-GenAI’ and we were maybe a little early on that technology cycle. We actually found really good value and use

of technologies like this in our nursing products where we could use virtual simulation and learning solutions.

It's a great way to remove risk and increase access to authentic learning environments and help create clinical competence, where it's difficult to have clinical practice. So these simulations, these new technology solutions, are really valuable there.

And now we're using GenAI. We're looking at ‘focus and adaptive learning’, remediation intervention, so that the students on their learning journey are getting improvements all the time. That is another great area, a great use case for AI, GenAI and ultimately, agentic AI.

The virtual twin! I'm really, really excited about this concept where you’re programming computer simulations with your genetic profile, your physiology… It's combined with continuous monitoring as well as your specific healthcare history.

And then you're introducing advanced modeling and simulation to improve diagnosis and treatment planning, as well as focusing on things like preventative medicine, predicting disease and offering more choice, early in disease progression.

Now, I have a personal connection… in my PhD and postdoc many years ago, I focused a lot of what we call ‘in silico’ simulation of tumor biology. So you've got ‘in vivo’ and in vitro. So this was ‘in silico’: all done on a computer! and it was all done using the earliest versions of GPUs.

At that time, the GPUs were small so we could simulate a small tissue compartment with a single tumor. We could simulate a lot of detail – everything from blood flow to drug delivery. That was really interesting.

Obviously today's GPUs are orders of magnitude more powerful so much more complex biological systems can be modeled. And this is really exciting in my estimation and in fact very doable. And I think we'll see this begin to emerge at least over the next five years.

Not every use case has the same level of risk and so we have to look at that very specifically in a rigorous way. Most importantly: we don't aim to replace clinical reasoning and judgment. But we have to assess algorithms on their ability to follow expert instructions; our expert in the loop. And to identify all the relevant clinical contexts and deliver what we call clinical intelligence.

It's not replacing clinical judgment or reasoning, but we're making sure that our algorithms can think and act like a clinician and not replace them. That helps make sure that we can provide explainability and transparency which is an absolute must around the ethical and responsible use of AI. We have to make sure it's safe, most importantly.

And then we also perform a series of intense efforts with our licensed physicians on staff and this is really to assess whether a solution has the potential for bias or the potential to cause patient harm. And we test and validate everything thoroughly.

And here's another really important criterium that really goes to the heart of governance and ethical and responsible AI: don't just turn on AI features in the wild! We don't just launch a product feature in an existing product and say: here's a button, try this!

We have to work with our with our customers and our partners and make sure that we roll out solutions in a controlled way. So we'll take additional clinical feedback from the front line and make any additional adjustments until the solution is trusted and demonstrated to be safe.

If you go back to the early introduction of LLMS and understanding that these are ubiquitous models and that they can do a lot of things. These generic, large language models began to become embedded in our everyday life, including workplace settings, including healthcare settings...

The problem is: on their own they aren't very explainable and transparent. They add value, yes. They can be powerful – but they are trained on the web and all of its darkest corners, so that obviously poses some fundamental risk.

We've developed many ways to rehabilitate your generic LLM, things like grounding, preventing hallucinations, making them more explainable… I would expect at least some form of guidance around how generic LLMs should be used in clinical settings. In fact you're already seeing generic LLMs implementing more restrictions, guard rails and warnings, which I think is positive.

When you look at where the technology is evolving, I think there's going to be a distinction between your “agentic AI workflows with the capacity to make changes automatically” versus AI that is designed to have the oversight of a human in the loop at each step that is verifying…

The regulatory authorities and the FDA may focus more on agentic AI, designed to act independently. In all cases, increased explainability and transparency will be operative. Things like citing any sources used, identifying potential risks and recommending mitigation should be the focus basically for all the AI solutions.

We feel trust is key and establishing trust is a shared responsibility across all stakeholders, in the ecosystem of clinicians, hospitals, care settings, vendors and service providers, as well as regulatory authorities... We all need to work together here!

This trust in the technology, this confidence in technology, it could be a potential failure mode. LLMs are designed to answer questions and to respond with what seems like increasing levels of confidence.

We always stand there is a risk of hallucination, that's been around for a long time, and that's certainly a risk. And we're doing a lot of things to mitigate that risk.

But that the purported confidence, the blazing speed at which these LLMs can answer questions, including clinical questions, it creates a powerful psychological effect. And another risk, potentially even bigger risk, may be this notion of de-skilling, or over reliance, on this new technology.

Especially in a clinical setting, especially without the right clinical judgment at those crucial clinical moments when it matters most. The machine seems to know what's going on here, and so you move forward… This could become a widespread failure mode as more of these tools become available. At Wolters Kluwer we talk a lot about how to create AI solutions that don't fatigue the user, don't encourage them to let their guards down.

Instead, focus on determining the right clinical context when nuances evolve. Make sure you don't overburden the clinician with too much information but encourage engagement and support the clinical reasoning and judgment of the clinician at each step.

The former incarnation of this is ‘ghost IT’. This isn't a new concept, but it's maybe a little bit more nuanced and subtlety within the directions of LLMs. I don't think everybody's completely aware of some of the potential risks. First and foremost, you have to revisit your standard IT governance: ghost IT and shadow IT, this stuff has been around for a while and it's not completely new. So what we have to do is, you have to add a dedicated AI governance

framework. That's really going to help you understand where the risk is and what compliance controls you are going to need. How do you evaluate safety and clinical relevance...

You're also obviously going to be putting emphasis on the ‘trusted zones’. The closer you get to Protected Health Information, the bedside, patient care, point of care, the more stringent the standards.

These technologies can be very useful across the board in our everyday lives and some of the work we do in a general workplace setting. So finding that right fit and being able to measure that is key.

Here's another key: the ‘culture of no’ or ‘delaying adoption’ that's going to almost guarantee that you will get shadow AI almost assuredly.

You have to be able to engage your workflows. You have to be able to identify those key areas where technologies like this can add value. Somewhere in the ecosystem, there are opportunities to add value. And so you perform surveys, you engage with your employees, and you find those opportunities to deploy and find ways to make GenAI safe and effective in the workplace. And that really gets to education and raising awareness. You can't emphasize this enough. You have to train and engage your staff.

You must teach them about the risks. Many people don't understand; don't put any personal identifiable information into an LLM. We have a joke in technology about LLMS, or ‘the Las Vegas of technology’: what happens in an LLM, stays in an LLM, potentially forever. And that's a risk that a lot of folks don't really understand or appreciate. But you also want to support the growth and maturity of your staff, of your workforce, as these new solutions develop and increasingly add value. Those are a number of key points to make sure that shadow AI does not become a major risk in your organization.

Absolutely – you have to innovate! I think there are two organizational patterns that have developed that can help address some of these. You want to innovate fast, but it's heavily regulated. You have to be compliant!

So the two concepts that I think are very interesting is, first one the ‘Team of Teams’ and second: ‘Two in a Box’.

1. You need dedicated technologists that are expert in cloud, AI, scaling enterprise solutions and managing costs and all that really important, IT focus. That's your engine for innovation.

But they're not going to be experts in compliance or experts in delivery of care, or experts on how to measure patient risk, or many of the other factors that are necessary for working in heavily regulated industries like healthcare.

So you create multiple disciplinary teams right from the get go, or what we call: a ‘Team of Teams’; stakeholders with different roles that have different decision rights, but all working together, and this is typically in an actual, physical room.

Remember those days? Really working together in person has also been key. Folks in one room, voicing challenges, but also bringing their unique craft experience to the table. So those stakeholders are there in day one. You don't want to bolt on compliance when you're in the terminal stage of delivery. That is not the time to try to figure out how to meet compliance requirements.

This is about developing a collective understanding of the mission and purpose, the stated goals, and how to ensure compliance at every step. This is really key. And then, once you have alignment, you begin the actual work.

2. Now you have to go into operational mode, and this is where we focus on what we call ‘Two in a Box’. And this is the core of our Expert AI strategy: we pair a clinician, nurse or pharmacist with an AI engineer. What could be simpler, right? Small, intimate and focused. That is the kernel of the process.

And they begin ideating, testing, how to break down a complex problem. They're experimenting hand in hand. They develop that intuition around the art of the possible and after a few cycles, the relationship begins to formalize. The expert defines the box, the guardrails, the safety measures and the instructions and guidance for attaining, for example, a good clinical outcome.

The AI engineer must stay within this box, while they turn the technical dials and the AI innovation really begins to accelerate. So the ‘Two in a Box’ develops intimacy. It creates empathy and a more aligned vision. People are aligned on their purpose, and that box can scale to multiple experts and AI engineers as needed, but always the two together. What sense does it make to have an AI engineer build a complex solution that they really know nothing about? You got to have a ‘Two in a Box approach’ and this has been an incredibly effective strategy for us.

This is very intimate, very focused and very experimental. You have to look at a lot of new variables, the risks, the value, the clinical relevance, all these things that are coming together.

It's really feeling much more like science than it is technology and it's really interesting and exciting. Again, we developed the strategies to make sure we stay focused and can remain focused on that trusted, verified, explainable and transparent solution.

When you think about LLMs and what they're really good at is this notion of LLM reasoning – and they're improving that – but they're better at following expert instructions than expert guidelines, and that comes from the expert.

The AI engineers can take that expert guidance and those instructions or what we call chain of thought and they can improve the ability of the LLM to analyze and break down complex problems, giving you that better outcome, more accurate, more clinically relevant and more safe, explainable and transparent, and that is absolutely key.

First of all, you have to look at what are you trying to achieve. We talk a lot about owning the outcome. You really have to define the problem statement. We talk about a process and focusing on the ‘jobs to be done’. We're talking about agent, we're talking about workflow...

You have to develop that customer intimacy. And it goes way beyond your traditional maybe UX and CX, where you're looking at customer journey mapping. This is really starting from first principles and it's difficult. It's not quite product management, it's not quite go to market, but it involves all these things.

And most importantly, it involves experts that authentically understand the jobs to be done, the workflow, and what are the outcomes, and what are the desired outcomes. And they should also be able to measure the potential for risk and whether or not a solution even makes sense to build or whether or not it can be effective. That's number one.

Number two. Where do you find these experts? We're very fortunate at Wolters Kluwer. We have over 7500 clinicians that work on our UpToDate solution for clinical decision support. We also have in house licensed clinicians. Folks that are domain experts that are really driving forward Expert AI.

And you can work with customers and you can find other ways to really understand those ‘jobs to be done’. That is really key and we found that that is the focus point really from first principles on day one: designed for the professional, by the professional!

We've moved beyond the idea of just adding GenAI features that add modest productivity gains.

In order to do that authentically, you have to get much closer to the customer and understand how to add that value and make sure you're doing it safely and effectively.

Also, one other comment and note: AI is changing so fast and it's really intense, that probably is true. But it has changed in ways that make that talent profile subtly different as well. The well pedigreed experts in AI, building teams of these PhDs, you may need folks with that skill set, but that isn't the universal skill set you need now.

You really need to focus on talent that demonstrates curiosity, a willingness to learn, a willingness to experiment, reason and plan under uncertainty. Not everybody's comfortable with this.

Some people like that. Like saying; here's the requirements document. Let's set up all of our story points and we’ll go into our agile SAFe mode and we'll deliver an outcome. At the end, we'll certify it and launch it.

You want people that can work in really ambiguous environments, take instructions and guidance and feedback from multiple stakeholders and be very focused and disciplined because it’s all very new and we're learning at the same time.

But I think that is really a key thing that may be overlooked in terms of how to develop the right talent. How to build the right organization and where you should focus on in terms of your own workflow.