Value Set Quality And Emerging Novel Approaches

Stephanie Broderick (00:04):
We’re going to go ahead and get started with our next session. So thank you everybody for joining us at the Clinical Architecture Data Quality Theater. My name is Stephanie Broderick and I am the EVP of Strategic Initiatives for Clinical Architecture and responsible for partnerships, business development, strategy, industry relations, academic outreach. So wear a number of different hats and I’m really excited to present with our amazing panel here. And so I’m going to go ahead and have them introduce themselves and then I’ll talk a little bit about what we’re going to be covering today. So Jim?

Jim Shalaby (00:44):
So I’m Jim Shalaby. I’m the CEO of Elimu Informatics, and I specialize in terminology.

Stephanie Broderick (00:51):
And Victor.

Jim Shalaby (00:52):
How about now, that better?

Stephanie Broderick (00:55):
They say hold them up kind of up like this.

Jim Shalaby (00:58):
How’s this? Yeah. Want me to sing a song? Okay. I’m Jim Shalaby, I am the CEO of Elimu Informatics and I specialize in terminology. We’ll be talking about value sets in just a little bit.

Dr. Victor Lee (01:11):
And hi, my name is Victor Lee. I am VP of Clinical Informatics at Clinical Architecture. I’ve been at Clinical Architecture for seven years. I run a content team. I’m a physician by training, so I refer to myself as a geek trapped in a doctor’s body.

Stephanie Broderick (01:28):
So before we go into this topic, Jim, I’m curious, where did you come up with Elimu? Does that mean something?

Jim Shalaby (01:34):
I can’t take the blame for that. It’s my business partner Aziz Boxwala. I can spell his last name if you’d like, but it actually means “knowledge” or “insight” in Swahili. And originally when we named the company, we were thinking of something with “cognitive” in it, but that was taken up by two other companies. And so Aziz had the great idea of Swahili.

Stephanie Broderick (01:59):
Got it. All right, interesting. Learn something new every day. Alright, so today we’re going to be talking about value sets and we’re going to start off with Victor and he’s going to talk about some example value sets and some challenges around the value sets. And then Jim is going to talk about some analysis that he’s done, some work that he’s done looking at can you manage value sets with AI, what are value sets, what are some good ways of maintaining them? And then we’re going to finish off with some potential things that Clinical Architecture and Elimu are thinking about doing together and that we want to share with the group and see if there’s interest in some type of joint offering between our two companies. And so with that, I’m going to turn it over to Victor.

Dr. Victor Lee (02:49):
Okay. So I wanted to tee off this presentation and kind of set up Jim to talk more in depth about value sets, but I wanted to give an example of how you might want to think about the quality of value sets. We’ve been talking about data quality in these data quality theater presentations, but I wanted to illustrate how if you’re going to think about measuring care quality and improving care quality, you might want to think about the quality of your value sets. So the example I gave, and I’m really trying to protect the innocence, so there’s a value set in VSAC that you see on the screen. And this is an ACE inhibitors value set and it sounds pretty straightforward. ACE inhibitors, they might be used for heart failure, hypertension, and there are a variety of different kinds of terms that you can see up on the screen.

(03:41):
There’s actually a total of 345 here, and I didn’t have space to list all of them, but you can see that there are ingredient based multi ingredient, some with brand names. So this looks like a typical ACE inhibitors value set. This may or may not be used in a quality measure, may be used for other purposes, but again, the names have been protected or the names have been hidden to protect the innocent. Okay, so it looks like a pretty comprehensive value set. There’s 345 members here. But if we go to the next slide, sometimes the numbers can be deceiving because Clinical Architecture has also authored an ACE inhibitors value set. There’s an offering called CA Elements is short for Clinical Architecture Element Set Foundation. But you can see that the scope is basically identical because we’ve bound our terminology to RxNorm, and there’s a whole bunch of term types, which I won’t get into the abbreviations there.

(04:41):
But basically we have the same kind of scope and focus, both the value set that I provided as an example in VSAC and the Clinical Architecture value sets have multi ingredient drugs. But you can see the counts are very different. And so when you start picking apart the differences, we see that there are a lot of omissions in the value set in VSAC because there are a lot of brand names that are missing, there are semantic branded drug components, that’s the SBDC, and a lot of multi ingredient drugs that are not captured. So if you’re asking a question about whether a patient is taking an ACE inhibitor, there might be things that are missing in your analysis if you don’t have a complete and comprehensive value set. And so there’s 842 in the Clinical Architecture element set, 345 in the one in VSAC.

(05:35):
Now that’s not necessarily to say that the VSAC value set is wrong because they might have specifically designed it for a purpose, but the question is if you’re going to use a value set that you didn’t author, you want to understand what was their scope, what are the inclusion and exclusion criteria, what was the purpose of the creation of that value set? And be mindful of that as you apply value sets for whatever quality improvement or analytics, pop health, clinical decision support, whatever use case you have, just be mindful of how it was created and how it’s maintained as well. So I just wanted to tee up that conversation about thinking about the quality of your value sets before you measure and improve patient care quality.

Stephanie Broderick (06:18):
Great. So I’m going to turn this over to Jim.

Jim Shalaby (06:21):
I got the clicker. Can everybody hear me okay? Yes. Okay, great. So we had very similar experiences. I wanted to better understand what the current quality of value sets are in a very large repository, VSAC. So I looked at the value sets. They are extremely valuable constructs. So one thing I want to just qualify this with is that they’re used everywhere. Sarah uses them all the time. Cory uses them all the time, pretty much a major bread and butter for terminology use in decision support and analytics. They’re intended to be designed for specific purposes. Okay, that’s something to keep in mind because we’re going to look at some slides where that can sometimes be a little tricky to use. They’re relatively easy to maintain if designed correctly, and the operable word here is designed correctly, okay? There are over 15,000 value sets in VSAC. So it sounds like a lot.

(07:22):
And you can access these value sets freely from the National Library of Medicine from VSAC. It’s a repository that is openly available to everyone. The current state of VSAC value sets is that they are published, they go through a routine maintenance. If any of you do have VSAC accounts, you’ve probably received emails saying your value set is getting a little stale, you need to refresh it. And if you don’t, they will continue to warn you and eventually you may find your value set retired. So there are potential roles for AI and creation and maintenance of search and discoverability and value sets. And I’ve been looking at that, I wanted to experiment because 15,000 value sets and growing, are they duplicates of each other? Are they the same? Do they contradict each other? How can I reuse some for other purposes? Those are all questions that I’ve received from clients and I’ve had to assess myself, but AI has a lot of potential for being able to help in this area. So I’ll go to next slide. There we go.

(08:33):
So what’s the difference between value sets? Just a quick primer. There are intentional and extentional value sets. And how many people here know the difference or have heard intentional versus extensional? Okay, so about half of you or a third of you. Alright, so it’s pretty simple. So the intentional value sets are rules defined. If I want to define a value set as beta blockers, I may instead of enumerating every single drug that’s a beta blocker, I might pick a therapeutic class beta blockers and say, bring me back all the drugs underneath that. Diabetes, I may pick a SNOMED code, say bring back all the descendants of diabetes. An extensional value set is an enumeration of every code. The rules based or intentional value set definitions are typically much more amenable to updates. They’re easier, they’re easier to maintain, but not everything can be done as intentional.

(09:30):
Whenever possible we try to use intentional value sets in our definitions. Extensional value sets are sometimes necessary. If someone asks me to create a value set of all antibiotics used in treatment of outpatient, I’m not going to find that in a hierarchy. I’m going to have to create that and may have to create that by hand. Extensional value sets are sometimes necessary when codes of interest cannot be easily defined or when the ontology doesn’t group things in the right way for what I want. The ontology could be perfectly fine, just I have a different way of looking at it.

(10:10):
So what’s the current state? 15,000 value sets in VSAC, but out of those in the analysis that we did, only 1,450 were defined as intentional. So that’s 9.6%. Okay, the rules-based value sets, and it’s true that some of them probably could not be defined as intentional, but I reviewed them all and the majority of them could actually be defined as intentional rules-based value sets. Out of those only 687 were intentionally defined SNOMED. So I was just looking at SNOMED, not all value sets, only 687 were defined. Approximately 3,200 out of 3,700 SNOMED based value sets could potentially benefit from being defined intentionally, the rules based, so I could redefine them. And actually at leaving we ended up in VSAC redefining a lot of them just for maintainability.

(11:14):
The value set quality is highly variable with respective frequency of updates. So some of them are kept up to date quarterly, yearly, every six months. The domain is important to keep in mind. SNOMED, you can probably get away with annual updates, every six month updates, LOINC or RxNorm. Let’s take RxNorm as an extreme. An annual update means you’re going to be missing a lot of medications. Okay? Intentional value sets can require more time upfront to define. It takes a little more work to figure out the rules and how to manipulate the relationships between concepts to define a value set. But once you do, it’s much easier to maintain.

(12:00):
So I started asking the question, can AI help? Okay. A lot of the value sets, the reason I picked medications for this example is that medications ontologies are what I call synthetic ontologies. These are manmade constructs and medication is ingredients packaged in a single form, usually by very well-defined strengths. And so they’re easier to understand and create rules around than other domains. They’re synthetic ontologies, but it’s very mechanical. It takes a lot of time to create those. So I looked at AI and I tried an experiment to see can an AI engine create for me at least the beginning of a value set. And then I raised the bar progressively to give it harder and harder prompts to be able to see if it could produce the results I wanted. So let’s look it an example. And I did use Chat GPT and I used Bard. I wanted to see if there was a difference. It’s now Gemini.

(13:05):
So a simple example, the prompts at the bottom in quotes are what I gave the AI engine and I had to tweak it for different engines, but essentially I said create, I want to start with very low bar hypertension is treated with beta blockers as one of the drugs and options. And I asked it, can you create RxNorm value sets of all systemic beta blockers? I don’t want a eye drop, I don’t want something that’s done topically, I want something that’s used to treat hypertension and I wanted to see if it could produce that for me. And I said, include the descriptions in the ATC class. Does anybody here not know what ATC classes are? Okay, so World Health Organization has therapeutic classes for drugs, beta blockers, alpha adrenergic, a variety of ACE inhibitors. And so you can use those. You can say give me the children or the descendants of that class, and you can put restrictions on it and say, I just want the systemic ones.

(14:03):
I just want ones that are used for hypertension. And you can write rules. They’re very powerful. You can write decision support rules or you can write value set rules that can actually define these. So I said give me the ATC classes because I wanted to define intentional rules. That’s very helpful. Sometimes it’s very hard to find these classes. It did a very good job. It listed all the ATC classes for me as well as the RxNorm codes that go with it. That looked great for my first test. My next test, but these were ingredients, it just gave me ingredients. There’s a drug called Timolol. It’s a beta blocker. So everybody know what Timolol is? Okay. It’s a beta blocker drug. It can be given orally for hypertension. It can also be put in your eye to control glaucoma. So I didn’t want glaucoma drugs, I wanted hypertension drugs.

(14:58):
So ingredients are not the right level for my value set. So I wanted to raise the bar and say give me hypertension drugs that exclude nonsystemic, just include systemic, it understood what I was asking. It could not produce what I needed. Okay? So it understood systemic, it referred me back to RxNorm and said, you figure it out. Okay, so what I was really looking for, I’m sorry the clicker is a little jumpy today, but what I was looking for is this. I expected the results to include term types are basically different levels of description for drug. Instead of just saying Timolol, Timolol oral tablets is more specific. It excludes ophthalmic preparations. I want to say 10 milligram, five milligram, those are all different term types. So I wanted specific term types. I didn’t want ingredients and I gave it that prompt. I said give me those term types. It acknowledged the term types, which is good. What it couldn’t do is bring back the results. It didn’t understand ontology. Now that’s promising because it knew it didn’t have the information, which is step number one. It didn’t produce the results, but eventually I can train it on an ontology and it should be able to produce those results.

(16:19):
So value sets use in AI I think has a lot of value. I did a lot of other experiments with it. I put a link, we have a blog on my site, on Elimu’s site that discusses experiments that I did with AI. I included all the prompts. You can copy and paste those prompts into your Chat GPT or Gemini and run it. You can tweak it, try different scenarios and see what results you get. And it’s a nice experiment to do. It doesn’t take long to do. It takes maybe an hour, an hour and a half or so, but there’s a lot of potential for discoverability. The main points that I think value will be introduced by AI is discoverability, maintainability, creation and comparability. So as far as discoverability, I’m searching for a value set in VSAC, there are 15,000 I can envision using AI to identify value sets like mine.

(17:13):
So here are examples, here are some drugs, find me other value sets like that. I can envision it being used for maintainability where discovering RxNorm in ATC. Other concepts similar to this that I can actually use to create my value set, which it’s a big lift. It takes sometimes hours to identify what you’re going to put in a value set before you even try to create it. As far as creation, it can actually, with a little training with the ontology, be used to create the value sets. So it’s conceivable that with drug value sets that I can at some point after training, trust it to create value sets for me without doing a lot of hands-on. Comparability is important, the 15,000 value sets, we have to reduce this. There are a lot of duplicates, a lot of overlap. I can run an AI engine and tell it, give me overlaps, give me overlap diagrams of how much this value set is represented by 10 other value sets in VSAC and decide whether I want to clean them out.

(18:12):
Maybe they need to be different, but maybe they are duplicates. Okay. So in conclusion, just the general theme is that efficiency improvements, I feel I can improve my efficiency in value sets significantly. And I’ll say this out loud, I think Victor may or may not agree with me. I think I can improve my efficiency with value sets, especially complex ones in certain domains anywhere between two and fivefold. For medication value sets, with experience I’ve been doing in some of the use of AI, I’ve been able to drop it down from a value set that might take 30 minutes to do to one that might take five minutes to do. So pretty significant differences in amount of time also for maintainability. And then there are some value sets where you just have to do by hand, training has not gotten to the point where it can do that. Useful to Terminologists.

(19:10):
I think it’s very useful to terminologists. I think there were questions that, the way I started doing this is that I was asked can we replace terminologists and can we replace tools with AI? Do we ever need value sets? Can we just ask AI prompts and get rid of the middleman? I think we’re going to need terminologists for quite a while. Okay. But it is a great tool for terminologists. It’s not going to generate value sets out of scratch, not today. Okay. For simple value sets such as the ones in the examples, there’s much potential for improvement. And again, potential for automation. Most of the drug value sets that are class-based or therapeutic term type based probably can be done with a great deal of automation in the future. For more complex value sets, it’s an aid to the terminologists, but it’s going to be quite a while before it reaches that point where you can have full automation. The value set experiment. I just want to provide you the URL. You can look at the blog I wrote. It’s a little bit long, but it’s because I give you the search results and you can try the prompts yourself and I’d encourage anyone to experiment. If you’re interested and you want to look at it and experiment with me on this, I’d be happy to do a sidebar and show you what these experiments yield.

(20:35):
And back to Stephanie.

Stephanie Broderick (20:38):
So Jim, before I go into this, I just want to ask a question. So you basically drew the conclusion that yes, it provides some value, it can help, but terminologists are still needed. And we’ve had conversations around some of the work that you’ve done with clients. And can you talk a little bit about your thoughts around creating metadata, the discoverability of value sets, and kind of tee me up for this topic as to why we are considering doing something. Because obviously AI, the world isn’t ready for AI to manage terminology and value sets aren’t going anywhere anytime soon.

Jim Shalaby (21:24):
Sure. I think it’s going to be interesting even in the future when AI can produce value sets, that metadata is going to play a very important role because the value sets can be very context specific. So for example, if you asked me to create a value set for beta blockers today and didn’t tell me how you’re going to use it, I can create one. Okay? I can include in it all ingredients. You’ll get Timolol eyedrops and you’ll get Timolol oral, you’ll get other drugs in there that are beta blockers. However, if you told me I want beta blockers that are used in treatment of hypertension or used in heart failure, I’ll reconsider and put, I’ll explain in the metadata what the use was and reduce the value set down. So I would still do that and the metadata can be very powerful for that as long as it’s searchable. The idea behind the metadata that expresses what it’s intended use is and also when it was last updated. And if the history of metadata in case the use has ever changed, then VSAC I found value sets, many value sets, where they were started years ago for one purpose and then repurposed for something else. But the value sets still has shadows, echoes of its older use, which are creating problems today. And so yeah.

Stephanie Broderick (22:40):
In the conversations that we’ve had, so we’ve got a couple of things. We’ve got work that Clinical Architecture is doing where we’re managing value sets, right? We’ve got our CA Element Set Foundation. We’re finding that the value sets that we’re authoring are more comprehensive, more accurate up to date. Then in the work that you’re doing, Jim, you are finding that people are reinventing the wheel. They can’t find what they need so they’re creating their own. And so Clinical Architecture and Elimu are talking about a joint type of offering that would utilize Clinical Architecture’s Symedical as the tooling platform, and then services from the organizations for creating and maintaining of value sets, doing mapping, modeling, cleanup, quality assessment, and basically a managed service for organizations who struggle with managing the value sets, creating very clinically curated value sets. And then obviously between the two organizations, a tremendous amount of clinical informatics expertise. So I want to turn it back to you, Jim, and just talk a little bit more because you approached us with this idea. What else would you add to what I’m describing?

Jim Shalaby (24:03):
No, I think that’s accurate. I think that the need for clinical informaticists who understand not just the value set but how it’s going to be used and have the domain expertise to be able to leverage it to its maximum and also to understand when it should be reused and when it shouldn’t be reused for other purposes is going to be really important. I think we understand that collectively within our companies and are able to manage and create some very context specific value sets that will not paint you into a corner. And a lot of it is metadata, but that metadata is how we express our expertise to make it more discoverable. The knowledge of figuring out what metadata to put on it and what is an allowable change in the future and what isn’t is where the subject matter expertise comes in.

Stephanie Broderick (24:57):
Great. Victor, would you add anything?

Dr. Victor Lee (25:01):
I guess my only comment is just to follow up on Jim’s comment about whether large language models may reduce the work required to author and maintain value sets. And I think we’re both right from our own perspectives. I think there are may be some differences in staffing, processes, tooling, that led me to the conclusion that I don’t know if we gain much efficiency from using large language models, large language models like Chat GPT or other similar LLMs I view as sort of complimentary to our existing processes because as Jim mentioned, intentional value set definitions can be a little bit more work to define upfront, but maybe more scalable. And my team, which is a team of all clinical people, we’ve authored intentional value set definitions and we’ve optimized the ability to also recalculate them and scaled them over time. And so for us, adding LLMs doesn’t necessarily reduce work.

(26:20):
It’s actually a little bit of additional work to kind of cross validate what an LLM has found in comparison to what we’ve authored within Symedical. But I think it’s just a difference in processes and tooling and personnel and whatever may account for the differences. But I do think there’s a lot of promise. One more thing I’ll say is that both of my parents were accountants before they retired. And the calculator, the invention of the calculator did not put them out of business. And in fact it might’ve improved their efficiency. The invention of spreadsheets did not put them out of business, but it might’ve helped them improve their efficiency and other applications came along and it didn’t put them out of business. And I don’t see LLMs putting clinical informaticists out of business either. And I think we are going to see novel applications of large language models and maybe other AI technologies which will merely help people do their jobs better, more comprehensively, more efficiently. So that’s my two cents.

Stephanie Broderick (27:25):
So I’m going to move on to questions, but before I do, we do want to leave kind of a call to action. So if this type of an offering, a managed service for managing value sets so that they’re comprehensive, up to date, clinically curated, accurate, is of interest, please let us know. Please come and see one of us, one of the three of us, and we’re happy to talk to you about that. So do we have any questions? I should just give you a mic, you have the first question every time.

Audience Speaker 1 (28:03):
Thank you, Stephanie. Yeah, so, Joe Bormel, first off, fabulous talk. If I was on stage, I’d say everyone, “let’s applaud” and my experience looking at VSAC is consistent with your research, Jim. So I thought that was phenomenal. And then my personal experience building over a thousand value sets from a hundred technical expert panel members that told me how they needed to be built and how they needed to perform are consistent with what you’ve been talking about, Victor, in terms of if you can make them intentional, you’d better. So I have a question though. And the question is that I thought I was comfortable and excited and knowledgeable about how LLMs are used. What I’ve been hearing definitely at this conference and elsewhere is people are using RAGS in addition when they have appropriate sources and they’re also using part of that whole pipeline often includes knowledge graphs to carry some of the context through, etc. There’ve been a number of presentations that have done that. I’m sure you’ve thought about that issue. So I’d love your comments.

Dr. Victor Lee (29:08):
I’ll defer to you.

Jim Shalaby (29:09):
Okay. Yeah, so I think there’s a lot of potential there. I think it’s moving into the next generation of knowledge representation. Today, we still think in terms of value sets as being a component of something bigger, such as I’m going to use it in a rule or I’m going to use it in a query definition for a report. But when we start talking about knowledge graphs, the separation between terminology artifacts and knowledge representation starts to get blurry. It’s not really blurry. It converges right? The context is now explicit instead of implicit. And that explicit context is perfect for RAG type modeling and leveraging. And it moves us to, I think it moves us to a new type of knowledge representation that we’re not there yet today, but the potential for there I think is pretty powerful. And I think it’ll still need human supervision, but it won’t be, Jim Shalaby reviewing the quality of a value set. It’ll be you reviewing the quality of a knowledge graph and seeing within that context, is that representation complete?

Stephanie Broderick (30:23):
Any other questions? No?

Jim Shalaby (30:33):
So if anyone wants to play with this, again, there’s a URL you can try. I’d encourage you, try Chat GPT, try Gemini. There are two other engines that might be of interest as well that you can experiment with and see if you can get it to do what you want. And it’s really interesting and I wouldn’t criticize it on not giving you back the results. I would look at seeing, did it understand your question and is it answering the right way?

Stephanie Broderick (31:01):
Alright, well, if there are no more questions, I want to thank our wonderful panelists, Jim Shalaby and Dr. Victor Lee, and thank all of you for coming today. Thank you so much.