The 6 Cs Framework for Assessing Value Set Quality

Dr. Victor Lee:
Good morning everyone. Thank you for joining our discussion on Value Set Quality. My name is Victor Lee. I am Vice President of Clinical Informatics at Clinical Architecture. I’m a physician by training, but I am a full-time geek and I work on value sets. We author a variety of value sets covering all different sorts of clinical domains, and we have some best practices to share for you to share with you. And these are all derived from my team’s experience with authoring and maintaining value sets. What I’d like to do for today’s session is to talk a little bit about what value sets are and how you might want to use them. And then I’m going to propose a framework for assessing value set quality, and it’s composed of 6 Cs, so fasten your seatbelts and we’ll get started. But first I want to talk about what value sets are.

Basically they’re just collections of codes. They can describe both clinical as well as administrative concepts. And what I’ll be focusing on are more of the clinical concepts, the sort of roll-ups that you might be used to in, for example, quality measurement, but I’ll talk about that in some other use cases. This is an example of a coded concept, and at a minimum, a coded concept could be composed of a triplet. So a code system like IDC 10, a code in this case, D 57.1, as well as a description. And so at a minimum we’re talking about a triplet, but there can be other things that are attached to these codes as well. So there can be attributes. So if we’re talking about ICD 10, there might be an attribute for whether a concept is billable or non-billable. If it’s a SNOMED CT code, we might have a semantic tag, which might be important for disambiguating, what a concept might mean.

So like a calcium, is that a medication, is it a laboratory test? Is it a chemical substance? So these kinds of attributes can further characterize what these codes are. So that is a single code, but then a value set would be composed of multiple codes. And this is an example from the value set authority center, also known as vs A. And this is a sickle cell disease value set. And as you can see, it’s comprised of multiple codes and they all share something in common because they’re all related to sickle cell disease. They’re all also in this case, coded with ICD 10. So that’s a simple example of a value set. And let me give you some examples of why value sets might be important for an analytics and research use case. We might want to ask ourselves, for example, do I have adult patients with uncontrolled diabetes?

And to do that, I might want to first start by limiting my population to the adult population. I might have an age parameter that doesn’t require a value set because it’s a number and you can easily filter and query based on age. But then when we talk about diabetic patients, and in this case we’re talking about type two diabetic patients, we would have a value set of concepts and we might have to decide if we’re using SNOMED CT or ICD 10, or we could bind to both terminologies, but we’ll want to restrict the concepts to those just composed of type two diabetes. Perhaps I’m not interested in type one diabetes. I might not be interested in diabetes in citus, so I would need to make sure I have a value set that precisely captures the codes related to type two diabetes. And then of course, I might need a different value set to represent the concept of hemoglobin A1C, because that’s how I might determine if a patient has uncontrolled diabetes mellitus.

So you could see that it would be useful to have a value set that comprises the concepts that roll up to type two diabetes, and therefore I don’t have to individually enumerate every single code that pertains to type two diabetes. There might also be multiple codes that characterize a concept of a hemoglobin A1C lab results. And so rather than having to individually enumerate, I can point to a value set and assess whether a patient has diabetes and assess whether the hemoglobin A1C exceeds a certain target. So that’s an analytics and research use case. We might also want to use value sets for graphical trending, and in this case, I might be specifically interested in random glucose measurements and as we want to compare apples to apples so that labs that relate to random glucose are going to appear on this graph. But if I have for example, a fasting glucose measurement, I might not want to trend those values on the same graph because we’re talking about apples and oranges.

So a value set might be helpful for me to constrain the numerous lab results within a patient record so that I can get a trend for one particular type of value. I mentioned quality improvement. So I assume that many members of the audience might be familiar with quality measure reporting, and it’s not my intent to go through the details of this flow diagram, but as an example, we might be interested in whether patients with heart failure are receiving appropriate therapy with a beta blocker. And so I might need a value set that characterizes all the concepts that roll up to the condition heart failure as a denominator. Then I might want to see whether patients are receiving a beta blocker as appropriate therapy. I might need a value set that captures these concepts of heart failure as well as a beta blocker. And if we want to be mindful of what a beta blocker therapy is, we have to think about not just generic ingredients, but perhaps also brand names, routed medications with different strengths and formulations.

And let’s not also forget that there might be multi ingredient drugs. So a beta blocker might be a multi ingredient medication in combination with, for example, a thi eye diuretic or in combination with the other things. So we need to be aware that multiple things could be considered beta blocker therapy. And so a value set might not be complete if you’ve considered all these different kinds of terms. There are also clinical decision support use cases where value sets would be very useful to be able to guide therapy to optimize outcomes. In this example, we have an alert that’s firing for perhaps a physician taking care of a patient. And it’s saying that the last digoxin level was out of the target range and the digoxin dose has recently changed. And in order to provide this kind of recommendation, we would need to understand two different things. What was the last digoxin level, and also what was the dosage of the Digoxin medication that was administered? So we’d need a value set for the lab as well as the medication, obviously in combination with logic, but a value set would be critical for being able to deliver this kind of guidance.

We can also provide, we can use value sets as part of clinical documentation improvement initiatives. And I’ll just kind of cut to the chase here, but if you looked at this sample patient record, there’s actually something missing from this patient record because as you look at the coded problems and the medications that this person is taking as well as the lab results, what you’ll eventually conclude is that the patient actually has heart failure and it’s missing from the problem list. But as we logically evaluate the patient’s record, we would need to understand that, okay, this patient is receiving some medications that are used to treat heart failure. There are some lab results, particularly the B type natriuretic peptide or BNP. Those are very specific for patients with heart failure, and therefore we can draw a conclusion that the patient likely has heart failure, but it’s missing. And so you could imagine that to enumerate all of the individual codes that would provide the clues, the evidence that the patient actually has heart failure, there are many, many codes that roll up to all these different concepts that you would need to evaluate. And so value sets can be incredibly important for being able to perform this type of logical reasoning.
Let’s talk a little bit about some unintended consequences. When value sets behave badly or when we poorly construct value sets, we might not get the results that we’re looking for. So in this first example, we have an alert for patients who have coronary disease, and the purpose of the alert is to recommend beta blocker therapy. Well, we actually might get a false negative if we’ve included timolol eyedrops. Now, timolol is a beta blocker, but I think what we really meant was is the patient getting systemic beta blocker therapy? And an ophthalmic preparation actually does not qualify for appropriate care for patients with coronary artery disease. So if you’re including too many things in a value set, you might miss an opportunity to optimize patient care. In this next example, we have a clinical trial where patients with on multiple antihypertensive therapies might be enrolled in a study.

And these are real life examples. So there was a patient who was receiving minoxidil for hair loss and minoxidil while it was originally approved as an antihypertensive medication. In more recent years, it hasn’t really been used for that purpose, but what we discovered was that it would promote hair growth. And so a topical formulation of minoxidil actually is not used for hypertension, and it might’ve been that someone created a value set for use for that purpose. It might be that for determining allergies and intolerances, there might be one use case, but from the perspective of treating hypertension, we might not be interested in minoxidil or topical minoxidil. And so you might get a false positive or negative. In this third example, we have a COVID-19 vaccine scheduling tool, and this was a few years ago, but there was an algorithm that was used to prioritize patients who were immunosuppressed.

And so one indicator of immunosuppression is steroid therapy, but there was a patient who was getting dexamethasone, which is a steroid, but this person was getting joint injections. So again, we’re not talking about systemic dexamethasone, but the inclusion, the over inclusion of just all formulations of dexamethasone does not actually help you serve that purpose. And so again, we can get inaccurate analysis if we’re not careful about the contents of our value set. And if you’re interested in seeing more examples of value sets gone wild, here’s a reference and you can learn much more about it. Okay, so we’ve talked about what value sets are. We’ve talked about how they can be used, and we’ve given some examples of how inappropriately constructed value sets can yield results that you might not expect. And so we’ve kind of taken our collective wisdom around best practices for value set, authoring and maintenance.

And we’ve come up with a framework for assessing value set quality. And what we’ve done is we’ve come up with these six C’s, which you can see here, and I’ll go over them in a little bit more detail. We’ve broken them down these six C’s, we’ve categorized them into two different domains. There’s an accuracy domain, which you might say you could kind of objectively evaluate as being correct or incorrect, as well as a usability domain. And the CS under the usability domain are a little bit more subjective depending on your use case, but I’ll go through each of them in a little bit more detail. But we’ve got completeness, correctness, currency, clarity, congruency, and consistency. So let’s talk about each one of these in a little bit more detail. So what we’ve done is actually take some examples from VSAC, which I mentioned earlier.

It’s a value set repository hosted by the National Library of Medicine. And I want to show some examples, but also kind of protect the innocent because the purpose is not to be disparaging here, but I wanted to show some real life examples of value sets that we’ve seen in VSAC, just for the purpose of illustrating why each of these six Cs might be relevant. So the first C is completeness. It’s under the accuracy domain. So in this case, there is a value set that’s been posted to VSAC and it claims to be an angiotensin receptor blocker value set. But if you start looking at the contents of this value set, what you’ll notice is that there are medications that are missing from here. So if you’re trying to use this value set to determine if there’s appropriate therapy for maybe a heart failure patient, or maybe if you’re using it to determine allergies and intolerances, there’s an issue here with completeness because you’re not actually getting all the angiotensin receptor blockers.

And I’ve listed a few of these drugs that are actually not present. And so the question is, is this the value set that you should be using? And so you might want to assess for the completeness of a value set before you actually use it. The second C is correctness. So it’s kind of the flip side instead of are you missing things, correctness addresses the other side, which is are you over including things? Are you committing errors or are there errors of commission as opposed to errors of omission? So in this case, we have a value set that claims to be a collection of oral antibiotics, but if you start looking closely, these items that are highlighted in red are not oral agents. So we have an ophthalmic solution. We have a brand name also for a non-oral, we have a medicated pad, we’ve got another ophthalmic product.

So this is actually the opposite of omission. It’s an error of commission. It’s putting more than what you thought was going to be in there. So be mindful of whether it over includes. The third C under the accuracy domain is currency. So again, we’re trying to protect the innocent here, but this is a very recent screenshot and you can see that the last time the steward of this value set updated the definitions was many years ago. And so the question is, what is your expectation for how often a value set is updated? And could things have changed since the last definition version? So that’s the third C under accuracy. So if we switch gears now and we talk about usability, the fourth C in the sixties framework is clarity. And so as a consumer of a value set, it’s really helpful to know what can I expect to be contained within this value set?

And this is, again, it’s a screenshot from VSAC. And depending on where you get your value sets, the interface might look different. But typically I think it’s helpful if there is some human readable description of what the contents of the value set are. What can you expect to be included and excluded from that value set? And in this case, I guess someone decided to upload a value set and not really characterize the expectations around the scope of that value set. So having clarity on that can really help a value set consumer decide if that’s the right value set for them.

The next one is congruence. And this one might take a little bit more explaining, but it’s also under the usability domain. But congruence really relates to whether the terminology bindings are in line with your expectations. So if we think about the United States core data for interoperability, they make recommendations on code systems that ought to be used for the interoperable exchange of patient data. So there are typical terminologies like SNOMED, LOINC, RxNorm, this is a set of conditions. And you can see these are kind of related to chronic obstructive pulmonary disease. If you’re exchanging patient data and you’re interrogating data related to patient problems and diagnoses, typically you might expect SNOMED CT to be the target code system. And I’m not saying this is wrong, but just notice that the code system that they’ve used here is UMLS, the United States it’s, it’s the UMLS mesosaurus. And so I’m not challenging whether this is correct or not. I’m just asking people to ask themselves if the target terminologies are the ones that you expect to use. So obviously, if you’re interested in having the UMLS codes for COPD, this may be perfectly fine. And if you’re doing other use cases where you need to be adherent to USCDI recommendations, then this might not be the value set for you.

The last one is consistency. This is the last C, and it’s under the usability domain. And what it really pertains to is what the experiences of the consumer, particularly if you’re going to use multiple value sets from the same steward. So the idea of consistency is does the steward apply a uniform approach to authoring and maintenance of value sets? So oftentimes, value sets are created by teams of people. And if you have different team members who are using different methodologies, they’re not adhering to a consistent editorial policy that may result in different value sets from the same steward, having a different look and feel. And sometimes it can be helpful if you have value sets from a steward where there is a uniform approach, whether it be related to how the inclusion and exclusion criteria are defined, or even from a style policy perspective in terms of how the scope is characterized with inclusion and exclusion criteria. So it makes a difference from a usability perspective to have a consistent look and feel for the value sets and have an understanding of what you can expect in the members of a value set. And by the way, if this bears any resemblance to squid game, that was purely intentional.

And so I’ll conclude by just sharing a couple of references with you. This was kind of a high level overview and there’s much more information on the clinical architecture website. There are two white papers that you can download if you’d more detail around best practices for authoring and maintenance. And the examples that I showed to kind of illustrate the six Cs, we actually did a small research study and we went into VSAC. And we did a lot of comparisons just to kind of validate the six Cs framework to see if there was any utility. And actually thinking about whether the six Cs framework would help you understand if a value set that you’re consuming is actually the right fit for you. So you can get the research brief here. And thank you for your time. I appreciate this discussion. I’m happy to take any questions you might have about value set quality.