Value Sets in a Distributed Learning Context

Victor Lee, MD (00:04):
Good afternoon everyone, and welcome to the Clinical Architecture Data Quality Theater. My name is Victor Lee. I’m Vice President of Clinical Informatics at Clinical Architecture, and it’s my pleasure to introduce our first speaker of the afternoon, Joe Bormel. He’s a friend of mine, he’s a Physician Informaticist. He’s trained at Johns Hopkins University, a Harvard University. He has done work for many agencies, including the Office of the National Coordinator, which is when we collaborated on a project and I met him about 10 years ago. Basically, he’s kind of a big deal. And what that means is when Joe Bormel talks, we should listen. And so Joe is going to talk about value sets in a distributed learning context, and I look forward to the presentation. So Joe, I’ll hand it over to you and we’ll take questions at the end.

Joe Bormel, MD, MPH (00:59):
Thank you, Victor. Yes, 10 years ago, and I have gray hair and you don’t, I’m not going to ask. Okay, so. Hello, I’m Joe Bormel. It’s probably better. Yeah. Thank you. The question from the audience, don’t you recognize you have hair? It’s like you shouldn’t be saying anything. So. Hi, I am Joe Bormel. My views today are based on published research, public information, and I have no conflicts of interest to declare perhaps except that my wife works for the FDA. So I use the term value sets in the title of the slide. Is my volume sounding okay? Because I’m not, feedback wise, I’m not hearing myself clearly. Okay. So I use the term value sets in the title, and so let me go ahead and define that since the whole talk is about value sets. So value sets are lists of codes and corresponding terms from standard clinical vocabularies that define clinical concepts to support effective interoperable health information exchange.

(02:10):
So last week I was discussing the content of this discussion with a friend on building value sets with Harold Layman at Johns Hopkins, just describing the work involved in building more than a thousand value sets in our case for the National Institute of Diabetes and Digestive and Kidney Diseases (NIDDK), which is the diabetes, digestive disease, and kidney disease. And he was working on the N3C, which is an NCATS, the National Covid Cohort Collaborative. And he also built over a thousand value sets and many of them were on the same topic. And in the course of some recent AMIA events, I found that other NIH groups were also investing a lot of hours into building value sets.

(03:04):
And it seemed like we might be able to have some refinement. Everyone was generally putting them into the VSAC, but finding them in the VSAC did not mean you could just, that you would necessarily meet your needs. And so people would build new ones, which has been a consistent theme throughout the talks here this morning. So many programs develop unique approaches due to the context differences and resource limitations. And in his keynote at the MCBK meeting, Dr. Richard Sherman, the scientific director of the National Library of Medicine, recently presented a compelling vision for leveraging research across institutes. Dr. Joshua Denny, Andrea Ramirez, Tiffany Callahan, Emily Paf, Jenna Norton, and others have elaborated that we really need to enhance data, getting the data, data quality to enhance biomedical research. And part of that is to be able to bring together information that’s done in different research projects, which you can’t do if you don’t have the data elements defined the same way and used the same way across programs, at least within context.

(04:18):
So furthermore, when you go into the advanced computational phenotyping and reasoning and using knowledge graphs, which is increasingly becoming part of every part of research, especially in the genomic space, this just becomes absolutely essential. You can’t do these joins unless you have harmonized data. So this presentation will address the challenges in data quality and propose solutions to improved data reuse, development, developmental efficiency, etc. And we’re going to focus on the NIH’s role in research and care delivery and explore the ways to strengthen the NLM’S roles. So does that sound good? Great, thank you. So, three quick things. We’re going to talk about comprehensive shared eCare plans. We’re going to talk about value sets and how they relate to data quality. And then we’re going to talk about going into the future. So the first question you have to ask is, what is the scope of the problem?

(05:23):
What’s the best way to get started with solving a problem? And if the problem is data quality, one way you could think about where we should start is, what patients are having the most issues and what are those issues, clinical issues. Chronic conditions has been identified as a huge component of this. 80+ percent of medicare recipients have multiple chronic conditions, and so in terms of starting at the head of the Pareto curve, that’s a good place to go. So I’ve got a color key here, in blue is the top three conditions amongst the chronic conditions. And then by the cost to the system, the top three are shown in green. Many of you are aware that kidney disease is a huge expense, and part of the issue with kidney disease is it’s concentrated amongst less people. So it has high cost density per person, but also it has a high opportunity to improve by putting attention because there’s less folks and so that you’d have to intervene with.

(06:37):
So storing, retrieving and using existing data is problematic around these and other conditions. And part of the problem is that the data for patients is distributed Charlie just said across 36 EHRs or whatever the number is, and it’s across providers and across institutions, and no one record is complete. Another part is that the requirements, the standards that are chosen are often not standardized. We’re going to call that normalization. And then mechanisms to do those normalizations are themselves not normalized. So you can’t ask a question about what we know about cystic fibrosis in the different institutes and be able to reliably pull that data because we’re not standardized on the data.

(07:33):
I think we covered the major points there. So this wouldn’t be a talk about data quality if we didn’t talk about why we need data quality. Data care suffers when data is inaccurate, incomplete, inconsistent, duplicated, and non-current. That’s a nice succinct definition on data quality or the lack of data quality. And the same is true for research. So our study today, we’re going to look at the complexity of dealing with multiple chronic conditions, which hopefully we just introduced why that’s important. And so this is several of the NIH programs set out to do exactly that, to be standardizing data quality at least within the domain of their projects. And so this slide talks about the multiple chronic conditions, eCare plan project, and part of the work, there were three work streams, part of it was to clarify the essential elements of taking data from EHRs and bringing the content together, facilitating the core functions including care delivery planning and research with cross system data quality improvements well beyond the data quality that exists today.

(08:48):
And that inspired the project, which started five years ago. It’s now in its final year. So shown in this picture on the right is the architecture used in the product. You’ll see we’re showing three different sites, could be three different institutions. The information going brought together by a FHIR server served up as Smart on FHIR apps to patients, to caregivers, to care team members, to providers to specialists, and also to human services health and human services community members. So that’s that stack. The goal with all of this is I just heard Jenna Norton, the program director, saying in a presentation two hours ago that the goal was to enable social risk, adaptive care, as well as social needs, informed and targeted care. Which was a really nice succinct way of talking about the goals here.

(10:05):
One of the things that, so I got a nice introduction. I worked for Cerner for six years. I worked for a company called Quadrant for 12 years, had a lot of clients, a lot of large clients, and the EHRs essentially were aimed at removing the paper and the paper charts from the legacy models of care delivery. And then to take on some of the core critical functions like medication management, medication reconciliation, and just the basics of ordering, scheduling, documentation, etc. A big part of EHRs that was lacking, which was attention to what health concerns the patient had and carrying them as such, what the patient’s goals were and what the state of those goals were, interventions, etc. So part of the goal of the project, this NIH multiple chronic conditions eCare plan, was to actually go to the level of care planning on top of what EHRs were doing.

(11:13):
And as I mentioned, there were three work streams to this, developing the value sets, the data elements and value sets, and a library of them to develop an implementation guide and to develop, pilot some applications in that space and do so for a few main conditions, chronic kidney disease, type two diabetes, cardiovascular disease, and chronic pain. And then subsequently during the project, long COVID became a target as well. So breaking down the data silos in healthcare has been a consistent theme of the data quality as one of the problems is access to the data. So a healthcare system where complete data, patient data is accurate, consistent, deduplicated and current is a shared vision. The challenges currently is that data is fragmented across multiple institutions and stages. And the NIH’s ongoing initiatives aims to achieve data completeness between its institutes, its centers, its offices and research programs addressing these deficiencies.

(12:31):
And that concept of data completeness has been brought up in probably every one of the data quality presentations. It’s a really interesting notion, and it also came up in a paper that was in Nature medicine just two weeks ago that was published by the NIH all of us folks. And essentially in this slide, it both calls out what I said earlier from Richard Sherman that the NIH recognizes that they want to enhance data across NIH. What’s shown over here is from the multiple chronic condition eCare plan project, just an elaboration of topics that we discussed in the course of developing the projects and the dates that we did these, how we were going to model goals, patient goals in the project, how we’re going to model type two diabetes so that it was complete from the standpoint of technical experts in the field who and the technical expert panel is more than a hundred clinicians who contributed.

(13:41):
And so we went through all of the issues and we did that with the patient care work group of HL7, getting to the issue of community building, which we’ll come to at the end when we wrap this all up. So this was the NIDDK group at NIH. This is from the all of us folks and down here is the N3C pipeline I mentioned earlier, this is the National Covid Cohorts Collaborative. And what they’ve been doing is they also have been using value sets, not the same ones as the others, but their values, their whole approach, I can see the value set mappings here in their pipeline and this is an incredible initiative. It’s the most amazing thing. It’s more than 70 organizations, mostly Epic clients who had output their data from a common clinical data model through the OMOP process to central enclave and then do research against that. So this is just an amazing tour de force when you have clean data, you can pump out incredible quality research quickly and reliably. And so yet another fabulous demonstration of some of the bright spots where data quality is being done well. And there was a link in here, this is the unique YouTube ID to Emily Path talking about pragmatic and data quality. It’s a fabulous paper and I recommend following up on that.

(15:25):
So as I discussed in the introduction, current practices lead to redundant work and information integrity problems. So in this slide I’ve got two examples that address how we can improve on that using HL7 FHIR mechanisms with extensions. And we can come back to that in the Q&A if we want to come back to that slide 13. So that brings us to the next section. Good. Which is specifically how do value sets relate to quality? How do they help create quality? And for that, again, data quality has a simple concept in terms of the value of it. It enables streamlined processes, it supports improved analysis and accurate comparison. And so it’s a nice concise description of what we are asking data quality to be doing for us. And so there’s a metaphor here because Charlie Harp told me that I have to use metaphors or I’m not going to get the swag.

(16:33):
So imagine a house. The standard coding systems are the bricks. Data capture processes are the construction methods, data governance is the blueprints, and data cleaning is the quality control. So value sets are like the architectural plans, they’re crucial for the overall design, but they’re completely dependent on those other parts. So you can’t just focus on value sets and expect things to work. So I wanted to give you an example of an actual value set, and actually it’s been sort of fun today. We’ve had several presenters who from previously worked at NCQA, so that worked out well. But this is what a value set looks like in this case for cystic fibrosis. This is from VSAC, we were talking about that earlier. And then this is the actual codes under cystic fibrosis. And what you have to realize is for some applications, for some intentions or some contexts to have a diagnosis of cystic fibrosis is enough to, for example, do billing.

(17:36):
But if you want to refer someone to pulmonology because of a cystic fibrosis diagnosis or if you want to refer them to endocrinology or the GI departments, there’s a lot of manifestations that are specific to cystic fibrosis for which you could code down at that next level. And so it helps clarify that value sets are really important because they provide the ability to go to a level of granularity and as well as rolling up. So what’s the goals of using value sets from a FHIR implementer perspective? Value sets offer several benefits they’re shown here, and I can see that Kate’s already finished reading the list. Good. And the concrete example is that for cystic fibrosis, there is clear and consistent data representation for a lot of people at this conference and certainly for Clinical Architecture, the ability to run your codes against a server and validate them so that you are putting known clean data in or data that’s appropriate for the context of the data elements. So there’s a fairly short list of what’s needed. We actually went beyond that in the MCC care plan project, which is shown in this next slide.

(18:57):
So from an implementer’s perspective, value sets ensure clear data representation, but from the comprehensive shared eCare plan, they go a step further. Value sets enable reliable and trustworthy user experiences, which is impossible without standardized data. For instance, if a patient has chronic renal disease, this will consistently show in the active problem list independent of other state issues because it’s modeled to support that. And also it’s modeled to support ensuring that the provider, when they see someone with, for example, chronic kidney conditions, knows for example whether the patient is on dialysis, is on the transplant list, all those sorts of contextual things that you can do when you can rely on your coding and your standardization. So lastly, we are going to speak to the future, and this was a slide that we at Cognitive Medical Systems put a lot of effort into creating clarity and estimations and delivering high quality work on schedule.

(20:10):
And this is showing you that in 2020 we identified, we just did one domain, which was clinical chronic kidney disease, and this was a number of value sets and then total code counts within that. So in planning for the next year, we made the assumption that we’re going from one condition, chronic kidney disease, to three others, type two diabetes, cardiovascular disease and pain. And pain is tightly associated with why would somebody, for example, be in an opioid management clinic? Those opioids were typically for some kind of pain. And so it looked kind of like we had three conditions compared to one, so we might have three times as much work to do. And what we actually discovered was it was about 27% more work to do the ICD-10 codes than what would’ve been predicted. But other codes specifically one example you can see in the LOINC set, we had essentially 10 times the number of codes that we had to deal with.

(21:20):
And part of that has to do with in those domains that we were now covering, there was questionnaire and questionnaire responses that all had LOINC codes that had to be managed. I’ve had a long talk with our friends from Elimu last night about the issue of whether the LOINC codes are going to be in SNOMED or not, that whole piece. And so that gives you a sense of that. So how do you tame the value set hydra? What kind of practices, what kind of advice can we recommend for you? And so the highlights and key takeaways are shown on this slide. And specifically in the interest of time, I can’t go into a lot of details, but attention to the clarity on concept breadth, granular level synonyms and lexical variants are a big issue. And then the strategies that we discovered and used are shown in this table. And then there’s some additional other thoughts for you.

(22:25):
This, I may have jumped a slide here. Yeah. Okay. So value set reliance. So there’s some key considerations. The critical factors to create good value sets are shown under the list of features here, and then the presence or absence and product strength of the features are elaborated out in terms of the data quality impact of those features. Again, this was a 15 minute talk, so I am not going to be able to go into detail, but you can send me an email to get the slides. And there’s descriptions on the slides. It’s fairly complete. In the interest of getting to the questions and answers, I’ll just kind of jump forward at this point. The last major point I wanted to make has been made a number of times by a number of places is the issue of why community building is just so critical. And it was interesting this morning during the plenary, Hal Wolf talked about its people process and then technology and people love to start with technology or just ignore the people issues.

(23:38):
And as you walk on the floor, people have been well trained over the last 15 years to say people process and technology in that order. And so community building is all part of the people aspect. And also in the context of our project that extended those people extended to the federal committees and actually Jenna Norton went long on that in her talk about coordination with CMS and the VA and National Institute of Aging. So these are all examples of the people part of people process. And so standards don’t work in isolation. Standards only succeed if adopted and implemented and we bridge the gap between fragmented efforts and functional gains by actively engaging the community, which was probably more than half of the work. And so we worked with the HL7 patient care work group committee. We helped them with their domain analysis model, they helped review our implementation guide.

(24:42):
We met every couple of weeks. I showed back on slide 10 that we went into each of the care domains. We went into each of the domains around goals and health considerations and other issues. It was absolutely essential. And then beyond that, the community building also extends to implementation and testing. So to foster widespread adoption and create implementation guides that underwent community review was absolutely critical. And then documenting all that. So all of the work that we did in those spaces, all the meetings and the video recordings of the meetings is all online. So people who are coming after us to build on this actually have all of that background. So also I wanted to call out, especially because I’m in this booth, I’m indebted to a lot of organizations. As you’ve heard, I’m very indebted to the work by a lot of the NIH institutes.

(25:40):
I’m also absolutely indebted to Clinical Architecture. In the course of my research, I found that Carol Graham presented at the HL7 and FHIR days, I think it was back in 2020. And so I’ve got the link to that video. It’s absolutely outstanding, and Carol’s still with the company couldn’t be here today. So in conclusion, we’ve seen that data quality is a cornerstone for achieving cost effectiveness, improved care quality, broader access to services, and robust research in healthcare. Value sets enhance this foundation as our exploration has demonstrated. Effective planning for value set creation is not only crucial, but it’s also achievable. Organizations like the NIH recognize the importance of context of use and they’re actively investing in solutions to maximize, reuse and impact. Prioritizing these elements can create a thriving ecosystem for value set progress across the landscape. So with that, I thank you for your attention. I thank the community that’s taught me so much in the background that I referenced earlier in the talk. Please drop me a note for feedback. There’s an QR code to provide feedback that’s there also, if you want to get a copy of the slides. Back to you, Victor for questions.

Victor Lee, MD (27:19):
Great, thank you so much, Joe. We do have time for some questions, so if anyone has a question, please raise your hand and I’ll bring the microphone to you.

(27:33):
Okay. I don’t see any hands, but I have a question for you, Joe. Actually, this is a two part question. The first part is just for the people because you mentioned VSAC on a couple of occasions. So part one is could you maybe talk a little bit about what VSAC is, what does it stand for and how do people use it? And then the second part of the question, and that’s just for people in the audience who might not be familiar with it or people watching the recording. The second part of the question is related to what we were discussing before this presentation, which was around context of use. And sometimes when you go to VSAC and you’re trying to look up a chronic kidney disease value set and you see six or seven of them, how is it that a user should differentiate between them and do you have recommendations for a value set steward for how to accurately describe their value sets? So I’ll leave you with those two.

Joe Bormel, MD, MPH (28:28):
Outstanding. Thank you. So the first question is what is VSAC? Value Set Authority Center. It’s a tool that’s offered by the National Library of Medicine. I believe it was initially stood up in the meaningful use days for electronic clinical quality measures. I was told by Steve Poznak, we were all told by his session this morning that we’re not ever using the term meaningful use again, it’s got too much baggage. It speaks to the legacy that in order to do the electronic clinical quality measures that were needed for attestation of meaningful use and for quality, there needed to be an authority for those value sets. And so it was stood up to do that. Value sets clearly existed before that for at least 20 years. Value sets are an integral part of the HL7 FHIR. There’s value sets for that that HL7 maintains, so the VSAC is from National Library of Medicine and it gets its funding and support through CMS’s the principal recipient.

(29:30):
It’s used in quality measures for their programs. But that speaks to your issue about stewardship, which is sort of the second question, the context and context of use. So the context of use for electronic clinical quality measures, which was where the initial value sets the VSAC was initially built for, was really for quality measures. So I’ll spare you the long form version of quality measures, but quality measures, the issue of whether or not they’re feasible, whether they’re representative, are all really important issues. And there’s a process that the National Quality Forum (NQF) has in order to actually validate measures. And then there’s organizations that we’ve talked about like NCQA that are part of that whole value chain ecosystem. And so the principal users of the VSAC are typically not NIH researchers doing this kind of program, or at least historically they were measure developers for quality measures.

(30:39):
And so the initial set and the major stewards that you saw in the early days, ten years ago, even five years ago, were really the context was quality measurements. We’re now into this realm where we’re realizing that in order for us to exchange data and reliably know if somebody has one of the chronic conditions or several of them and what stage it’s at, for example, and what the standard codings are for the other diagnostic and therapeutic associations, you really need to have a discipline like value sets. And so there’s a variety of approaches and schemes people are using and it’s an evolution I think that says I can do on short notice.

Victor Lee, MD (31:27):
Okay. Great. Thank you so much, Joe. Any other questions from the audience? Okay. Thank you so much for attending and thank you again, Joe, for a great presentation.

Joe Bormel, MD, MPH (31:37):
Thank you.