AI-based Clinical Decision Support in High Acuity Settings

Steve Emrick (00:04):
Welcome everyone. Can everyone hear me okay? Okay. For a booth area? All right, good. Well welcome. It’s the last panel of the day, 5:00 PM We’re ready to go. Save the best for last. Thank God. True. It’s true. We have a great panel today. So thank you all to the panelists for participating and your thoughtful engagement here on what I think is going to be a very exciting topic on AI-based clinical decision solutions in high acuity settings, emergency departments, ICUs, et cetera. So I’m going to introduce our panelists, starting with Kelly Sager. Kelly Sager is a technology business leader with over 20 years experience developing and commercializing healthcare IT solutions, including 10 years as an entrepreneur establishing and leading corporate owned high tech startups at GE, Becton Dickinson and Danaher. Kelly specializes in launching and scaling innovative new technologies with expertise in AI and ML software as a medical device, EHR integration, software engineering, new product development and commercial strategy.

(01:18):
Kelly currently serves as Vice president and General Manager, Clinical Decision Support solutions at Beckman Coulter Diagnostics, a Danaher Operating Company, where she leads the division focused on delivering AI ML clinical decision support solutions that predict or diagnose the risk of acute conditions. Kelly holds a BS in computer science from the University of Michigan and an MBA in marketing management from Northwestern University. Dr. Julia Skapik. Dr. Skapik is a board-certified internist and Chief Medical Information Officer at the National Association for Community Health Centers, NACHC. At NACHC she heads the development of technology enabled public health and quality improvement projects, a next generation HIT data infrastructure and warehouse unified data dictionary for community health and a curriculum for informatics and human-centered design. Dr. Skapik holds leadership positions in HIT standards development through her work as the board chair of HL7 International and as a Director in HL7 Europe.

(02:19):
Dr. Skapik came to NACHC after positions in both private and public sectors, including serving as the Chief Health Information officer for cognitive medical systems as well as Senior Medical Informatics Officer at the Office of the National Coordinator of Health IT. She is currently a practicing community health primary care physician with Neighborhood Health of Virginia and a PRN hospitalist with Inova Health System. Dr. John Lee. Dr. Lee is a well-known Clinical Informaticist and practicing emergency physician and is board-certified in both emergency medicine and clinical informatics. He has served as Chief Medical Information Officer at two healthcare systems and has a passion for data-driven healthcare. He’s an active member of the clinical informatics community. In 2019. He was awarded the prestigious physician executive of the year recipient from HIMSS and AMDIS. Debra Umlauft. Debra Umlauft is a Product Director for Cardiology Solutions at GE Healthcare and brings a wealth of experience from clinical and administrative roles in prestigious medical centers. Having served as a clinical product manager and director for industry leaders like Fujifilm, TeraRecon and GE Healthcare. Debra is a driving force in healthcare workflow technology innovation, passionate about addressing fragmented data and pathways. Debra is committed to revolutionizing cardiology products and processes to optimize patient outcomes and enhance health system efficiency.

(03:50):
Alright, so I wanted to tee up this discussion a little bit today with some trends that we’re seeing in healthcare that intersect with technology and data, and I bucketize them, it’s not meant to be exhaustive, but there’s things in those first two buckets that are influencing the last bucket. So we see things in labor and economics, there’s a shortage of providers, there’s just not enough providers to care for people that need it, and that’s probably not going to change. The cost of care delivery is increasing and health systems are seeking big ROI investments, especially in digital. If you can’t show an ROI within 12 months, then it’s hard for them to pull the trigger on that investment. We’re seeing value-based model shape up that’s driving the combination of clinical and claims data. In care settings, we’re seeing decentralization of care. All care doesn’t happen in the hospitals anymore.

(04:45):
We’re seeing a shift to home care and hospitals are left with sicker patient populations to care for. In addition, there’s more patients with chronic conditions to keep out of critical care, and that requires different methodologies or remote monitoring that rely on data. And then on the technology and data side, we’re all searching for efficiencies, innovation and revenue generating opportunities. There’s a huge push for interoperability and data quality and really what we’re going to focus on today is the trend of automated AI-based clinical decision support that incorporates multimodal data diagnostics and patient records, vital signs, et cetera. I have a bunch of trucks on the screen there. I think of health it as kind of the things that are under the hood that have to flow back and forth to ensure care delivery. And sometimes those things can go very well and sometimes they don’t go very well. But when you think about a truck, a self-driving truck, it has to know the weather, it has to know its tire pressure, the road conditions, there’s all types of sensors inputting all different types of data. So I think it’s a good analogy for where things are going today.

(06:01):
So some of the questions that we’re going to tee up here, we have all the technology. It’s easier than ever to deploy software to deploy AI models, but are we solving the right problems because our technology guided in the right direction? And finally, just like the trucks, if you open up the hood, what does the data under the hood look like? That’s what we’re going to talk about. Here we go. I’m going to start out with Dr. Lee and Dr. Skapik. If you could describe a few examples of some of the AI-based CDS that you’ve worked with and from your perspective, what is the need that they’re addressing?

John Lee, MD (06:44):
Well, I guess I’ll go first. I remember when I first started in clinical informatics, I thought that this would all be relatively easy. And I think emblematic of that was when the sepsis frenzy started 10 ish years ago, my director, emergency department director said, Hey, why don’t you just fire something for all the service criteria in the emergency department? And if you’re not familiar with service criteria, its a set of vital signs that occurs with sepsis. Unfortunately, this set of vital signs also occurs for probably about half the patients in the emergency department. So it wasn’t that helpful. And so that trajectory, in that trajectory, what we did was we started adding more and more logic and to the point where instead of having a four or five parameter logical alert, we ended up with something that had 33 parameters stitched together with just this massive boolean expression.

(07:48):
And then the epic predictive model came into play and that had I think 150 parameters. And then you think about the stuff that’s going on right now, I mean GPT-4 trillions of parameters. So that is kind of the trajectory of where we’re going and it’s becoming more accurate because of all the data compute power and the new algorithms that we have in play. But at the same time, I don’t think we’ve solved the workflow problem very well. It’s still very, very clunky. And I think, again, we’re getting there, but it’s, it’s not there quite yet. Julia.

Julia Skapik, MD (08:32):
Yeah, I have other things I could say about the sepsis model, which I also use, but I’ll skip over that and I’ll say, I think right now the big opportunities are automating things that may or may not require decision-making because there are some risks and problems with replacing that clinical mind with the machine that we may or may not understand how it works. That being said, in the community health center space, and I’ll bring this up, even though we’re talking about acute care, because community health centers represent underserved populations, and I like to take an equity first approach to a lot of technology. So if we’re not designing things for people who have access issues and health related social needs and we’re not using their data to make these things work, then the likelihood that those things don’t work for those people goes up and up. That being said, I’m excited about some of the opportunities to use AI to interpret physical findings. So a person could be applying an exam device to a patient in another room, another state, and then AI can help to interpret those signals. And I think that’s an appropriate use of AI. We can train on a lot of data, we can get a diverse population and then we can get some of that feedback and it doesn’t replace clinical decision making, it augments it. And I think right now augmenting clinical decision making is a real value add.

John Lee, MD (10:12):
Yeah, and with that in mind, I think one of the things that’s starting to happen is that there’s kind of a fuzziness between actual outright decision support versus I guess information delivery support. Because if you have the right information, then it’s easy to make the right decision. And so you’ll start seeing things like we have this tidal wave as this avalanche of data and information that we need to consume to take care of the patient correctly. And it’s very difficult sometimes to parse out and pick the wheat from the chaff, but now we actually have much more advanced tools like LLMs to be able to then determine what are the things that I need to know for this particular patient at this given time. And I would actually venture to say that I think that what’s going to also start happening is that those tools are going to start being delivered to patients so that they can actually make those decisions and be more aware of what they need to do as well.

Julia Skapik, MD (11:15):
Yeah, I think that’s right on because the reason a lot of clinicians are experiencing burnout is information overload and AI is an appropriate tool to use to visualize data more effectively, to draw out pieces of information that may be missed and to sort of highlight patterns. If a lab value today is slightly abnormal, but the last one was more abnormal, that’s a very different finding than if it’s slightly abnormal and before it’s always been normal. So helping to show those trends and highlight those changes is really useful in making decisions more efficient and making use of the record more efficient as well.

Steve Emrick (12:01):
Thank you Dr. Lee and Dr. Skapik. So now to Kelly and Debra. So both of you are at the forefront of innovation and developing and delivering technologies that can be deployed in these high acuity settings to improve care. So can we take a moment to tell us a little bit about the solutions that you’ve delivered to market? We’re going to start with Kelly and TriageGO, and then we’ll go to Debra with CardioVisio and other GE Healthcare solutions.

Kelly Sager (12:31):
Sure. Thanks Steve. So I work for Beckman Coulter, which is a Danaher company. We are a diagnostics company. So we realized some time ago that in order to be the best partner to clinicians in their diagnostic decision making, we really needed to evolve beyond just running lab tests and sending back lab results. We understand that lab results that our instruments produce are just one of a myriad of data points that a clinician needs to put together. And we heard Dr. Lee say avalanche of data. You’re exactly right. This is a perfect use case where AI can really help synthesize that. We can rapidly grab all the data out of the electronic health record, put it together with lab results, and then deliver better insights that can help you with that diagnostic decision making. So we have built a CDS platform that integrates with the electronic health record, and then that platform is set up to host AI-based decision support solutions.

(13:26):
We have one on the market today called TriageGO. There was actually a great presentation about it earlier today with two of our users from Hopkins and Yale talking about how they’ve applied AI to help with that very first decision that’s made in the emergency department, the triage decision, and really move from a resource-based model, which is kind of the classic model use in emergency departments today to an acuity based decision, which patient really needs to be seen first and make sure that patients who were previously maybe sitting in the waiting room waiting when they shouldn’t have been are actually pulled back to a bed sooner. Steve just put, that’s a screenshot of what it looks like embedded inside of the EHR. It uses all of that electronic health record information, uses vitals, and then makes an acuity base recommendation to help that triage nurse decide who should be seen first.

(14:19):
We’re also working on a portfolio of other decision support solutions. This is perfect examples were listed off around the ways that AI can really help, especially in the emergency department. I’m sure Dr. Lee, you can talk to this, there’s such a high cognitive load. Anything we can do to use AI to bring together the data quickly, pull up what are the most important relevant aspects of data. We’ve heard over and over from our customers that it’s not just a risk score itself that’s valuable, it’s all the contributors. And why are we saying this patient is at high risk for sepsis? Why are we saying they’re at high risk for a major adverse cardiac event? The AI can pull that out of the record and then bring those things to the surface and put them right there at the fingertips of the clinician. So clinicians aren’t spending their valuable time digging through the record trying to find all those pieces of information.

Steve Emrick (15:21):
Thank you, Kelly. Thank you, Kelly. So Debra, do you want to speak to GE Healthcare?

Debra Umlauft, MBA, CNMT (15:27):
Yeah, thank you very much. So at GE Healthcare, as far as workflow solutions, we really have a three-pronged approach to helping our patients and our customers through AI solutions. And we call it the D three strategy, and it’s really taking care of all parts of the workflow management of patients, and they can come through different ways. So when you’re taking care of patients in a high acuity center setting, one of it is you have to get these patients through as fast as possible. And some of it’s because it’s lifesaving. If somebody’s coming in for a stroke, you need to have your diagnostics as soon as possible, get that information as quickly as possible to the doctor. So you want to make sure that workflow solutions through disease states, having smart devices that can actually leverage advanced technologies to better scan, faster, diagnose, get reports, and automate some of the processes.

(16:17):
My particular product is called CardioVisio, and it’s a workflow solution, and what we’re doing is it’s multimodal consolidation of information. So as far as workflow solutions and optimization, we need to integrate it into the areas where you’re using the solution. So that would be either through our Centricity PACS or it could be through Epic, but we want to make sure those solutions are available through the workflow that you’re already doing. We don’t want to make extra clicks, extra workflows for you, make it as simple as possible, but it’s for the care and management of people who have really chronic conditions and it’s multimodal. So you can imagine it’s hard enough with just some data such as blood pressures and things like this, but patients with atrial fibrillation and chronic conditions, they have much a lot of data that’s coming from different sources. It can be PDF files, it can be images, it can be devices and things like this. And a lot of them are coming from AI devices, but we need to consolidate this information that’s coming from different sources and very unorganized today. So how do we organize it and bring that together? And that’s what we’re doing at GE Healthcare. We’re making sure that we can automate these processes, make it better, organize it, then help them make clinical decisions using clinical decision support, which is directed towards evidence-based care, so the guidelines that are supported by the American College of Cardiology.

Steve Emrick (17:36):
Thank you, Deborah. So as you were talking about these solutions, I was thinking that all the triangulation of the multimodal data that you’ve discussed is something that clinicians have had to do on their own in their head for a long time, but I wanted to take a minute and get Dr. Lee and Dr. Skapik’s reaction to anything that Kelly mentioned or Debra mentioned.

John Lee, MD (17:56):
Well, I think it’s a continuation of what we said right before a few minutes ago in that the, there’s this concept and kind of a cognitive bias that decision support in these systems is something that pops into your head or pops onto the screen. And what we’re seeing, at least the places that are doing this right, are actually delivering the information that you need so that then you can make the right decision. So your TriageGO product, what it does is it coalesces a ton of information and gives you basically one piece of information that’s an abstraction of this patient is going to die, this patient is really sick. And then you have all the others. And if you think about that’s really important in the emergency department, it’s not telling you what to do, but it’s giving you that abstraction and that consolidate and your product, it’s taking a lot of these cardio diagnostic tools. It can be a complete mess and you need to have kind of a summarized, abstracted, single source of truth or close to truth. And that’s another thing that I would point out. We don’t have to be perfect on this sort of stuff because our analog world is way far from being perfect. So I would contend that we need to just get good enough, and I think these AI solutions are definitely getting us there because we have the data, the compute power, and the algorithms now to do this.

Julia Skapik, MD (19:36):
And I think one of the opportunities for AI also is to help us identify sort of information mismatch. So I can’t say how many patients I admitted from the emergency room who are billed to me as a chest pain rule out, and the patient’s like, no, I said it’s abdominal pain.

John Lee, MD (19:55):
That never happens.

Julia Skapik, MD (19:57):
I know no shade right on the ED, but being able to highlight and actually incorporate the patient’s own words into that documentation so that we get a better story can help us avoid diagnostic pitfalls and potentially harm to the patient. So I think all those pieces together can really add a lot, especially in the middle of the night. Right.

John Lee, MD (20:19):
Yeah. And I think the newer more modern tools like LLMs and NLP will really help with that, where I think we’ve only reached that inflection point of usability within the past few years. And I think prior to that, we were still largely relying on 1970s, eighties ish rules-based expert systems, even though the rest of the economy, and everybody else was using things that are far more sophisticated and complex.

Julia Skapik, MD (20:50):
Thinking of the EKG readout?

John Lee, MD (20:52):
Well actually that is actually more complex than some of the stuff that we were using in the EMRs, like my CERS example. It’s heartening to me that we have these things available.

Steve Emrick (21:08):
Awesome. Thank you. So when I think of the basics of interoperability and data quality and the challenges that we have today, even with things that are relatively more simple, like there’s a lab test and if you look at how that lab test might be mapped to a loin code, there could be great variation across 10 different facilities or hundreds of different facilities and the problems that causes downstream. So I just wanted to take a minute for Kelly and Debra, when you deploy these solutions, you might have to tailor them a little bit to the needs of that particular health system or that hospital. Can you shed a little bit of light on to what degree you tailor these solutions and how that kind of works?

Kelly Sager (21:52):
I can go first. We have found that this is incredibly critical. When you train a machine learning model, the training dataset, we get it, we harmonize it, we clean it up. The training dataset looks great. In the real world, the data is often spotty, lots of missing fields, and that really is one of the critical factors that can affect the performance of a machine learning model. If you’re not prepared to handle data missing this, it can be a big problem. And practice patterns are different. We find talking about the sepsis example, you brought up that there’s a lot of differentiation, hospital to hospital around how often they order lactate, for example, do they use lactate at all? If you don’t take those things into account, then you can get very different performance hospital to hospital. So what we do with TriageGO, we actually custom train the ML model based on each hospital’s data. And we find that that’s really the best way to get really good performance because it’s customized for the patient population, customized for the practice patterns of the clinicians, and then we can really guarantee it’s a solution for them.

Debra Umlauft, MBA, CNMT (23:07):
In regards to the consolidation of medical data, I don’t know how many people are from the medical imaging world, but I lived through the consolidation of PAC systems back in the day. And just for the medical imaging piece of it, trying to map that with all of the variable private DICOM standards was one issue. Now, if you add all of the different LOINC codes and FHIR codes, all of those things are not standardized as much as people want to do. They are not standardized. So the deployments are quite complex because when you have one organization, they can be standardized but not always between departments. And now if you have consolidations of medical centers, now you have multiple centers who do not have consolidated information now trying to consolidate information. So it becomes quite complex to try to normalize that data and it, it’s quite a challenge. So each one of our deployments are actually really a custom job because there really is not a standardization of all data across all medical centers. And this is really is a challenge.

John Lee, MD (24:15):
I actually think that there was one statistic I recently saw that said that writ large of all the lab data that exists in our national healthcare system, something like only like 20 or 30% of it is actually mapped to a LOINC code.

Julia Skapik, MD (24:35):
Yeah, and I remember I watched a really fascinating presentation from Simone at Mayo who was trying to bring in external COVID test results during the pandemic. Only 13% of hundreds of thousands of lab results that they received had enough provenance and metadata to be usable in the clinical system. So you can imagine if we try to apply AI to something, which 13% of the data meets the criteria, what are we going to get? We don’t necessarily know.

John Lee, MD (25:05):
Yeah, if you try to add two plus two and the numbers that you’re given is 1, 3, 4, 5, 6, 7, 8, 9, 10, it’s hard to do.

Steve Emrick (25:14):
Well. So I’m hearing that there’s, while we’re able to deploy these technologies, there’s maybe a gap in the foundations in terms of data quality that’s not yet addressed. And that seems to be a repeating theme. And I’m wondering how big is that gap and how far are we away before we’re able to trust these solutions a little bit more? And it’s easier to deploy them across different facilities or settings without such a big lift.

Julia Skapik, MD (25:49):
So it won’t surprise you that my answer’s going to involve standards. I think AI can help us to use standards better and more effectively, and it can tell us when we’re not applying standards to mappings, to messages and how to put those things together more efficiently. But there does have to be a human component right, because people are entering most of this information. So doing a better job of capturing the data and standardizing it will result in the better performance of AI. But I really loved the points about customizing to the home data set because I think that’s really important. I’m sure most of the people in the audience saw the article about the rapid drift of clinical decision support in AI context, and that’s pretty scary to me. I mean, even if I learned to trust something today, will it still give me the same results in 3, 6, 12 months from now? I don’t know if it will or not based on some of what we’re learning.

Kelly Sager (26:56):
That’s why monitoring model performance over time is critical. You really need to proactively monitor it and then know if you need to retrain the model because that is a concern.

John Lee, MD (27:08):
And I think you touched on it a little bit. I think ironically, the biggest opportunity I think in the short term that we have to improve things is actually use our AI tools to improve the underlying data that we’re feeding into the system. Unfortunately, I don’t know if this is your experience, the people who end up making a lot of the resource decisions don’t understand that and they want to jump to the shiny object and what is it ForwardHealth. Are you guys familiar with ForwardHealth? The big basically Edison box that I don’t know what they’re going to do with it, but instead of focusing on that, I think our organizations really need to focus on the data quality.

Julia Skapik, MD (27:53):
Yeah, I think my answer to the c-suite people on that one would be to help them understand that the data contains so much value in it, and the better the data, the more value that’s there and the opportunities that come out of using that data. Most of the data in healthcare is not used for anything. It’s just sitting there. It’s taking up server space.

John Lee, MD (28:15):
So how can you tell a story to somebody who doesn’t know? Let me give you an anecdote.

Julia Skapik, MD (28:21):
With pictures.

John Lee, MD (28:22):
Well, pictures and simple stories. I was called into my Chief Medical Officer’s office once and he said, John, I don’t know why you’re so fixated on analytics and dashboards. I already know what I need to do, so don’t bother me with any sort of information.

Julia Skapik, MD (28:39):
And that’s when you say, here’s what percent of the time you did it.

Steve Emrick (28:51):
Okay. Well, I want to take a moment to thank our panelists, and I also want to take a few minutes to open up the floor for any questions. I will shuffle around the mic for anyone who has a question for our panelists. There must be a question out there.

Kelly Sager (29:10):
So while people are thinking of questions, I think you just made an excellent point at the end that I just thought would be worth expanding on. Here’s how often you actually did it. This is another key metric that we have realized to really measure the AI, is to look at outcomes, what actually happened, and then close the loop and find out was the AI prediction actually useful? Does it actually tie to the outcomes in the emergency department? The triage use case that we work on triage don’t actually know what happened to the patient later and they have no way to then educate themselves on was that a good decision or was that a bad decision? Because no one is closing the loop for them. And so I think this concept of closing the loop and being able to bring that information back, that’s another opportunity that AI solutions can play a role in, is helping to make predictions and then close the loop and report back metrics to find out how accurate was it, AI.

Julia Skapik, MD (30:08):
Yeah, in our EHR, I actually kept a list of patients. I wanted to know what happened to them, because otherwise I would never find out.

John Lee, MD (30:16):
So let me be a little bit provocative then here for you, Kelly, what if you instead of say employing your product, you created a program where you reported back to each of the nurses on the quality of their triage assessments. You do? Yeah. Well, okay. We don’t do that. I wish we did. That’s a great idea.

Kelly Sager (30:44):
Support card. We have nurse report cards for each nurse that helps show their agreement with the model because there was another great point earlier. It’s the human plus the AI. That’s the powerful combination. There’ve been studies that show AI alone and humans alone are both inferior to human plus AI together. And so we actually do that. We actually produce report cards to the nurse and show them, here’s how often you’re agreeing or disagreeing. We really don’t want to see a nurse agreeing 100% of the time. It means that they’re not using their clinical judgment, nor do we want to see them only agreeing 30% of the time. And then we show them, here’s what actually happened. Here’s the percentage of time that you down triage, meaning you didn’t listen to the AI, you triaged lower, but that patient went on to be admitted to the emergency room. So maybe the AI knew something you should listen. That’s really key.

Julia Skapik, MD (31:36):
Yeah, and I think I’m buying in a little bit to the AI acronym as augmented intelligence as opposed to artificial intelligence. I mean, I think in the example that you just gave, sorry that you just gave, John, that I would try to sell my C-suite on saving time and money first with AI making clinical tasks less burdensome. My dream is information is missing and AI just goes out and finds that information and puts it right in the place that I needed to make that decision. And I don’t know that there’s a lot of decision making really happening there. It’s just going and finding what I need, but that will save me time and it will make me make better decisions. And so I think if we focus on reducing wasted effort that we’ll start getting our money back right away and then we can delve into CDS if we have some naysayers that are sort of holding us back.

Steve Emrick (32:34):
But it’s true. There’s definitely an opportunity to leverage these kind of data lifecycles to improve learning across the clinical system. So thank you. I think Anne has a question.

Audience Speaker 1 (32:46):
So this morning when we were chatting, we chatted a little bit about the intersection of clinical decision support and clinical quality measurement. So now we can have an intersection. Where do you see that intersection of clinical quality measurement, clinical decision support and AI feeding into that?

John Lee, MD (33:09):
I actually happen to be working on a project with that where the clinical data that feeds some of the quality metrics are actually available within the transactional database so that people know real time whether their patient is at risk for a falling out of some sort of metric. It’s not quite there yet because the data is of dubious quality. But that’s I think where we want to eventually go where the reporting and the decision support all kind of get squished into one sort of set of data that the data can be used for both reporting and the data that can be used for transactional care.

Debra Umlauft, MBA, CNMT (33:56):
I think all of this data is so useful when it comes to analytics. So when you have all of this information, you maybe do quality reporting, things like this, but how do you know that you’re better? How do you know that you’re getting worse? What is the outcomes of people? I think without analytics driving and supporting the information, you have to be able to measure and support what you’re doing. So I think to answer your question, I think the analytics, for me anyway, I see that being very important component of being able to measure what you’re trying to do

John Lee, MD (34:35):
And to feed you a little bit of an HL7 softball. All that stuff has to be normalized so that if something bad happens at my hospital, it means the same thing in terms of data wise that happens at another organization. So we could benchmark each other because ultimately it’s not just quality. The reality is that it’s comparative quality and you need to know where you’re falling out and where you’re doing well compared to your peers.

Julia Skapik, MD (35:07):
And I’m sure there are folks in the audience who have PTSD from clinical decision support designed to support meaningful use stage two quality measures, because that’s a good example I think, of how not to do decision support. That being said, in FHIR, the standards for clinical decision support and quality measurement are in fact the same pair of standards. They’re in FHIR clinical reasoning, they use CQL, and I think that you’re going to get a much better quality measure outcome if you’re augmenting that data entry and that activity with clinical decision support. And I think the measures have gaps in them too, and helping a person be able to say, sorry, you’re wrong. It’s not right for this patient. Hopefully people are going to look at that data and use it to make measures better and make patients perform appropriately.

John Lee, MD (36:02):
So yesterday I did a talk on how AI is surrounding us, and the example that I used was the keyboards that we use on our phones right now. If you type in THW, it will actually put in THE because there’s not many words that start with THW, but most of us don’t understand that there’s an algorithmic process behind it. That’s where we need to get with our decision support and information delivery tools so that it’s so seamless that it’s the Arthur C. Clark, any technology sufficiently advanced that is indistinguishable from magic. That’s where we need to go and we need to improve our data quality to do that.

Steve Emrick (36:48):
Amazing. Thank you all. Is there any other questions from the audience? All right. Well please do give a big round of applause to our panelists. This has been a very enriching discussion. Thank you so much. Thanks for having. Thank you.