Closing the Blind Spots: Designing Devices for the Real World

In three… Okay. Good morning, good afternoon. Welcome, everyone, to, Blosing the Blind Spots, Designing Devices for the Real World. I'm Sean Smith. And… I can figure out how to advance my slides. I will be in good shape. Hmm… There it is. Sorry. So, we're here to talk about, What are we trying to talk about? We are here to talk about closing the blind spots. I wrote an introduction. So I'm just gonna introduce today's speakers, because obviously I'm incompetent, and

But before I do that, some housekeeping announcements. Here's what you need to know. This session is being recorded. All registrants will receive the recording within 24 hours. Slides may be available, I understand, they are. We will distribute those along with the recording. For certificates of attendance, this is important. Please drop your name as you would like it to appear in the chat on your certificate of attendance. And… your email address with that, and we will send you a certificate of attendance. Feel free to share those certificates of attendance on LinkedIn. or your social media platform of choice. Use the chat when communicating with me and the panelists. Use the QA button for submitting questions to our panel.

This webinar is the first collaboration. Between Project MedTech and MedTech Leading Voice. It's sponsored by Clarimed. Thank you to our sponsors, collaborators, speakers, and especially all of you for attending today. Quick reminder, please subscribe to MedTech Leading Voice at MedTechLeadingvoice.com. I'll drop that in the chat, and the website for Project MedTech. One final reminder, in 8 days from now, ClareMed will be having the guest speaker on our Friday in-focus call every Friday. MedTech Leading Voice Exchange does an in-focus call with industry leaders, it's a live webinar, everyone is welcome to participate, so I'll drop that in the chat as well.

somewhere along the way, we're going to do a three-question poll, and after the call, there will be a survey. Please, participate in all of those. Now, for today's speakers, I'd like to introduce today's speakers. Welcome, speakers. Kelly Kendall, CEO and Chair of… chairman of the board of Claramed. Kelly is one of the Claremed is one of the fastest growing… Human Factors Consultancies in MedTech, helping companies align product design with both regulatory expectations and real-world user needs. Dr. Amelia Hufford is co-founder and Senior Vice President of Clinical and Regulatory Science Operations at 3Aware. Amelia specializes in leveraging real-world evidence.

To uncover insights that traditional controlled studies often miss, strengthening both device design and regulatory submissions. Dr. Steven Levine, Senior Director of Virtual Human… modeling and Executive Director of the Living Hard Project at Dassault Systems, Steve pioneered the application of aerospace and automotive simulation technology to medical devices, creating the world's first commercially available beating heart simulation. It's amazing, you should watch the video on YouTube. Dr. J. Vaishnav, Director of Regulatory Affairs at Cannon Medical Informatics and former FDA reviewer. Jay is gonna kick us off today with an introduction. Jay brings a unique perspective from both sides of the regulatory table, understanding how to transform compliance challenges into strategic advantages.

Again, a reminder that join us next week, and I'm going to stop sharing and turn it over to… Jay, to get us started. Thank you.

Welcome today. And thanks for putting this together, it'll be fun. So let me get up screen sharing, and Alright, one sec, sorry. Yeah, there we go. Alright, so I get the fun job today, I get to talk about the problem, and then we get to listen to smart people solve it, so, or talk about how to solve it, so yeah. But here's the problem. Really, the theme of my talk is that you… your device is only as good as the population you validate it on, so that when you test it on a few, but you deploy it to many. What you end up with is blind spots in the real world. And so, why do I care about blind spots? Well, I'll just tell you a little bit about me. Sean, thank you for the introduction, and I've spent the last 25 or so years bridging science, innovation, and regulation, mostly in medical imaging and AI.

I've worked at, NIST FDA, I started as a physicist, I've worked at NIST FDA, startups and multinationals, and, I've, also, we have this book that, has, Chapter 11 on human factors and, Real World Evidence, Chapter 29, in case people are interested. This is on, through the Regulatory Affairs Professional Society. We have another book on, medical imaging, that one has a free chapter on AI, so I encourage those to check people out, encourage people to check those out if. they find these subjects interesting. So, but in my time in tech and med tech, something I've just seen over and over again is that great technology stumbles because the validation didn't actually reflect the real-world use population.

So, every device, whether or not it has AI in it, every device, performs safely only within the populations that it's validated on. And so what happens when you test narrowly, or homogeneously, is that you actually build blind spots into your device, and when I was at FDA, one of the first things I was educated on was to look very closely at this. The evidence the company provided had to match the actual intended use population that the device was labeled for. And if your study leaves people out, you can't claim that your device works for them, and so if you only tested the tip of the iceberg and there's a whole iceberg beneath it, that's something you need to be really aware of, because it's not just a regulatory problem. It's a patient safety and a business problem, it's this big shark here. So,

that's really the framing, I think, for this panel. And that was abstract, but here's what it looks like in practice. So, there was a review article that looked at 21 public dermatology datasets. These are data sets on which, you know, many models were developed. And they had over 100,000 images. Now, of those 100,000 images, this gray here, only 2%, under 2% actually reported the skin color. or ethnicity. And of that 2%, do you know how many patients actually had darker skin? It was 11 out of 100,000. And… So, that was the data set. Out of 100,000 patients, we only had 11 who were confirmed to have dark skin, and people trained those models, and they tested them, and they had these wonderful AUCs of above 0.9. But then somebody said, let's actually test this model. Of course, real life is not, like, this data set, right? Real life has all sorts of skin types, and so, the models, they looked really great when you tested them on this, you know, data set, but

when you actually took them to a more diverse dataset that reflected real-world populations better, well, it didn't do that well for anyone. You see, the AUC was 0.62, but actually, for Black and Hispanic patients, the AUC was 0.5. That's like a coin toss. So that was for diagnosing melanoma, and it completely missed it. It was as good as chance. So blind spots in your dataset become failures of your model, they become blind spots in diagnosis, and so that's sort of what happens when you validate narrowly, but you try to scale your device broadly. You know that FDA has authorized a lot of AI and CAD in other spaces, one of which is my medical imaging, but so far, dermatology is a little bit stuck because there just hasn't been enough representative or robust data to support its safe use in the intended population.

And here are two papers, you can read more about this if you want. And… you know, we talked about AI, but this isn't just an AI problem, it's just that AI is kind of like gas on a fire. But let's go back to a much older technology, which is pulse oximetry. Pulse oximetry isn't new, you know, it's that thing you stick on your finger that measures your SpO2. They've been on the market for decades. Now, historically, who were they validated on? They were validated on people who signed up and volunteered for validation, who were mostly light-skinned individuals. The problem is infrared interacts differently when you have melanin in your skin.

And so during the peak of COVID-19, we actually learned the consequences of that, and there were studies showing that Black patients were actually 3 times more likely to have occult hypoxemia. That means they had really dangerously low blood oxygen levels that were missed by the pulse oximeters. And since that time, FDA has been, you know, aware of this, has now convened advisory panels, and they're reviewing new performance standards, and my thing that I want to highlight, this is a completely different technology, but it's the same lesson, that when your validation doesn't reflect the real world, you have the potential to create patient harm.

So, over my career, which, as Sean said, has been on both sides, or really all sides of the table, because I was in academia too, so, here's some pitfalls that I've seen repeatedly. Companies, or entire sectors sometimes rely on retrospective data sets, or ones that are homogeneous. Or what they do is they validate in very narrow conditions. They validate using experts, clinical experts, or at only academic centers, and not at a community center with a really messy workflow. And they assume that if it works in dev, that it's safe, it's equitable, it's regulatory grade, and often it's not. So, as I said, when I was at FDA, the very first thing that was kind of put into my head was always make sure that the data reflects and represents the intended use population.

So, as I… I just keep one of hammering home, your validation defines your device. If you tested your stent on men, you can't say that it works as well on women without checking that. If you test it on adults, you should think about pediatrics. You tested on readers with, you know, 5 years of… who are physicians with 5 years of post-fellowship experience. I don't think you can assume that, you know, a nurse or a resident in a busy emergency department will have the same performance with your device. So, there are many things that you have to think about, and Narrow or homogeneous validation leads to blind spots in the real world.

Now, as I said, AI is kind of gas on that fire. Why? Because your training data actually get baked into your device, they're actually part of your device. And we see this now that FDA is reacting to this, and other regulatory bodies are reacting to this. For example, in the good machine learning practices, TMLP, there's an emphasis on representative data and on real-world validation. So, what does all this mean if you're an innovator? Well, the stakes of this are very real. If you're a patient, then blind spots can mean a misdiagnosis or delayed care. If you're a regulator. it means that your market authorization more and more relies on you having population-level evidence. That's something you have to look at as far as your regulatory risk. If you're an investor or payer, you gotta think about, you know, is the device that you're going to pay for equitable and generalizable, its performance?

And finally, reputational risk. If you're a company, and you deploy, like, this device to a hospital, and they come back and say it doesn't work on half their patients, it's hard to regain the trust of a physician once you've lost it. So, yeah, once the clinicians see that blind spot, it's difficult to rebuild trust and reputation. So these blind spots, they aren't just something technical, they're really… especially if you're a startup, they're an existential risk. Okay, so what should every… we all take away, and like I said, I really… my job is really to frame this problem. If you don't test it, you can't claim that it works.

if you don't train AI on your populations, your AI won't know how to diagnose them, so we saw that with melanoma. It had never seen melanoma in darker-skinned patients, it didn't know how it presented, and it wasn't able to diagnose it. So, how do we actually build validation that scales across the real world? Because it's hard, it's expensive. And today's… what we're going to talk about are some of the tools to help make it easier, so human factors engineering, real-world evidence, and in silico modeling. Alrighty, so that is all from me.

Thank you so much, Jay. Next up, I'm going to introduce Kelly Kendall. Welcome, Kelly, from Claramed. Again, Kelly is the CEO and Chairwoman of the board. I don't… I don't know how you say that. Anyway, Kelly, Let's dive in.

Great, Sean, thank you so much. Sean, are you going to show… throw, show my slides, or do you want me to…

No, you can just share your screen, that's fine.

Hold on one second here…

Sorry. Video's off. While she's looking for that, I'm just gonna launch the poll.

Trying to find…

So, if everybody will kindly participate in our three-question poll while she's sharing her screen, we'll run this for a minute and a half, and…

Can someone confirm they can see the right thing?

Yes, yes, thanks.

Okay, good. Well, Jay, thank you so much for teeing this up for us here. I couldn't agree more, in terms of Yep. needing to have representation, and the challenges with validation, and what comes after that. And it's really easy. for us to think about human factors, and associate it with validation. And it's really important, as you said, to… Ensure that we have a representative population of, you know, the intended users. And not only the intended users, but the probable users, of the technology that we develop, and the agency is definitely attuned to that. And, you know, when we think of validation, we think of it from a…

risk standpoint, right? And I think what Jay lays out really nicely is… This is a… This is a business. problem. It's bigger than just a regulatory problem, right? Because it's really about Market adoption. And what does market adoption mean? It means market sizing, and it means commercial success, right? So, often we put HF validation into this. regulatory box, this checkbox, and I think it's misunderstood. It's certainly an important step, and when I think about it, and why I build a business, that is with usability and human factors as its foundation, is because I…

I think it just makes business sense, so I am not an expert. You will hopefully dial in when Kay speaks, on next Friday. She is an expert. But, you know, from a logical standpoint, not only do we have an opportunity where human factors validation, can sometimes be seen as a pain point. But really, it's about thinking about usability and human factors throughout the development to reduce risk, but to optimize design and optimize your market potential. And so what I'm hoping to talk about today is you know, and I'm sure I've seen some of the people that registered. I'm preaching to the choir, right? How do you think about

usability, holistically, through design and development, and really, I think all of the tools that we're going to talk about today it's not a one-time thing, right? And they all are very complementary to getting to the heart of the problem, which is… How do you… How do you truly understand, the environment the potential user groups and the opportunity, because what I see is not just risk, I see opportunity cost. And then, you know, my next slide… If I can get to my next slide… is, how do we get here, right? Because no… everything I just said is not rocket science, right?

Everybody wants to believe that they're doing this. But the reality of how development starts, is you've got really smart people. Engineers, scientists, clinicians, That see a problem. Or have a technical solution to what they believe to be a problem. But for each and every one of us, whether we like it or not, our worldview is narrow, right? And… we tend to hang around with people that have similar experiences to us. And so… From day one, you've kind of got this confirmation bias of, you know. it's what I need to do my job, or the person that I spoke to at the hospital down the street agrees that this is something that's needed, or what have you, and

the work, because it's extra, it's seen as extra, doesn't really get done to understand what are broadly the pain points, what are broadly, potentially, the application? And we go down this path. of development, of getting… to… a prototype. And… We put it in the hands of those people we know. And we get… That feedback, and we haven't really pressure tested in all of these different ways, to see are there other opportunities? Is there a way to optimize this design to work for a more diverse population? Have we considered all of the elements that

Jay spoke up, and… while we… Think about, you know, design input, and, industrial design does Pressure test to some… degree. It's really about this kind of broad understanding of environment factors, broad understanding of, ethnographic information, that… really needs to be pressure tested really early. And it's not… just about… designing the best product. It's really also about conversations with investors. Right? And I know that that is something that's really important to many early, development teams, and it's something that, you know, even within a larger organization.

you've got to… you've got to fight for your innovation within a broad portfolio of individuals, that may also have a good idea. How do you justify this? And I think that often this… is put to the side, or we look at the space based on what other people have done, and what other technologies there are. So… By the time we often see a product, We are at Design Lock. Right? Money has been spent. Promises have been made. we don't want to make those changes, but now we're putting them, and as Jay pointed out, there's requirements to put it into diverse populations to ensure

Safety parameters are met and mitigate risks. And… now… It's the first time that we're truly testing on this broader data… data set. To go back and fix something, The cost is much higher. And more than the cost, It's time. Time to market is the most important thing, right? From an investment standpoint, from a competitive standpoint, from an opportunity standpoint. And we're putting that at risk. Often, we are challenged with, how do we mitigate those risks? Yes, are we able to work through that? There's different things that we can do from a training standpoint, and other tools, that we offer. However.

We miss the opportunity, because sometimes those risk mitigation activities Also… Increase your cost of your go-to-market. Delay your launch, because now you need a commercial training program that you didn't think you needed. Or we have to narrow the label, and the intended user groups so that we can Put this forward and have regulatory success with the agency. So… again, I think I… I've said it, I'm kind of preaching to the choir here. Well… we are still happy and always happy to help people with their challenges, and as they're facing, building their strategy for human factors validation. What we do want to reinforce is

We can make life a lot easier for you, both in, setting you up for success with your validation program. But even more so. You know, optimizing the investment that you're making early on. Help you fail fast, so that you're narrowing and optimizing your design. Such that you really have something, once it gets on the market, that Your users want to use, that truly solves their problem, and that you've optimized your market sizing, so that you can maximize your market penetration, that you do have a competitive product. that we understand the opportunity for Gen 2, right? The power of understanding that opportunity

once you're launching Gen 1, means speed to market. for the next generation, it means you're gonna hold that market position. So my message, and I think the message of Amelia and Steve will be, you know. this is an iterative process. This should be an iterative process. We should be testing our hypothesis. We should be learning and confirming and tweaking, right? So, by the time we get to validation, design lock. We are… clear. We know what our risks are, right? And not just because, you know, we've done some clinical work in a controlled environment, and didn't see any problems there, but because we truly pressure tested the variations and the variables, and without guidance of the users, understanding what the pain points are, so that we can address it. We can address it in the

our instructions for use. We can make sure the packaging is clear. We know what the opportunities are to differentiate, and… That's what we hope to achieve, because the fear is early on that it's going to slow us down. it's going to cost more money than what we have, but… The reality is, investors want to invest in something that is going to have market success. And… The cost to do this iteratively pales in comparison to the cost of having to fix the problem later. Are missing out on a market opportunity. are missing out in terms of timing of your market launch, or making a promise to your investors and spending the last bit of investment dollars that you have to get to this point and having to go back. Is there money left for you to rethink it?

And we do see this. We see products that, you know, had an R&D timeline of 3 years to market turn into 11. So, each one of these tools… That we're going to talk about today, whether it be usability, real-world evidence, or simulation, helps to narrow that risk. Right? Helps you understand The world that you're going to be introducing your product to. Understand the limitations of your intended users. Understand the context of the environments that they need to use these products in, because it's very different than what you might experience in a lab.

Kay often talks about, kind of, emergency use environments, and so does my other colleague, Alper, right? The considerations there are extreme. And we need to be building tools that understand all of those factors. How can you pressure test that? You're not pressure testing it in a clinical environment. And that's what we aim to solve, is giving you clarity.

So, on that note, we have to move on, but what I… No problem. I put in the chat, you know, Claramed welcomes conversations, from anyone, but they have a new startup program. If you're interested in the startup program, please write Startup in the chat. Jay, I need to ask you to stop sharing your screen, because apparently people are able to toggle back and forth between the two screens. So, there's that. And… The other thing is, is I know there are a lot of consultants on the call, Claramed does have a partner program. If you're interested in being contacted about the partner program, just write partner in the webinar chat, and someone will reach out to you for an exploratory conversation.

in the coming days. Thank you so much, Kelly, for that, and I'm gonna move to Amelia. And.

And I'll stop sharing my screen.

And, right, and stop. Right. That's good.

Okay. Can I get a thumbs up if I shared my screen perfectly? All right, well, thank you so much, and thank you, Sean, for the invitation to speak today with this panel. I always like to start off with a, a quick definition, just to clarify the differences between real-world data and real-world evidence, though I'm needing to have to do this less and less now. So… We had a conversation, backstage with… among the panelists, and… and I, you know, I think… Kelly, you, having a pharma background, the concept and understanding of real-world data and real-world evidence is not new, but within the med tech community, it's becoming, more and more mainstream, but a couple years ago, it definitely wasn't the case.

So, I like to throw this slide up to show folks that there are lots of different sources of real-world data. And basically, the definition of real-world data is any data relating to patient health status or the delivery of care. That's routinely collected from a variety of sources, and routinely collected You know, collected during clinical practice, routine care. That's… that's the key distinction besides clinical data collected as part of an investigation. And very common sources of this data include electronic medical records. Device registries, claims database, patient-generated data, so, both in the form of wearables, as well as other data, such as, patient-generated out, outcome measures.

Active surveillance systems is another one that is cited, by the FDA in a recent draft guidance. As well as data platforms. And the distinction between real-world evidence and real-world data is really the real-world evidence is the assessment, or the analysis of the data that, speaks to a device's benefit or risk. So it's, it's the analysis of the rural data to, to answer, you know, a study question or objective. So, as I was listening, honestly, to the great presentations by Jay and Kelly, I was thinking about how, you know, real-world evidence

And my presentation today dovetails. And I think, you know, with regard to Jay's presentation, I think the comment that the device is only as good as the patients you investigate it on is so true. And when you put devices out in the wild, so to speak. and you see, you know, maybe the testing and validation doesn't reflect the real-world scenario, you do have the potential to create harm. And one thing that I think the med tech community is still struggling with is, you know, the gold standard being sufficient. And I, you know, I definitely don't think that there's perhaps a need to do away with, you know, traditional clinical studies, but those are narrow in scope by design, because they have to control sources of error and bias.

And real-world data may provide more accurate insights, from larger, more heterogeneous populations because, you know, the world is in a controlled environment. And I think, honestly, heterogeneity is seen as a limitation of real-world data. But in the context of representation, it can honestly provide a better and more comprehensive understanding of how devices perform in clinical practice. And there are ways to control for these, you know, these limitations, so to speak, and one is to monitor selection bias, and that's true, honestly, in every study. But it's not impossible when utilizing real-world data.

And that's another piece I wanted to discuss, is that I think there's perhaps a misconception that real-world data studies, real-world evidence, is of lesser quality, and there's a lot of, great. frameworks and guidances. I'll talk a little bit later about NEST, but, FDA has put together, guidance that discusses, it's currently in draft right now, but it discusses how you can, treat real-world data. How you can ensure data quality and data relevance. And really, the bottom line is that, you know, treat it with the same scientific rigor that you would any clinical study, and you can really, you know, utilize this data to,

to answer, unanswered questions that you just can't get from traditional studies. I could talk on and on about that, but there are other reasons for the use of real-world data. One is that they may be impractical or unethical to conduct. For example, if, you know, in assigning treatment assignments. If the benefits, you know, definitely outweigh the risks, and then the ethical considerations by having a randomization and controlled arm. Also, honestly, just in operational practicality, if you're needing to collect long-term data. That's very hard to do in a traditional clinical trial, and there are ways that you can approach that with real-real data that can be less burdensome to clinicians, to patients, and, to manufacturers in the way that it's collected.

Honestly, we talked about real-world data informing the benefit-risk profiles. Again, you don't really know, how… The… all of the different variables that exist, as you're running a traditional clinical study, and monitoring those variables, looking for unanticipated adverse events, looking for, perhaps, design implications. That you were unaware of, such as skin color. That doesn't necessarily apply to the broader population. All of those things, you know, are informed by real-world environments. Also, real-world evidence can expedite both regulatory decision-making, and I'll talk about this. I like to show, some case studies where real-world evidence has been used.

Not only, you know, from the… the safety aspect, but also from the… the, the… The aspect of… of… Broadening your access to medical products. in a sustainable way. Not every, you know, clinical trials are very expensive, and it's hard to, even from an operational standpoint, manage many, especially if you're a smaller medical device manufacturer, startup manufacturer. There's a lot you can utilize real-world evidence for, and I really think FDA And, and, is, is… Honestly, kind of pioneering this effort with some of their…

their programs and, and, through NUST. I'll… I'll talk about that a little bit, but, Real-world evidence can be a sustainable approach to collecting data and leveraging data and fully capitalizing that data to, you know. speak to R&D, to speak to expanded market access, label expansion. Etc. One other bullet point that I want to add to this slide after today's talk really is dovetailing with Kelly's presentation that, you know, this is an iterative process, looking at the device. through the entire product life cycle. It's iterative. You're gonna learn along the way. And human factors and usability applied holistically, I think, is a great way to reduce risk, as Kelly mentioned, and optimize the design.

And collecting that data, you know, unsolicited from the broad users using real-world data, can… can, I think, pressure test designs. Device designs, so… Again, really, really interesting to have us all together today on the same platform. Just quickly, I wanted to talk about, Nest. I mentioned FDA is a, you know, really kind of pioneering the use and the understanding of real-world data in the med tech community. Fda, several years back, awarded MDIC, which is the, Medical Device Innovation Consortium. They're essentially a public-private partnership that is… is tasked with advancing, regulatory science and medical devices. We'll, basically FDA awarded

NEST, the ability to create… or sorry, awarded MDIC the, ability to create NEST, which is the National Evaluation System for Health Technology, and they're actually tasked with advancing the use of real-world evidence and medical, med tech for regulatory decision-making. And they've created frameworks, as I mentioned, for data quality and methods for utilization of real-world data. And they also have a coordinating center that, are, are trying to do, various initiatives. So they have one initiative called the Nest Mark, and, and this basically is helping to.

advance technologies in the U.S. by expediting the use of real-world data for regulatory decision-making. And they're working with CDRH, and, and, data partners to make this happen. Finally, two, they're… they're trying to find test cases, and they are, there's actually a call for this, and I… I think two of their test cases have been very exciting, but there definitely needs to be more. So, these are just a couple of citations here, but one test case, this is from 2023, Where, Johnson & Johnson, an ablation catheter, was able to, successfully expand their, label indication utilizing real-world data from electronic health records. So this is, this is, you know, a perfect case example of how

The collection of data can be used to both see how your devices are used in the wild, in this case off-label, but then leverage that data to assess safety and performance for that indication, and then expand That, that label, so now, they can successfully market towards that use. And then another example is, the, DaVinci, surgical systems. They, this one, I think, was last year, yes. Where NEST, was able to assist, intuitive, labeling modifications. So they were also able to utilize real-world data to expand, the, the indications for the DaVinci and ex-surgical systems.

Again, capitalizing on learning how the devices are used in the real world, and then leveraging that data to expand market access. I'm gonna just quickly talk about one case study where another way, this is from our company, working with an orthopedic manufacturer. They have a device that's currently on the market in the U.S, but they're wanting to expand to a new market in the EU. And, their device is a total knee replacement system that was, 510K cleared by FDA. However, it's a Class 3 implant in the EU, so they need clinical data, in order to get a CE mark for this device.

And they had a clinical study that had a 5-year follow-up, but as we talked about, long-term Data collection and traditional clinical studies is really hard, and they had almost half of the patient sample withdrew early or were lost to follow-up. So, one thing that was important was that the EU notified bodies or regulators required that not only do they have long-term follow-up, but they also have patient-reported outcome measures. So, there's a lot of reasons for why patient-reported outcome measures are now becoming increasingly recognized by regulators as valuable. However, there's also reasons why,

the clinicians and hospital systems themselves want to collect this data, and a lot of times that results in reimbursement. So, CMS and private payers offer financial assistance incentives for practices to collect this information. And it's also important now if you're… for certain, disease-specific or specialty-specific certifications, as well as, quality reporting programs. So, the initial concern was that if we want to utilize a patient-reported outcome measure as a As an outcome, a primary outcome. Can we get that from rural data? And the answer is yes, you absolutely can. We were able to, look at patients, and this is just a screenshot, showing one particular, problem, which is called the CUSJR score. We can follow patients over time. They come… they basically are sent this survey

annually, and we have access to that data, and again, this doesn't increase burden on the patient, because this is part of their standard care. It doesn't increase burden on the clinician, but we can actually tap into this and access this so that manufacturers can utilize this in their regulatory filings.

Great.

So I think I'll go ahead and turn it over back to you, Sean.

Thank you. I was just getting ready to say that we need to move on to Steven. Thank you, Steve. What… thank you, Amelia. Thank you, everyone. Let's… let's keep going. Steve, so, Amelia, stop sharing your screen, just so we don't have that…

True.

situation again.

Okay. Alright, so, you see mine.

Yes.

Working. All right, great. Excuse me. So, so thanks, Sean and everyone. This is really, you know, such an important topic. You know, we all know that there's so much innovation out there that's possible today, but the burden to bring it to market is so difficult. And that's really what I've kind of spent a lot of my time for the last 10 years or so working on. I won't… I won't go too much of the background here. Sean mentioned You know, that some of the work that I've done has been recorded on video, and you can get access to it. So I'll just give you the key information that you kind of need to know so you have context here. So, I don't work for a startup. I actually work for a very large multinational company called DeSosa Stems.

Now I… I helped, and now lead what we call virtual human modeling, which is really to take The history of the company, which is really an engineering company, so if you… if you rode in a car today or used a cell phone, that was pretty much… pretty well guaranteed to have been designed and tested by our tools. So if you're in a car, they no longer build prototypes, physical prototypes, and put physical dummies in those cars and test them. They do it all digitally. We call that a digital, or I like to call it a virtual twin. Because it's more than digital, it's conceptual. Because you can do innovation on a virtual twin. So we have,

30, 45 years now of history of developing these twin models, basically complete functional replicas of very complicated devices. Or machines, or buildings, whatever it happens to be. But there was a kind of a blind spot, which is appropriate for this call, when it came to the human body. And so, so about 10 years ago, as you'll see, I… decided to see if it was possible to bring the same technology into the world of healthcare. And now you can see at the bottom, we now, have one of the largest, Repositories of clinical evidence data, and we just heard about the importance of data driving these models. I'm going to take us up front with something that I call the Living Heart Project.

The idea was to see if we as a population knew enough about something as complex as the human heart, and lots of reasons I started with the heart, not least of which is that it's still the number one cause of death. So even though it's not very sexy to talk about heart disease, it is still the number one hospital burden, cause of… of loss of work or income, and the cause of death. And so despite all of our advances, we're still not quite winning the game. So I wondered if the problem that we couldn't model the human body was because we… what we measure gets broken down into small pieces, because it's so complex, and we never reconstruct it back into the whole. There's no vehicle where doctors can bring everything they've learned

Together with the people who are maybe doing research on tissue mechanics, and say, hey, I've learned how the heart responds, let me tell you how to build your model. There's no vehicle for that, so I thought I would create one. And see if we could then create a virtual twin of the heart, and then understand. And so, so that was 10 years ago, no, 11 years ago. I knew that the community was broken up into the research community, the industry community, the clinical community, and as we've talked a lot about the regulators, I knew that was important as well. So what I'm showing you here is actually the current, or maybe it's a year or two out of date, but these are actually all the members of the Living Heart Project.

The organizations across these four different disciplines that have agreed to come together, bring their experience, their data, their knowledge, to seek to actually build… not only build it, but could test it and refine it to make it really represent the real world. And so, they're the real heroes. My job was to kind of organize the cats, and it turns out, we actually know a lot about how it works. When I was able… brought them all together, within one year. We had all of the information necessary, and we were able to build a virtual twin of the human heart. Here you see it. It's actually multiple models. The heart's an electromechanical response.

But now you actually have a completely digital version of a human heart. And as you can see, alluded to here, the members now have access to that to test their treatments, their devices. They can introduce diseases. They can represent any population you want. Because you're not limited to what patient walks in the door, or what patient signs up for the trial. You can actually create the patient you need. And that was not lost on, on the members of the project, they've been very careful, and again, you can understand what's happening, because you don't just measure it, you can actually see it. These colors represent, in the case of the electrical systems, you can see the electrophysiology, you can see the stresses on the tissue in the heart if you have heart failure, you can look at the hemodynamic behavior, you can look at

leaks in your valves, all that information is not obscured, you can actually resolve it. And these are just some examples of the ways that that manufacturers have now used the heart. So instead of testing on animals, or then eventually trying it out on patients. they now have virtual populations that they can actually test their devices, and they can see what's happening, can say, oh, if it's causing that. change to the blood flow, I need to tune that and make it a little bit different. And they don't need to wait until a year or two of evidence, after they have the product in clinical trial, which is, of course, as we've talked about, extremely costly if you try to do it then. You can get that taken care of right up front.

So as I said. the population, because we have such a large group, they've tested it on virtually everything that a heart could actually, C, and virtually every kind of treatment. Does it work? Here's just… here's one example of Medtronic, who presented that by using the computational modeling for their leadless pacemaker, they were able to bring it to market 2 years sooner, with 256 fewer patients. Saving them $10 million in clinical trial costs and translating to $400 million worth of business. And of course, they were in a race against St. Jude, who became Abbott, who's the competitor here, who lost the race, of course. Here, they've presented, this is actually… they shared… this is the animal model that they used, the virtual twin. They… they kept the human model proprietary, but you can see that leaderless pacemaker in there.

Abbott, on their side, came to us and said, wow, can you help us compete? So we said, okay, great, what do you want to do? And we worked on a dual lead pacemaker, because that was only a single lead, so now we helped them to create a little network inside the heart. And so they actually leapfrogged Medtronic and was able to get the dual lead pacemaker approved before, before Medtronic. And so, that evidence was not lost on the FDA, who said, how do we make this more institutionalized? And the idea is to take this process, which is very iterative when you're actually testing on humans. And say, let's test on the virtual design, virtual human, same way they do with cars and planes. Can we do it so that by the time we get to the trial, we've already run it, we know what we're seeing, we're largely validating.

And then you get, of course, reduced time and patient risk. And so this got the year of Jeff Shuren, who was a CDRH director at the time. So he personally funded a project where we refused the heart as an entire population in an in silico clinical trial, and here's the announcement. It was a 5-year project in 2019, and a lot of his interest was reducing the ambiguity of the review. We've heard a lot of people talk about is the evidence for your population accurate enough? And often that translates to long delays, more iterations. We've heard 3-year timelines go out to 5, 7, 11, because the FDA is not confident

this was his belief that it was the most, effective way to reduce that time. What he was proposing is, how do we get the average of 9 months for an FDA review down to 2 weeks? You want to get to the point where the evidence is clear, it's all there, everybody knows, and it should only take 2 weeks. And that was our goal. And so we spent 5 years, could go into this in great detail, working, actually, we worked with the MDIC to bring in the members of the Living Heart Project, brought in the community, the clinical team, we had a clinical advisory council. a medical, board, you had a scientific advisory council, and then, then of course the FDA Review Council.

And so, we had a total of about 75 people working together, and we put together this process where we began, with, actual clinical data. So we start with, in this case, early feasibility data. We reproduced the mitral clip, trial. Because we wanted to see if we could… if the FDA would… if we submitted the synthetic evidence from the virtual population in the in-silocal clinical trial, would they draw the same conclusion? Could we provide credibility? So we first had to, create virtual twins of the real people to convince them the models were great, and as we heard earlier, The next step, if you train your AI with high-quality data, you end up with a high-quality AI model. And that's what we showed to the FDA, that AI is powerful as long as you have good input data that you can review, it's properly

cultivated, and not a black box, and you can test it. And that's what we were able to do. By creating the population. And again, here's a case where you can define the population. You want children, you want older people, you want gender, you want different genomic variations. All of that can be baked into it. So you can get your inclusion criteria, all of that before you've ever actually spent a penny on On a clinical trial. And then, with the AI model, you can then say, okay, what is my best population to run, first an in-silco clinical trial, so you then run it on your patient population? Once you're happy with your design, then you can run your real clinical trial.

And so, we were successful in the project, and we actually published those guidelines With the FDA last year. So this is now accepted as evidence, in the, in the medical device world. And so, so we believe this is the way of really addressing a lot of those early design issues, because you have your entire population available to you before you ever… you could even before you ever build a prototype. You could do it all digitally the way the automotive industry. And part of the convincing argument is, and I'll just go through this quickly, but, can we really predict how a patient will behave? Well, one of the things you probably know is that we don't really do a lot of testing on children.

Which is ethically great, except if you happen to be a cardiovascular surgeon having to operate on children, the fact that there's no clinical trial telling you how to use technology means you're having to guess on… by… operate on real patients. And so, we worked with Boston Children's Hospital to see if we could overcome that problem. I don't have time to go through these in detail. These are two use cases that we did early on with Dr. Hogenson and the team. On the left is a patient that had previously been operated on, and that team unfortunately guessed wrong, and the child was dying. All the hepatic blood flow was going to one lung. The doctors didn't know, they had one shot to save their life.

They brought 5 doctors together, had 5 different surgical ideas. They didn't know which to do. They tested on the virtual twin. Turned out none of them would have worked. No matter what they chose, they would have actually lost this child. So we sat down with the twin and designed one that would work, or at least predicted to work, and they did perform that surgery, and the clinical results were exactly as predicted, and you see them here. On the right is a case where they actually had to design an implant, which are now done in advance, very complex three-dimensional structures, figure out exactly what you need to do instead of figuring it out after you've opened the child, and guessing.

And now they report back, they get engineering precision. They have a laser in the operating room that actually puts… takes the three-dimensional object, it turns it into a 2D, template, they just cut it out, put it in, and it works perfectly. And so, much so that they've now done over 1,600 patients, so this is no longer a trial. 40% of all surgeries. This is the top cardiac hospital children's hospital in the world, and 40% of those kids now go through this process, because the doctors have realized they get better results. They've collected the data, it's now reimbursable. So, hopefully that gave you a sense of how we can

Really take care of that patient issue, using clinical data, using whatever evidence you have, taking it up front, taking everything you know and applying it so you can get the right design, and get that trial out to be successful the very first time, and more quickly. So I appreciate it. I'll actually be, if anybody's going to be in San Diego at AdvaMed, I'll be talking about this at that conference on October 6th, so love to follow up with you on that.

Huge thank you to all our speakers today. Thank you very much. I'm gonna wrap things up. Please remember to take the survey. join us next week, next Friday, or every Friday for Friday in Focus Call. A great opportunity to expand your network. I encourage everyone to connect with each one of the speakers. Or your favorite. And, on, on LinkedIn, and, I would like to summarize all of this by saying that when great minds come together to solve a common problem, we come up with better solutions than when we each work in silos. And so, I love today's webinar. I think it was an excellent example of the cross-disciplinary, application that we need more of in medical devices.

And I want to thank all of you again. Have a great… Rest of your day.

Great. Thanks, everybody. Thanks, Sean.

Thank you.

I'm getting it right.