Fishbone Fatigue, risk that actually reaches the patient

Hello, everybody, and welcome to Friday in Focus. I'm Sean Smith. So today's topic, fishbone fatigue, a risk that actually reaches the patient with… Taylor, help me pronounce the last name properly.

Deering her.

daringer. And, I met Taylor, last year, and then again this year, at the risk, Patient Safety Conference, by Bijan, first in Amsterdam, and then this year in Boston. And, he gave a standout presentation on, on risk. And I talked to him about it, and I was like, you know, I just felt like, it went… a lot of it went over my head. And so what I did was I recorded the session, and I took the slides, and I said to him, let's try to repackage this for, a broader audience.

That… that isn't quite as… Focused on risk as maybe a risk expert is. And so, from that, we came up with this idea, an interactive session on why complete risk files fail in the field, and how anchoring by harm instead of sequence of events makes Complaints usable, signals trustworthy, and audits calmer. So, with that in mind, that's what, he's gonna be covering today. A quick update while we're waiting for people to log in. Tuesday of next week, QMSR Unpacked. This is gonna be, a tour of the menu of everything QMSR-related. You will not want to miss this event.

like, 5, 6 different sections, of QMSR, so it's gonna be really good. Then, we're gonna have, yours truly. What does Medusa have to do with any of this? Well, tune in to find out. And, the following week, we're gonna have, Marcus… from, Compliance Studio, and he's gonna be talking about when AI replaces regulatory professionals, what happens. I think that ought to be interesting, and, probably spark a lot a lot of conversation. With that, I'm going to stop sharing. Oh, and we have a bunch more speakers lined up, I just haven't…

Posted them yet. To stop sharing, and introduce… Taylor. So, raise your hand if you have ever drawn a fishbone you did not believe in. anybody, feel free to join in. Yeah, there we go, I see some hands. Yeah. On that note… Let's… Taylor, tell us… tell us about your childhood. Tell us about Where you grew up, and did you want to be a risk expert when you, med tech risk expert when you grew up, or did you want to be something else?

Sure, sure. I actually, grew up in a nowhere town in the middle of Wisconsin. Right now, I reside in the Milwaukee area, and when I grew up… I was just mostly a nerd, I would say. So, I, found myself, like. Kind of excelling with math and science, and also had an affinity to fitness, sports, things like that. So, kind of combining those things, I found myself in the biomedical realm, and yeah, that mathematics piece really lent itself into eventually minoring in statistics, and then bringing that into risk management. Definitely did not see myself, becoming a risk, professional, I guess. I…

I think whenever I think about risk professionals, my mind goes to… I don't know if y'all are familiar with the movie Along Came Polly. Ben Stiller plays a risk guide on the financial side, but that's one of the things that kind of pops into my head, and I didn't really see myself being a Ben Stiller character. But yeah, No, I'm very happy with where I ended up. Coming out of college biomedical, you're kind of thinking, oh, I'm gonna design really exciting devices, but it wasn't until I really got into it that I realized that my skill set really isn't on the design side, it's more on the, process implementation side. So what I'm more,

what I'm more comfortable doing is learning systems that work in other industries, as well as other companies, and then implementing them somewhere that they haven't been. And then also understanding where the challenges are. So that's a lot of where my, topics really Crosses, you know. level setting with everyone in the industry, understanding where the problems exist, and then how do we… not only what the end state is, but how do we get there? I think that's a lot of things that are missing in presentations that I've seen, is not really providing a roadmap to get there. So that's where I like to really focus as a practitioner.

That's great. Why don't we go ahead and get started? Before we do, any other risk professionals on the call? Just, put it in the chat. All you.

Okay, awesome. Yeah, so, like Sean so eloquently introduced, we met at Amsterdam, at the IMSC conference also this year, and one thing that I am terrible at doing is marketing. I do not have a marketing mindset. So I often get very deep in the weeds, and oftentimes. about 5 or 6 slides in, gets to the point where the audience's eyes are glazed over, because I'm usually focusing way too much in the details, and not necessarily the overall picture. So what I really like, what Sean helped me out here in doing is repackaging it, like you said, into something that everyone really experiences in the quality world, kind of on a day-to-day, if they're dealing with investigations of any kind, is fishbone fatigue, right? And everyone knows that when we're dealing with CAPA investigations, complaint investigations, NCMRs, anything on the post-market side, oftentimes it

devolves into really using one of two tools, and that is Fishbone and FiveWise. And what we wanted to kind of do is show how the risk management file can be used as a entry point into these investigations, rather than the endpoint, which often happens. And, yeah, we'll get into that, but… The first poll that Sean had here, another one of these is, how many fish bones have you drawn that you didn't believe in? And then feel free to use the chat to respond, or I know that we have reactions as well, so I'll walk through these. Is it one or two that you didn't believe in? A few, or you've lost count? That's…

Feel free to raise your hand or share a reaction if you feel like that you fall into one of those categories. And I know I've felt this way, one of the things that I've found with fish bones is Oftentimes, when we… are doing these investigations, it comes down a lot to bias, right? If that quality professional has a ton of experience in the manufacturing world, oftentimes they're going to be focusing more on that manufacturing side, because that's where they're comfortable. But we really want to create a risk management file that acts as an introduction into areas that people aren't familiar with, creating a system that can be built upon, rather than creating individual investigations that rely on people in order to make them robust, rather than the process itself.

So why do fish bones feel safe? So fish bones, were developed, quite a ways back, also known as the Ishikawa diagram, and these were developed as part of very controlled processes, right? So this is something that's typically used in a manufacturing setting, in order to identify causes that can occur as a result of a failure. And the reason why this gets, troublesome in the post-market world is there's many factors that we can't control in post-market surveillance. Not only are there factors that we can't control, like user interactions, environmental considerations, positions that are involved with our products, but also the data that's coming in is often very hard to interpret.

Right? We're not often getting what the individual cause of a post-market surveillance complaint is. Oftentimes, we're only learning, you know, a general statement as to what the patient experienced, as well as the final patient outcome. Were they given over-the-counter medication? Were they actually treated in the hospital, or was it just something that they're reporting, some type of an irritation? So those are the two pieces of information we typically get, and oftentimes we aren't given a full understanding as to what went wrong with the device. So Fishbone can organize thinking for us in these investigations, but it doesn't create evidence, and that's important to keep in mind.

So, with Ishikawa or Fishbone, a clean diagram can still be a weak investigation. So, again, if in the first part of your product development lifecycle, where you're developing product before launch, you haven't developed a strong evidence structure, that evidence isn't created at the back end, right? This thin evidence can occur. You're forced to find an individual root cause, so that's the whole point of this diagram, is to identify a single cause as to why this could be happening in post-market surveillance, and with a stronger risk management file structure, it can also help us look at interactions, and we'll get into that later on in the slide.

Silo bias, again, this is where we have, you know, individual specialties. Or people that are doing these investigations are gonna hone in on the thing that they're most familiar with. And then also hidden uncertainty. So the structure is not the same as proof. Alright, so in this slide here, we're going to kind of walk through a little bit of an example. So you be the investigator here. Basically, we have a complaint that comes in, and essentially, we've identified that a patient is experiencing harm as a result of a burn from wearing a device. So, basically, all we know about this complaint is

The device was overheating some way, and the patient was burned. So what line item does this complaint belong to? So we can look at 4 different line items from our risk management file that include sequence of events, insulation breach, surface overheats during normal wear, temperature, control fault, patch becomes too warm, RSK3, worn beyond label duration, heat trapped under patch, and RSK4 being external heat source applied over the patch. So, you all could, put in the chat, unless, Sean, do you know if there… do we have a poll function here, or not? Okay, well, if you could put in the chat, A, B, C, or D really quick.

Or just blurt it out.

Yeah, yeah, feel free.

Steve…

B, okay? That's a good one. I think I'm more akin to, like, the device failure, so I'm, you know. Following RSK2, the B, so some kind of fault with the device, patch becomes too warm. See in another C, Alright, and the point of this is to highlight that it really could be any of those four. So, if we have, Sorry, I got this chat over here. If we have all of those line items that really relate to the same complaint, this event can, be produced with those four different signals, right? So, oftentimes what will happen in complaint investigations, and we kind of saw it in this example, is when one person identifies a single risk line item, see in the previous example, oftentimes,

Complaint investigators will refer back to a similar complaint that's happened in the past, and just use that same risk line item throughout entirety, which gives us a false positive in terms of a signal that's being experienced. Now, that exact example happens all the time in our medical device Systems, our risk management systems, and When there's no clear home for a complaint, people are going to be picking individual places that they can live by themselves. Alright, so we're gonna kind of walk through Typical structure of risk management files, and why they fail in post-market surveillance so often.

This first point here, failure is not the same as error, right? So, if we have individual sequence of events identified as our risk management line items, what oftentimes can happen is we have a very siloed look at how hazardous situations can occur. Iso 14971 risk management requires us to look at how hazardous situations can occur from both normal use and fault conditions. If we only have failure mode effect analyses and sequences of events defining our risk management line items. then we're kind of missing a big side of this, big side of what risk management means. So, failure is when a component breaks, right? So we've defined a specification as to what we expect our component to do, and it doesn't perform to that specification.

Error is very different. It's something that we need to consider as well, is when the system behaves as designed, so we have some type of specification that actually performed as intended, but the outcome is unsafe. And this can happen a lot of times in software, for example. Software, human factors, cybersecurity, these are all vectors for some type of a hazard to occur. but don't fit in that typical failure mode effect analysis structure. I know Shannon Hostie provided a really great presentation on human factors at the IMSC conference, and I remember from her presentation, she explicitly said, you know, when software was

started to get used in medical devices. It was just generally, accepted in the industry that, well, software doesn't fail, because it's code. Code can't break. So therefore, it doesn't result in a hazardous situation. That was just what the industry practice was. And that's because we were trying to fit this software error into our failure mode effect analysis framework that's been developed, you know, beyond… since the 1950s. So, this is something that we had to learn as an industry and identify that, well, actually, if we Improperly implement requirements. If the requirements aren't properly defined for our system, then that can also result in hazardous situations.

So even though a failure might not be found, it's not the end. And this can oftentimes be the result of, in a Ishikawa diagram or some type of investigation. Oftentimes, if we don't find an individual failure in the process or the product, oftentimes we'll rely on operator training or some type of a residual risk warning statement as a means of solving that issue. And that's a good indication that you're not considering all the interactions, and only are focusing on the failures. Okay, so one thing that really changed my mind recently, I was looking at a new, risk management technique, and I was learning, it's something we all should be doing constantly, is looking for new sources of learning.

Iso 14971 defines a hazard as a potential source of harm. And what I find, the issue with that definition particularly is that we often attribute that to a single cause, right? So when we talk about potential source. Our mind goes to an individual cause, and that's kind of where that root of sequence of events starts. Is because that hazard is defined as a potential source of harm. But if we can flip our thinking here. Rather than frame it as an individual cause, and more frame it towards a system condition, so a condition of the system that may result in harm. That can help us frame how we should structure our risk management file, and get away from individual sequences of events.

All right, and then this slide is just kind of demonstrating that risk lives in the handoff, right? So each of these individual buckets can be evaluated for risk, at their own level. And this kind of is derived from systems engineering, where we have a system defined, and then we go down through a V model, where we go through our subsystem, and then eventually component to implement requirements. But we know that when the patient experiences the device, there are many different factors that are interacting with each other real time, that could result in hazardous situations. So if I'm looking at specifically device failure modes, or doing a user risk, use-related risk analysis.

I can do those in a silo, but then also, when all of these come together, those interactions and how each of those individual boxes interact with each other can also result in unique hazardous situations that we need to consider. Alright, so then… I like this slide a lot, it kind of, like, level sets, there are a lot of busy slides there. And really what it comes down to, is two really basic questions, right, that risk is really trying to answer. And this is also something that I find, the FDA can agree with. I've had experiences in the past where I've presented to the FDA and had a chance to speak with, you know, some of their

personnel off to the side, and I just straight up asked, what is your perspective of ISO 14971? Do you like it? Do you not like it? What's… Their response wasn't very positive. They identify ISO 14971 as something that can help you do risk management, but they also identify it as a means of manipulating probabilities in post-market phase that is beneficial to the manufacturer. So, with sequence of events, particularly, and we'll get into that, if we anchor everything by the individual cause, oftentimes we're splitting up probabilities in a way that could not be defendable in an audit. Another thing that the FDA shared with me is, one of the worst things that you can do, is

When you identify, severity 5 harms, and defining acceptability, for severity 5 harms, you don't necessarily want to have a probabilistic outcome as to what is acceptable, i.e, how many deaths is acceptable. Of course, the very first answer to that question is, no amount of harm is acceptable for our device to cause. It is on our risk policy to define what is justifiable for what's in the field. So. Again, going back to that simple thinking, is… How often do people get hurt when using our device, and how badly are they hurt? And if we can answer those two questions simply, it's going to be a lot easier in audit situations to first kind of start at that high level, and then drill down when appropriate if new signals have been identified.

All right, so the equation we should cross out, I like this slide. It summarized the busy slide that I had very nicely, so thank you for doing that, Sean. So risk equals probability times severity. I'm sure everyone's seen that as a result of these FMEAs that we've all created. Right? FMEA uses, typically, severity, occurrence, and sometimes detection to come up with some type of an RPN score. So we kind of get jaded and identify that, oh, risk can just really be simplified into two factors multiplied together. But ISO 14971 explicitly defines risk management as the combination of these two factors and not multiplication on purpose. That being said.

When we're talking about combination, it gives the manufacturer flexibility as to how to weight these two factors. So oftentimes, when we create a risk profile, and have severity on one axis and probability on the other, you'll see that it's not exactly symmetrical. Oftentimes, we weight more towards severity. And… just to remind everyone, P1 and P2 here, the chance… P1's, the probability that the hazardous situation occurs, and then P2 being, the probability, that that hazardous situation actually leads to harm. So it's important to define here. And one thing that, Mr. James Pink, who gave a fantastic presentation over at IMSC as well.

What he likes to say, too, is we want to put the manufacturer strictly… in control of what P1 is, right? Because that is where the manufacturer really has control. Before a hazardous situation occurs, what can I control? And that's something I'd like to highlight as well. Okay. All right, one harm, one line item. So now we're kind of getting into the before and after problem solution. So before, having anchored by sequence of events, this is what I've been leading up to previously with all the other examples and all this other talk, is… one complaint mapping to four nearly identical rows. That's the example we saw earlier, where we saw multiple line items could be chosen for one individual complaint, and that requires the investigator to reverse engineer a cause, and then… that the complaint never gave them, right? We only have

What generally happened to the patient, and then also the patient outcome. And then the investigator has to guess as to what lineup best fits. And that results in false signals, split probability, like I mentioned with the FDA, that's a high audit risk. And then the after is if we anchor it by harm. So each risk, rather than being defined by an individual sequence of events, now gets, identified by an individual hazardous situation harm pair. And that, complaint then maps to those harms. Those are the pieces of information that we actually get from complaints most often. And then once we have those mapped to hazardous situation harm pairs, that gives us true signals that we can then investigate on specifically.

All right, so then, kind of as an add-on to that, I wanted to share this ISO 14971 diagram, which everyone is probably very familiar with. We have hazards, sequences of events leading into hazardous situations, and then that hazardous situation leading to harm. We also see that P1 and P2 definition there as well, and then showing that the overall probability of occurrence of harm. It's P1 times P2. And then risk is defined as severity of harm, probability of occurrence of harm. The reason why I bring this diagram up, is it gives us a very linear perspective of what risk management should be, right? We're looking at hazard on one end, harm on the other. Oftentimes, if you look at a risk management file, you'll see that same structure. Very linear flow, and this kind of

looks like, oh, I should be… Putting individual lines based on individual sequences of events. So that's why I would like to show this slide, because… or this diagram, because you can kind of intuitively understand why we set it up that way to kind of match what this diagram provides. Now… If we want to flip it and do a hazardous situation harm pairs, I… drafted a little bit of a diagram here. I would love to get feedback from everyone here. As to what they think of this one. But basically, at the top, we have hazardous situation harm pairs, and then underneath it, we have individual nested hazard categories. So there are multiple hazards against a state or condition of the system that would result in a hazardous situation.

And then we have individual sequence of events that live inside those individual hazards. Now, when we look at all of those that can lead to a hazardous situation, combined can create a P1, and what I liked about this diagram specifically is it gives a different direction for P1, and that's significant, or that's supposed to depict that this is Something that manufacturers have control over. Once we get into the P2 world, a hazardous situation has already occurred. And one thing that James Pink is also, I gotta quote him saying is, one of the worst signals that we have as medical device manufacturers is when a patient gets hurt.

Because that's already beyond the point that we can help them, right? So the only time that type of training becomes useful is when we identify that it needs to be reduced, right? We… we hope that we never get to that point where we identify a trend in P2. So I do like to separate, show P2 to the right to show we don't really have as much control there, and we really should be focusing all of our attention on P1 to reduce the probability of that hazardous situation occurring at all. I think one other thing I wanted to mention on this slide… So, here we're showing hazardous situation leading to harm.

one hazardous situation can lead to multiple harms. So, this is just showing one hazardous situation harm pair, but for this specific hazardous situation, we might have severities ranging from 1 all the way to 5, if you have a categorical scale, right? So. Keep that in mind as well, but this is really just demonstrating the entryway to our investigation phase, right? So, from complaints at the top, we go down into our hazardous situation-harm pair, and then if we identify that there is, some type of signal as a result of the data that we're getting from the field, then we can drill down into our individual states or conditions of the system and sequences of events in our investigation phase. So it acts as a entryway into that investigation, rather than

Something that we gotta sift through many individual documents to come up with a unique and, illustrative fishbone diagram.

I hope I'm not the only one.

Yeah.

know what SOE stands for.

Oh, sorry, yeah, SOE here is sequence of events, I apologize.

Thanks.

I should have put that on the slide. And then I guess, yeah, just to really hammer home, like, the control in P2, what does a control and P2 look like? Oftentimes, these are some type of a protective measure that's put in place to alert the user that the medical devices entered a condition that could lead to harm, right? So, maybe for this example that we have, where we're talking about patient experiencing burn, maybe the device is able to, have… that will have, like, a thermocouple, or some type of a temperature monitoring sensor. on the device, and as it increases in temperature, all of a sudden we get a LED that blinks, right?

So, that LED that blinks. Is, technically, a control that we have in place in the form of a protective measure to alert the user that there is a hazardous situation. Now, one of the things that's required to prove, risk control, reduces risk is… We need to have objective evidence that there is, or I should say, verification of effectiveness, right? And… this control is extremely hard to prove as effective, because once the device gets into the field, we're not exactly sure how the patient is going to interact with that device, right? So, once they see that the LED is blinking. Then they have to go and look at the IFU. What does this LED mean? What does that troubleshooting mean? You know, is this something that I have to take off right away, and then now they're…

diving in. Is every single patient gonna do that? So… In comparison to our P1, our, our P1s are more defined on the… system requirement side, which are much more, you know, obvious to have that objective evidence for effectiveness of those controls, versus, again, once we get into the field on that P2 side. Alright. Now, we're gonna do a little bit of an exercise here. So, rather… or so, going back to that. Original example, where we have a patient, wearing our device, and they're exposed to an excessive, form of energy.

That results in a burn. And again, we had those four risk line items that could be attributed to that individual complaint. So the ask here is to identify the hazardous situation and harm pair that this complaint that would be linked into. So let's collapse those four line items into one. So if you guys could take a little bit of time. Think about… You know, the complaint that's coming in, and then identifying a hazardous situation or harm pair that would be applicable to that specific complaint. And if you want to share it in the chat, I'll give you guys a couple minutes, and we will… See what comes through.

So while they're thinking about that, I want to see if anybody has any questions for you. I know Chris had dropped a question into the chat. Which I'm gonna let him read, per se.

Okay.

There were two different instances at a company I worked with that this came up. They had their share of recall. And, and, so one of them was. we figured out it with this infusion pump that was sold for U.S. hospitals. hadn't been thoroughly vetted for how… it wasn't designed for use by nurses, it was designed for use by engineers, right? So, they hadn't done enough of clinical trials to ask a nurse, okay, exactly how do you use this pump in an emergency situation? Maybe I have to bolus a life-saving drug for this patient, and it's more bolus than the pump will allow, but they're gonna die. Things like that, and that had happened more than once.

And none of it was required by the hazards for that project, or any of the requirements. And then the same for the system failure, which Cerner famously experienced multiple times, because they didn't take the time to do, like you said, those sequence events, they didn't do scenario-based what-if to say, okay, if we did this, what would happen? And they did, of course, after, just not enough, because it kept happening again. How do you, trace that?

Yeah, great question. So, What you're describing is, again, so… I don't know if anyone here has been at IMSC, I think. Other than Sean, but… This year they had… a speaker… from MIT, their Aerospace and Aeronautics Division, or department, I should say. Dr. John Thomas is his name, and he has a new technique, this is what I was alluding to earlier, called STPA, and it's Systems Theoretic Process Analysis. Now… When we talk about failure mode effect analysis. Again, this is a result of systems engineering from way back when, when we had very simple systems. That is, we had components that could fail, and we knew that if those failed, it resulted in an effect that resulted in some type of a hazardous situation, or loss of functionality, something along those lines.

And that's very simple, linear thinking. But now, we've gotten much more complex systems, right? We talk about interactions with human factors, like who's using the device, the nurses, talking about software, we're talking, you know, how those requirements can fail. Cybersecurity is the newest one. So, one of the things that I have to do as part of my job is not only look at individual component failures, that's one side, and again, that comes back into that failure versus error. So now. Cybersecurity, we'll take, for example. FDA is requiring us to look at cybersecurity from the perspective of two different factors that are different than risk management. That includes exploitability and severity. So how do I, as a risk management professional, knowing that cybersecurity can result in hazardous situations.

bake that into my ISO 14971 required risk management process. So, in terms of what I've seen be effective as the systems exist today, taking it from an FMEA is rooting everything into the DFMEA. So, right now, And the example I'd like to bring up is, if we create a DFMEA based on our requirements, everything that can result as an error From that, not a specific failure of that device can still be rooted in a failure mode in the design. So, for example, on the process side of things, when we're doing design transfer, there's a lot of things that can fail in our process that would be very akin to

how the design would fail. So, the way that I… create that trace is through the DFMEA. So when I'm creating my PFMEA, I create a linkage directly into a DFMEA failure mode or individual line item that is attributed to that specific type of a failure. So that effect path still goes up. So in a way, you're kind of creating a trace matrix from your DFMEA into all of these other documents that can result in other failures or errors. So there's the PFMEA, like you said, there's the URRA, the Use Related Risk Analysis. Then there's going to be our… some type of cybersecurity risk analysis document, and then also our software hazard analysis. So really, if we link into our DFMEA all the way through all of those, then that can still siphon up into our hazardous situation harm pairs as well.

So that's kind of how I tackle that with the current system. I think that there is going to be a new process in the form of STPA, that actually forces us, rather than going down, like, a fault tree analysis, which… I'm sure everyone is familiar with that top-down approach, but oftentimes what ends up happening with vault tree analysis is just ends up being a summary of all of our individual failure mode effect analyses doesn't really look at the interactions like we're talking about. So what STPA does is it looks at all of these scenarios, right, the context that's required to evaluate, which I think is what your question was, Chris.

And then it makes us look at it in the form of control loops. So rather than having that top-down or bottom-up thinking, where it's a line to the failure, it's, you know, what is the control action that's happening, and what's the feedback that's given? And that helps us create or identify additional controls that should exist. And that's… I mean, I think that is going to be a game changer, and I'm really excited about it. That's something I hope to speak on in the future, showing true examples for a medical device and how it can be implemented. If you're interested in that specific topic, feel free to look at MIT's Aerospace and Aeronautics Division, our department.

they do have previous dissertations, on STPA applied to medical devices. It demonstrates how STPA not only identifies all of the failure modes that were identified in FMEA, but also identifies new ones, and it shows you know, really cool diagrams doing. So, that would be, if you're interested, and I can type this out in the chat too, it's, Vincent Balgos is the… main author of that one is Vincent Balgos et al.

Thank you very much.

Yeah.

So let's return to the exercise for a moment. And, I don't know if somebody wants to… Chime in on this. Or… If we just let Taylor… Yeah? Maria? Hi.

Maria E. Donawa, MD: Yeah, risky. Risky me. Maria E. Donawa, MD: Okay, so, this is, this kind of reminds me of the… what I see in various, risk management matrices, or risk… whatever people call them, they have different names, where they're starting with hazards, and I'm sure you've seen all kinds of Maria E. Donawa, MD: problems there. I always think in terms of there's no harm unless there's a hazardous situation. Maria E. Donawa, MD: So, and of course, in clinical practice, you can have various hazardous situations, so I think I understand that you're saying Maria E. Donawa, MD: You have, various sequence of events, so different things can happen, but then…

Maria E. Donawa, MD: You have, the hazardous situation, in my view. Maria E. Donawa, MD: would be the patient is exposed to excessive thermal energy. That's the problem that can happen, that can cause the harm, which is the minor burn. Maria E. Donawa, MD: I don't know… I'm not certain…

That's great.

Maria E. Donawa, MD: Okay.

That's exactly what this hazard situation harm pair would be boiled down to, is excessive thermal energy, minor burn. And then we can use our hazard categories, in order to You know, identify… Individual sections of our risk management file that could go into deeper investigation. So for example, with thermal energy, there could be a mechanical type of a thermal energy, like heats getting trapped, heat sink failure, battery, something like that, where it's a physical interaction, versus, Some type of a software failure that's resulting in, some type of a processor to, Ex… be exposed to more work, and therefore resulting in more energy being exposed. So it's kind of breaking it down that way to create individual states of the system that might result in the hazardous situation harm. But you summarized it perfectly.

Patient exposed to excessive thermal energy is the hazardous situation, and the minor burn is that harm. And once we have that pair, then we have something that we can link complaints into, and then do our trending.

Maria E. Donawa, MD: Question?

Yeah.

Maria E. Donawa, MD: I'll try to be brief.

Sure.

Maria E. Donawa, MD: What I see are problems with companies defining the actual hazards.

Yes, absolutely.

Maria E. Donawa, MD: So, actually, I see problems everywhere in this whole… in this whole process. So… Maria E. Donawa, MD: Are you… you may have already made comments about that process, because I think what happens is, once you get into the more practical aspects, you may have to go back and say, oh my goodness, that's a hazard, I missed that before, I didn't include that. Maria E. Donawa, MD: And so, yeah, it's… Maria E. Donawa, MD: Oh, I'm sorry. Aside from that… No worries. I'm being bad now, but… Maria E. Donawa, MD: I don't know if you're gonna talk about this. Maria E. Donawa, MD: the validation of the effectiveness of the risk control measure. You may be.

Maria E. Donawa, MD: think, you know, you may be thinking about other aspects, but one of the pet peeves I have is that Maria E. Donawa, MD: companies very often will not indicate that, for example, a clinical study has been used to validate the effectiveness of some particular risk control measure. Maria E. Donawa, MD: I see that a lot, that companies have not done that, okay? I'm done for now.

So what you're saying is that you have seen that they haven't used clinical trials as a means of validating the effectiveness of a risk control measure.

Maria E. Donawa, MD: No, they've used it, and they haven't documented it in the risk…

Oh, okay.

Maria E. Donawa, MD: No, totally used. I see. But it's, it's, they'll say, oh, clinical evaluation report is a validation method, which drives me crazy, because in my view. Maria E. Donawa, MD: and I've had big internal debates about this, the clinical evaluation, exercise, and report is not itself the validation. It talks about the validation. I mean, the whole idea is to say, okay, you're done. Maria E. Donawa, MD: But the actual… Maria E. Donawa, MD: validation itself, this is fine. This risk control measure is good. Maria E. Donawa, MD: find it, and now I've checked it in clinical studies, whatever it is.

Maria E. Donawa, MD: For me, that is the validation, and very often, let's say 50% of the time, It's not documented.

Beautifully said. I'll start with that one, and then I'll go back to the hazards. So, for risk control measures, you know, oftentimes we are struggling to define how that fits into our design control process. And oftentimes, we use that design controls as a means of validating these risk control measures. And we'll just use our design control, techniques as a means of identifying how we're going to validate those things. But oftentimes, risk control measures can be more general, or Not as specific as these design requirements are. And risk control measures should really be viewed, from my perspective, at the level of the patient. How the patient experiences the design of the device during use that would prevent hazardous situations from occurring.

I 100% agree that the clinical validation or clinical studies, clinical trials, are a great way to essentially assess the user needs at a higher level, and therefore the risk control measures as well. I agree that it's poorly documented, and the solution right now, from what I've seen, and feel free to correct me, is again, loop it into design controls, have some type of a statistical technique based on risk. for our design process, that is following confidence reliability tables, looped into our severity and or probability, and then using the design, verification, and validation as a means of demonstrating effectiveness. Whereas

It really should be done at the clinical setting, at the user needs sides of things, not necessarily the individual device side of things, too. See how those interactions kind of work in the field. So I agree with you 100% there. Oh.

Maria E. Donawa, MD: Nice thoughts, yeah.

I would also, going back to the hazard, also, I love seeing these comments, please keep them coming, I would love to respond to them as well. And we can… because sometimes when I get into these rants. I… I'm not exactly clear on my points, and would love to circle back on some. But in regards to identifying hazards, I agree that In terms of 14971 and the technical report 24971, it doesn't necessarily give us a great means of identifying how to define hazards. The only thing that really exists, of course, is in Annex A as a like a checklist to identify what hazards may exist, but it doesn't really give us a framework for defining those hazards, and you'll see hazards defined very differently from one company to the next, usually based on how the very first one was defined. So, again, I would refer back to that STPA process where

They actually have a very specific and stochastic way of defining what an individual hazard is, and it does it very systematically. And again, it views it from the perspective of the condition. One of my favorite examples of this is there's a… A article written by I can't remember his name, but I will provide it in the comments after this, or maybe Sean can help me send it out to the attendees. But basically, it was the banana peel problem. And, very simple, you're walking down the street, and I'm sure many people have heard this example, and it's rainy outside, slippery, you're looking at the buildings, all of a sudden there's a banana peel there, and you slip on it.

And now we have to define the hazards, hazard situations, and harms. Well, oftentimes, when we say, define the hazard, everyone says it's the banana peel. But in this case, if we did that, we're going…

Maria E. Donawa, MD: -Oh, frozen.

Back to our sequence of events, right? We're going back to anchoring. And in this, yeah, in this example, we're talking about Potential energy. Basically, gravity is the condition that causes this hazardous situation to occur. And that sequence of events is the banana peel exposed us to that condition, which was a harmful occurrence for us, would be, like, break… that article. I think it's really eloquent in showing how hazards and hazardous situations can be defined.

Maria E. Donawa, MD: Interesting.

Okay. So, I think we did a good job anchoring on our patient exposed to excessive thermal energy as a hazardous situation, and then the harm being a minor burn here. Any other things before we leave this slide that we've been on for about 15 minutes? Okay. Excellent. So, I know we're kind of… Really. coming up on time here, so I want to go kind of quickly through the rest of this. So, we've talked a lot about different things. I really want to give a little bit of time for comments and questions. So, just gonna highlight, where FMEA is strong, right, is it helps us identify individual mechanisms, individual sequence of events, can help us come up with individual controls to, stop those events from occurring, and then, of course, we have that objective evidence in the form of verification of effectiveness.

and implementation and effectiveness. And it's not really useful as a front door, right? So that's where we have that hazard analysis as a means of Creating those higher-level buckets, and then we can use that as an entryway into investigations, as well as into our FMEAs. Alright, new complaint, where do you look first? So the patch fell off after 48 hours of exercise, cutting the usable monitoring time, right? So we have some type of a monitor that we're wearing, and after 48 hours of exercise, it falls off. So where do we look? Well, a fishbone would require us to look through the six M's individually and use our risk management file as a means of filling in that information. So we're kind of looking one by one, but

what is much more useful is kind of using that design control bridge as a means of identifying evidence, right? So, of course, starting at that intended use and environment, going down through design inputs. having our verification and validation, and then finally ending up at our labeling and training. If we followed our design controls effectively, we can use our requirements as a means of a nice entryway into You know, what is working effectively and what is not. Okay, so again, I love this slide, too. It very… and the boxes are nice, too. Kind of shows, like, the overarching one being our hazard analysis at the system level.

Anyone can read it, anyone can understand it. It summarizes individual hazards. It doesn't get bogged down in the sequence of events, right? Let's leave that for the FMEAs. The FMEAs are going to give us our individual sequence of events and our individual controls to stop those things from happening, but at the higher level, we can rely on system requirements and higher level hazard definitions to help not only, auditors digest that information, right? Because if you give an auditor thousands of lines of FMEAs, they're gonna not be very happy as to us demonstrating that we're reducing risk over the system of our device. They can see individually, you know, causes that have been reduced as a result of mitigations or controls that are in place, but they really want a summary at a higher level, showing at a system level, how we're reducing risk across a specific way a person can be harmed.

And then, of course, those FMEAs, also help us document requirements BNB evidence. This is also something that would live in some type of a trace matrix as well. All right. And then, this is just going to be very high level here, so from complaint codes to signals you trust, right? So we need to have something that links into our MHA, our hazard analysis, and obviously, to build on that, we really should have some type of a coding scheme, so something that tells our complaint investigators, hey, you gotta file or fill out these IMDRF codes anyways, let's create some type of a key, some type of a…

translation tool that helps us go from IMDRF codes into individual hazardous situation harms. And that's something that's typically done in a cross-functional manner with quality engineers, subject matter experts, and then also clinical. to help identify where these individual complaints can fit into our risk management file. Once you've done that, you want to make sure that you're trending two things. So this is something that's often missed in EUMDR, is that we're required to identify statistically significant increases in frequency and severity. So, in order to do that, not only are we looking at complaint rates, or other statistical tools to identify increases in frequency. We should also be looking at, for a single bucket of complaints, how the distribution of severity goes across that specific bucket. So how many

severity 1 versus Severity 3 harms am I seeing for a single type of complaint? And is that distribution changing over time? That's something that a lot of people miss. Okay, Now, this is just kind of, again, like, how do I get to that endpoint, right? How do I get to a nicely siloed, investigation-ready hazard analysis document? This is just, you know, individual stages that you'd be looking for. So first, working with your clinical team, CMO, or consultants to come up with harms that actually make sense for your device. Maybe they don't perfectly nest into IMDRF, but you're going to be relying on, you know, physicians, AHPs,

healthcare practitioners to identify what harms exist for your device type, and then you're going to, be creating hazardous situation buckets, like I said, creating those two individual buckets, hazardous situation harm pairs. Everyone has failure mode effect analyses, usually, so now my next step is to take all of those individual sequence of events and attributing them to the correct buckets. Again, from that previous slide, building that complaint coding scheme, so having that translation tool to get from complaint to our hazard analysis. And then the fun part, we kind of… touched on this very, very high level, but essentially what we want to do also is use our systems engineering as an entryway into our evidence pathway, and then using a trace matrix to close the loop from our evidence all the way into our FMEAs.

And then, keep the fishbone, move the anchor. I love this slide, too, that Sean put together. Basically, Fishbone is a tool, and all tools have their benefits and their detriments. But really, if you have a strong anchor, if you have a strong risk management file, these investigations will be much more meaningful than if we are trying to recreate the wheel every time we initiate one. Okay, and that's it. Sorry, I won't try to… go through those pretty quickly. We have a few minutes left, Sean, how do you want to handle these last few minutes? Do you want me to just kind of walk through some of these comments?

I got a question, if you… if you could answer it.

Yeah, go ahead.

Alright, thanks.

Hi, Paul.

I love… Hey, buddy. I love the depth of… your knowledge…

Okay.

But I'm a sales guy, and I was lost in.

Yeah.

your offers. I know you said you're not good at marketing, but…

Yeah, very bad.

How do I use this? I mean, I understand, because, I mean, you could do this in Excel, you know, do you have a product? Do you, you know, how does this engage with, you know, Greenlight Guru? I mean… Yeah. Tell me, where do you locate yourself in the market?

Yeah, I'm just a lowly, quality engineer. Just trying to share what I've experienced, and.

Okay.

I would say, yeah, the only…

You don't have an offer other than your knowledge base with other people could hire you?

He works for a manufacturer.

Yeah, medical device manufacturer. I don't have any incentive for selling.

Got it. Okay. Thank you.

You're open with that.

But it's great.

Maria E. Donawa, MD: Lovely.

Yeah, yeah, totally, totally. No, I get it.

One, one thing, one of the participants here has been just… given some great feedback. It's Alex. I don't know, can we bring Alex on? I'd love to have, some of these comments addressed.

No, I don't know if he wants to come on, but…

Okay, that's fine, that's fine. I can just walk through some of his questions.

Yeah?

Okay. Or, some of his comments, I should say. So, first one that I saw from him was, probably relating, back to fault tree analysis. He says, specifically, disagree, fault tree analysis and Boolean math are essential. for turning sequence of events into P1 probabilities. So I, I, I… I agree with that statement. What I'm saying with, P1 is So, oftentimes I'm asked, how do I handle probability of occurrence of harm in pre-market versus post-market? What I particularly say is breaking it up in a P1 and P2 pre-market, if we don't have a predicate device to work off of data. Is a very… It's a very complicated thing to do, and this is something that, I took from…

Kaplan and Garrick Nuclear Engineering, very long time ago, and they identified that any estimation that we have, which every single fault that occurs in a fault tree analysis, is an estimation. And if we concatenate that P1 into P2, That's going to result in two levels of uncertainty, and if we look at a fault tree, for example, every individual probability that we estimate has a level of uncertainty, so every time we multiply those together to come up with a final probability, we have a Exponentially increasing rate of uncertainty. So… That's why pre-market, what I like to say, use a general probability of harm estimation, because that's all it is, is an estimation. Use your risk acceptability matrices, your probability categories, semi-quantitatively, so that you can

come up with some type of an estimation to identify priority, right? We're not trying to get exact with our probabilities in pre-market. What we're trying to do is identify where our design needs additional controls. and use our risk acceptability criteria to help guide us when acceptable… when acceptable is acceptable, right? So… That's kind of the response I would have to that. I guess the only thing that I defer, or I… I would, I guess, kind of fall away from that line of thinking is everything has P1 probabilities, is instances of errors, right? And errors do not follow a probabilistic model. If we look at human factors, we can't probabilistically determine how a nurse or a physician is going to fail.

In software, we can't probabilistically determine how a software requirement's gonna fail. In cybersecurity, we're not gonna be able to probabilistically determine when there's going to be an adversary attacking your device. So all of these things fall away from that fault tree analysis, fall away from FMEAs in a way that it just doesn't fit. So. Again, I would just defer to that new method. I'm… looking to investigate, come up with a presentation on STPA, which kind of brings all of these together and creates, and puts context into the analysis, rather than individual failures. Okay. And then… Yeah, so 1059,

Kinda coming close here… Yeah.

I think we're… I think we're gonna have to wind down. Can you stop sharing your screen?

Yeah, for sure. Sorry, I'm just getting really amped up on the topic.

banking. But, I want to thank everybody for their time, especially, Taylor for a brilliant presentation. I, you know. am not a risk expert by any stretch of the imagination, and I really appreciated his insight and his effort in bringing this to us. I also appreciate all of your comments and questions. I will share the chat file in the Q&A with Taylor. I encourage you to reach out to him on LinkedIn and connect with him. Again, he's not selling it. anything. And, he's just, he's just showing his knowledge, so,

you know, congratulations, very well done. If you enjoyed this presentation, please feel free to send me a message if you did not, for whatever reason, please feel free to send me a message. It'll be confidential, and I, appreciate all of, all of your, contributions today. Next week, obviously, is the 3rd of July, for those of you based in the United States, that means the kickoff to a holiday weekend. I, therefore, invited myself to speak. And what I invited myself to speak on was something very near and dear to all of the consultants on this call, which is, how the heck am I gonna make more money? And so, with that in mind.

we're… You know, I've worked with hundreds of consultants, done hundreds of webinars, do the weekly roundup every week. I see what works on LinkedIn, I see what doesn't. I see why people are successful, and how people become leading voices. I'm basically gonna give you a roadmap next week on how to become a recognized and respected expert in your field. Free of charge. Please show up, and I look forward to it. Those of you who are already MOVX members. We're gonna be kicking off a new year-long program. You're already enrolled. It's gonna cost double for everyone else. So…

Congratulations on joining early and often, but the, yeah, looking, looking forward to it, and Maria, did you have a closing thought?

Maria E. Donawa, MD: No, only that it was so, so useful and so thought-provoking. Thanks for finding and making the connection with Taylor, and thank you, Taylor, for a really Maria E. Donawa, MD: Very, very, well done presentation.

Appreciate that.

Nice job.

Thank you all. I hope you have a great weekend. Thanks, Alex, for joining. And, I look forward to, talking with all of you soon. Bye.

Bye, everybody.

Maria E. Donawa, MD: Everybody.