Human Factors Engineering in MedTech: Why Usability Matters From Day One | Kay Sim

Hey.

Hi! I'm just re-hooking my computer up. I need to turn on the camera. A minute. And then I'll tell Kay to see if she can come in a little earlier. When'd you fly home?

Sunday.

What has been the best thing you've eaten? on this trip.

Hang on, let me get… These headphones working. They're not…

Sorry?

Oh, my headphones aren't working right, I need to fix them.

Oh, okay. Oh. Man, my mask doesn't work.

Can you hear me?

Yes.

No, the sound is still coming out of the computer. This is like… why… why don't these stupid earbuds work? Oh. Like, I really need them to work right now. Position.

Take a breath, you can…

What's the troubleshooting step?

They're plugged in. Are they virtual?

They're supposed to connect. Settings…

My mouse isn't working.

How about now?

I could hear ya.

I think the sound is coming out of the computer, I don't think it has anything to do with these stupid earbuds.

I cannot attach my Apple AirPods via Bluetooth. to my, computer, so what I do is just call in on my phone. And then the sound comes in through my AirPods, which have really good noise canceling. And then I call in on the computer without audio for the video. It shows me twice, but… I've just always done that to use the AirPods, because they have nice noise canceling, you know?

Yeah, it's just useless. It's just absolutely useless. The fact that they would have Apple products that don't work together. It's just… Oh, you have a Mac? Yeah.

Oh, I was gonna say, I have a PC, but… The fact that it's not working on the Mac is… Cool. There's 12 people. Already, let me ping Kay.

people.

Okay, just in K. I'm just gonna be in the office, but I'm too… And sleep well, so if you're lazy. How do I… change my background. Virtual.

Good morning!

Hey, good morning.

So why are you?

Sorry, it's loud here.

Oh, conceive. Are you not home?

I'm not home, I'm in Portugal.

Oh, for the event! Sorry. Yeah. I couldn't keep my own schedule straight, so… I can't give anybody a sketch. Hi, Victoria, I'm…

Hi, Michelle.

Yeah, I don't think we ever met.

No, it's nice to meet you.

What happened to my… Camera. Sorry. Alright? At least I'm home this week.

That's nice. Where's home?

East Coast, Florida, East Coast.

Okay.

So, where are you at?

I'm in Westchester, Pennsylvania, which is very close to… Philadelphia?

Ugh.

And Sean.

Michelle, where are you in Florida?

Space Coast?

like, near, Coconut Creek?

No, you know where's Panz Inlet?

Yeah, okay.

Yeah, yep.

I'm in Jacksonville.

Wow. Well, this is about only an hour, hour and a.

Yeah, it's a little over an hour away.

Yeah. Jacksonville is very congested, that's why we decided.

I'm closer to St. Augustine, so I'm.

No.

But… Yeah.

Okay, okay, yeah, so we live in a very small town, the speed limit 35, and there's only Two… well, one major street, speed limit 35.

Fun.

So, just a quick update from, Portugal. So, I'm here for the, Complier AI and MedTech event, and one of the questions that just came up was, how can… how can companies use AI to, understand user needs better early in the development process? And I was like, I got some opinions on this. So, I was… I was kind of vocal about that. the event is going well. Should have a post up later today from this morning's session, so everybody, please give that some love if you find the time today. Trying to promote

Everyone to panelists.

Sean Kay's just getting her… sound to work.

Okay. I'm just promoting people. If I promote you twice and you don't want to be promoted, just… decline. If I miss promoting you, and you want to be promoted, just raise your hand electronically, or something. And, and I will… Promote you. There he is. Okay, make co-host. Okay, Francesco is now co-host. Confirmed. Good. So he'll be promoting people, and I will just be… Here are trying to hold it together.

We have about 2 minutes before we start. Looks like a lot of people are showing up. How many of you were able to go to, the… human factors, webinar that we did last week with, Kelly Kendall, and real-world evidence in silico and FDA. anyone able to attend that? It was really good, the, replay is available, just ping me if you want me to send you the replay of that. And, it was… it was super exciting. event.

How's everyone today?

Pretty good.

Susan's got her hands up. Having a party over there.

Yeah, it's a real party. I had my office redone. And I had to rewire everything yesterday. And I want you to know I don't enjoy that.

Noted. I will put that in your permanent file. Susan does not enjoy rewiring. Okay, well, it's 4 o'clock here in Portugal. I think… Yes, that's what time it is. I'm a little confused about the time zones, but welcome, everyone, to Friday In Focus with MedTech Leading Voice Exchange. I am super excited about today's guest. I met Kay, about a month ago, I think. And we sat down and we had a long interview, about human factors. I now know more about human factors than probably 99.9% of the general population does.

And I'm excited about that. I'm excited to have her here today because she is a dynamic speaker, she has a lot to say, and I think that, I mean, just… well, I'll let her ask all the questions, but Okay, I'm gonna go ahead and introduce you, and just feel free to unmute yourself and start talking.

Is my sound coming in okay?

Yes, perfectly, thank you. I'm gonna go on mute.

All right. Thank you guys so much for having me today. I'm really excited to be here. Sean's told me a little bit about this group. He mentioned that you guys are a pretty frequent flyer for these type of sessions, so I'm excited to hear about some of your experiences and, Just what you've experienced from your perspective within your specific field, or your specific therapeutic areas, how much or how little you know about human factors, the current understanding you have of it, and then the current portions of it that remain a mystery, or kind of sticky points for you that you need support with as you go forward with whatever products

Therapies and clinical fields you work within. Alright. And then, Sean, am I okay to share my slides? Okay, let me pull that up. You'll have to forgive me. We're running study sessions today, so I've been plugging and unplugging out of a lot of different computers and camera setups, so my computer setup is a little bit less, standard than normal right now. Let me get those up. Alright.

Right now, as I switch it over, we'll see if it actually goes the way that I'd like it to. Can everyone see my screen okay?

Yes.

Okay, perfect. I won't… I won't be able to see you, so if you have questions, you'll have to interrupt me. We're down to one monitor, because my other ones are working on something else right now, or they tried to, anyway. All right, so kind of to kick us off for this, we're going to be talking about specifically why human factors, engineering within MedTech, why it matters from day one. So, thank you so much for coming in. Quick poll, how many of you have actually been involved in a human factor study before? I realize I'm asking this, but I won't be able to see any of you as this is happening.

So, a couple people raised their hand. Just drop and chat Y or N. Or, yeah, the thumbs up works. Looks like we're getting some answers here. Yes? Yes, I was involved as well.

Very good. And so, how you interact with human factors and usability engineering within your products, the different devices that you work on, can have a very big impact, no matter how big or small, or how much you're integrated into that process. So it sounds like some folks have some experience, so that'll be good. You'll have to forgive me. Typically, when we go into these things, we try to make sure that we're hitting the board for everybody, so whether you've had a lot of experience, if you have, you might recognize some of the stuff we'll talk about today. If this is newer to you, it might feel like drinking from a fire hose, so… but we'll get through it all together.

So, when we talk about what human factors engineering is, one of the best ways that I like to describe this to people within MedTech is describing it as the Russian nesting dolls. So, when you think of design controls, which is typically where some of our engineers live within, they're very familiar with that. That's a very comfortable space for them, they understand what those regulations and requirements are. That's our biggest Russian nesting doll. Then if you go to one down from that, we have risk management, and all of that ties into.

Okay, you may want to advance your slides.

Oh, I did, maybe it didn't go yet.

It didn't go. I'm clicking my computer. Sorry.

It looks like we're not seeing the presentation, we're seeing your.

Aww…

your other view.

That's okay, that worked.

Okay.

Hang on, I know how to do this. Da dot. Present from current slide…

By the way, Zoom would fail the human factors tests.

The trick is, you have to do… It is easier. I can share my screen so you could see everybody's…

Oh, yeah, Victoria, that might be great, that way I can see the responses. Would that be alright with you?

Absolutely.

Teamwork for the win.

Okay. I have to see how to share screen. Share screen…

Beautiful.

I was sick.

Perfect. And then we're gonna go to the slide with my little penguins. Huzzah! All right. So, as I was mentioning, design controls is our bigger Russian nesting doll. Then, when we look at it, risk management becomes your next one down, because all of those risk management activities are tied to what's happening within design controls. Oftentimes, when people are newer to human factors, the points of confusion come from, where does this fit in? I don't understand this. Historically, we haven't done this. Maybe this isn't part of our current workflow. So… By presenting it this way, you see how human factors is kind of within risk management. Everything that we're doing is about safety and efficacy. We want to make sure that these devices are safe and effective, and that they're also compliant. So the user needs to be able to use these in the context of use for that actual end user.

So we kind of become this, like, strange subset of risk management, where some of the tasks that we're looking at may be very closely related to the tasks you would see in design controls, but it's much more about who is doing it, where they're doing it, and does it make sense to them? We can see something built by one person and used by another, and they may not speak the same visual language, they may not have the same You know, background information, so it becomes a lot trickier for them to interpret how they're supposed to operate within a system. So, my next question for you is, when you hear human error, do you think of the user or the system? What does that mean to you when you hear human error associated with a device?

both. It's… the system doesn't enable the user to safely use the system.

We do, we get a little bit of both for those.

It's how the user interacts with the system. And I'm, the, convener of the user interface aspects for the fourth edition of 60601-1.

Oh, very good. Yeah, and now, when we think of that kind of box that the FDA shows us of how user interface works, that workflow and where those connection points are, that's where we start to see some of the breakdown of this. Are we seeing the breakdown within the system? How the person interprets that within their system? What does it look like for them? There's… oh, this is a funny one. Marcus put into the chat that there's stupid users and stupid designs. And it's hilarious because we see those things happen often. That classic, example of the doors, where we try and push or pull the door because it's aesthetically beautiful, it's a big glass door with one vertical handle, but it matches on both sides, so you're not sure if you're inherently supposed to push or pull.

So it's one of those things. It can be the environment that you're in, the context of use for that. If you live in a certain country, and all of the code… the doors are fire-coated so that it's an outward to get out of the building, no matter what that door handle looks like, you're gonna know which way you're going. In the States, we often see that that's not always the case. Might be some older coded buildings, it might just not have been something that was done before, so… Whether you're opening or closing to exit, you get a little bit more confusion in that, too. So, you can start to see kind of a breakdown of how does this person interact with the system? Where do we see the fault points? Is it their perception of the system? Is it their processing of what the system is telling them? Is it a failed action where they thought that they did something, but maybe they didn't push the button hard enough, or maybe the toggle flips the wrong way?

Yes, and Susan's got a comment in here for you, Sarah. So, the user interface, so whether that's a GUI system or a physical device, it might be functioning as intended. But that the actions that you take with that interface can result in an unanticipated use by the end user. So, that's where we start to look at things, too. Is the device operating as intended by the manufacturer? And if that's a yes, you're down one checkpoint. And then if we look at, but is it operating the way that the end user is expecting it to operate with that similar result? So that's where we see some issues pop up sometimes with negative knowledge transfer, if somebody is operating within something, assuming that it will act similarly to their iPhone or their tablet, but this is a GUI system attached to an in vitro diagnostics device. So you can start to see where there can be some really wonderful overlap to make it a quick

Uptake on a new device, and it also can cause a few issues if it's something where they've never seen it before, or they expect it to do something the way that it's shown. Let's see… And then I see Marcus has a question about something that was seen in one of his HF studies. Oh, the ISO symbol for storage conditions. These can be a really tricky one. So, when it comes to the ISO symbols, they are regulated. Those are determined by somebody else, and we have to test those oftentimes if they're associated with critical tasks. So making sure that the readers and end users can interpret what that means.

Let's see. And for this one, it looks like… There were storage conditions from 5 to 30 degrees Celsius. And then the question for them was, what does that mean? And the response they were getting was to store it at 30 degrees Celsius plus or minus 5 degrees. So yeah, you can see where this becomes a bit of an issue. Is it the range of 5 degrees C to 30 degrees C is acceptable, or is it 30 degrees Celsius with a plus or minus differential of 5? So that's going to dramatically change what's the correct answer for that, because one of them has a 25 degree range. In which you can go, and then the other one only has… A 10 degree range.

So we can move to the next slide, Victoria. We'll move on from our penguins. So… Why human factors matters at every stage? There's a few reasons. One of the big ones that we always understand quite clearly is the regulatory compliance. Human factors and usability is required by the FDA. MDR states that best of practice is what they're expecting when it comes to devices, and that ties into IEC2… IECISO 62366, which is kind of our blueprint for how we do usability and human factors testing. So that's the portion of it where it's required, and we're obligated to do that. The other portion of it, though, is kind of a business bonus for us. So, that's going to be your market adoption. It's going to reduce training burdens if you've done human factors and you've vetted this with your end users.

It can improve users' confidence in the product. That, in turn, just ups the delight factor. It makes people want to interact with your product more. And then you see people leaving institutions, going to new institutions, and they're asking, ugh, why is this the device that we have here? I much prefer this one. Here's all the benefits for it. So you've inherently got people going out and doing marketing for you because they've had a positive experience with your device. The big one, and this relates back to our Russian nesting dolls of how we see human factors and usability within design controls, is the risk reduction. So, because you've done this testing with end users prior to this device going to market.

you're gonna see a lot less issues when it comes to unforeseen use errors. So, you'll be able to, kind of avoid additional recalls. you'll see a reduction in adverse events. Again, these are things you really see on the back end of things, and it's hard to gauge what those percentages can be, but again, that adds, bolsters your reputation as a company. I think we're on slide 4. Is that what looks like it's right for everybody?

Yeah, and I'm just putting a message in the chat. Feel free to speak up, you don't need to use the chat, you know, unless you just want to write a written comment, but if you want to… if you want to interject something, you know, we're open to that here.

Yeah. And then, the other portion of this, before it even gets into the market, is the risk of rejection or delayed launch. So, if you go in and you skip out on your human factors, or if you have gaps within your human factors and usability engineering portion of your DHF, or your submission files. you do run the risk of a rejection from the FDA, and that can end up in a delayed launch, that can have a lot of negative impacts to, you know, your manufacturing, and any other contracts that you have out for products, especially if you're doing bundles for contracts, too. Has anyone seen a submission delayed due to the usability within their products? Or a product in general?

Yeah, for me, working in the human factors space, I've certainly seen this, and sometimes it's a balance, isn't it, Kay, between the commercial strategy of the company, so you might do a validation study, find that there is some significant use errors. But it's up to the manufacturer to determine what they want to do with that information, and if they want, you know, if they deem it a big enough risk to actually implement a change. And I've seen it where the manufacturers made the decision to not implement a change that would address the significant use error, and the FDA has came back and not liked that, and asked them to make the change, and therefore do the validation

again, so… Yeah, certainly something we've seen in the past, and as consultants, we try our best to convince manufacturers of the importance of addressing those risks, but at the end of the day, yep, it's up to the manufacturer to determine what they want to do there.

Absolutely.

Have also seen, And we've recovered some programs where they haven't included all of the user groups that are a part of the workflow, and when they go to submit, that is pointed out by the agency, and they have to go back and redo that study. So I think really understanding the user group, and making sure that we're inclusive, or we have a rationale as to why we don't need to include, a certain user group is critical, and I've seen those get pushed back.

Yeah, and oftentimes when you see a missing user group, that can be discovered during the actual testing itself. You may run into a situation where you're talking with a physician, and you're asking them to perform tasks. And you're seeing a lot of use errors, and in your debrief with them and your root cause probing, you're asking, can you walk me through this? What does this typically look like for you? How does this workflow work in your current institution? And what I've seen happen in those situations is they say, oh, well, I don't do that. And now you've got a gap. So you've got to identify who's the user that does that. Is that an institution-specific? Sometimes it's state policies. EMTs are allowed to do certain things in certain states and not others. So you have to be careful when you're looking for those user groups where you're doing your testing. So that is something that we've seen pop up. And when it comes to the…

residual risk is what Anne-Marie was talking about there. So when you get to the end of your… validation testing, everything that you go through, all of the use events, whether they were close calls, difficulties, use errors, you start rationalizing through those, picking them apart, assigning those root cause analysis, and then you get into the residual risk. And that's executed by the manufacturer. And within ISO 14971, that risk management regulation, it leaves it up to the manufacturer to determine the residual risk. So, being able to go through and dissect that, and figure out what does this mean for us as a company, does this need to change, is this risk acceptable? And then you go into your benefit-risk analysis of

Is this going to cause more harm than good? What is the harm? What is the positive that can come out of using this therapy? So, that becomes kind of another entire layer of the conversation prior to your submission. And one of the things that we always put forth to clients whenever we're doing residual risk, we kind of bucket things into different categories. So, you get your list of use errors that you're looking through, and you need to understand What can change? what that benefit-risk analysis is, and then what they're willing to move forward with for submission. So sometimes they are going to want to chance it and do the submission. Timeline, budgetary concerns, you never know what those different, pushing factors are, but

They'll do the submission, and the FDA may accept it, the FDA may reject it. So whenever those types of situations are happening, we always try to put forth the conservative, the suggested, and the riskier approach, and explaining specifically to the manufacturers what your options are, where things can go wrong, how you recover from those things, if it is a rejected submission, so that no matter what, you're making the most informed decision for yourself and for your product. Another example of when I've seen delays for submissions has been validations that went through on products that were not production-level equivalent. So, running into a validation test where you're still working with more of a prototype stage version of something, rather than a final version of something, that can be really risky, because the numbers that you're doing for validation are a minimum of 15 per distinct user group.

And if you're running that on something that's more of a prototype and doesn't match your production level equivalent for your final device. that's a quick way to get that relegated down to a formative test pretty quickly. So that's another one that we've seen pop up in the past. So, some of those little tricky bits that are a little bit more embedded within the regulations that we try to keep our eyes out for to make sure that we're setting people up for success when they're going into that validation test phase.

Hey, Kim. Okay, sorry, you mentioned, formative tests.

You wanna just give us a little, like.

Idea of what that means.

Sure! So, there's two major kinds of tests within human factors. The first one we do is formative testing, and the way to kind of remember this is that a formative test informs the design. This is the stage within your design and development, and even in concept stage, where you can make those tweaks. That's where we're looking at potentially iterative prototyping, you might have a few versions of the device, you may still be parsing through some of your user needs to figure out what features features are most important, least important, what may be just kind of, like, bells and whistles that end users don't really need, but it seems like it's something that's kind of edgy on the market for it.

So that's your formative bucket, and when it comes to that, there's not a set number of how many of those that you're obligated to run, but we do recommend doing those, because if not, what ends up happening is you're flying blind into what we call a human factors validation test. Now, in the EU, they call that summative testing, and that's how you'll hear it referred to in IECISO 62366, but those two things mean almost the same thing. your summative test and your HF validation test, that's where your user groups jump. So instead of 6 to 8 for your formative as a minimum per distinct user group, when we jump up to HF validation and summative, it becomes a minimum of 15 per distinct user group. And that's the… that's the test where we're looking for safety and efficacy.

So all of your qualitative questions that are more about feelings, likes, dislikes, preferences, a lot more of that lives in your formative testing, because when it comes time for your HF validation and your summative testing, I just want to know that this is safe and effective for use. And then there's usually a handful of subjective feedback questions we ask at the end, and it's really just around Do you feel like this was safe? Did you have any issues? Was the information clear and interpretable for you? So… That's kind of the big difference between the two, and oftentimes we'll see if people have lower budgets, they're less likely to invest in those formative tests, but it's really important because it actually saves you money on the back end. Because if you're running those formative tests, by the time you get to validation, you've seen end users with your product, you know what features are important or not important to them.

You've seen the errors that they're gonna have. So when it comes time for your validation report, there will be less unforeseen use errors, less explaining away, and your residual risk will be a lot easier to write. Leo, do you have a question?

Yeah, so I've never actually sat through and run the actual usability testing, so what type of testing do you do for the safety and effectiveness? Summatives.

Great question. So, the first thing that you do when you're building that type of protocol is you look at your use-related risk analysis, or your company might do a UFMEA. And what those are, are your use errors that can happen, so they're more related to the context of use, less related to the device design itself. So within those risk documents, you go through and you pull out your severity levels that rate on the high scale. Depending on your company, if you're on a 1 to 5 scale, I've seen companies go a little bit more conservative, where they do anything rated from a 3 to 5 severity. They list those as critical tasks.

I've also seen it where companies only do 4s and fives as critical tasks. But whatever your risk management and quality management team determines is a high severity within that scale, those are all the tasks that you're required to test for HF validation and summative testing. The other portion of that.

on the cast.

It is based on the severity level, yeah.

But the task-related items.

Yes. So, those severity levels determine your critical tasks versus non-critical tasks, and then you also want to test your mitigations as well to make sure that the mitigations are, in fact, doing what they're supposed to do and reducing the risk.

Yeah, of course, the mitigations make sense, because that's the risk management aspect of it. Okay.

And Susan, did you have a question?

Yeah, so I had a comment and a question. So the comment, quickly, was this complexity around combination products. human factors, where they don't do it based on severity, it's based on compromised medical care, so you run into a whole other level of, oh wait, now what do I do? Which then that leads to the question I have, is, there is no statistics being applied when you're deciding whether or not your human factor study was successful or not. So, at what point, other than you're trying to justify your use-related risk analysis, how do you judge whether or not that one use error is something that's unacceptable versus, no, one use error, that's okay?

Yeah, and you're right, we don't do probability within HF, and we don't do percentages of success versus failure when it comes to how well or poorly people did on whatever task it is. So for me personally, whenever I've had this pop up, I go through and do the same kind of mapping through. You're ending up testing all of your tasks in that situation to make sure that everything's covered and you're not missing anything, and then I would still include in my residual risk, or my benefit-risk analysis, the ratings of severity that are tied to that. So. we're delineating whether something is a high severity, whether it is a critical task, and then you have to interpret and parse out what that potential harm is that comes from the specific use error made. So, when we have the use errors defined and your success criteria defined in your HF validation testing.

whatever the failure mode is, isn't necessarily the harm that's going to be the highest rating for that use error. There may be something that ends up being more of a delay of therapy than it is an actual negative Or highly negative impact to the patient, like delay in therapy versus not being able to get a medication. at all, for an emergent use, those are two very different things. Actually, we'll use that as an example. So, if you have an EpiPen that's your auto-injector that's delivering a medication, delay in therapy? Critical for that, because anaphylactic shock is happening quickly. However, if you have something that is a…

Daily dose, of a certain amount of… like, 2.25ml syringe that you're getting daily, you have a store of them within the fridge, you get multiples. Your delay in therapy for that, it's gonna be a lower risk, because it's something that you're doing daily, there's backups inherently already in place. If there's an issue, you can phone the physician, and if you get that dose within a certain amount of hours, you're not going to see the negative impact. So for that, it becomes really based on the use error that happened, the potential harm to the patient coming out of that, and then understanding the context around that that we don't really have within the protocols. There may be systems in place, like phoning the pharmacy, the…

issue for a misfire dose or an underdose. Those things are already in place within the healthcare system so that they're able to step in and help those patients. that we're not considering, because we're not able to control those things as the manufacturer. So, there's additional safeguards put into place. Like, if you think of the Swiss cheese model, where the arrow gets through all the cheese pieces, because the, like, one horrible thing after another just lines up. Those cheese pieces are basically your lines of defense, and we control and test as much as we can from the manufacturer perspective, but

because this is a system and there's other people involved, there's inherent fail-safes within those portions of the process, too. So those are things you have to look at within your residual risk as well, what that typical process is that we just may not have control over. So it's not as strong of a justification, your residual risk, but those are things that are in place.

So when you're doing that, it's a… you're basically turning it into doing a risk assessment, even though they call it, you know, use-related risk analysis, but it's not. It's a hazard harm analysis. But when you're doing that activity, you're still asking those questions of, well, how likely is this thing going to happen? Because if it could occur frequently, even if it's low risk, that's also unacceptable. So you're basically doing a real risk management activity out of it, even though that's not exactly what the categories are when they talk about that risk.

It is, and we don't actually capture that in the URRA. That discussion that I just had about benefit-risk analysis and residual risk, that ends up going into your HF submission report. So after your HF validation or summative reporting that you do, you'll table out, here's what happened, here's the data, here's the use errors, here's some potential root causes. The actual residual risk definition and justification and research that you do into that typically goes into a separate document that's called your HF Submission Report, and that's kind of the big one that you want to hand over to the FDA as.

here's our human factors documentation for this product. You can… you can put it in the HF validation report. It's gonna make your HF validation report really long, but sometimes it's helpful for the context of use for the device, especially if it's a complex product, because then you can see that clear delineation, and from a quality engineer perspective, too, they like to see less information repeated in multiple documents, which I can agree with, too. So I can see the benefit of having it in either one. But in your actual use-related risk analysis, you're only going to go so far with that. We're not going to get into probability there. We're not going to get into some of those fail-safes that are outside of our control, because we can't take credit for them, because they're outside of our control. Those only come into play when we're talking about residual risk and benefit-risk analysis.

Hey, there's a few more questions that…

Oh, yeah. Came in.

In the chat. Oh, I see one about the FDA still calls the usability requirements as a guidance.

So, that's correct, and I think part of that is also a guidance, because And we've heard, Hannah, do you speak on this, one of the FDA team members at conferences before, where he talks about it's a guidance, because it's our job to interpret that and make sure that we're interpreting it correctly for the devices that we have that we're implementing. Which I think is a really beautiful way of discussing it, because he puts it forth as This is a government… Bret, like… Guidance document, and we've put it out there, but… They kind of take the stance of. which I can appreciate, they reserve the right to know more tomorrow than they did today, and they also reserve the right to leave things a little bit vague, because technology could render things within the guidance documents as irrelevant as things progress. So, they have to implement enough of a structure where

We can hold everyone accountable to make sure that they're doing the things that are best for end users and the patients themselves for their care. But they also don't want to limit the manufacturers in a way where they're holding your arms behind your back and not allowing you to progress through as technology develops. We've seen that a lot in how software has developed, how quickly some of that stuff can change. If we wrote regulations to that, the timeline of getting those things written, reviewed by, you know, cross-functional teams, across different therapeutic areas to make sure everyone agrees to get those published, by the time one of those got out, you'd be, you know, five regulations behind on the next one. And I think

Similar to risk management, one of the things within human factors is we want to just reduce the risk as much as we possibly can. We want to make sure that this is as safe as possible, that we've found all of the potential unforeseen risks that we couldn't predict. That we're making sure that what's going out is going to be successful for those end users, and that ultimately the therapy or care that the patients are getting is something that's going to benefit them. And as far as… where the line truly exists, like home devices versus hospital devices, that's a tricky one. When we look at things that are used by

I think it's less about the environment of use, and it's more about the end user. You have to take into consideration how educated your end user is in the medical field, how educated they are with this device specifically. And then it gets into the context of use after that. If you're looking at a highly specialized surgical robot that is only going to be used for a very specific type of brain surgery, your end users are probably going to be a very finite group. the education level within that, the precursors of what they've had to do to get there are going to be quite high. So you're working in a very controlled environment, even though it's a high-risk device.

And then you can see something that's… you know, a lay user would use. So, the general population, people that have no medical training at all, but it might be a very, very simple device that's very low risk. But that's intentional, it's because they don't have medical training. And I always joke when I say lay users, because I am also considered a lay user as not having medical device training. So it's not meant to be a vicious word, but it does sound a bit cruel sometimes when we talk about it like that. But you have to make sure that you're setting those people up for success and reducing those risks as much as possible. That's why we see things like auto-injectors having needle guards and shields, and some different devices even have the needles retract, because we want to make sure that for people that aren't trained in how to prevent sharps injuries, that they're inherently being taken care of by the device.

Honestly, that's one of the reasons why emergency medicine is one of my favorite areas to work within, because the context of use is varied, chaotic, and you have to build it for the worst case scenario, no matter what. When you get into some of these more surgical, clinical, in-institution procedures and devices, you have a much more controlled environment. Don't get me wrong, there can absolutely still be chaos in those environments. But you're working with more understood lighting, situation, teams often, and the communication between them. So, it's really interesting to see The level of… foolproofing, I guess is the way to best put it, that goes into different devices that are used in different environments.

V… I just saw a question pop up on sample size. Let me see if I have my books with me. So… I do. there is an article, and I want to say it's from probably 2003, that often gets cited for the law of diminishing returns. That's where we pick our sample sizes for this. We are not going to get statistically significant sample sizes within human factors. It's just not the way that it works. If it was, you'd be looking at numbers like you would have for clinical trials, which is not the case for this. So… And I'll find out what the article exactly titled is, and send that through afterwards. But within that, it shows you, within the Law of Diminishing Returns, how you get to identifying most of the use errors. So when we look at sample sizes that have anywhere from 6 to 8, or I've seen 5 to 8 be for formatives as well.

I believe that reduces, I want to say, about 80%. You can identify 80% of the errors that would happen within that user group for that. And then when you hit the 15 number, you start to see a little bit more of an incline, but after 15, that law of diminishing returns drops, and you're not gonna… the likelihood of identifying more use errors that would be common within the representative population drops after that. Now, the FDA is very specific about that minimum of 15 distinct users per user group for HF validation testing. That's not something you want to chance your arm on, by any means. Now, how you define your user groups is going to be grouping them by…

education level, the tasks they perform, that way you might be able to group in, like, nurse practitioners and RNs. They may end up being one user group, but multiple job titles. But if you have something where you have a physician and a nurse, and they're performing very different tasks, you're likely not going to be able to combine those user groups. Now, when it comes to formative testing, again, because the idea is to run multiple formatives, you have a lower number requirement. And I have seen anything from 5 to 8 accepted. It just depends on how much of those unforeseen use errors that you want to try and capture. If this is a new field for you, new therapeutic area, new device type, it's probably a good idea to lean more onto the larger numbers within your formatives to identify some of those use errors that you're not expecting.

If this is a field where you are a lot more comfortable in, you may be just running 5 at first within each user group, because you're more interested in getting more information across 4 user groups, rather than diving into one specific user group. Now, there is wiggle room to do analytical testing as well, or even one-on-one interviews. That's still helpful information, it's just not going to give you that mass overview to be able to identify those unforeseen use errors. So we would call those, like, key opinion leaders, or even doing sit-down interviews with different physicians. Those can be really, really helpful pieces of information to help you build out what you want your formative test to be. They may be helping you with user needs, different features, and that's really helpful to get that information and use it as input.

But it's not going to give you the same output that you would if you were running formative tests. The… Usability testing of medical devices. This is a good one that walks through different formative tests, and it gives you a little bit of background on the numbers that are required for that, too. This is something that I always have on my desk. I have a lot of the books that I like to reference. One of the things that I think is really wonderful about Human Factors is because it is a more… recent science, if you will, within the field, the information is very attainable. The publications that you can see in that, the history of human factors, it makes it so that if you're not sure why something is the way it is, you can go hunting for that, and you can find that information in the root source to those things.

I'm trying to see if there's other questions. Let's see, do you need a new summative? 2… Verify and validate your risk controls if the original summative was not successful, and have And you had to implement additional risk controls. This is a good question. So this depends, which is usually not the answer everyone wants to hear. So if you have a summative test or an HF validation. And let's say there's… Use errors that you determine are unacceptable, and you have to put in a new mitigation or a new risk control. The first thing you want to do is identify what those things are. So, the other part I should caveat with this is, if you have unacceptable risks, you're not going to want to submit that to the FDA. You need to address the unacceptable risks.

And whether or not you're gonna have to run testing on that depends on what those things are. So, understanding that… if you have an unacceptable risk, because it's still going to lead to serious harm, what's your new mitigation or risk control? Is that something that impacts the user interface? Is that… something that is within the physical build of the device. Is that something within the instructions? Is that something within the training? You hope to see something that gets implemented in the design if it's an unacceptable risk, and it seems that high. The other thing is understanding that maybe there was a use error, but the change to user interface is undiscernible. And what I mean by that is, it might have been a use error that happens because of a functional failure during the test… during the test… during the testing of the device itself.

If there's no change to user interface and the actions the people are performing are still the same, the end users, you may be able to get away with not having to do additional summative testing on that. If it's something where you've made changes to your instructions for use. I've seen it in the past where those are accepted and not required to be retested. This is where you'll run into… not hot water, but it depends on who your reviewer is as well. If you're looking at someone from CDER versus CDHR… CDHR, you're going to have a difference of opinion. So. The FDA reviewers that come more from that combination and drug space background have a little bit of a different interpretation of things. They're more likely to see, oh, your instructions for use has been updated, you need to run a new validation for that, because that is considered a change to user interface.

So, some of it depends on who your reviewer is, and just understanding how you phrase that to people as well. Making sure that you put all of your cards on the table, give them all of the information that you have so that they can make a clear assessment. And if you're not sure, you can always call the FDA. You want to schedule a pre-submission, you want to schedule a review with them, make sure that you understand or you're communicating to them, here's what we saw, here's our plan of approach, do you find this to be acceptable? Like I said, they always reserve the right to change their mind or know more tomorrow than they did today, but it's a good place to put yourself in if you're looking to move forward in a very risk-averse manner, so that you have a more calculated pathway going forward.

Kay, what's your thoughts on, using a comparative analysis or threshold analysis approach in that situation? So…

No, it's a good question. It's very… it's very device-dependent. So, for things that are… clear competitor device. Like, you can point directly to, this device works absolutely the same, very similar user interface. This has been a procedure that's in practice for 50 years, very minimal changes. You have a leg to stand on there. However… Then you can move into gray territory as well. So maybe this device Has a bit of that historical information, but it's got a new component to it, a new technology. It's doing two different therapies at once, and it wasn't previously before. you start to see a little bit of a chink in the armor there. You may be able to get away with

doing the threshold analysis, you may be able to get away with some of that comparative work, and pointing to standard of practice, this is within workflow, this is the anticipated method, like, if you have clear procedures for, you know, a heart transplant, and you can point to what those techniques are, and how this doesn't impact that, and how it's common knowledge. Yeah, you might have a leg to stand on there, but how many of those things are going to be varied from institution to institution? That's where it gets a little bit trickier. And if you full-on have a device that you're linking to as a competitor that may have gotten you into the 510K process and out of de novo, but they don't quite work the same, you're really not going to have a leg to stand on as far as threshold goes, especially when you start to try and break that down, and those user interface components are wildly different, or you have different

system components. Like, maybe this has software and the previous version didn't. I wouldn't even go down that road. That's not gonna lead you anywhere but heartbreak and a long time of comparing things that don't exist.

Excellent, and the reason I ask is, with Timo's… Timo's question around, do you need a new summative study to verify and validate your risk controls if the original summative was not successful? So where I've seen, you can use a comparative analysis or a threshold analysis in those instances. Is to look at, okay, what are the changes that you're making? If the changes aren't too different, and you've got evidence from previous that those tasks were done successfully and didn't cause any harm. then you can justify not having to revalidate. However, if there was a very critical task, so let's say

It's an at-home hemodialysis system, and the patient was unable to remove the bubbles, from the venous line, or the arterial line, then you've implemented a change. You can say that you believe that the change is going to mitigate that risk, however. the only way you can evidence that is by doing another summative study to actually… to prove that. So, sometimes you can use them, yeah, threshold analysis, I think it's the 2017 FDA guidance that talks a little bit about that for NDA submissions for combination products, but you can also use it for, medical devices, too. And then there's another question here, do you mind me jumping in, Kay? Because it's quite a good one. It's about, are you utilizing the FDA 2022 draft guidance to evaluate modified devices? So it's kind of, you know, on a similar topic there. How often are people using formatives to support justifications that critical tasks are not impacted?

Yeah, so for this one, this is gonna depend on… It depends on the critical task that comes through. I've seen it where if it's a legacy device, and there's been a modification, and you're doing this proactively, like, you've gone in, you've seen a legacy device be updated, there's a change to user interface made, and you're not sure if you're gonna have to do an additional validation for this 2.0. I've seen people go in and submit for the change, and run a conservative formative, maybe with some extra numbers, just to do a temperature check to see, is this going to be a problem? I've seen that be successful, but on the proactive end of it. If you submit, you get a rejection, and then you try and run a formative to say that you've de-risked it, you're going to get a no from that one. So, again, that comes down to your planning. Do you want to go in and have this in your back pocket?

in case they ask for it. It is the more conservative approach, but it may save you some heartbreak down the line if you've already run that formative test and you can point to it.

Okay, I was just gonna jump in as well, there was, there was an… Julian, and I think a bit of conversation back and forth between Julian and Susan. But Julian asked, how critical is performing a summative study with the final product from a validated process versus a representative How do the FDA see that?

Yes, so we get a little bit of wiggle room with production-level equivalent. You may not have your manufacturing lines fully set in place, and that's okay, but we do expect, you know, the appropriate tolerances, materials, physical appearance, the interactions that come out of that. We don't want to be sending something in that's more like a breadboard or a black box than the actual end Device itself, because that can have an impact on your critical tasks. So… Trying to think. Some of the issues I've seen with that is… Where somebody walks into their validation test, and they've got the production level equivalent of the physical device, but what they didn't realize is they decided to go in with their updated software that's not actually going to be going out to the world with this

physical product. So what ends up happening in that situation is you've just invalidated your validation test, because the software component that goes with the hardware component is not what's going to be out on the market. So you've just run an entire validation test for something That's not going out yet. So it's things like that that you want to be careful with. Like I said, by no means do you have to have your entire manufacturing line done, approved, everything in its exact place, because the FDA understands that if you do run into an issue and things have to change, or mitigations have to be made, the costs that can be associated with having to do those larger manufacturing changes that come afterwards.

And I know in a previous life, when I was in manufacturing, we had software, and it was the first, like, digital product that we had, developed. And then there was always questions, like, how many iterations in the software can you do until you need to revalidate? Because, I mean, making multiple small changes can actually turn into quite a big change. So there was… Kind of an unknown to when do we actually really need to revalidate the software altogether.

Yeah, and it's a hard… it's a hard line to draw. I think that's related to how much of the user interface has changed that's perceivable to those end users, how much of that impacts their typical workflow, and then just the severity of the tasks associated with the changes. There may be small changes where you've added, you know. a different hover, color, clickable portion to the buttons that may be perceivable, but it doesn't impact the workflow. All those things are physically in the same spot, the organization is in the same spot, they can still navigate everything from Software Update 1 to Software Update 7 on that. But it's when you start moving critical functions.

that are associated with those high-severity tasks, that's when you want to start reconsidering, do we need to test this again? Because if an emergency happens, and they can't find, you know. X menu within that IVD system, or within that pump system, you're not going to be able to navigate out of a really bad situation, and that's where, when it goes wrong, it goes bad very, very quickly. I'm trying to see if I've missed other questions. They come in, and then the next one's gone.

I think we've covered, a good chunk of them now, Kay, so yeah, if you're happy to move on to the next slides, that was a great conversation.

network.

Thank you so much.

So, these are the common misconceptions that come with HF and usability testing. I've heard this directly from the mouths of very intelligent, very well-informed people within different departments in med tech, that they believe that this is only for high-risk devices. That is not the case. I mentioned some of the lay users and how low-risk those devices are. Those are still devices that we have to test to make sure that it can go out to the public and they can operate it safely and effectively. Misconception number two, you can test it at the end. There's no bigger nightmare that plagues me than somebody handing me a device that's in design freeze and saying, can you human factors this?

Because at that point, I am… I am set up for failure, because I have to tell someone that their baby is potentially ugly, and it's never something you want to do. They've spent a lot of time and hard work and research, and they believe in this product, and I believe in them, but sometimes it's just those things that they haven't considered, and it's who is the end user, where are they using it, what is their frame of reference for how this product works? So if you wait until the very end, it's almost like the difference between writing a paper. You want it to be perfect, so you do everything you can to get it to that A-plus before you submit it, versus having some of those checkpoints along the way to find out if it needs to be adjusted in the meantime. Leo, what have you got for us?

I just want to say one thing. Saying lay users is one thing I want to avoid, because it's not just lay users, it's also the clinicians and professional users you have to address, too.

Yeah, absolutely.

All the users, really.

And those clinicians may be very well versed on what this product or this background is, but not always. Sometimes we're introducing new technology that's very unfamiliar to them, so we have to integrate them and treat them as if this is the first time they're seeing it, too. You're right, there is still a lot of training and understanding that can go into that for clinicians. And I think that's where misconception 3 comes in. It's just common sense. It's common sense to the person who has the blueprint of how to build it. It's not always common sense to the person who you've just handed this box to out of context and said, go for it! I've seen that in testing. Again, brilliant ER doctor comes in, 30 years experience, they've seen everything walk into their emergency room. I handed them a package and said, right, here's your prompt, here's the situation, I'm gonna have you perform the

as you typically would, hand them the package, and they looked at me like I was nuts. And that was a situation where we find out that that's not always the end user that does some of the setup for whatever that, you know, execution of the testing is. So…

And who reads the user manuals? mice.

No one.

My son is the exception.

Oh, I love that! You've instilled good values in him! We always have a little moment of celebration whenever we get readers in for, for usability testing, because it warms our hearts, because we're not used to seeing it. I've even had clinicians come in when we do testing with them, and certain parts of it, you know, you get to your usability testing, and you may have knowledge tasks that are required for comprehension and findability, so I have to get them into that IFU to make sure that what the manufacturer is saying makes sense to them, and they're communicating on the same, like, language level.

And… I've had them full-on tell me, oh, I would never read this. I don't even know where these are in my institution, I don't read directions, and they'll have a good laugh, and they'll tell me when they do it, too. And I usually just have to say, oh, I'm so sorry, I know it's not part of your typical workflow, but I need to make sure that what we're communicating to you makes sense to you, and you know what to do with that after having interpreted it. So it does take a little bit of extra nudging sometimes to get them into those test sessions where we have to evaluate parts of the IFU, but you're right.

Well, part of the issue is the requirements require IFU requirements, and Service manuals, and symbols that no one understands.

Yes.

And there's tons of symbols.

Yes. The crossover of graphic design and actual ISO symbology does not quite exist, in my opinion. I think some of them are pretty spot on, but the amount of things that they have to have these symbols for, it just doesn't always make sense, and it's really hard to get them through some of those. The amount of use errors that I've seen pop up in testing where they think that the manufacture date is the expiry date, and I've had people look at me and be like. what does that mean? And they're pointing at the hourglass, and I'm like, I don't know, what do you think it means? What does that mean to you? So sometimes we have to just walk them through that and understand it, because we don't have a choice in what those ISO-regulated symbols are.

That's why a lot of times you'll see the symbol, and the definition, and the entire thing written out next to it. So it'll have the hourglass, and it'll say expiration date, and then the full date, rather than just the symbol and the date, because we want to ensure that they do understand what that is, where it is, and the difference between that and the manufacture date. That's when I see.

Which then needs to be translated in every language.

And that's why they don't want to put the text next to it. So you end.

I know.

kind of competing section of how many pages is this IFU gonna be, like… you know, from a design perspective, the goal is, you know, the IKEA visual language, where you don't have to put all the text in it, but because we have these mandated symbols, and it's going to be a user if they're getting it wrong, we still have to include that information in it.

Exactly.

So, I'm gonna interrupt for a moment, because I can, and I would like to say, Kay, this is fascinating talk. You didn't get through all your slides, so if anybody would like the slides, please ask for the slides by writing slides in the chat, and I will make sure that you get a copy of the slides. Number two, we're gonna have to have you back, along with other members of the Claramed team. Now, I understand that Claramed, is, making an offer to consultants. to, work with them in some sort of partnership arrangement. It's not written in stone exactly what that's going to be, but we invite you to, explore partnership opportunities to refer clients, either them to you or you to them, that need human factors work done. So if you have

clients that have or may need human factors, please write a partner in the chat, and we will reach out to you, and, arrange a call to talk about the potential collaboration. Lastly, they're also starting a new program specifically for startups. So if you're interested in learning more about their startup program, please write startup in the chat. That'd be really useful. And because, you know, the sooner startups start doing this testing, start considering human factors, the less expensive it's going to be for them. That's the takeaway message that I want you to remember from this call. The sooner you start human factors, the less expensive it is. The later you start, the more expensive it is. Keep that in mind. Now.

Lastly, I just did a post on LinkedIn, and I want everybody to like it, so that we knock the algorithm out of the park. Please help me with that. That's live from our event here, from the session this morning. And, let's see, join us next week for State of the Art and Strategy with Robert… than Buxtall, and… Jelly, and I'm sorry, Schraerin, I can't say his last name. Those are my announcements. Kay, I'm gonna let you bring it home. You did a wonderful job today, and please, Closing thoughts in the last two.

Yeah. Thank you so much for having me today, and for being part of the discussion. I always really, really enjoy that. There's so much of human factors and usability that is interpreted based on the specific products you have and their context of use. Everything is about context. The user, the use environments, the people that are building it, and it's something that I just really personally enjoy. So, if there's anything that you want follow-up questions on, absolutely message me. I'm definitely a book nerd when it comes to human factors, and I'm big into making sure to help share resources that are available to make sure that you're informed and you can have that,

Very intelligent and well-informed conversation with your own colleagues, too.

I'd like to thank you all for joining us today, for your time, and for your participation. Those of you who asked questions, and those of you who just listened, it's wonderful to see so many people here, and special thanks to Claramed for participating in this, and Kay, you… particular. And everyone from Clarimed, thank you so much.

Okay.

Yes, thank you.

Okay…

Thank you, James.

Thanks, Kay, it was really good.

any…

Appreciate it.

Bye.

Bye!

Hi.