Beyond the Protocol: A Realistic Resource and Planning Guide for Human Factors Validation Success

Hello and welcome, everyone. My name is Sean Smith from MedTech Leading Voice. I'm super excited about this morning's webinar. Which is, beyond the protocol, a Realistic Resource and Planning Guide for Human Factors Validation Success. presented with ClareMed. Let's see. Yeah, this one filled up really fast. I was just super excited to see everyone here. I don't know if you saw my post this morning, but I'm gonna put a link in so you can show some love to the presenters. Few quick housekeeping announcements. One, we are recording this session. Everyone who registered gets the recording and a PDF of the slides.

So if we drop at the half hour, you're still covered, or you're not able to attend, too. Questions, please put your questions in the Q&A for the speakers. Otherwise, they might not see them if you put them in the chat. Use the chat function, however, to communicate with me. That's fine. And… let's see… If you would like a certificate of attendance, please drop the certificate. in, the, chat, along with your name, and as you would like it to appear on the certificate and your email address, and we'll get those out in a day or two. We have to make them by hand, so please be patient. Let's see…

And, the last thing is, you know, we have a couple upcoming webinars, one tomorrow on cybersecurity. I'll drop a link in for that, and in two weeks, a clinical trial webinar, what to do when enrollment stops. Both are going to be really good. I encourage you to join them now, let's see, with… I'm going to introduce today's speakers. So, Denise Forkey is, Director of Human Factors and Advisory at Clara Med. 25… she has 25 years in medical Product development, the last decade in human Factors consulting across everything from wearables to robotic surgery systems. Leah Bergquist is Associate Director of Human Factors Engineering and

Boston site lead at Claramed, also 25 years in product development for devices and combination products, including R&D, core team lead at a large global manufacturer. Kira Favica, I hope I said that right, spent her career, until recently as a Senior Human Factors Design Engineer at Medtronic. she brings the view from large manufacturer side on how cross-functional teams understand the human factors process, and I understand, or misunderstand it, and I understand that she's… going to medical school. So, congratulations to Kira.

Yep.

On that. Let's see. That's it from me. I'm gonna let Denise take it from here, and be quiet for a while.

Oh, son. So, just to give you guys some background, first of all, thanks for attending today. We're excited to share, the webinar, and if you have any follow-up questions or want to speak with us, we're more than happy to have follow-up conversations as well. So, we… Kira and I were at a lunch at HFES with other human factors professionals, and part of that conversation was, you know, I really wish my management understood what is actually involved in human factors. Validation studies, and anyone who's run a human Factors study knows the importance of planning, resourcing, project success, and lack of attention in this area certainly is where studies get repeated, submissions get questions, and trend lines slip.

So, the work that's done up front really makes sure that your HF Validation Study succeeds, but it's mostly invisible to those who are outside of human factors, and so today we really wanted to highlight some of the planning and resourcing needs for project success, and, Are looking forward to sharing that with you in the webinar. So first, we wanted to just throw up, kind of, a really brief infographic about the human Factors and Usability Engineering process. Not all organizations understand the importance of having human Factors personnel involved in the product development team, but Having a human Factors and usability engineering focus within product development and incorporated into the project plan can actually help ensure that it becomes a really core consideration

And is not an afterthought after the fact. So, human factors and usability isn't just assessed at the human Factors Validation stage, but it actually needs to be incorporated, assessed, reworked throughout the product development cycle in order to make sure that you're de-risking your human Factors Validation. And really, I think the main importance of this cycle and the human Factors process is making sure that we collect information feedback from users, because we as engineers, as human factors professionals, as management, can't really predict exactly what that user experience is, and how this device will be used by them. So, making sure that we're gathering that feedback along the way is really important to designing the right product.

The right user interface, and one that is easy and safe to use. So, this is just a graphical representation of two different projects, and it's founded in, kind of, the realities of what happens when you implement human factors and usability engineering late. So, if we look at the purple graph, we'll see that Implementing the human Factors and Usability Engineering process into product development early may, Increase the cost up front. But it helps you to refine the user interface so that the validation study passes on the first time. So, this kind of early part of the curve is where you're getting the product in hands of users, you're iterating, you're changing the UI, you're making sure that it's working for them.

And then you pass that validation on the first try. As opposed to the pink graph, where maybe you don't put a lot of focus in human Factors up front, but then you get to the validation, and you have a late realization of product issues, ones that don't match the user needs, ones that The UI is confusing, and at that point, it's really late in the process, so you may have to do expensive retooling, you may have to make major hardware or software changes. Or maybe you're on the totally wrong track. And that can actually affect your costs substantially. in your ability to make changes, because it may be that you can't make the design changes you need, but instead you have to implement labeling and training changes, and maybe institute more support costs along the lifetime of the product. So it's always easier to make changes while they are cheap, and the early-stage work de-risks your human Factors validation. So, as we were kind of talking about how we

de-risk human Factors Validation. This was a key point that Kira and Leah and I really wanted to make sure that we made, so that we're not losing the forest for the trees. So, as we mentioned at the beginning of the webinar, there may be a misunderstanding of resources needed, planning, cost to run a successful human Factors Validation. This work is invisible many times to those not involved in the process, and they think that there may not be a need for a dedicated timeline, or budget, or incorporated… or human factors incorporated into the project plan. But, Human Factors actually requires planning, it requires lead time, and mental preparation, even before the first session is run.

People outside of human Factors may think moderating sessions is just like moderating sessions for an R&D study, or preclinical, or clinical study, so anyone with that experience can just immediately step in. I, for one, came from more of a preclinical background, and the first human Factors study I was involved in was really eye-opening to me, because with the different goals of the studies. You actually end up using a totally different methodology. And so someone in a clinical or preclinical world where they're used to over-supporting the user to make sure the product is used perfectly. then steps into the human Factors role, and they end up biasing the study. So, making sure that you have the right people to be

executing the Human Factors study is really an important and key principle that we'll talk about today. And finally, there's a perception that human Factors Validation study data is only quantitative, similar to a verification study that you can quickly write up the data within a few days after the last session is completed. But that actually is where a large portion of the work begins. You actually need to do more of a qualitative assessment of the human factors Factors data that's collected, and this involves evaluating what the root cause might be for those use errors and use problems, as well as a residual risk assessment of the findings.

So it's not a study where you say, oh, well, a 95% success rate passes. It's definitely a lot more to that, where we're involving The risk management personnel, we're looking at the data, we're looking at the root cause to really evaluate, is this product safe to use, commercially? And there's a cost of misunderstanding the resources and planning, and those costs, I think, are not unfamiliar to many of us in the medical device space, but they certainly are ones that we like to avoid. So it could mean that you need to repeat the study if you have untrained moderator, unusable data. You're gonna have to run it again, and that second study is going to cost way more than some small tweaks that you need to do in the first study to make sure you have the right people running it.

Or there could be a delay, you could stall your submission, where the FDA is in the middle of reviewing your data, and they have lots of findings or questions for you, where you either have to navigate that process, or potentially collect additional information to repeat that study. Or worse yet, maybe you didn't find use errors because the study personnel weren't trained in kind of looking for those use problems, and you end up discovering a really major problem when it gets into the field, and then your product has to be recalled, and the trust that the general public and the users have in your product, it gets degraded. And so, there might be delays to adoption of other technologies that your company runs. So all of these things are things that we're really trying to avoid.

So today, we're going to go through 6 common mistakes. These are not the only mistakes, but they're definitely some common ones, and we want to make sure that you can also prevent those from happening, so we'll give you some key tips and tricks there. We'll also talk about how to appropriately plan the resources. And plan to minimize those issues in human factor studies, and we'll go through an argument to take to your R&D management for the time and resources that the Human Factors study actually requires. So, the six common mistakes that we'll be talking about, for planning and resourcing are poor project scope and planning.

Mismanagement of project and resources, lack of attention to recruitment. Not dry running or running a pilot for your protocol, use scenarios and moderator script, using poorly trained personnel, and underestimating the time and the skill necessary for doing the post-session analysis and reporting. And at this point, I'm going to pass it over to Kira.

Great, thanks, Denise. So, let's think about when you're running a human Factors study, the first mistake that you would need to try to avoid is poor scoping and planning. So, with scope, this might look like miscommunication between your product development and human Factors team about the status of the UI development, or any changes that have been made since your last study, and then misaligning on what aspects of the UI you're actually expecting to get feedback on. Or even what type of feedback you're looking to get. So you might show up to the study with the wrong people, or a team who aren't on the same page, which can make a mess of your data collection, and ultimately with poor plotting.

poor planning, you might just misset expectations for the timeline of the study execution, deliverables and results. So as we'll see with all six of these mistakes, like Denise mentioned, you can imagine any one of these issues could lead to inaccurate data or incomplete data collection, or delays to the study. what can we do about it? So once it's been decided that you need to run a formative or a validation study, ideally, as you're making that decision as well, you need to align with your study team on the goals and the purpose of the study. So why are you even talking to users? This can look different for a validation than it would for a formative, or studies, for changes to a device versus a new product development.

And you need to think about what product or prototype you're testing, are you testing the whole UI or just certain parts of interest, to make a design decision or evaluate certain risks? What features have already been tested? What hasn't? And so understanding this what and why of the study are critical for developing how. What's your protocol gonna look like? How much time do we need to prepare it? How many How much time do we need with users, and what your actual execution's gonna be. So, these are the first conversations that need to be decided as a team, so that you can go to management with clear expectations of the time and resources needed.

So, for a case study that we can talk about. On the next slide. I was recently pulled in to run a formative study on a patient monitor for a project team that already had their timeline set around their software development of the monitor, without fully understanding the time needed to plan the study. I was unfamiliar with the product being pulled in last minute, so before we could think about what the study was even gonna look like, or when we could run it. or if we were meeting timeline, I needed to work with the R&D team to close these gaps in our understanding of the UI and the current progress of the product development to define our scope, which is our what, and the goals, which is our why.

So then on the next slide, we can talk about the impact. So, what this means is aligning on what aspects of the user interface were actually under evaluation for this study. The first question that often gets misunderstood when you're working with a development team is what actually counts as user interface and impact to your user workflow. So it's something not to be taken for granted to align on. Once you understand the scope, only then can you determine which users and how many to include, how long is your session gonna take, and what type of use environment do you need to test it in. None of your contracting with a recruitment or facility partner can even get started until you have those questions answered about

how much time you need from your space and from your users. And that can take, you know, weeks to align on your scope and weeks to then align on your contracts and get those approved before you're even starting active recruitment. So it's important to take all of these steps, in order and In a timely way. So, some challenges for this situation, is that, I came to the team needing to close the gap on the understanding of the UI, but they didn't have any definitive tracking of what changes they'd made to the UI. They had a full list of all the software changes that they'd been making since their last study and their last version of the software. But there was no reference to what impacted UI and what didn't, so it was tedious to go back and forth with the team to determine which software changes were internal and which were actually showing up in the UI. This is made even more difficult, like I said, when there's confusion over what gets flagged as a UI change, when things like

font size or color or small wording changes to things on the screen were considered small and not impactful. And that's something that needs to be evaluated with your Human Factors team. So, with this method, we did have to adjust our study contracts after already getting started, which put risks to the start of the study being delayed, and changes to the scope protocol all the way through the week before we ran the study. So in summary, once we had a general idea of the what and the why with our scope and our purpose, we could set a clear timeline and begin our longer lead time items with the contracting, and do that successfully, and then working on the protocol development, your how, you can do that with fewer surprises and modifications needed along the way.

I will pass it along for our next mistake.

Thanks, Kira. The next mistake that we often see is mismanagement of the project and its resources. So that may include a whole slew of things. We captured several of them on the right there, but there may be a facility availability conflict, too little test Test articles available are not considering that the right configuration of the test article was built, maybe you might have a customs problem, maybe you forgot to go to IRB, or that you didn't account for enough time for data analysis before a submission. And that can really lead to, again, incomplete data and or delays in your submission. So what are some solutions for that?

One, make sure you're actively managing your project, having periodic check-ins to make sure that you've cleared any roadblocks the second that they come up, and confirm that the study goals are being met before you even start your work. Map out your pre-session activities. Do you have a recruitment screener in your IR… do you have to do an IRB review? Make sure all your documents are lined up, including protocol, script, data collection sheets, and your informed consent. And as Denise mentioned before, it's important to do a dry run and a pilot, both of them, to make sure that you have any documentation

Documentation updates ahead of your first session, that needs to be done. And planning for test articles is a big one that we see. Oftentimes, people think, oh, we're testing 15 people, we only need 15 test articles. Well, don't forget about dry runs, don't forget that people might forget, drop a product, don't forget… I once had an entire box of study articles damaged during shipment, and the second box was fine to get started, but we had to recreate a whole box, so… think about some of those backup, scenarios that you might need to plan for in, for sure, customs. A lot of, at least on our side, we have a lot of clients that are global, and we always ensure that they make sure that they're shipping their product, through customs with plenty of time to spare.

Next… Planning your facility and study use environment, whether you're doing your product… your study internally or externally, making sure that you have your location set and have assessed the cost and availability of that location, that your equipment and supplies are ready to go and on-site. And some of those things are what you might think of, little nuances that you need to make sure that you have your use environment set up appropriately. Are you… are you in a home use environment? Are you in a clinic office? Are you in a surgical suite? Do you have all of the components to adequately simulate those?

Those situations. And then making sure you're planning for different project activities. Again, recruitment's involved. Who's writing your pre-study documentation? On your day of sessions, do you have a backup moderator or observer in case someone hits a snafu or is ill? Who's going to be turning your… the room around to make sure it's ready for the next participant? And, and yeah, who's doing that data analysis reporting in the end? It's not often efficient to have the same team do reporting. Next. Mistake number 3 is a lack of attention to recruitment, and I think this is a big one. Making sure that, you know, if there's a poorly written screener, you might have the incorrect population recruited.

you may not have allotted enough time for recruitment to happen. It often can't happen in a week, and often not less than 4 weeks, depending on the type of Person you're recruiting for. And then you may have had inadequate incentive, or not enough people to recruit for no-shows. So how do we have… what are some solutions for paying attention to recruitment? One is to start early, integrate that recruitment lead time in your study timeline, and not run it concurrently. Make sure that you set yourself up to start that right away. And then you can also make sure that you set inclusion and exclusion criteria with enrollment targets that are clear for your recruitment team, so you don't have a mismatch of getting the wrong participants on the day of study.

And, you know, they can reach out to the interested participants early from their database, even if you don't necessarily know the exact study dates, they can start to look in their database, and they can start to tell you if the recruitment is even feasible in the area. So, yeah, again, gathering that recruitment feedback early. they'll let you know if you have a really hard recruit, and you need 6 or 8 weeks, is it even possible? Some recruitment firms will even say, no, we don't think we can recruit this, and they'll let you know that you need to look elsewhere or in a different geographic area to be able to

To get your population that you need. So, start that work early, for sure. And, yep.

Did I hop in with the poll?

Sure.

So, I'm gonna share the poll results with everyone. So we have nearly 200 people on the call, half participated, so what we see is 31% from Medjek's startup. 34% from manufacturer, 25% consultancy, and then we have, 23% working in human Factors Usability. 32% in regulatory affairs, quality. R&D, engineering, consultant, advisor, other. So there's that. Now we're gonna run poll number two. If I can. To this rate. And, please participate, let's see… polls and quizzes… Share results… Hmm…

Okay, while I figure out how to launch the next poll, I'm gonna let them continue. Just a gentle reminder, if you have questions, please use the QA. function, and we'll try to get to them time permitting. If not, they'll follow up afterwards. Thanks.

Thanks, Sean. Sarah, getting back to some solutions for recruitment, plan to over-recruit, I don't know that I've been involved in a study where everyone came in on time and were the right people that we needed, so plan for no-shows and cancellations, and have backup participants on hand, ready to go, so if someone doesn't show up or they give you a call in the morning, can you get that person in on the ready? And that really closely relates to communication with your recruitment team and your study team. Who is the go-to person on day of? And, well, leading up to day of, but also during day of, when you have to have someone readily available on both sides. The recruitment team may have got a cancellation, needs to let the study team know, and vice versa. There may have been a no-show, and the study team needs to let recruitment know that they need to quickly

Get… get a replacement for those… those participants. I think that's it. So, a little case study here on recruitment. We once had a client that was doing an HF validation study for type 1 diabetes, patients and caregivers. There was 30 adults, 15 adolescents, and 15 caregivers with split, experience with and without pump, you know, insulin pumps. The challenge that the client gave us is that they were on an accelerated timeline. The sessions were long, four and a half hours. That included several, hours of training, and, Am I missing the word?

Help, Denise. delay, what's that? The break between training?

Decay.

Oh, decay! I kept thinking of delay. DK. So yeah, training, the decay, and then several hours of the session conduct. And where it really hurt us was with those, adolescents, and even the caregivers of those adolescents who are trying to work around a school schedule. And we kind of had a uniform group… a unicorn group with adolescents without pump experience, because the standard of care for an adolescent who is newly diagnosed with diabetes is for them to go on a pump quite quickly. So it's hard to find someone without pop experience. And then, originally there was a… we knew that there was a single geography recruitment would… would be a problem.

So how did we solve for these problems? One, we recruited across, two different locations and ran the study, in parallel. We added evening and weekend sessions to allow for better flexibility for the adolescent participants and their caregivers. We expanded the inclusion criteria, so we did educate our client about the challenges of standard of care in the United States with those adolescents not having surrogate ex… I mean, pump experience, and we put together a rationale as to why we did not need to have diabetics with… without pump experience… experience. It could be any adolescent without pump… without any experience, or without diabetes.

We did an over-recruit, 3 to 6, participants per user group, because we understood that there was a large cancellation rate for that type of group. And then, yeah, we made sure that we were ready to backfill cancellations within 24 hours to keep on that timeline. And our key learnings from this study was definitely involve recruitment early, so you can learn about their database and their timing and scheduling, and get their evidence for whether their inclusion criteria and exclusion criteria is going to be a problem. And then, again, recruitment will help you know whether those no-show rates are gonna… what kind of effort is gonna need to be done to over-recruit for over no-show rates in that target population.

And you can learn from that. Alright.

Alright, so mistake number 4 that we'll talk about is not dry running your protocol and use scenario setup. So, without that, you might find that your session time is too short for your workflow, that you're missing equipment that you need to support your prototype device, or create the simulated environment the way that the user expects, and you might find that your prompting is biased or confusing, or just your intended study workflow isn't panning out the way that you thought it would once you have a user with you. So, what's the solution for this one? Pretty simple, just practice. Just like you would in R&D, where you do benchtop and feasibility testing, before you jump into formal data collection, with human factors, we first perform dry runs. So, you're walking through your protocol, reading your script out loud, acting out tasks with your equipment in real time to make sure that you have a good workflow, good prompting.

And the right equipment to properly simulate your use environment. This can be done really early on with more or less formality, and it can be done multiple times. But when we think about our last dry run, it should be what we call a full pilot session with a user. So, that could be an internal mock user or an actual recruited user, depending on your needed fidelity, and, available resources, but that pilot session is going to be your best method to fully test your script and your datasheet, and even how to prepare to respond to potential close calls or use errors, or even equipment mishaps that you can anticipate. And it's a great way to get your whole study team aligned and prepared together.

So, dry runs and pilot sessions are… Sorry.

Can I just jump in real quick?

Yeah.

So one of the things also about the dry run is it can be very helpful if you have someone from your clinical team, or someone who has a really good experience of what's happening in the field with that user population, to help make sure that those prompts make sense, to help make sure that it's representing a real-world situation. So, I just wanted to jump in real quick to help people understand that that's a really key thing That can be useful early on in looking through your moderator script.

Yeah, and the same as we've mentioned multiple times, if you're doing this pre-validation, pre-formative, for different types of devices, this could look different. There could be different things that are useful to do here, such as bringing in your clinical team. So in general, your dry runs and pilot sessions are going to be super helpful, but also only if you give yourself enough time to then make the necessary changes before your study begins. So, with another case study that we have, for a different validation of a patient monitor, we were set to run a 30-person validation with a consulting firm, and the plan was to spend Monday familiarizing the consultants with the product and finalize the protocol dry runs, as they hadn't had an opportunity to get their hands on the device yet. And this was followed by two pilot sessions on Tuesday morning, and the first study sessions

that afternoon. So, clearly, this is an example of an ideal timeline, as the study team should have been introduced to the product well before it's time to run your pilot sessions, so that they can have that to aid in their development of the protocol and reduce risk of any possible delays if you do find any issues. And additionally, we didn't have a ton of time then to polish our updates, and we were putting a global team in a tough position, where those protocol approvals needed to be completed, you know, in the middle of the night, because we only had a few hours between the start of our sessions. But what this did show was that even in a really tight timeline, we prioritized the inclusion of dry runs and pilot sessions, which proved to be well worth it. So with this rollout,

Due to some travel delays, which are always a factor in study situations. It was 6 p.m. on Monday, and we were still working through the dry runs and had an open question of how to set up the screen on the monitor. So, with these monitors, different facilities, or even just different users might have certain orientations or layouts on the screen that they prefer. And this hadn't come up on the phone between R&D and the Human Factors teams until the Human Factors teams was able to explore the UI on their own, and then it was an open question of whether the layout should be maintained between users for consistent data, or unique to each user to match their typical use. And so then come Tuesday, when we ran the pilot sessions with the users, and we had actual recruited users in this case.

We were able to run through our full protocol and update some of our prompting to reduce confusion and bias, and then also try out these different monitor layouts. And I'd say another huge benefit to these pilot sessions is that because it's not part of your actual data collection process, you can even pull back the curtain a little bit and get your user feedback on how the session went, and what would be the most effective way for them to be working through those tasks as realistically as possible, and they can help you get the right protocol in place. So, in summary, from this scenario, we learned that the value of dry runs and pilot sessions, in testing your script and setup, when it's the last chance to finalize your protocol, before you collect your data, and ideally, we would run those informal dry runs and be familiarizing your study team with the physical product well before, those pilot sessions even start.

and giving yourself enough time after your pilot session, to make effective updates to your protocol and have time to deliberate there. But between the dry runs and pilot sessions, your whole study team can practice the workflow and set expectations about what a full session will look like, and what your data might even look like, which will help you be as effective as possible once your real participants are ready.

Thank you. So, the fifth mistake that we'll talk about is using poorly trained personnel. So, this could mean a trainer that doesn't understand that they could introduce bias while they're training a participant before a session. It could be moderators or observers who lack an understanding of how to do root cause probing, or may introduce biasing questions into the session. It could also be personnel who have too little understanding of the user interface, or they're not trained in the protocol or the use-related risk analysis. So some solutions to this would be to use human factors professionals to run studies. At a minimum, we feel that a human Factors professional as a moderator is a key, important step.

They understand how these studies are run, they understand their role, how they're supposed to roll out use scenarios instead of walking people through task by task. And they help make sure that any of those follow-up questions are asked in an unbiased manner, and subjective feedback is collected, root cause probing questions are asked to make sure that you have the information you need when you do that. Root cause analysis after, and the residual risk assessment after. It's important to train the study personnel on the study protocol, on the use-related risk analysis, on the moderator script, the data collection sheets, so that they're very familiar with what they'll be doing in the study and how it will be done, so that you have the data you need at the end.

And then finally, it's important to train the study support personnel and the trainers, because everybody involved in the study, whether it be the person greeting the participant who's answering their questions about what's going to happen in the study session, or whether it's the trainer who's giving them information, or whether it's people who came on site and they might want to talk to the participant in the middle of the training decay, we don't want them biasing the And so making sure that they're aware of, kind of, what's allowed, what's not, how these studies are run is really key. So, we'd like to share a case study, and I would say that this is not an abnormal case study by any means. We regularly see

clients coming to us and saying, hey, we tried to run the study ourselves, with our R&D personnel, our clinical personnel, and the FDA came back with questions. So, in this case, it was a robotic surgery system where the company tried to run it with their own internal personnel and R&D personnel. They ended up… the moderator script had biasing prompts. The FDA was not happy with those. There was inadequate root cause probing. And so the data was incomplete, and it was biased. So the FDA flagged these things, and what ended up having to happen is the the manufacturer had to repeat the study, and had to do it in a very quick timeline to make sure that they had what they needed, for the FDA. And it was a little bit under the gun, because they were trying to work with FDA to review the protocol under their SIR, and they had to come back, they gave us feedback, and then we ended up

having to implement additional changes prior to running the session. And so, that can actually take A little bit of time when you're under the gun, and it's much better to plan the study up front with the right personnel, with the right questions and methodology, so that you don't have to be under the gun collecting data later. So in this case, we feel that if they had used appropriately trained human Factors personnel, they probably would not have been in that situation, and so it's a critical key learning, in our opinion. The last mistake that we'll talk about today is understanding the time and the skills necessary for doing the analysis and writing up the report.

Some of the mistakes that we've seen are you have non-human Factors trained personnel writing up the data, they don't understand how it should be written, they're writing it up more like an R&D verification report. 95% of them were successful, and so it passes. Unfortunately, that's not going to fly with the FDA. Also, it takes time to do root cause analysis and to evaluate residual risk, and make sure that you have enough time to do that at the back end of the study. Your report can't be written up within one minute. excuse me, one day or two days. So make sure you have that time. And then also, maybe you do a preliminary release of the data, and people start using that for making decisions, but unfortunately, the data changes because the data's not been locked down.

So, some of the, Some of the things that you can do to prevent this from happening is to plan for qualitative analysis time in your reporting. And making sure that you've identified an appropriate amount of time. Typically, it's weeks instead of days, to do a report, and so making sure that you have that time necessary. Also, map out all of the reporting activities. Ideally, you would do this up front in the project, but at the beginning of the reporting timeframe, it's important to check back in and say, okay, what is reasonable? What can the team expect? And so, making sure that you have the right number of personnel and the skill level for those activities. And then communicate that timeline with your team, with management, up front, so that they know when they can expect things, and they're not preemptively using data that they shouldn't.

So, we've discussed all of these different mistakes. Many of the preventive measures include aligning your team, your management, making sure that you know what your user groups are, what your UI scope is for evaluations, making sure you're mapping out pre-session activities and resources and post-session activities and resources, starting early with recruitment, integrating the recruitment into the study schedule. Integrating recruitment into, kind of, the collection of information. testing the moderator script and data sheets, using an experienced moderator, and setting expectations early related to report writing and delivery dates. So all of these preventive measures actually help with

Staying on schedule, reducing time delays, making sure you have high-quality data, and making sure that you're not incurring additional cost to the project. where insufficient data or inappropriately collected data can impact them. So we've talked a lot about all of these things that you can do, and about the time necessary for completing the project, and Leah will be presenting kind of a sample slide of a timeline that you can use as well. But what you can do is there are options for flexibility. The options for flexibility include maybe getting a dis… if your timeline is fixed, the management's like, nope, this is a critical

key milestone for our organization. It makes or breaks us as a business. then you can ask for additional support personnel to help with the project, you can ask for additional study teams, maybe running concurrent sessions. You can ask for additional geographies. Maybe you can't recruit the number you need as quickly in one geography, so how can you do that in multiple? You also might involve someone like, a consulting firm. We work a lot with clients who have a very firm schedule, and there is a need to hit a timeline, and so helping them creatively figure out how they can do that while not impacting the quality of the data is something that we do on a regular basis.

And then you can get additional recruitment firms, you can broadly train people that are not in Human Factors to help support you, give them that human factors background, train them on moderating, train them on what data looks like and how you need to collect this data and have supportive evidence for your reporting. So, the key is really to understand those trade-offs, and to balance the time needed, the costs. That are, incurred, as well as the quality of the data that will result, because the quality of the data will directly impact those other two, and may drag the cost and the timeline out if you don't have sufficient quality.

So there's really this strong trade-off between time and money. You can pay the price and dedicate the timeline for the quality data, or you can risk the cost of repeating the study, regulatory delays, or even recalls. You pay now, or you pay even more later. If your timeline is fixed, the cost shows up elsewhere in the money and the resources it takes to produce a complete, unbiased data package for regulatory submission.

So, let's think about, making the case to management in these situations where, if you're, human factors engineer, at a manufacturer, or if you're just on the R&D team, and you are proposing that, you know, a study needs to be run, how do you make that case? So, first of all, you want to focus on selling the outcome and not the cost. So, thinking about all these common mistakes that we just discussed, and all of the details that go into successful planning and execution of a study, really focus on the need to have appropriate resources to ensure success. As opposed to getting bogged down in all of those details, and the cost of each one.

Number two is show the trade-off. So, be clear about what decision needs to be made for time versus money, and again, be clear about how that time is the critical resource for success. And then lastly, you can bring the evidence. So, please lean on the human Factors expertise that you've learned, and that you've acquired, and knowing that it comes from folks who have ran a lot of different types of studies, seen a variety of feedback from different regulatory bodies, and also seen success and failure in product launches into the field.

Kira. So, earlier, Denise mentioned the timeline, and if there's, like, for me, I really love to see a timeline, especially in my earlier career, to be able to, you know, not be making this up when, you know, have the management office scratch their head when I say it's going to take 3 months to do a human factor study. Why? So we wanted to give you something to walk away with as a, you know. Can this take long… less than 3 months? Yes. Can it take 6 months? Yes. It depends on the ease or simplicity of your study, but I think as a general rule of thumb, it takes a good 3 months to be able to get through all of the scoping and planning and recruitment and…

data analysis. And included in here as part of your reporting is, you know, alignment with your team on the results and getting that approved in your quality system. So there's a lot more that you have to remember that goes into the timeline, so hopefully this helps you have something to To defend your… your case with your management team. And so what to take back to your team if there's any… if there's someone that couldn't make this call today? I think we… we've all realized that most of the work that makes HF validation successful is… seems to be invisible to the people who are funding it, unless you let them know. So educating them is important.

In the common… Failures are planning failures, so… Too little lead time, too few resources, untrained study personnel, don't forget to, Do some of the solutions that we suggested for those items. And then plan and resource it once. As Denise mentioned in the financial trade-offs, repeating a study, is quite expensive, so planning it right and doing it right the first time can save your team a lot of money just in the funding, but also, you know. Getting to the field earlier to be able to make those, have income come in as a commercialized product. Time matters. And bring the trade-in and evidence to management so that the spend and the timelines are defensible.

That's it. Denise, I think you're the next one?

I think it's Sean.

Okay.

You ready for us, Sean?

There I am. So, sorry about that. Yeah, so I've been posting some notes and some links, just a reminder to, put your questions in the Q&A. Now, we've prepared a MedTech usability prep kit. If you'd like the prep kit, you can… Either ask for the download link, or just write prep kit, and we'll send it to you. And the same thing for the 7-point readiness assessment. If you'd like to spend 30 minutes. with, someone from the Clarimed team, one of the SMEs there, walking through your particular project, it would be time well spent, and they're gonna go through the 7 aspects that you need to be considering.

So, yeah, and then just a reminder that everyone is gonna receive the slides and the recording, and we're gonna, Finish up shortly with a, a survey at the end. Thank you for everybody that answered the poll questions, and now we can go to Q&A, if you'd like.

Excellent. We do have some great questions that you guys have submitted. Thank you for submitting those. Feel free to send some additional ones as well. We'll be able to answer some of them live here, but we won't be able to get to all of them, so we'll try and reach out and answer your questions after the webinar if we can't get to them live. So, the first question that I think we should address is from Haley. Thank you for submitting your question. For small startups with budget restrictions, how can they ensure the FDA will accept the usability Validation Study while reducing costs by performing some of the HF activities in-house?

What are key aspects or stages to check in and make use of HF professionals? So, I can go ahead and start the answer, and then I'll open it up for, the other panelists. So, at ClareMed, we regularly work with small startups that maybe don't have the budget, to hire a full human Factors consulting firm to execute the study. And so, one of the key things that I think is important is I always recommend having professionals run the study, but if you absolutely cannot, there's budget constraints, then you'll have to figure out how internally. You'll have to come up to speed on things that are necessary for running a study. That could mean educating yourself, it could be bringing in human Factors consultants, like I do regularly, work with other small startups.

to say, okay, let me walk you through some of the requirements, let me help you understand, kind of, what are the key points that you need to consider in your protocol, in your moderator script. You can have them reviewed by professionals. And then I always recommend, though, making sure that you have a trained moderator, because the trained moderator will definitely make or break your sessions, and if you cut on that cost, you will end up paying later, because the FDA will find the issues, and they'll find the problems. So, at a very bare minimum, I would say make sure you have a trained moderator.

But there are other ways that you might be able to save on costs, like recruitment. If you have a large network of of users that you already know, or your team already knows, maybe you can support some of the recruitment efforts. Or maybe there's other ways that you can save on costs. Each one of these is kind of an individual study consideration, and so I would encourage you to reach out to professionals and to ask questions and to kind of gather your feedback before you embark on this journey. Leah or Kira, did you have additional questions or answers to that question?

Yeah, no, I think that probably the biggest, you know, compromise that we end up seeing is, to your point, educating people on kind of what the content of some of the documents needs to look like, and then doing a review of the final document to highlight any areas of concern. So, we do the recommendation. you write the protocol, the UFMEA or URA, any of the other supporting documents, and we can come in and do a quick check to make sure that there's not any major flags.

So, the next question is from Benjamin. Thank you for your, question. So, thank you for sharing your hard-earned learnings. If senior management are unwilling to consider HFUE appropriately, where would you be most comfortable compromising on your timeline? So, I think, Compromising on the timeline means that you'll have additional resources needed, potentially additional costs needed to help make sure that you're still gathering that high-quality data. So, if you, before you're ready to start your validation, if you already, reach out and start getting, recruitment, support, and start kind of working on that super early, so that as you're ready to really scope out the project, you can

run through that quickly, then that's one way that you can shave off on time. There may be other ways to kind of start working on that protocol sooner, working on the session sooner, running parallel sessions. Running, getting additional support in report writing. Having people involved in the report writing, kind of, in parallel, so that they… you can kind of bolster and work off of their, efficiencies to try and make sure that it gets completed. Leah and Kira, do you have additional thoughts for Benjamin?

I think I would just emphasize that, recruitment isn't really something you can budge on. With the recruitment timeline and how long it takes for your recruitment partner to find, the users that you need, you can't… force them to do that any faster, and you also obviously need to end up with the correct number… correct sample size for your study. But the thing to think about is that, you know, unlike some other types of engineering testing, it's not something that you can readily redo. So if you get through your first participant and realize this didn't go well, we need to make some changes, that participant's a wash, and you're gonna have to re-recruit, you're gonna have to re-plan. So…

if that's, you know, another case that you can make at all, that it's not… human factors in that sense does not parallel other types of R&D testing, where it can be more easily, repeated or, adjusted. It has to go right the first time, or you're gonna redo everything.

Alright, the next question is from Shanie. We are preparing to launch usability testing soon, but project timelines are tight. What is the single most compelling metric or artifact you use to convince project managers that post-session data analysis and root cause analysis require as much calendar time as the testing itself? Maybe, Leah or Kira, if you want to take that as a start?

Sure. You know, I think when… when… whenever I've been challenged in the past. It's, you know, providing evidence as to why it's going to take so long. You know, what is… what are the pieces of data that you're collecting? What are the type of data? Is it quantitative and qualitative? If they or teammates have to be involved in it to make it move faster, maybe you can ask for help from their team if they want it to happen faster. So I think just really explaining and educating What the type of data that needs to be, and how it impacts the entire team, and submission process goes a long way, and… and if, you know, when it…

depends on your personality type, but I love to get people involved who want things to move faster and help them own the responsibility of making it move faster, so, that's probably, you know. corral the team, to rally, to make it move faster if it really does have to. Of course, be careful in making sure that people are trained, adequately trained to make those analyses, but… I think that's probably my approach. Other people might not be that way.

No, I think, kind of along a similar vein, rather than, you know, necessarily including people in the process afterwards, but informing them ahead of time of what it looks like by actually training them and fully taking them through the process, I'll often, before a study, either with the project team and the management, so that they understand what we're working with and what timeline we have and why. Or with the, you know, R&D team members who are gonna come and potentially note-take, or observe, or be part of the process. I… carve out a significant chunk of time in bringing them up to speed in what Human Factors is. What does it mean to collect qualitative data? What does it mean to have a biased… biased prompting? What does it mean to… what's the root cause? Like, why are we collecting all of this data, and what's the use here? So…

to try to make the process less invisible. Unfortunately, we have to do the work to make it visible, to those people that are involved or in charge of decision making. But I do find that, you know, if I take somebody through an example of, you know, study results that I've gotten. For a study with 10 to 15 people, and, you know. 20 tasks, and once they see how much information that is that needs to be parsed, that can help.

Yep. Better answer.

So, we might have time for one last question, So, this is from Sarah. What is the difference between summative usability and human factors studies, and is the latter acceptable for Class 2 devices that go through 510K? So, do you guys want to answer it, or should I go ahead?

for it.

Okay, so a summative usability evaluation is a term that's used in IEC 62366-1, the International Usability Standard. And human Factors Validation is a term that's used in the FDA guidance. And so they typically are one and the same. They're both trying to evaluate the final product to ensure that it's safe and effective for use. For the users and the intended use environments and the intended, for the intended use. So, a lot of times they can be used interchangeably. The human Factors Validation, FDA has specific requirements, like they typically want to see a 15… at least 15 users per user group.

Whereas that's not clearly defined in the usability standard. So, there are times that you might have two separate studies, but generally they're trying to accomplish the same thing. And that's the same thing for a Class 2 device going through 510K. So I think we're at the top of the hour. As I mentioned before, we're happy to answer any follow-up questions, happy to meet with you if you have a need to meet to go through your validation plans, and we'll definitely follow up with the The other questions that we weren't able to answer live on the call.

Thanks, everyone.

Thanks, guys.

I'd like to thank the speakers, and I would like to thank all of you who showed up today, and even those of you who just registered and are going to watch it later. Thank you so much for your participation, and Show some love. Thanks, bye.

Bye, thank you.

Thank you.