What the session covered
Hosted by Sean Smith, this session featured Denise, Kira and Leah discussing the planning and resourcing work that sits behind a successful human factors validation. The speakers argued that much of this work is invisible to management and to teams outside human factors, which leads to unrealistic timelines, insufficient budgets and repeated studies. They set out six common mistakes: poor project scope and planning; mismanagement of project resources; weak recruitment planning; failure to dry run the protocol and use scenarios; use of poorly trained personnel; and underestimation of the time needed for qualitative analysis and reporting. Case studies covered a patient monitor, a diabetes study with adolescents and caregivers, and a robotic surgery system that had to repeat its validation after FDA questions. The speakers offered practical fixes, including early recruitment, pilot sessions, trained moderators and root cause probing. They also described the cost trade-off between planning now and paying later, gave a three-month planning rule of thumb, and suggested how to make the case to management. Q&A covered startup budgets, fixed timelines and summative usability versus human factors validation.
Figures
Figures the speakers cited
Source: Kira Favika
Chapters
Jump to a moment
Times run from the start of the recording.
Slides 14
1 / 14
Take these with you
Six checks
Full recording
57:29 · follow along with the transcript belowPlays from YouTube when you press play. Cookie settings
Transcript
Edited for readability · 9,153 words00:06 · Denise Forkey
So just to give you guys some background, first of all, thanks for attending today.
00:10 · Denise Forkey
We're excited to share the webinar and if you have any follow-up questions or want to speak with us, we're more than happy to have follow-up conversations as well.
00:20 · Denise Forkey
So we, Kira and I were at a lunch at HFES with other human factors professionals and part of that conversation was, you know, I really wish my management understood what is actually involved in human factors validation studies.
00:39 · Denise Forkey
And anyone who's run a human factors study knows the importance of planning, resourcing, project success.
00:46 · Denise Forkey
And lack of attention in this area certainly is where studies get repeated, submissions get questions, and timelines slip.
00:54 · Denise Forkey
So, so the work that's done up front really makes sure that your HF validation study succeeds, but it's mostly invisible to those who are outside of human factors.
01:05 · Denise Forkey
And so today we really wanted to highlight some of the planning and resourcing needs for project success and are looking forward to sharing that with you in the webinar.
01:18 · Denise Forkey
So first we wanted to just throw up kind of a really brief infographic about the human factors and usability engineering process.
01:26 · Denise Forkey
Not all organizations understand the importance of having human factors personnel involved in the product development team.
01:33 · Denise Forkey
But having a human factors and usability engineering focus within product development and incorporated into the project plan can actually help ensure that it becomes a really core consideration and is not an afterthought after the fact.
01:51 · Denise Forkey
So human factors and usability isn't just assessed at the human factors validation stage, but it's actually needs to be incorporated, assessed, reworked throughout the product development cycle in order to make sure that you're de-risking your human factors validation.
02:12 · Denise Forkey
And really, I think the main importance of this cycle and the human factors process is making sure that we collect information feedback from users.
02:24 · Denise Forkey
Because we as engineers, as human factors professionals, as management, can't really predict exactly what that user experience is and how this device will be used by them.
02:38 · Denise Forkey
So making sure that we're gathering that feedback along the way is really important to designing the right product, the right user interface, and one that is easy and safe to use.
02:52 · Denise Forkey
So this is just a graphical representation of two different projects and it's founded in kind of the realities of what happens when you implement human factors and usability engineering late.
03:06 · Denise Forkey
So if we look at the purple graph, we'll see that implementing the human factors and usability engineering process into product development early may increase the costs up front, but it helps you to refine the user interface so that the validation study passes on the first time.
03:25 · Denise Forkey
So this kind of early part of the curve is where you're getting the product in hands of users, you're iterating, you're changing the UI, you're making sure that it's working for them, and then you pass that validation on the first try, as opposed to the pink graph where maybe you don't put a lot of focus in Human Factors up front, but then you get to the validation and you have a late realization of product issues, ones that don't match the user needs, ones that the UI is confusing.
03:57 · Denise Forkey
And at that point it's really late in the process.
03:59 · Denise Forkey
So you may have to do expensive retooling, you may have to make major hardware or software changes, or maybe you're on the totally wrong track and that can actually affect your costs substantially in your ability to make changes.
04:15 · Denise Forkey
Because it may be that you can't make the design changes you need, but instead you have to implement labeling and training changes and maybe institute more support costs along the lifetime of the product.
04:28 · Denise Forkey
So it's always easier to make changes while they are cheap.
04:31 · Denise Forkey
And the early-stage work de-risks your Human Factors validation.
04:35 · Denise Forkey
So as we were kind of talking about how we de-risk Human Factors validation, this was a key point that Kira and Leah and I really wanted to make sure that we made so that we're not losing the forest for the trees.
04:53 · Denise Forkey
So as we mentioned at the beginning of the webinar, there may be a misunderstanding of resources needed, planning costs to run a successful Human Factors validation.
05:04 · Denise Forkey
This work is invisible many times to those not involved in the process.
05:08 · Denise Forkey
And they think that there may not be a need for a dedicated timeline or budget or incorporated or Human Factors incorporated into the project plan.
05:17 · Denise Forkey
But Human Factors actually requires planning.
05:20 · Denise Forkey
It requires lead time and mental preparation even before the first session is run.
05:27 · Denise Forkey
People outside of Human Factors may think moderating sessions is just like moderating sessions for an R and D study or preclinical or clinical study, so anyone with that experience can just immediately step in.
05:42 · Denise Forkey
I for one came from more of a preclinical background and the first human factors study I was involved in was really eye-opening to me because with the different goals of the studies, you actually end up using a totally different methodology.
05:57 · Denise Forkey
And so someone in a clinical or preclinical world where they're used to over-supporting the user to make sure the product is used perfectly, then steps into the Human factors role and they end up biasing the study.
06:11 · Denise Forkey
So making sure that you have the right people to be executing the Human Factors study is really an important and key principle that we'll talk about today.
06:22 · Denise Forkey
And finally, there's a perception that human factors validation study data is only quantitative, similar to a verification study that you can quickly write up the data within a few days after the last session is completed, but that actually is where a large portion of the work begins.
06:43 · Denise Forkey
You actually need to do more of a quality qualitative assessment of the human factors data that's collected.
06:49 · Denise Forkey
And this involves evaluating what the root cause might be for those use errors and use problems, as well as a residual risk assessment of the findings.
07:00 · Denise Forkey
So it's not a study where you say, oh, well, a 95% success rate passes.
07:06 · Denise Forkey
It's definitely a lot more to that where we're involving the risk management personnel.
07:12 · Denise Forkey
We're looking at the data, we're looking at the root cause to really evaluate is this product safe to use, use commercially?
07:21 · Denise Forkey
And there's a cost of misunderstanding the resources and planning.
07:25 · Denise Forkey
And those costs, I think, are not unfamiliar to many of us in the medical device space, but they certainly are ones that we like to avoid.
07:34 · Denise Forkey
So it could mean that you need to repeat the study.
07:37 · Denise Forkey
If you have untrained moderator, unusable data, you're going to have to run it again.
07:43 · Denise Forkey
And that second study is going to cost way more than some small tweaks that you need to do in the first study to make sure you have the right people running it.
07:51 · Denise Forkey
Or there could be a delay.
07:53 · Denise Forkey
You could stall your submission where the FDA is in the middle of reviewing your data and they have lots of findings or questions for you where you either have to navigate that process or potentially collect additional information to repeat that study.
08:10 · Denise Forkey
Or worse yet, maybe you didn't find use errors because the study personnel weren't trained in kind of looking for those use problems.
08:19 · Denise Forkey
And you end up discovering a really major problem when it gets into the field.
08:24 · Denise Forkey
And then your product has to be recalled, and the trust that the general public and the users have in your product, it gets degraded.
08:33 · Denise Forkey
And so there might be delays to adoption of other technologies that your company runs.
08:39 · Denise Forkey
So all of these things are things that we're really trying to avoid.
08:43 · Denise Forkey
So today we're going to go through six common mistakes.
08:46 · Denise Forkey
These are not the only mistakes, but they're definitely some common ones.
08:50 · Denise Forkey
And we want to make sure that you can also prevent those from happening.
08:54 · Denise Forkey
So we'll give you some key tips and tricks there.
08:58 · Denise Forkey
We'll also talk about how to appropriately plan the resources and plan to minimize those issues in human factor studies.
09:06 · Denise Forkey
And we'll go through an argument to take to your R&D management for the time and resources that the human Factors study actually requires.
09:17 · Denise Forkey
So the six common mistakes that we'll be talking about for planning and resourcing are poor project scope and planning, mismanagement of project and resources, lack of attention to recruitment, not dry running or running a pilot for your protocol, use scenarios, and moderator script, using poorly trained personnel and underestimating the time and the skill necessary for doing the post-session analysis and reporting.
09:49 · Denise Forkey
And at this point I'm going to pass it over to Kira.
09:53 · Kira Favakeh
Thanks Denise.
09:54 · Kira Favakeh
So let's think about when you're running a human factors study, the first mistake that you would need to try to avoid is poor scoping and planning.
10:02 · Kira Favakeh
So with scope this might look like miscommunication between your product development and human factors team about the status of the UI development or any changes that have been made since your last study, and then misaligning on what aspects of the UI you're actually expecting to get feedback on, or even what type of feedback you're looking to get.
10:20 · Kira Favakeh
So you might show up to the study with the wrong people or a team who aren't on the same page, which can make a mess of your data collection.
10:28 · Kira Favakeh
And ultimately with poor planning you might just miss that expectations for the timeline of the study, execution, deliverables and results.
10:37 · Kira Favakeh
So as we'll see with all six of these mistakes, like Denise mentioned, you can imagine any one of these issues could lead to inaccurate data or incomplete complete data collection or delays to the study.
10:47 · Kira Favakeh
So what can we do about it?
10:51 · Kira Favakeh
So once it's been decided that you need to run a formative or a validation study, ideally as you're making that decision as well, you need to align with your study team on the goals and the purpose of the study.
11:01 · Kira Favakeh
So why are you even talking to users?
11:04 · Kira Favakeh
This can look different for a validation than it would for a formative or studies for changes to a device versus a new product development.
11:14 · Kira Favakeh
And you need to think about what product or prototype you're testing.
11:17 · Kira Favakeh
Are you testing the whole UI or just certain parts of interest to make a design decision or evaluate certain risks, what features have already been tested, what hasn't, and so understanding this what and why of the study are critical for developing how what's your protocol going to look like, how much time do we need to prepare it, how much time do we need with users, and what your actual execution is going to be?
11:43 · Kira Favakeh
So these are the first conversations that need to be decided as a team so that you can go to management with clear expectations of the time and resources needed.
11:52 · Kira Favakeh
So for a case study that we can talk about on the next slide.
11:57 · Kira Favakeh
I was recently pulled in to run a formative study on a patient monitor for a project team that already had their timeline set around their software development of the monitor.
12:07 · Kira Favakeh
Without fully understanding the time needed to plan the study, I was unfamiliar with the product being pulled in last minute.
12:14 · Kira Favakeh
So before we could think about what the study was even going to look like, or when we could run it, or if we were meeting timeline, I needed to work with the R&D team to close these gaps in our understanding of the UI and the current progress of the product development, to define our scope, which is our what, and the goals, which is our why.
12:31 · Kira Favakeh
So on the next slide we can talk about the impact.
12:34 · Kira Favakeh
So what this means is aligning on what aspects of the user interface were actually under evaluation for this study.
12:42 · Kira Favakeh
The first question that often gets misunderstood when you're working with a development team is what actually counts as user interface and impact to your user workflow?
12:50 · Kira Favakeh
So it's something not to be taken for granted to align on.
12:54 · Kira Favakeh
Once you understand the scope, only then can you determine which users and how many to include.
13:00 · Kira Favakeh
How long is your session going to take, and what type of use environment do you need to test it in?
13:05 · Kira Favakeh
None of your contracting with a recruitment or facility partner can even get started until you have those questions answered about how much time you need from your space and from your users.
13:17 · Kira Favakeh
And that can take weeks to align on your scope and weeks to then align on your contracts and get those approved before you're even starting active recruitment.
13:27 · Kira Favakeh
So it's important to take all of these steps in order and in a timely way.
13:33 · Kira Favakeh
So some challenges for this situation is that I came to the team needing to close the gap on the understanding of the UI, but they didn't have any definitive tracking of what changes they'd made to the UI.
13:47 · Kira Favakeh
They had a full list of all the software changes that they'd been making since their last study and their last version of the software.
13:54 · Kira Favakeh
But there was no reference to what impacted UI and what didn't.
13:58 · Kira Favakeh
So it was tedious to go back and forth with the team to determine which software changes were internal and which were actually showing up in the UI.
14:05 · Kira Favakeh
This is made even more difficult, like I said, when there's confusion over what gets flagged as a UI change, when things like font size or color or small wording changes to things on the screen were considered small and not impactful.
14:19 · Kira Favakeh
And that's something that needs to be evaluated with your human factors team.
14:23 · Kira Favakeh
So with this method, we did have to adjust our study contracts after already getting started, which put risks to the start of the study being delayed and changes to the scope protocol all the way through the week before we ran the study.
14:38 · Kira Favakeh
So in summary, once we had a general idea of the what and the why with our scope and our purpose, we could set a clear timeline and begin our longer lead time items with the contracting and do that successfully and then working on the protocol development.
14:52 · Kira Favakeh
Your how you can do that with fewer surprises and modifications needed along the way.
14:58 · Kira Favakeh
I will pass it along for our next mistake.
15:01 · Leya Bergquist
Thanks, Kira. The next mistake that we often see is mismanagement of the project and its resources.
15:09 · Leya Bergquist
So that may include a whole slew of things.
15:11 · Leya Bergquist
We captured several of them on the right there, but there may be a facility availability conflict, too little test articles available, or not considering that the right configuration of the test article was built.
15:27 · Leya Bergquist
Maybe you might have a customs problem, maybe you forgot to go to IRB or that you didn't account for enough time for data analysis before a submission, and that can really lead to, again, incomplete data and/or delays in your submission.
15:42 · Leya Bergquist
So what are some solutions for that?
15:45 · Leya Bergquist
1.
15:45 · Leya Bergquist
Make sure you're actively managing your project.
15:48 · Leya Bergquist
Having periodic check-ins to make sure that you've cleared any roadblocks the second that they come up and confirm that the study goals are being met before you even start your work.
15:59 · Leya Bergquist
Map out your pre session activities.
16:02 · Leya Bergquist
Do you have a recruitment screener in your ir?
16:04 · Leya Bergquist
Do you have to do an IRB review?
16:07 · Leya Bergquist
Make sure all your documents are lined up, including protocol, script, data collection sheets, and your informed consent.
16:14 · Leya Bergquist
And as Denise mentioned before, it's important to do a dry run and a pilot both of them to make sure that you have any documentation updates ahead of your first session that needs to be done.
16:28 · Leya Bergquist
And planning for test articles is a big one that we see.
16:31 · Leya Bergquist
Oftentimes people think, oh, we're testing 15 people, we only need 15 test articles.
16:37 · Leya Bergquist
Well don't forget about dry runs.
16:39 · Leya Bergquist
Don't forget that people might forget to drop a product.
16:42 · Leya Bergquist
Don't forget, I once had an entire box of test articles damaged during shipment and the second box was fine to get started but we had to recreate a whole box.
16:52 · Leya Bergquist
So think about some of those backup scenarios that you might need to plan for in for sure customs.
17:00 · Leya Bergquist
A lot of at least on our side we have a lot of clients that are global and we always ensure that they make sure that they're shipping their product through customs with plenty of time to spare.
17:13 · Leya Bergquist
Next, planning your facility and study use environment, whether you're doing your product, your study internally or externally, making sure that you have your location set and have assessed the cost and availability of that location, that your equipment and supplies are ready to go and on site.
17:31 · Leya Bergquist
And some of those things are what you might think of little nuances that you need to make sure that you have your use environment set up appropriately.
17:40 · Leya Bergquist
Are you, are you in a home use environment?
17:41 · Leya Bergquist
Are you in a clinic office?
17:42 · Leya Bergquist
Are you in a surgical suite?
17:44 · Leya Bergquist
Do you have all of the components that adequately simulate those situations?
17:50 · Leya Bergquist
And then making sure you're planning for different project activities.
17:54 · Leya Bergquist
Again, recruitment's involved.
17:56 · Leya Bergquist
Who's writing your pre study documentation on your day of sessions?
18:00 · Leya Bergquist
Do you have a backup moderator or observer in case someone hits a snafu or is ill?
18:06 · Leya Bergquist
Who's going to be turning the room around to make sure it's ready for the next participant and, and yeah, who's doing that data analysis reporting.
18:13 · Leya Bergquist
In the end, it's not often efficient to have the same team do reporting.
18:19 · Leya Bergquist
Next mistake number three is a lack of attention to recruitment.
18:23 · Leya Bergquist
And I think this is a big one.
18:25 · Leya Bergquist
Making sure that you know if there's a poor, poorly written screener, you might have the incorrect population recruited.
18:33 · Leya Bergquist
You may not have allotted enough time for recruitment to happen.
18:36 · Leya Bergquist
It often can't happen in a week and often not less than four weeks depending on the type of person you're recruiting for.
18:44 · Leya Bergquist
And then you may have had inadequate incentive or not enough people to recruit for no shows.
18:50 · Leya Bergquist
So how do we have.
18:51 · Leya Bergquist
What are some solutions for paying attention to recruitment?
18:55 · Leya Bergquist
One is to start early.
18:58 · Leya Bergquist
Integrate that recruitment lead time in your study timeline and not run it concurrently.
19:03 · Leya Bergquist
Make sure that that's you set yourself up to start that right away.
19:06 · Leya Bergquist
And then you can also make sure that you set inclusion and exclusion criteria with enrollment targets that are clear for your recruitment team.
19:15 · Leya Bergquist
So you don't have a mismatch of getting the wrong participants on the day of study.
19:20 · Leya Bergquist
And you know they can reach out to the interested participants early from their database.
19:24 · Leya Bergquist
Even if you don't necessarily know the exact study dates, they can start to look in their database and they can start to tell you if the recruitment is even feasible in the area.
19:35 · Leya Bergquist
So yeah, again, gathering that recruitment feedback early, they'll let you know if you have a really hard recruit and you need six or eight weeks.
19:44 · Leya Bergquist
Is it even possible?
19:46 · Leya Bergquist
Some recruitment firms will even say, no, we don't think we can recruit this.
19:50 · Leya Bergquist
And they'll let you know that you need to look elsewhere or in a different geographic area to be able to get your population that you need.
19:58 · Leya Bergquist
So start that work early for sure.
20:02 · Leya Bergquist
And yep.
20:03 · Sean Smith
Did I hop in with the poll?
20:05 · Leya Bergquist
Sure.
20:06 · Sean Smith
So I'm going to share the poll results with everyone.
20:10 · Sean Smith
So we have nearly 200 people on the call, half participated.
20:15 · Sean Smith
So what we see is 31% from MedTech startup, 34% from manufacturer 25 consultancy and then we have 23% working in human factors usability, 32% in regulatory affairs, quality R and D engineering consultant, advisor, other.
20:37 · Sean Smith
So there's that.
20:38 · Sean Smith
Now we're going to run poll number two.
20:42 · Sean Smith
If I can do this right and please participate.
20:49 · Sean Smith
Let's see polls and quizzes, share results.
21:09 · Sean Smith
Okay.
21:10 · Sean Smith
While I figure out how to launch the next poll, I'm going to let them continue.
21:15 · Sean Smith
Just a general reminder, if you have questions, please use the QA function and we try to get to them time permitting.
21:24 · Sean Smith
If not, they'll follow up afterwards.
21:26 · Sean Smith
Thanks.
21:27 · Leya Bergquist
Thanks Sean.
21:29 · Leya Bergquist
So, getting back to some solutions for recruitment. Plan to over recruit.
21:35 · Leya Bergquist
I don't know that I've been involved in a study where everyone came in on time and were the right people that we needed.
21:41 · Leya Bergquist
So plan for no shows and cancellations and have backup participants on hand ready to go.
21:46 · Leya Bergquist
So if someone doesn't show up or they give you a call in the morning, can you get that person in on the ready?
21:53 · Leya Bergquist
And that really closely relates to communication with your recruitment team and your study team who is the go to person on day of and well leading up to day of but also during day of when you have to have someone readily available on both sides.
22:06 · Leya Bergquist
The recruitment team may have got a cancellation, needs to let the study team know and vice versa.
22:11 · Leya Bergquist
There may have been a no show and the study team needs to let recruitment know that they need to quickly get get a replacement for those, those participants.
22:22 · Leya Bergquist
I think that's it.
22:23 · Leya Bergquist
So a little case study here on recruitment.
22:26 · Leya Bergquist
We once had a client that was doing an HF validation study for type 1 diabetes patients and caregivers.
22:35 · Leya Bergquist
There was 30 adults, 15 adolescents and 15 caregivers with split experience with and without pumps, you know, insulin pumps.
22:44 · Leya Bergquist
The challenge that the the client gave us is that they were on an accelerated timeline.
22:49 · Leya Bergquist
The sessions were long, four and a half hours.
22:51 · Leya Bergquist
That included several hours of training and why am I missing the word help, Denise?
23:04 · Leya Bergquist
Delay.
23:05 · Leya Bergquist
What's the break between training.
23:09 · Kira Favakeh
Decay.
23:10 · Leya Bergquist
Oh decay.
23:11 · Leya Bergquist
I kept thinking of delay decay.
23:13 · Leya Bergquist
So yeah, training day and then several hours of the session conduct and where it really hurt us was with those adolescents and even the caregivers of those adolescents who are trying to work around a school schedule.
23:27 · Leya Bergquist
And we kind of had a uniform group, unicorn group with adolescents without pump experience because the standard of care for adolescent who is newly diagnosed with diabetes is for them to go on a pump quite quickly.
23:41 · Leya Bergquist
So, so it's hard to find someone without pump experience.
23:45 · Leya Bergquist
And then originally there was a.
23:47 · Leya Bergquist
We knew that there was a single geography.
23:49 · Leya Bergquist
Recruitment would, would be a problem.
23:51 · Leya Bergquist
So how did we solve for these problems?
23:56 · Leya Bergquist
One, we recruited across two different locations and ran the study in parallel.
24:03 · Leya Bergquist
We added evening and weekend sessions to allow for better flexibility for the adolescent participants and their caregivers.
24:13 · Leya Bergquist
We expanded the inclusion criteria.
24:15 · Leya Bergquist
So we did educate our client about the challenges of standard of care in the United States with those adolescents not having surrogate, I mean pump experience.
24:26 · Leya Bergquist
And we put together a rationale as to why we did not need to have diabetics with without pump experience.
24:35 · Leya Bergquist
It could be any adolescent without any experience or without diabetes.
24:41 · Leya Bergquist
We did an over-recruit three to six participants per user group because we understood that there was a large cancellation rate for that type of group.
24:52 · Leya Bergquist
And then we made sure that we were ready to backfill cancellations within 24 hours to keep on that timeline.
24:58 · Leya Bergquist
And our key learnings from this study was definitely involve recruitment early so you can learn about their database and their timing and scheduling and get their evidence for whether the inclusion and exclusion criteria is going to be a problem.
25:13 · Leya Bergquist
And then again, recruitment will help you know whether those no-show rates are going to.
25:18 · Leya Bergquist
What, what kind of effort is going to need to be done to over-recruit for over no-show rates in that target population.
25:25 · Leya Bergquist
And you can learn from that.
25:28 · Leya Bergquist
All right.
25:29 · Kira Favakeh
All right.
25:30 · Kira Favakeh
So mistake number four that we'll talk about is not dry running your protocol and use scenario setup.
25:35 · Kira Favakeh
So without that, you might find that your session time is too short for your workflow, that you're missing equipment that you need to support your prototype device or create the simulated environment the way that the user expects.
25:47 · Kira Favakeh
And you might find that your prompting is biased or confusing or just your intended study workflow isn't panning out the way that you thought it would once you have a user with you.
25:55 · Kira Favakeh
So what's the solution for this one?
25:59 · Kira Favakeh
Pretty simple.
26:00 · Kira Favakeh
Just practice, just like you would in R and D, where you do benchtop and feasibility testing before you jump into formal data collection.
26:08 · Kira Favakeh
With human factors, we first perform dry runs.
26:11 · Kira Favakeh
So you're walking through your protocol, reading your script out loud, acting out tasks with your equipment in real time to make sure that you have a good workflow, good prompting, and the right equipment to properly simulate your use environment.
26:26 · Kira Favakeh
This can be done really early on with more or less formality, and it can be done multiple times.
26:32 · Kira Favakeh
But when we think about our last dry run, it should be what we call a full pilot session with a user.
26:38 · Kira Favakeh
So that could be an internal mock user or an actual recruited user, depending on your needed fidelity and available resources.
26:47 · Kira Favakeh
But that pilot session is going to be your best method to fully test your script and your data sheet and even how to prepare to respond to potential close calls or use errors or even equipment mishaps that you can anticipate.
26:59 · Kira Favakeh
And it's a great way to get your whole study team aligned and prepared together.
27:03 · Kira Favakeh
So dry runs and pilot sessions are.
27:07 · Kira Favakeh
Sorry,.
27:08 · Denise Forkey
Can I just jump in real quick?
27:09 · Kira Favakeh
Yeah.
27:10 · Denise Forkey
So one of the things also about the dry run is it can be very helpful if you have someone from your clinical team or someone who has a really good experience of what's happening in the field with that user population to help make sure that those prompts make sense, to help make sure that it's representing a real world situation.
27:29 · Denise Forkey
So I just wanted to to jump in real quick to help people understand that that's a really key thing that can be useful early on in looking through your moderator script.
27:39 · Kira Favakeh
Yeah, and the same as we've mentioned multiple times, if you're doing this pre validation, pre formative for different types of devices, this could look different.
27:50 · Kira Favakeh
There could be different things that are useful to do here such as bringing in your clinical team.
27:55 · Kira Favakeh
So in general, your dry runs and pilot sessions are going to be super helpful, but also only if you give yourself enough time to then make the ness and necessary changes before your study begins.
28:05 · Kira Favakeh
So with another case study that we have for a different validation of a patient monitor, we were set to run a 30 person validation with a consulting firm and the plan was to spend Monday, Monday familiarizing the consultants with the product and finalize the protocol dry runs as they hadn't had an opportunity to get their hands on the device yet.
28:25 · Kira Favakeh
And this was followed by two pilot sessions on Tuesday morning and the first study sessions that afternoon.
28:31 · Kira Favakeh
So clearly this is an example of an unideal timeline as the study team should have been introduced to the product well before it's time to run your pilot sessions so that they can have that to aid in their development of the protocol and reduce risk of any possible delays if you do find any issues.
28:47 · Kira Favakeh
And additionally we didn't have a ton of time then to polish our updates and we were putting a global team in a tough position where those protocol approvals needed to be completed in the middle of the night because we only had a few hours between the start of our sessions.
29:01 · Kira Favakeh
But what this did show was that even in a really tight timeline, we prioritized the inclusion of dry runs and pilot sessions, which proved to be well worth it.
29:09 · Kira Favakeh
So with this rollout, due to some travel delays, which are always a factor in study situations, it was 6pm on Monday, and we were still working through the dry runs and had an open question of how to set up the screen on the monitor.
29:23 · Kira Favakeh
So with these monitors, different facilities or even just different users might have certain orientations or layouts on the screen that they prefer.
29:31 · Kira Favakeh
And this can't come up on the phone between R and D and the Human Factors teams, until the Human Factors teams was able to explore the UI on their own.
29:39 · Kira Favakeh
And then it was an open question of whether the layout should be maintained between users for consistent data or unique to each user to match their typical use.
29:48 · Kira Favakeh
And so then come Tuesday, when we ran the pilot sessions with the users and we had actual recruited users, in this case, we were able to run through our full protocol and update some of our prompting to reduce confusion and bias, and then also try out these different monitor layouts.
30:03 · Kira Favakeh
And I'd say another huge benefit to these pilot sessions is that because it's not part of your actual data collection process, you can even pull back the curtain a little bit and get your user feedback on how the session went and what would be the most effective for them to be working through those tasks as realistically as possible, and they can help you get the right protocol in place.
30:23 · Kira Favakeh
So, in summary, from this scenario, we learned that the value of dry runs and pilot sessions in testing your script and setup when it's the last chance to finalize your protocol before you collect your data.
30:36 · Kira Favakeh
And ideally, we would run those informal dry runs and be familiarizing your study team with the physical product well before those pilot sessions even start, and giving yourself enough time after your pilot session to make effective updates to your protocol and have time to deliberate there.
30:53 · Kira Favakeh
But between the dry runs and pilot sessions, your whole study team can practice the workflow and set expectations about what a full session will look like and what your data might even look like, which will help you be as effective as possible once your real participants are ready.
31:12 · Denise Forkey
Thank you.
31:15 · Denise Forkey
So the fifth mistake that we'll talk about is using poorly trained personnel.
31:20 · Denise Forkey
So this could mean a trainer that doesn't understand that they could introduce bias while they're training a participant before a session.
31:29 · Denise Forkey
It could be moderators or observers who lack understanding of how to do root cause probing or may introduce biasing questions into the session.
31:39 · Denise Forkey
It could also be personnel who have too little understanding of the user interface or they're not trained in the protocol or the use-related risk analysis.
31:51 · Denise Forkey
So some solutions to this would be to use human factors professionals to run studies.
31:58 · Denise Forkey
At a minimum, we feel that a human factors professional as a moderator is a key important step.
32:05 · Denise Forkey
They understand how these studies are run, they understand their role, how they're supposed to roll out use scenarios instead of walking people through task by task.
32:15 · Denise Forkey
And they help make sure that any of those follow up questions are asked in an unbiased manner and subjective feedback is collected.
32:22 · Denise Forkey
Root cause probing questions are asked to make sure that you have the information you need when you do that root cause analysis after and the residual risk assessment after.
32:34 · Denise Forkey
It's important to train the study personnel on the study protocol, on the use-related risk analysis, on the moderator script, the data collection sheets so that they're very familiar with what they'll be doing in the study and how it will be done so that you have the data you need at the end.
32:52 · Denise Forkey
And then finally, it's important to train the study support personnel and the trainers because everybody involved in this study, whether it be the person greeting the participant who's answering their questions about what's going to happen in the study session, or whether it's the trainer who's giving them information, or whether it's people who came on site and they might want to talk to the participant in the middle of the training day.
33:15 · Denise Forkey
We don't want them biasing the sessions.
33:17 · Denise Forkey
And so making sure that they're aware of kind of what's allowed, what's not, how these studies are run is really key.
33:25 · Denise Forkey
So we'd like to share a case study.
33:27 · Denise Forkey
And I would say that this is not an abnormal case study by any means.
33:32 · Denise Forkey
We regularly see clients coming to us and saying, hey, we tried to run the study ourselves with our R and D personnel, our clinical personnel and the FDA came back with questions.
33:45 · Denise Forkey
So in this case it was a robotic surgery system where the company tried to run it with their own internal personnel and R and D personnel.
33:55 · Denise Forkey
They ended up the moderator script had biasing prompts.
33:59 · Denise Forkey
The FDA was not happy with those.
34:01 · Denise Forkey
There was inadequate root cause probing.
34:03 · Denise Forkey
And so the data was incomplete and it was biased.
34:08 · Denise Forkey
So the FDA flagged these things.
34:10 · Denise Forkey
And what ended up having to happen is the manufacturer had to repeat the study and had to do it in a very quick timeline to make sure that they had what they needed for the FDA and it was a little bit under the gun because they were trying to work with FDA to review the protocol under their sir and they had to come back, they gave us feedback and then we ended up up having to implement additional changes prior to running the session.
34:40 · Denise Forkey
And so that can actually take a little bit of time when you're under the gun.
34:44 · Denise Forkey
And it's much better to plan the study up front with the right personnel, with the right questions and methodology, so that you don't have to be under the gun collecting data later.
34:57 · Denise Forkey
So in this case, we feel that if they had used appropriately trained human factors personnel, they probably would not have been in that situation.
35:05 · Denise Forkey
And so it's a critical key learning in our opinion, the last mistake that we'll talk about today is understanding the time and the skills necessary for doing the analysis and writing up the report.
35:18 · Denise Forkey
Some of the mistakes that we've seen are you have non human factors trained personnel writing up the data.
35:24 · Denise Forkey
They don't understand how it should be written.
35:26 · Denise Forkey
They're writing it up more like an R and D verification report.
35:30 · Denise Forkey
95% of them were successful.
35:32 · Denise Forkey
And so it passes.
35:34 · Denise Forkey
Unfortunately, that's not going to fly with the FDA.
35:37 · Denise Forkey
Also, it takes time to do root cause analysis and to evaluate residual risk and make sure that you have enough time to do that at the back end of the study.
35:48 · Denise Forkey
Your report can't be written up within one minute, excuse me, one day or two days.
35:54 · Denise Forkey
So make sure you have that time.
35:56 · Denise Forkey
And then also maybe you do a preliminary release of the data and people start using that for making decisions.
36:04 · Denise Forkey
But unfortunately the data changes because the data has not been locked down.
36:09 · Denise Forkey
So some of the things that you can do to prevent this from happening is to plan for qualitative analysis time in your reporting and making sure that you've identified an appropriate amount of time.
36:23 · Denise Forkey
Typically it's weeks instead of days to do a report, and so making sure that you have that time necessary.
36:32 · Denise Forkey
Also map out all of the reporting activities.
36:34 · Denise Forkey
Ideally you would do this up front in the project, but at the beginning of the reporting timeframe, it's important to check back in and say, okay, what is reasonable?
36:43 · Denise Forkey
What can the team expect?
36:44 · Denise Forkey
And so making sure that you have the right number of personnel and the skill level for those activities and then communicate that timeline with your team, with management upfront, so that they know when they can expect things and they're not preemptively using data that they shouldn't.
37:03 · Denise Forkey
So we've discussed all of these different mistakes.
37:06 · Denise Forkey
Many of the preventive measures include aligning your team, your management, making sure that you know what your user groups are, what your UI scope is for evaluations, making sure you're mapping out pre-session activities and resources and post-session activities and resources, starting early with recruitment, integrating the recruitment into the study schedule, integrating recruitment into kind of the collection of information, testing the moderator script and data sheets, using an experienced moderator, and setting expectations early related to report writing and delivery dates.
37:46 · Denise Forkey
So all of these preventive measures actually help with staying on schedule, reducing time delays, making sure you have high quality data, and making sure that you're not incurring additional cost to the project where insufficient data or inappropriately collected data can impact them.
38:08 · Denise Forkey
So we've talked a lot about all of these things that you can do and about the time necessary for completing the project.
38:16 · Denise Forkey
And Leah will be presenting kind of a a sample slide of a timeline that you can use as well.
38:23 · Denise Forkey
But what you can do is there are options for flexibility.
38:28 · Denise Forkey
The options for flexibility include maybe getting a if your timeline is fixed, the management's like, nope, this is a critical key milestone for our organization.
38:37 · Denise Forkey
It makes or breaks us as a business.
38:40 · Denise Forkey
Then you can ask for additional support personnel to help with the project, you can ask for additional study teams, maybe running concurrent sessions, you can ask for additional geographies.
38:53 · Denise Forkey
Maybe you can't recruit the number you need as quickly in one geography.
38:58 · Denise Forkey
So how can you do that in multiple?
39:01 · Denise Forkey
You also might involve someone like a consulting firm.
39:06 · Denise Forkey
We work a lot with clients who have a very firm schedule and there is a need to hit a timeline.
39:12 · Denise Forkey
And so helping them creatively figure out how they can do that while not impacting the quality of the data is something that we do on a regular basis.
39:21 · Denise Forkey
And then you can get additional recruitment firms.
39:23 · Denise Forkey
You can broadly train people that are not in human factors to help support you, give them that human factors background, train them on moderating, train them on what data looks like and how you need to collect this data and have supportive evidence for your reporting.
39:41 · Denise Forkey
So the key is really to understand those trade offs and to balance the time needed, the costs that are incurred, as well as the quality of the data that will result.
39:52 · Denise Forkey
Because the quality of the data will directly impact those other two and may drag the cost and the timeline out if you don't have sufficient quality.
40:03 · Denise Forkey
So there's really this strong trade-off between time and money.
40:08 · Denise Forkey
You can pay the price and dedicate the timeline for the quality data, or you can risk the cost of repeating the study, regulatory delays, or even recalls.
40:17 · Denise Forkey
You pay now or you pay even more later if your timeline is fixed, the cost shows up elsewhere in the money and the resources it takes to produce a complete, unbiased data package for regulatory submission.
40:34 · Kira Favakeh
So let's think about making the case to management in these situations where if you're human factors engineer at a manufacturer or if you're just on the R&D team and you are proposing that, you know, a study needs to be run, how do you make that case?
40:53 · Kira Favakeh
So first of all, you want to focus on selling the outcome and not the cost.
40:57 · Kira Favakeh
So thinking about all these common mistakes that we just discussed and all of the details that go into successful planning and execution of a study, really focus on the need to have appropriate resources to ensure success as opposed to getting bogged down in all of those details and the cost of each one.
41:17 · Kira Favakeh
Number two is show the trade-off.
41:20 · Kira Favakeh
So be clear about what decision needs to be made for time versus money.
41:24 · Kira Favakeh
And again, be clear about how that time is the critical resource for success.
41:30 · Kira Favakeh
And then lastly, you can bring the evidence.
41:32 · Kira Favakeh
So please lean on the human factors expertise that you've learned and that you've acquired and knowing that it comes from folks who have ran a lot of different types of studies, seen a variety of feedback from different regulatory bodies and also seen success and failure in product launches into the field.
41:55 · Leya Bergquist
Kira?
41:57 · Leya Bergquist
So earlier Denise mentioned the timeline and if there's like, for me, I really love to see a timeline, especially in my earlier career, to be able to, to, you know, not be making this up when, you know, have the, the management office scratch their head when I say it's going to take three months to do a human factors study.
42:14 · Leya Bergquist
Why?
42:15 · Leya Bergquist
So we wanted to give you something to walk away with as a, you know, can this take long?
42:20 · Leya Bergquist
Less than three months?
42:21 · Leya Bergquist
Yes.
42:22 · Leya Bergquist
Can it take six months?
42:23 · Leya Bergquist
Yes.
42:24 · Leya Bergquist
It depends on the ease or simplicity of your study.
42:27 · Leya Bergquist
But I think as a general rule of thumb, it takes a good three months to be able to get through all of the scoping and planning and recruitment and data analysis.
42:39 · Leya Bergquist
And included in here as part of your reporting is, you know, alignment with your team on the results and getting that approved in your quality system.
42:46 · Leya Bergquist
So there's a lot more that you have to remember that goes into the timeline.
42:49 · Leya Bergquist
So hopefully this helps you have something to, to defend your, your case with your management team.
43:04 · Leya Bergquist
And so what to take back to your team if there's any, if there's someone on that that couldn't make this call today?
43:10 · Leya Bergquist
I think we, we've all realized that most of the work that makes HF validation successful seems to be invisible to the people who are funding it unless you let them know.
43:19 · Leya Bergquist
So educating them is important.
43:22 · Leya Bergquist
And the common failures are planning failure.
43:26 · Leya Bergquist
So too little lead time, too few resources, untrained study personnel.
43:30 · Leya Bergquist
Don't forget to do some of the solutions that we suggested for those items and then plan and resource it once.
43:38 · Leya Bergquist
As Denise mentioned in the financial trade-offs, repeating a study is quite expensive.
43:44 · Leya Bergquist
So planning it right and doing it right the first time can save the team a lot of money just in the funding, but also, you know, getting to the field earlier to be able to make those have income, come in as a commercialized product, time matters.
44:00 · Leya Bergquist
And bring the trade-off and evidence to management so that the spend and the timelines are defensible.
44:09 · Leya Bergquist
That's it.
44:11 · Leya Bergquist
Denise, you're at the next one.
44:16 · Denise Forkey
But I think it's Sean.
44:17 · Leya Bergquist
Okay.
44:19 · Denise Forkey
Are you ready for us, Sean?
44:23 · Sean Smith
Here?
44:23 · Sean Smith
I am so sorry about that.
44:27 · Sean Smith
Yeah.
44:28 · Sean Smith
So I've been posting some notes and some links.
44:30 · Sean Smith
Just a reminder to put your questions in the Q&A.
44:37 · Sean Smith
Now we prepared a MedTech usability prep kit.
44:42 · Sean Smith
If you'd like the prep kit, you can either ask for the download link or just write prep kit and we'll send it to you.
44:53 · Sean Smith
And the same thing for the seven-point readiness assessment.
44:57 · Sean Smith
If you'd like to spend 30 minutes with someone from the ClariMed team, one of the SMEs there walking through your particular project, it would be time well spent and they're going to go through the seven aspects that you need to be considering.
45:18 · Sean Smith
So yeah, and then just a reminder that everyone is going to receive the slides and the recording and we're gonna finish up shortly with a.
45:35 · Sean Smith
Survey at the end.
45:36 · Sean Smith
Thank you for everybody that answered the poll questions.
45:40 · Sean Smith
And now we can go to Q&A if you'd like.
45:44 · Denise Forkey
Excellent.
45:44 · Denise Forkey
We do have some great questions that you guys have submitted.
45:48 · Denise Forkey
Thank you for submitting those.
45:49 · Denise Forkey
Feel free to send some additional ones as well.
45:53 · Denise Forkey
We'll be able to answer of them live here, but we won't be able to get to all of them.
45:58 · Denise Forkey
So we'll try and reach out and answer your questions after the webinar if we can't get to them live.
46:05 · Denise Forkey
So the first question that I think we should address is from Haley.
46:11 · Denise Forkey
Thank you for submitting your question.
46:13 · Denise Forkey
For small startups with budget restrictions, how can they ensure the FDA will accept the usability validation study while reducing costs by performing some of the HF activities in house?
46:24 · Denise Forkey
House.
46:25 · Denise Forkey
What are key aspects or stages to check in and make use of HF professionals.
46:32 · Denise Forkey
So I can go ahead and start the answer and then I'll open it up for the other panelists.
46:39 · Denise Forkey
So at ClariMed, we regularly work with small startups that maybe don't have the budget to hire a full human factors consulting firm to execute the study.
46:51 · Denise Forkey
And so one of the key things that I think is important is I always recommend having professionals run the study.
46:58 · Denise Forkey
But if you absolutely cannot, there's budget constraints, then you'll have to figure out how internally, you'll have to come up to speed on things that are necessary for running this study.
47:08 · Denise Forkey
That could mean educating yourself.
47:10 · Denise Forkey
It could be bringing in human factors consultants like I do regularly work with other small startups to say, okay, let me walk you through some of the requirements.
47:20 · Denise Forkey
Let me help you understand kind of what are the key points that you need to consider in your protocol in your moderator script.
47:28 · Denise Forkey
You can have them reviewed by professionals.
47:32 · Denise Forkey
And then I always recommend, though, making sure that you have a trained moderator, because the trained moderator will definitely make or break your sessions.
47:42 · Denise Forkey
And if you cut on that cost, you will end up paying later because the FDA will find the issues and they'll find the problems.
47:51 · Denise Forkey
So at a very bare minimum, I would say make sure you have a trained moderator.
47:55 · Denise Forkey
But, but there are other ways that you might be able to save on costs like recruitment.
48:01 · Denise Forkey
If you have a large network of users that you already know or your team already knows, maybe you can support some of the recruitment efforts, or maybe there's other ways that you can save on costs.
48:15 · Denise Forkey
Each one of these is kind of an individual study consideration.
48:19 · Denise Forkey
And so I would encourage you to reach out to professionals and to ask questions and to kind of gather your feedback before you embark on this journey.
48:29 · Denise Forkey
Leah or Kira, did you have additional question or answers to that question?
48:35 · Leya Bergquist
Yeah.
48:35 · Leya Bergquist
No.
48:35 · Leya Bergquist
I think that probably the biggest, you know, compromise that we end up seeing is, to your point, educating people on kind of what the content of some of the documents needs to look like and then doing a review of the final document to highlight any areas of concern.
48:51 · Leya Bergquist
So we do the recommendation, you write the protocol, the, the uFMEA or URRA, any of the other supporting documents, and we can come in and do a quick check to make sure there's not any major flags.
49:08 · Denise Forkey
So the next question is from Benjamin.
49:11 · Denise Forkey
Thank you for your question.
49:14 · Denise Forkey
So thank you for sharing your hard earned learnings.
49:17 · Denise Forkey
If senior management are unwilling to consider a HF/UE appropriately, where would you be most comfortable compromising on your timeline?
49:29 · Denise Forkey
So I think compromising on the timeline means that you'll have additional resources needed, potentially additional cost needed to help make sure that you're still gathering that high quality data.
49:44 · Denise Forkey
So if you, before you're ready to start your validation, if you already reach out and start getting recruitment support and start kind of working on that super early so that as you're ready to really scope out the project, you can run through that quickly, then that's one way that you can shave off on time.
50:08 · Denise Forkey
There may be other ways to kind of start working on that protocol sooner, working on the sessions sooner, running parallel sessions, running getting additional support in report writing, having people involved in the report writing kind of in parallel so that they, you can kind of bolster and work off of their efficiencies to try and make sure that it gets completed.
50:35 · Denise Forkey
Leah and Kira, do you have additional thoughts for Benjamin?
50:40 · Kira Favakeh
I think I would just emphasize that recruitment isn't really something you can budge on.
50:47 · Kira Favakeh
With the recruitment timeline and how long it takes for your recruitment partner to find the users that you need, you can't force them to do that any faster.
50:57 · Kira Favakeh
And you also obviously need to end up with the correct number, correct sample size for your study.
51:04 · Kira Favakeh
But the thing to think about is that unlike some other types of engineering testing, it's not something that you can readily redo.
51:11 · Kira Favakeh
So if you get through your first participant and realize this didn't go well, we need to make some changes, that participant's a wash and you're going to have to re recruit, you're going to have to replan.
51:21 · Kira Favakeh
So if that's, you know, another case that you can make at all, that it's not human factors in that sense does not parallel other types of R&D testing where it can be more easily repeated or adjusted.
51:37 · Kira Favakeh
It has to go right the first time or you're going to redo everything.
51:46 · Denise Forkey
All right, the next question is from Shani.
51:50 · Denise Forkey
We are preparing to launch usability testing soon, but project timelines are tight.
51:56 · Denise Forkey
What is the single most compelling metric or artifact you use to convince project managers that post session data analysis and root cause analysis require as much calendar time as the testing itself?
52:10 · Denise Forkey
Maybe Leah or Kira, if you want to take that as a start.
52:16 · Leya Bergquist
Sure.
52:18 · Leya Bergquist
You know, I think when, when, whenever I've been challenged in the past, it's, you know, providing evidence as to why it's going to take so long.
52:28 · Leya Bergquist
You know, what is, what are the pieces of data that are, that you're collecting?
52:32 · Leya Bergquist
What are the type of, type of data?
52:34 · Leya Bergquist
Is it quantitative and qualitative?
52:37 · Leya Bergquist
If they or teammates have to be involved in it to move faster, maybe you can ask for help from their team if they want it to happen faster.
52:46 · Leya Bergquist
So I think just really explaining and educating what the type of data that needs to be and how it impacts the entire team and submission process goes a long way.
52:59 · Leya Bergquist
And if, you know, depends on your personality type.
53:06 · Leya Bergquist
But I love to get people involved who want things to move faster and help them own the responsibility of making it move faster.
53:13 · Leya Bergquist
So that's probably, you know, corral the team to rally to make it move faster if it really does have to.
53:19 · Leya Bergquist
Of course, be careful making sure that people are trained, adequately trained to make those analyses.
53:24 · Leya Bergquist
But I think that's probably my approach.
53:29 · Leya Bergquist
Other people might not be that that way.
53:32 · Kira Favakeh
No, I think kind of along a similar vein rather than, you know, necessarily including people in the process afterwards, but informing them ahead of time of what it looks like, like by actually training them and fully taking them through the process.
53:48 · Kira Favakeh
I'll often before a study, either with the project team and the management so that they understand what we're working with and what timeline we have and why, or with the, you know, R and D team members who are going to come and potentially note-take or observe or be part of the process, I carve out a significant chunk of time in bringing them up to speed in what human factors is.
54:10 · Kira Favakeh
What does it mean to collect qualitative data?
54:14 · Kira Favakeh
What does it mean to have a biased bias prompting?
54:17 · Kira Favakeh
What does it mean to what's a root cause?
54:19 · Kira Favakeh
Like, why are we collecting all of this data and what's the use here?
54:23 · Kira Favakeh
So to try to make the process less invisible, unfortunately, we have to do the work to make it visible to those people that are involved or in charge of decision making.
54:36 · Kira Favakeh
But I do find that, that if I take somebody through an example of study results that I've gotten for a study with 10 to 15 people and 20 tasks, and once they see how much information that is that needs to be parsed, that can help.
55:01 · Leya Bergquist
Better answer.
55:03 · Denise Forkey
So we might have time for one last question.
55:08 · Denise Forkey
So this is from Sarah.
55:09 · Denise Forkey
What is the difference between summative usability and human factors studies?
55:15 · Denise Forkey
And is the latter acceptable for Class 2 devices that go through 510?
55:22 · Denise Forkey
So do you guys want to answer it or should I?
55:24 · Denise Forkey
Go ahead.
55:26 · Kira Favakeh
Go for it.
55:27 · Denise Forkey
Okay.
55:28 · Denise Forkey
So a summative usability evaluation is a term that's used in IEC 62366-1, the international usability standard, and human factors validation is a term that's used in the FDA guidance.
55:47 · Denise Forkey
And so they typically are one in the same.
55:50 · Denise Forkey
They're both trying to evaluate the final product to ensure that it's safe and effective for use for the users in the intended use environments and the intended for the intended use.
56:04 · Denise Forkey
So a lot of times they can be used interchangeably.
56:07 · Denise Forkey
The Human Factors validation FDA has specific requirements like they typically want to see at least 15 users per user group, whereas that's not clearly defined in the usability standard.
56:21 · Denise Forkey
So there are times that you might might have two separate studies, but generally they're trying to accomplish the same thing and that's the same thing for a Class 2 device.
56:32 · Denise Forkey
Going through 510 so I think we're at the top of the hour.
56:39 · Denise Forkey
As I mentioned before, we're happy to answer any follow up questions.
56:46 · Denise Forkey
Happy to meet with you if you have a need to meet to go through your validation plan and we'll definitely follow up with the the other questions that we weren't able to answer live on the call.
57:00 · Leya Bergquist
Thanks everyone.
57:01 · Kira Favakeh
Thanks guys.
57:02 · Sean Smith
Like to thank the speakers and I would like to thank all of you who showed up today and even those of you who just registered and are going to watch it later.
57:12 · Sean Smith
Thank you so much for your participation and show some love.
57:19 · Sean Smith
Thanks.
57:19 · Sean Smith
Bye.
57:21 · Leya Bergquist
Bye.
57:21 · Leya Bergquist
Thank you.
57:22 · Denise Forkey
Thank you.
How to cite this session
Forkey, D., Bergquist, L., Favakeh, K. (2026, August 6). Beyond the Protocol: A Realistic Resource and Planning Guide for Human Factors Validation Success [Webinar]. MedTech Leading Voice, MedTech Leading Voice. medtechleadingvoice.com/watch/2026-08-06-beyond-the-protocol-a-realistic-resource-and-planning-guide-for-human-factors-validation-success













