Sex Bias in Medical Device Usability Testing

Hello and welcome, everybody. I'm Sean Smith with MedTech Leading Voice. Today's webinar is Sex Bias in Medical Device Usability Testing, presented with Claramed and DNV. Three quick things. We are recording, and the recording and slides will go to everyone who registered. Whether or not you stay for the full hour. If you want a certificate of attendance, please type, your name and email as you would like it to appear, into the chat. We're gonna use the chat function for communicating, with me, the Q&A function for asking questions of the speakers. They, I'm not going to be able to monitor the chat, but we will be monitoring the chat. We try to acknowledge everybody's questions.

And, There will be a quick poll and survey, which I'm going to talk about in a second. While we're waiting for everyone to join, two very quick announcements. Tomorrow at noon, Eastern Time Device Innovation to Commercialization Phase 1. Phase 1 concept, with Renee and Michelle from Lean RAQA. Friday at 11am, please join us for Compliant and Still Wrong, the Human Side of Risk Decisions. with Steve Gompertz. and Jean Bloom, who I met both of them at the Risk Safety Conference. This is not a human factors event, but it is not unrelated.

As you can imagine. So, I'm going to introduce the speakers. I'm pleased to introduce Aarti Swaminathan. I hope I didn't destroy that. Senior Human Factors Consultant at Claramed. She's done close to 100 human factors projects, and More than 1,000 moderated sessions in 8 years on products with sex-specific design considerations, from sexual health tests to home fertility devices. Diane Legere is a clinical evaluation specialist at DNV, the notified body. She spends her days looking at whether clinical evidence actually covers the people who will use the device, and she's here to give us the reviewer's side.

Aarti, over to you.

Great. Thank you so much for the introduction, Sean. So, let's dive straight into it. Sex bias in medical device usability testing. Why is that an important question to consider? It's really because the ramifications, what the ramifications are when you fail to consider that. Consider these two studies on this slide. In one that was conducted in 2024, across almost 200 different medical device trials, it was found that women represented only about a third of participants. On the flip side of the same coin, in another study, across 340,000 FDA device injury and death reports, two-thirds of the people harmed were women.

When a certain group or subgroup is underrepresented in your design, research, and testing, there's a high risk and high chance that that same group can easily become overrepresented when it comes to device failures. Now, the starting point for that, and a great starting point, is a 50-50 split of men and women. But really considering through bias, not just sex bias, but other types of it, involves considering beyond that, going a step further and looking at other demographic physiological characteristics, like lower grip strength, different anatomy, and so on. When you don't include those or pool the data that doesn't stratify or look at performance.

Across these different groups, the study might still pass and the device might ship, but there's a hidden problem that ends up stuck inside it. So, what does that passing study report tell you and not tell you? So, you might have a… report that says participants were recruited to an even split of men and women, all your critical tasks met thresholds for acceptance criteria, all the use areas were assessed, the residual risk was accepted, and all your results were pooled across the sample and summarized. On paper, this seems like a perfect report. But there's a lot of hidden questions within that report that may stay unanswered, and might end up tripping you up, not just at the review stage, but on market.

Questions like, did any task fail more often for users with a certain demographic or physiological characteristic, like lower grip or reach? On the patient side, did fit, insertion, or handling differ by anatomical considerations like body size or even life stage? Were… did you ever look at those task-level results stratified and compared by sex or by any of these other factors? If a subgroup struggled. In this study, was that root cause recorded against the design, or against the participant? These are questions, as I mentioned, that something that the FDA can question you on at the end, or something that might end up showing up as market failures when we fail to consider these ahead of time.

So, I'll pass back to Sean to talk through the questions.

So, we are doing a quick poll, and… Where is your device today, if you're working on a device? How did you set the participant mix in your last usability study, and has your team ever analyzed usability results separately by sex? And so we're going to take 90 seconds. I'd like as many of you as possible to participate in this. Because this really helps us frame going forward, so… Where is your device today from concept to on the market?

How did you set the participant mix in your last Usability Study? Matched numbers of men and women, stratified by age, body size, life stage. And then, took what require… what… what recruitment delivered. Love that. And, has your team ever analyzed Usability results separately by sex? Yes, routinely. Once on one study, no, don't know. Joe Siegrist, And thank you. Looks like lots of people are participating. Wave. almost reached half. I'm gonna stop at half, and so we just need a few more people to participate.

And, I'm gonna remind everybody while we're waiting for the last, person to answer the question, if you would like a certificate of attendance, please drop your name as you would like it to appear on the certificate, into the chat, along with your email address, and we will make sure you get it in the next couple days. So we've crossed the 50% fresh, right?

Right?

threshold, And I'm going to share the results. with you now. So the results are 26% early stage, 21% formative testing, 19% preparing. Validation or submission, and 33% are on the market. Question number two. Did you set the participant mix? In your last Usability Study, matched men and women, 14%, stratified by age, 30%. Took what recruitment delivered, 28%, don't know, 28%. Show 3. Has your team ever analyzed results separately?

Yes, routinely. Once on one study. No, 40%. Sex was not relevant, 25%. I'm not close enough to know and don't know. So the vast majority of the answer is no. I'm going to stop sharing and turn it back over. to Aarti Aarti Thank you.

All right.

Surprise? Any surprises in those, surveys there?

No, I don't think so. I think, it seems like a balance across different. options on almost all the questions. I think on question 3, in terms of analyzing usability results separately by sex, I think the yes, routinely responses were a little on the lower end, and I think that's exactly what I expect in what we typically see. It's also something that not everybody thinks to do or considers before the FDA or another regulatory body explicitly asks for that kind of data, either in a pre-sub or at the review stage.

Okay, fantastic. Thank you.

So let's jump into where that sex bias enters a device program. And how that impacts, you know, throughout the lifecycle. So, there are 5 different places throughout the lifecycle of the product where this bias can get introduced, all the way from the beginning of defining the intended users and doing that generative user research. And focusing, you know, at this stage, on the average, rather than kind of who are on the edges in terms of your demographics and abilities. In the design input stage and collecting reference data, we often see where, again, that tendency to focus on the majority or who the average is ends up leaving out the lower end of the spectrum in terms of

physiological characteristics, like height, weight, hand span, and so on. And even when using the reference data, a pitfall that we often see here is using reference data without considering where it came from, or what that reference data really represents, and whether that matches to who your population is. When it comes to task analysis and use-related risk. Oftentimes, we'll… won't consider any risks that are specific to certain subgroups, and fail to consider any or tie use-related risks to physiological considerations that can help us later recruit for the right kinds of people, and so on.

And at the end. or I say at the end, but really these last two items can kind of show up at both the formative stage and early stage research all the way through validation, which is recruiting the right participant mix and making sure that you're collecting, again, all those edge cases in terms of ability, and you're representing the lower end of the threshold in terms of your participant abilities. And on the flip side, when you're reporting or analyzing that data, not considering or not stratifying across these different characteristics. Now. really sex bias and any other kinds of bias can enter this Device program in all of these different places. The real places where this can get caught is those last two, with recruiting the participant mix and analyzing and reporting properly. And that's kind of where we also see, you know, according to the poll, we had a lot of people say that they did, you know, intend to recruit a mix of men and women across different

stratifications, like life stage and ability and height and weight and so on, but a fewer proportion answered that they stratified and analyzed this. So when we do both of those things correctly, that's where we can identify and then be able to go back and correct any of these other places where Bias might have been introduced.

So let me just interrupt for a second and ask Diane, you know, if this… What we're talking about here is consistent with what she's experienced. As a reviewer, and And any comments that she may have on that.

Yeah, I'm gonna say that, thanks, Sean, for the question. I haven't seen, task analysis so much, so the amount of effort put into the, sort of, the user profile as such has… I mean, this has been… I mean, for me as a… both a participant and a speaker here, it's been very eye-opening, and I'm gonna say, changing the way I will look at Usability studies. So, not… not seeing the strength in the task analysis. The rest, I'm gonna say, you know, mostly… Yes, recruiting a fairly good participant mix, but again. Aarti, you've given me lots to think about for my next reviews.

And sometimes that, you know, there's, for instance, when it comes to the FDA and your user-related risk analysis, they tend to dictate you know, exactly what data they want to see, what columns they want to see, and this might not always make the cut in the version that gets submitted to the FDA, but I think this can often be something that's an important background consideration that might, you know, that kind of inherently you think through as you're building the use-related risk analysis. And gets sort of reflected officially in your recruiting and participant mix, but it's more about… I'll say it's more about the activity than really the result when it comes to your URE.

Good point.

Alright, so we'll move on to the next section, which really explores three different kinds of bias, physical, computational, and interpretation bias. Start off with physical bias. Now, this is kind of the most tangible type of bias, and in some ways, can be the easiest to miss. This is one where the device works exactly as designed, and it still performs worse for part of the intended user population. Nothing is broken here. The design verification might still pass, and the specifications are met as a whole, but those specifications might have been written around a specific body, and oftentimes, historically, that body was implicitly male.

So, where does that show up? As an example, grip strength and torque. Think about the force needed to remove an auto-injector cap, or prime an inhaler, or fire a surgical stapler. On average, women have lower grip strength than men, and if the force requirement for a certain task sits near the top of what some intended users can generate, that is a use-related risk on a critical task. Another example with reach, span, one-handed operation. There are many handheld instruments and controls that assume a sort of larger hand. Female surgeons have often reported difficulty with operating laparoscopic and stapling instruments with one-handed. The device, again, still works, but the users in those cases often have to adopt a workaround, and that workaround is where use errors begin.

Another example, fit against anatomy. Now, oftentimes when it comes to bias, and I'm guilty of this too, we tend to focus on the user side of things. So, in a surgical tool, we focus on the surgeon's anatomical considerations and their abilities and so on. But in some tasks, it's just as important to consider the patient's end. Or if that person is different than who the user is, if it's not, like, a self-use product. So, Fit Against Anatomy is a great example of that, where, like, placement, insertion, seal site, and so on. For instance, respirators and masks often were historically developed on male facial dimensions, and may not seal properly on smaller faces.

Other wearables, continuous glucose monitors, things like that, can adhere and perform differently depending on body shape and tissue distribution. ECG electrode placement is a great example of this, where the instructions don't always account for breast tissue, or don't have clear and adequate instructions on how to place the leads and the pads on female anatomies. Other factors exist, and not just for devices used in medical environments. Long nails, for instance, can make it hard to press a small button, peel a blister pack, or even grip a small component. That's something that can easily impact the use of at-home devices.

But why does all this matter for Usability Testing? Because we can only find these problems and design for them if our participants represent the physical range of the intended users. If a formative evaluation sample skews male or larger-handed, the device might look fine as a whole, but then it goes into the field. So, when you define user groups, it's important to think about the anthropometric and physiological range, as well as the user roles. Moving to the second type of bias, which is computational. Now, the first type of bias was about the interaction between the body and the device. This one's a little bit harder to see, because it sits inside the output.

This is one where the output is less accurate for users, the training or reference population underrepresented, and the device, again, still meets its specification as a whole. And that last part matters just as much here as it did for physical bias. The accuracy claim is true on average, the problem is who that average describes. Let's go through a few examples here. First, pulse oximetry. A pulse oximeter is typically calibrated and validated against a reference group, and its performance is typically reported as a single sort of accuracy figure. But that reference group doesn't always reflect the full range of intended users. There's a lot of studies and reports that have shown that pulse oximeters, on average, perform worse for users with darker skin tones, for instance.

Now, this might be not a direct example of sex bias, but the structural issue is the same. It's one number that's reported for everyone, and creates, like, an issue for some of these specific subgroups. Another example is early AI ECG classifiers. Now, normal ECG values typically differ between men and women. But many historical diagnostic criteria in some of these early classifiers were developed from predominantly male populations. So, when the algorithms would learn from a data set weighted towards one group, it would inherit that weighting. The data that was specific to subgroups, like female bodies and so on, were rarely reported, so users had no way to know the model might be less reliable for them.

In general, reference ranges and thresholds are one of the quietest form of bias. Alarm limits on patient monitors, normal ranges on a wearable, diagnostic cutoffs on a test. These are all values that come from source data that nobody constantly revisits, and they get applied to every user. The bottom line here is use-related risk doesn't end when the user finishes the task. It continues when a clinician or a patient or caregiver acts on that number. Users tend to trust device outputs, and very few displays and devices really communicate how certain a read is for a given person with specific demographics and characteristics.

Alright, now the third and final type of bias here is interpretation bias. Now, the first two types of Bias lived in the device, you know, in the hardware or in the output. This last one is different. This is one that's in our own teams. That matters because root cause analysis is where a study finding becomes a design decision or doesn't. FDA Human Factors guidance expects us to investigate the root cause of every user, close call, and difficulty on a critical task. What we write there shapes whether or not anything changes. So, how does interpretation bias reach the report? It comes in, for instance, when a difficulty, say, is recorded as a participant error rather than a design cause.

Let's take, for example, a participant who can't remove an auto-injector cap on the first try. One note-taker might write, participant didn't pull hard enough. Another one might write, participant was unable to get the cap off. It's the same use event, but a completely different lens through which the path taken when root cause is later probed can end up being completely different. And that leads to the next point, which is. When root cause probing stops at the first possible answer. The participant might say in root cause probing, I just wasn't paying attention. And the moderator might move on. But participants will often blame themselves in these kinds of studies. Our job is to keep asking, what about the device might have made that easy to happen?

Another example is when a struggle is explained by the participant being unfamiliar with technology. This is where assumptions about who a participant is can quietly take the place of evidence. If we'd accept that explanation for some participants more readily than for others, including based on sex or age, that's when Bias enters the analysis. Once the root cause points to the user, the fix also naturally points to adding a warning, or a step in the instructions. But there's a reason why the risk control hierarchy in ISO 14971 puts design changes first, and information for safety last. Labeling is the weakest control that we have when it comes to medical devices.

With a small sample, trends that we note might not be statistically conclusive, and that may lead them to get easily dismissed, but qualitative patterns are still valuable, and exactly what formative evaluations exist to find. The good news is, this is one of the easiest kinds of bias to fix. The moderator, the note-taker, whoever writes the root cause, all simply need the same training. It doesn't require a device redesign or a new data set, it just requires a sort of shared language and understanding, consistent task reading, and a habit of asking, would I write this up the same way for a different participant?

To bring all three types of bias together, physical bias affects how the device fits the body, computational bias affects how accurate the output is, and interpretation bias affects what we conclude from our own studies. The first two, we can only find if we look for them. The third determines whether or not we act on what we find. Oh, I'll pass this over to Diane to give some insight on what happens to this in a clinical evaluation.

Thank you. I was just thinking about, as you were going through the different biases, that I have not paid enough attention to the training, that the Usability, testers themselves, have undergone. And again, thank you, Aarti, that. My, my evaluation of Usability, data is going to be very different, after… after this session. So, Sean, do you want me to just go ahead and read the question, or do you want to, do you want to? It doesn't matter, I can do it. I can carry on. So, where sex-specific data is expected to appear in the clinical evaluation, and what an assessor does when it is not there. So, clinical evaluations under the MDR, different, I think, I'm not a FDA reviewer, includes a

state-of-the-art assessment. And that should provide a comprehensive assessment of how the natural course of what the device is expected to treat, for example. can be impacted by sex. So sex-specific data is, as we know, expected in clinical investigations and any clinical studies. And notified bodies will always look for representative data. And we'll go back to that background or state-of-the-art assessment and looking to see if the disease process maybe Is more common in women. Or if it's more common in older patients, and then we'll look to see how that impacts the data that is presented. So a comprehensive assessment of how sex affects the intended purpose is expected by a notified body. And if it's not there, you would receive a non-conformance, and we would present that back to the manufacturer to provide and ask them to provide that data.

Including, and if necessary, we would have to put the review on hold to, to allow them time to go and get that data. Depending on how On how important, that data is to our ability to assess the benefit-risk, with that data missing. So when a notified what sorry, next question is what a notified body asks when the evidence does not cover the people who will actually use the device. So as above, it could it will absolutely result in a non-conformance and delay your review. But sometimes, often, the manufacturer and the assessor can agree that the missing data can be part of a PMCF, a post-market clinical follow-up, either study survey, whatever is appropriate for what is missing.

We will… we will create a plan. Sometimes we will shorten up the amount of time you need to review or send that data to us, again, depending on how long it will take to… to get that data. And that's an agreement we will make and put a condition on… on your certificate. So the notifying body will also ask for a justification. of how the current evidence supports the intended purpose. So that's all part of… part of what I've already, talked about. But I want to point out, Usability data under the MDR is not considered clinical data. It is important supporting data, and obviously considered in the totality of data when you put together your application, process, but it itself is not, considered clinical data. If you look in the MDR Article 2, that defines, what clinical data is specifically.

So the last question on the slide is, what post-market data and complaints have shown when the subgroup was never looked at? Yeah, so post-market data and real-world use has so much opportunity to show us subgroup data. You know, the unfortunate part here is complaint data is very limited. We all know it's very underreported. And even when it is reported, if you've had a chance to review some of the MOD reports, for example, you'll see that they're often not comprehensive reports, and follow-up often doesn't occur. So real-world data, though, collected from electronic medical records, for example, what I think this can show is how safety

possibly has been compromised for both patients and users, I think here's where we could really get a subgroup analysis when it's done well, and has the ability to show us data that's been buried in the larger subsets, or the missing subsets on Use errors, lower effectiveness, greater residual risk, maybe unexpected clinical outcomes. Great! That's, hopefully I've answered all those questions.

Thank you.

Thank you, Diane. Alright, so, having now looked at different kinds of bias, and how they show up in the device, and how they need to show up in our studies, I want to move upstream to see how it can be mitigated, because by the time we're running a formative evaluation, much of this risk has already been built in. So let's look at four different places where that happens. First, anthropometric reference data. Datasets like NHANES and CSER can be extremely useful sources for early-stage design considerations before you even get to a prototype. However, it's important to use such sources with their limitations in mind. Each one of these datasets describes a particular population at a particular point in time.

When using these, we need to check that the range covered actually covers the people who are going to be using our device, whether that includes women, older users, users in other markets, and so on. Second, strength and torque assumptions. Forces that we set are sometimes set against a population mean, but the mean isn't really the user we need to worry about. The user at the low end of the intended user population is. If your intended users include, for instance, older women, or people with reduced hand function, that's what the design target should be, not the average adult. Third, task analysis. Most of us already break critical tasks down by perception, cognition, action, you know, PCA.

This is asking us to go kind of one step further, and name the physiology that each critical task depends on. Strength, reach, dexterity, anatomy, skin, tissue, and so on. That list does a lot of work later. It becomes your inclusion criteria so that your study sample covers the physical range that actually matters for the device. This is also a really helpful tool or point where you can understand, say, for a surgical device, whether what the kind of physiology you need to represent is really at your user's end or at your patient's end for tasks. that depend on user anatomy or user physiology, you might want to… that might funnel into the inclusion criteria. For tasks where there is patient anatomy, that's where actual use studies or having representative models can then come from, or stem from.

The fourth is use-related risk analysis. Now, if we have reason to believe that a use error is more likely for part of the population, that belongs in the URA as an identified risk, with a risk control. It shouldn't appear for the first time in the discussion section of the validation report. That is when… by then, it's basically a finding. If you track that in the URA and start thinking of it from the beginning, it's something that we can still design out. Now, all of these basically connect back to all three biases. Physical and computational bias often start with reference data that don't match up to your intended users, and that nobody questioned.

Interpretation bias is much harder to fall into when something like the URA has already told the team what to look for and what to design against. Now let's talk about the two choices that we make in every study that decide what we're able to see. Who we recruit, and how we cut the data. If we get either of these wrong, the biases that we've talked about can be present in the device and still not appear in the report. So let's start on the left with recruiting. First. Writing inclusion criteria to the physiology the task depends on. Now, like I said at the beginning, a 50-50 split between men and women is a really great, important starting point, but it doesn't guarantee that you've included anyone with, say, small hands or low grip strength. This is where the task analysis from the last slide pays off.

If a critical task depends on something like grip strength, your criteria should make sure that you've recruited people at the low end of that range, not just people of a certain split, or a certain sex, sorry. Second, setting enrollment targets across various characteristics. Of these, life stage is one that teams most often miss. Take, for example, pregnant women. Pregnancy changes a person's reach, and even something like the available injection sites on the abdomen. Arthritis is another example, which becomes more common with age, affects more women than men, and that's something that changes dexterity.

Having now identified and, you know, set your enrollment targets, the next is… next thing to do is to over-recruit for the hardest subgroup first. It might be older women with reduced hand function, or it could be users of a rarely prescribed therapy. If you leave that group until last, you'll end up short, and the study timeline and your review deadline and so on will push you to just accept the gap. Finally, give recruitment the lead time it needs. When a schedule ends up getting compressed, recruitment is often the first thing that gives, and the sample just quietly becomes whoever was available. That's often how a well-designed and well-planned study, in theory, ends up with a biased sample in practice.

Now let's talk about analysis. First, stratify and compare task-level results by sex. Ideally, this is something that is pre-specified in the protocol, so that it is part of the plan, rather than something that's only done once someone asks. Human factor samples are small, and we're not really looking for statistical significance all the time. We're just looking for patterns worth investigating. Second, look at failure rates per task and per subgroup. An overall result can look clean while one subgroup struggles with one critical task. It's the same problem as the single accuracy figure we discussed under computational bias. Pooling can hide exactly the pattern that we're trying to find.

Third, record root cause against the design, and keep the probe going past the first answer. This is interpretation bias again. When a participant says, that was my fault, the next question should be, what about the device might have made that easy to happen? And this is something where, kind of going back to my earlier point, really is… impacted by the quality of training that the team receives. You know, how they interject at the session moderator stage, you know, how do interviewers react, how do they probe, what perspective do they have, and making sure that there isn't some kind of implicit bias that

Kind of frames how they ask a question, or whether they choose to or choose not to ask a question. Fourth, and finally, report the subgroup results even when it passes. A sentence like, performance on XYZ task was consistent across male and female participants tells a reader that you looked. If you don't have something like that, nobody can tell whether there was actually no difference, or just that no one checked. It's a small addition that makes the analysis visible to a reviewer, a client, or even your team in the future. And it's something that often we end up seeing the FDA question, or reaching the market and having issues there, because at that point, you know, people hadn't looked previously. I can't tell you the number of times we've had FDA feedback come back and say.

well, did you consider this subgroup? Did you consider that subgroup? I don't see any data on demographics here. Did the performance vary across these subgroups? Thinking about this in advance, tied back to your task analysis, knowing where those kinds of pitfalls might be, can really help to smooth your path later. None of this necessarily needs a bigger budget or a bigger sample. It just needs these choices to be made deliberately at the start, instead of by default at the end. But let's look at this from the lens of a case study, and look at an actual product and see how some of these considerations kind of went in.

So, the case study in this case is an at-home fertility device by Baya Fertility, which was an at-home insemination device used by patients with clinical guidance. Now, the tasks in this product were such, like, loading, insertion, deployment, removal, and so on, that all heavily depended with 90% of tasks on the female anatomy, and to a smaller extent on male anatomy. So there were no critical tasks, basically, that could just be assessed on a general sample. Standard reference data, in this case as well, did not describe the range of users the device had to work for, because the anatomical considerations that were critical to this product were things like cervix size, and, you know, whether or not participants had previously given birth, which can, again, change a woman's anatomy.

So, these were all considerations that had to be built into early-stage studies, as well as in validation, which sampled across age, ethnicity, participants, whether they'd given birth, collected data on things like cervical size, height, weight, so on. And some tasks with this kind of product could not be answered in a simulated setting at all, which kind of changed the study plan right from the beginning, rather than just at the protocol stage. This is another great example on user versus patient, where some of these tasks depended on patient anatomy, and so therefore required, like, an actual use component, rather than just having.

A range of inclusion criteria from the user's end, even though those were The same person, because it was sort of self-use. So, there were majorly 4 different types of studies that were done when it came to this product. So, starting at the beginning, we have formative listed as one here, but in reality, it was really a series of very tiny studies, 3 users, 5 users, little quick and dirty studies that focused on different, very specific aspects, from designing the handle to label comprehension of one particular part of the IFU, and kind of iteratively contributed to the design of the product. And one thing that I think is worth calling out here is not every study resulted in a change. There were some studies where the end result of the study was actually what we had before performed better, let's go backward and go back to our original design.

But these questions were important to be considered and answered because it helped them really narrow down on what worked and knew what didn't work, and it was something that they could then build into their HRV report and pull from if they ever got questions from the FDA. When it came to the actual, sort of, validation strategy, it was kind of like a three-stage, which is what you see here in 2, 3, and 4. So, users, to start with, all had to go through a self-selection. As I mentioned, it's an at-home product, so a lot of the intended users had to correctly decide if the device is for them and go through a self-selection study.

After which… People who correctly self-selected into, you know, declaring themselves as intended users went into one of two prongs for the full validation. The first prong was simulated use. This is kind of your standard validation with controlled conditions in a lab, where the entire workflow was run through, root cause probing on every difficulty, but importantly. Because of the simulated setting and, you know, the video recording and so on, participants demonstrated the use of this product on a model. To then build in some of those anatomical considerations and show that those were accounted for, the actual use prong then kind of did the same thing, where patients

or participants coming in essentially came and actually used the product. In privacy, they would go install the product, have a healthcare professional come and perform a physical assessment to see if a female participant had placed the device correctly in their cervix, see if, like, in terms of actual use and actual placement, whether it was accurate and effective. And so, ultimately, the validation data used data from both of these, from root cause probing and understanding the source of issues, to understanding almost, kind of, preclinical performance and seeing whether the device was actually placed correctly, to then give a full picture. And importantly.

all of this strategy, or most of the strategy, especially when it came to 2, 3, and 4, was something that the company decided and discussed early on with the FDA through pre-subs and understanding, okay, what were the considerations? So, some of the anatomical and physiological characteristics that were included were things that the company themselves identified from early formative research and risk analysis and so on. But some other criteria were things that the FDA specifically asked for early on in pre-subs, saying. well, when you do your validation, make sure that you're including people who've given birth, people who've never given birth, people who've given birth multiple times. We want to see data across all of these things. And that was something that is a… was a planning decision made at the start, rather than something that they had to scramble towards at the end.

So, what did this program ultimately teach us about sampling? First, that the anatomy that drives the critical task is also what should decide the inclusion criteria. Identifying that early on from task analysis, risk analysis, even early-stage formatives, and writing that down before recruitment opens is critical to a smoother path later. Second, sampling across age, ethnicity, and life stage are all protocol decisions with both a cost and a lead time attached to it. By planning this almost, like, a year in advance, and thinking through all these characteristics, giving enough time for recruitment to actually sample across these criteria, they were able to end up with a much more holistic and balanced

User group for their validations. Third, in this case, with this specific kind of use and considerations for this product, actual use data was able to answer questions that simulated use could not. And again, that was something that had to be identified a year earlier, discussed in a plan with the FDA, and kind of planned for way in advance. Finally, subgroup results were reported whether or not they deferred, which is what gave the submission kind of a clean answer to both obvious questions, but also questions that the FDA had asked for early on. Alright, I'll pull Diane back in to talk about where it gets caught and where it does not.

Okay, thank you, Aarti. The difference between a submission that reports a subgroup result and one, excuse me, that reports An average. So, so far, I have not had the pleasure of reviewing a usability study that is drilled down into subgroup results beyond reporting averages. And then the qualitative feedback, from users being tested. And Aarti already spoke to the root cause sort of probing, so I'm not going to add, anything more to that, because I think she did a great job in explaining, how that is very necessary, and the qualitative data Is very important, for us to see beyond, the averages.

Lori Gutierrez , that are reported to find data that's buried there. But I am going to add another bias to the ones that, Aarti added, and it's representational bias. Where we believe that a small sample size can be representative. I know that a lot of. manufacturers have used the FDA human factors testing, where I believe it's somewhere between 15, 20, 25 users is considered enough for the studies. The IC for. for MDR does not… we will accept human factors testing, so I know there's lots of people like, but the IC does not provide any guidance on the sample number. It's more of an open-ended Approach, and dependent on the engineering assessment, and looking for a justification for how those sample sizes will get you, you know, reliable, data.

So the next question is, what post-market surveillance and complaint data look like when the subgroup was never examined? So again, under the MDR, use errors are not reported unless they're related to a serious adverse event. However, MDCG 2023-3 includes a very specific section on use errors due to ergonomic features, and talks about the compromising, how these errors compromise, safety. There is a chance here with the well-designed post-market activities that you could also root out these issues. But often the reports… reported events are not enough, for manufacturers to trigger, you know, a capital or other review. So often these, these issues remain… remain buried and… and

You know, with the risk that we'll actually never find them, or they're… because we think they're another… another issue. I think the real opportunity comes from the user profile and things that, Aarti has already talked about, and really looking at the mental, physical, and demographic traits of the intended users, and also, patients, because often they're both users as well. So, what manufacturers can do in the clinical evaluation that costs little and answers the question early? Yeah, that's one thing we're always looking for, so I'm gonna say a stitch in time, definitely. With the current pressure of the MDR 2.0.

Looking for ways to lessen the burden for MDR certification, it's my belief that Usability Testing will be an area that will weigh more in assessments going forward. The other point is know your device. The expectation is that the manufacturer is the expert of their device, including how it is used, who uses it, the environment it's used in, how that impacts the use of your device, if you know your device is used by nurses. You know that the majority of them are women, and Aarti talked about grip strength and height, and how that can impact the use. the device. And aging nurses grip strength, both men and women. It's a known, physiological change that that lessens, in time, and the demographic of nursing staff is definitely aging.

So finding user interface issues early and not having to redesign or recall your devices is always less cost across. For the manufacturer, patients, and everyone. Okay, I think we're ready for the next slide, and also mindful of the time, so I tried to hurry up a little bit through there.

Yeah. Before we go to the next slide, I just wanted to chime in quickly on a really great point you mentioned on participant numbers, and I think that's something that, you know, as you said, even with the FDA, generally, the kind of base guidance is. 15 to 18 per user group, at the validation stage. But very often, depending on how novel a product is, I've definitely seen the FDA come back and ask for more. One of the really great examples of this was COVID tests, where early on in 2020 and so on, there was specific guidance that came out where it was, like, the FDA said flat out, we don't want, you know, 15 or 30, we want 100 participants.

You know, split across your different demographics, representative of the census, and so on. I've seen the same thing when the first sexual health test came out on the market, where the FDA and their feedback said, we want at least 60 participants. So, very often, depending on how untrained your users are, depending on how novel your product is, that number can definitely go Way higher up.

Yeah, I would add to that the new guidance just at the draft stage on SAT probes, also looking at changing their whole view on testing and and sample size there as well.

Yeah. Okay, so then… Going to, kind of, what is already being asked for, and I'll talk through a little bit of the FDA and pass it over to you, Dan, to talk about EUMDR. So on the FDA side, guidance… typically ask manufacturers to balance demographics and study populations. I touched upon this a little bit already, but it really depends on who your intended users are and profile. So, for instance, where your intended user is really gen pop and could be, for instance, anybody in the United States, I've seen the FDA come back and say, well, just replicate the census, you know, balance your demographics across race, age.

education level, and so on. For other more specific products, they'll ask to represent or over-represent certain groups, depending on what they're concerned about. FDA also asks for results to be stratified and compared by sex where relevant. This is something that we've seen pop up majorly when it comes to pre-subs and specific devices, where the ask kind of varies according to different products, rather than just being a general guidance for everyone. And lastly, the… a human factor submission that pulls everything, kind of invites the question in review when the clock is most expensive, so doing some of that work early on before you get asked can really save time and

Money.

Okay, the EU MDR looks at the clinical evaluation has to cover the intended patient and user population. An assessor looks at whether the evidence describes the people who actually use the device. And I'm going to say this is very specific to what's called GSPR1. And I'm going to read a little bit of it because of memory is not what it used to be. GSPR1, devices shall achieve the performance intended by their manufacturer and shall be designed and manufactured in such a way that during normal conditions of use, they are suitable for the intended purpose. I know that's kind of general, but that's sort of how legalese often is written.

And also talks about any risks which may be associated with their use constitute an acceptable risk. So, for clinical evaluations, GSPR1 is our… I mean, it's our main, requirement that we sort of look at everything comes from… from that. So, exceptionally important here. Post-market surveillance and PMCF are where an unexamined subgroup surfaces with a complaint attached. Sure, it can absolutely come there, and PMCF is going to be a much. more, sort of weighted activity, again, with what we expect from MDR 2.0 coming, and also appreciating that data sets are never perfect, and they're always missing something. So this is where we try to find, you know, with greater use and more real-world use.

A plan that can capture, those subgroups.

Right?

Thanks, Diane.

Yeah, so I'm not gonna go through this slide in detail, I'm just gonna flash it up here, but these are just some really quick five checks, things that you can do even next week, that checks to run against your current program that will take into account sex and other types of bias. If you would like this full checklist, just type checklist in the chat, and it'll come to you today. And I'll pass it back to Sean to wrap things up.

So we want to talk about what you take back to your team for things worth repeating to the people who are not on this call, right? A lot of people not on this call. And so what, you know, there's so much value presented, but a 50-50 split between men and women is the start of the answer. Polled results. can pass failure… can pass on the failure inside them. So be careful with pooled results, right? Two, risk enters, at design inputs, task analysis before the prototype exists. 3. A subgroup failure is only visible if somebody decides to cut the data that way before the report is written. So, think about your subgroups, right? FDA and notified bodies are asking for this now, and the cheapest place

to answer it is in your own protocol. Start early, start often. Next slide.

Here, here.

So we are very, very practically out of time. We have two minutes left. So I love all the questions that were submitted. We will follow up with you individually with responses to your questions. I'd like to acknowledge everybody who submitted a question. And move to the next slide, please. So, what I want you to do is, to type a 1 for the Inclusive Usability Checklist. You'll get that, sent to you. If you would like a 30-minute, plan to review… to review with Aarti, type 2, and. The fertility Device walkthrough type.

A 3. So, yeah, that's… more or less that, and I want to thank our speakers, Aarti. If you want to move to the next slide, I think there's… Maybe one more, maybe there isn't.

I think this is it.

This is it. Okay. So Diane, who has joined us from her vacation. I want everybody to applaud Diane for joining us on vacation. Very near Alaska at the moment, I understand. I hear it's beautiful. And Aarti, who got up very, very early in California to join us, so we appreciate it, we appreciate you, we appreciate everyone who attended, and, look for a lot of, really interesting, follow-up from this, because this was so detailed. And so helpful. I really encourage people to take this and share it with their team because I think this is really incredibly valuable information.

Thank you so much to both of you for your time. I'd like to give you a second to say goodbye to everybody, and then we're going to end the call.

Yeah, thank you so much, everybody, for attending and for… I've gotten a sneak peek at some of the really great questions, and I am very excited to respond to those in more detail over email.

Yeah, thank you everyone for your time and thanks Sean for putting this great topic together and Aarti I learned so much from being participating in this as well. So thank you.

Likewise. Thank you so much, Diane.

Take care, everyone. Until tomorrow. Bye-bye.