Designing Better Formatives: Simulated Use, Realism, and the Risks of Playing It Safe

Hello, and welcome. I'm Sean. And I can't get this bar to go away. There we go. Welcome, everybody. I am super excited. Today we have guests, Denise Forky and Davide Salinitri from Claramid, and we're going to be talking about designing better formatives, or as I've been putting it, formatives vs summatives. And, It's a fun topic, it's human factors, related topics. So, before we get started, just if everybody wants to just put in the chat, like, you know, how familiar they are with human factors and or,

you know, formative studies, any experts on the call, just let me know. And… yeah, I'm gonna go ahead and turn it over to, Davide, and have him… Tell us why we're here today.

Hello, hello, thank you very much, Sean, for the… for the introduction, and of course, the invitation to this… to this webinar. So, first of all, I would like to introduce myself, and give you a little bit of more information on who… who I am. So, as Sean said, I'm David de Salanitri, I'm a Senior Yuma Factor Consultant at LarryMed. I've been in ClimateMed for about 2 years, but in human factor for the last… 11 or 12 years, so I… the story of UMO Factors, I always like to… to tell how I got into Yumo Factor, because I didn't, you know, I didn't start thinking about human factors, I did my psychology, university, my psychology degree, did my bachelor, thinking that I…

was going to be the next Freud, and then something changed in me, and I didn't know what to do, and then I… I was chatting with this professor, at my university, and he was an ergonomic and human factor professor, and he said, oh, you know what? see a lecturer, and I fell in love since then. And that was 2011, and here I am now, pursuing my dream. So, yeah, I started Human Factor from Psychology. That is one of… of the root of, of human factor, and then… I did my PhD in human factors in… in the UK, at the University of Nottingham, then worked on, automotive first, and passed into medical. That… so I… I kind of…

merged two of my passions, that one is human factors, and the other one is medical. And to all my surgeon participant, I always tell them one of my dreams was to be a surgeon. And so, you know, medical, human factors, and here I am, doing what I like. So this is… this is me in a nutshell. Denise, I'll let you introduce yourself.

Thanks. Thanks, Davide. And thanks, Sean, for the invite. Nice to see everyone here. Looking forward to some great discussion today. I'm Denise Forky, I'm the Director of Human Factors, Strategy and Advisory here at CoreMud. Our clients navigate FDA feedback on human factors, figuring out strategy, as well as just mentoring clients and other team members on best practices or ways to customize the process. So, I have been in the medical device industry for almost 30 years now, hard to believe it went by really fast, but, in various capacities. So, I've worked in companies like Guidant, worked in service organizations like Licron and Surpass.

In various capacities, from preclinical research, doing more quality systems, to more recently, in the last 10 years, focusing solely on human factors and usability engineering. This last year, I actually helped my late husband navigate his cancer journey, and it was quite eye-opening for me, even as a human factors professional. And so I like to say that I have, like, a renewed energy and renewed vigor towards really making sure that devices are safe and can be used effectively, and so I'm happy to be in this field and continuing to work towards making products better for users and for patients. So, looking forward to talking to you all today.

John, do you want to come forward, or should we take it from here?

Yeah, let's go ahead and start. Let's hop into the presentation. I'd like to encourage the audience, if you'd like to come off of, lurker mode, to just raise your hand and you'll be promoted to presenter. If you have questions, you know, jump in, be respectful, and, You know, let's, let's, let's keep this fun and active today.

Hey, Sean, can I say something? Quick, quick question, where is Clarimet located? Where are you guys? U.S, Europe?

oath.

Both…

Yes, so we are… we have a global footprint. We have multiple sites in the U.S, from San Jose, Chadsford, which is near Philadelphia, Atlanta, and Boston, and then also have, sites in the UK, in Leeds and Cambridge, and consultants like Davide, who are in Spain, and an office in the Netherlands.

Nice, amazing.

Yeah.

And I might add to that, if you're… if a consultant, and you're on this call, and you're interested in having human factors testing done for any of your clients, please feel free to drop in affiliate program into the chat, and, you know, somebody from ClareMed will reach out to you and talk to you about, you know, referring business and, you know. Being a partner. Thanks.

Hey, so… I think we can… we can start with the… with, the main topic of this chat, and I'm calling it a chat because, as Sean said, I really hope that this, moves, from a presentation to our conversation, with all of you. So, if you have any questions, or anything, or any remarks, any examples, just feel free to, as Sean said, raise your hand, and we would be happy to, to answer, all your questions. So… The first, the main topic that we are talking about today is formative evaluation. And, of course, we cannot talk about formative without mentioning the summative evaluation, but the main, the main topic would be formative. So.

what are formative evaluation? Of course. You… you will have different experience with formative, summative, human factor process. Some of you may be very familiar with what a formative is and what a summative is, and some of you maybe are hearing this for one of the first times. Oh my god, what is a formative, what is a summative? So… Formatives are… They're both… we can call them tests, we can call them study, we can call them… I would say probably test is the… is the right… is the right term. They're… they're, used to understand medical devices and pharmaceutical devices. So they are used to understand something about the interaction that users have with those devices.

But there is a big difference between formative and summative, and this is the main reason, one of the main reasons we're having this chat, is that more and more, and Denise, can confirm that, clients come to us, and they say, we want to do a formative. And they say, well, that's perfect, great. How do you want to do it? Exactly like a summative. And we are like, okay, here there is… there is an issue. How do you want to do a formative exactly like a summative? There are two different things. Doing the same thing twice is not very smart if you are developing your device. So, the first thing we come up to talk with our clients is.

These two evaluations have different scopes and different purposes. formative. is what you do during your device design. So while you are designing your device, you use formative to kind of get that user opinion, user interaction, user understanding of your device. The summative evaluation is what you do at the end. When your device is designed and is ready to go to market, or you think is ready to go to market, then it's the final proof That that device is safe, it's effective, and your users are going to use it Without any issues, or without any issues causing critical harm.

I like to make this example. Fourth example. Formative evaluation is the training. Some of the evaluation is… is the match, is the game. You cannot… treat them the same way. Training as a purpose, the game as another purpose. You cannot treat the training like it is a game, because you wouldn't learn anything. The training is meant to teach you something, and to then prepare you for the game. And you cannot treat the game as training, because you will lose all your matches. So, that is basically the main difference between these two, these two evaluations. Here, there are some, as we show in this slide, there are some comparison between the two.

I spoke about the purpose. The core question? what is happening with my users, formative? What is happening with my users? What are my users thinking about this device? How can I improve it so the users are not led to make mistakes that then can be risky at the end? And the core question of the summative is, is this safe? Did I do everything that I had to do to make this device safe? When, this is very important. Formatives happen through all the process, from the beginning until the end. It doesn't happen right before the summative, it doesn't happen just once in the middle, it happens from the beginning until the end, and we're going to discuss about it.

we… we did formatives with… even with concepts. It doesn't even have to… you don't even have to have a physical thing there sometimes. Summative are done at the end. Again, requirements, we're going to discuss it. The device has to be market-ready to do a summative. So, It has to be done at the end, it cannot be done in the middle. Usually, You have more formatives, and you have, again, multiple rounds for formative, and you kind of change your formative depending on the result from the last formative, and you kind of go and change it. Summative usually is one at the end, where you get your user group, and you do the final evaluation. If something goes wrong, and you have to repeat it, you have to repeat the whole thing again.

as… as of now, you may have guessed that formative are the flexible bit, summative are the strict bit. So, formative are the ones that, allow us to be fluid on even the protocol. We can adapt the protocol, we can be We can change the protocol. We have some formatives where you change the protocol during the formatives. Sometimes we do formative of 2 days, change the protocol other 2 days, change the protocol another 2 days, and go on. Summative is a protocol that usually is… you do it, you approve it, you lock it, that is what you're going to do. Every bit that you're going to change during the study, for whatever reason, will have to be justified, and will have to be reported in the report. So, it's very strict, it's very,

Square, I would say. And the outcome, both reports, formative and validation, or summative reports. But they have different meaning. The formative report allows you to demonstrate that you do… you did your job in iterative design, and the summative report is what actually demonstrates to the FDA or the notify body in Europe. that your device is safe. You're presenting data that there is no critical Arm that can result from interaction with your device. So there is, there is a difference there. Talking a lot on this, but, this is basically the main… these are the main differences between the two evaluations. So, as you can see, there is difference. So, if someone comes to us and says, can we do a formative, we are prepared to go through all these topics and say, okay.

What phase are you in? Well, I'm at the end. I'm like. what do you mean you're at the end? You know, you should do formatives beforehand to understand how to design it. Or… no, I just want to see if my summative protocol is fine, and I won't have an issue in the summative. Again, not exactly the reason for… for informative. denise, did I miss anything?

I think there's just… I think you did a great job of summarizing that. I think there's just kind of two comments that I'd like to make. Number one, I like to think of formative evaluations as those evaluations that are helping to form the user interface into what that formal user interface will look like. Making sure that it's safe, making sure it's effective. People can use it. And second, if you don't do formatives, you're gonna have a very expensive summative that will turn into a formative, and you'll end up costing you lots of money in the long run. So, we like to encourage people to do those formatives, make sure that your product user interface is evolving appropriately, you're identifying, you know, potential use errors and risks, you're optimizing the usability of that.

product, making sure that you're designing the right device for the right user in the right use environment, and the right intended use, so…

There was a question in the Q&A, is it required to have plans for both evaluation types?

Yes.

Yes, it is a requirement. Please, go on the.

So… Yeah, so you're required to have a user interface evaluation plan. which can be pretty high level. You're not required to have an individual protocol for each formative. You are required to have a protocol for the summative. So, depending on your organization, the size, the culture, everything. else that kind of can tailor that. Basically, you can decide how you want to handle it, but at a minimum, you have to discuss your formative evaluation plan as part of a user interface evaluation plan. So, some clients, we kind of guide them to do, like, an overarching plan, and then, as it makes sense, have individual protocols for the formatives, just to help guide your team on what exactly your goals are, and how you're going to accomplish that goal.

I agree. I think it's… again, agencies want to see that you… you've done your job. That it's not just do something. engineering-wise, or designing-wise, and then put it in the market at the end. They want to see, did you include the use… did you get all the possible risks? Because that's one thing the formative do, that is. I'm going to introduce the users to my device, the potential users to my device. That allows me to… allows me to discover new and uncover new risks that I didn't think about. And the agency wants to see, did you do that? So, once you come to the end, to the summative.

And you say, these are the risk of my device, how do you know that those are the risk? without including the users first. So this is what Formative gives you. They give you that data that you can say, I did my job, I did my research, and not only literature, I actually got the users there. And this is what the results are.

So there was a question on whether or not the slides were going to be available. Yes, we're going to send the slides. Anybody who wants the slides, just write slides in the, thing. Also, at some point after the fact, day or two, maybe a week. I don't know, we're doing a PDF, packet of, designing better formatives and a comparison chart of formative and summative, so if you'd like that, PDF, just write PDF in the chat, and I'll make sure you get a copy of that. Let's see… .

There was a question in the chat here. Can you comment on how a summative HF study for CLIA waiver is different from a standard device 510K usability from Alan?

Yeah, waiver. Summative. Denise, if you want to add, please do. I mean, there are several aspects to, to get into, into summative. It really depends on the… on the product. I mean, Having, a general guideline of how to, design a summative, just from… The submission pathway, it's difficult, because it really depends what your device is, what risks there are, how many critical tasks. Any information you have beforehand. So, if there are any devices that are relatable or not. So it's… it's hard for me to say exactly how… to… to design the… or what the difference in design are without knowing exactly how the… the device is going to be. I… I know that clear waiver is, of course, is for in vitro diagnostic, and, usually it's for, it kind of tells you that,

It has to be… It's very intuitive. It has to be a very intuitive device, so usually you get The summative to demonstrate that. Denise, do you want to add something on that?

Well, I was just gonna opt to ask me to put him in contact with expert, and we can… she can kind of talk through the specifics on the case, and that way, we can provide a more Targeted answer to the question.

Great point.

Well, thank you, thank you for asking the question, Alan, we appreciate it.

I'd also like to mention, that we're doing, a full webinar, next week on, human factors for legacy devices, that the number of legacy devices are going through, EUMDR, final submission and are getting, stuck. on their human factors work, and that's gonna be a really interesting session next week. I put that in the chat earlier, I'm gonna put it in again. Please register for that. We have more than 300 people registered so far. Would love to see you there. That's it for me.

Okay, so, I think we can go forward. this is exactly what we spoke about. I mean, formative studies are not summative rehearsal. It can be very expensive to treat them like that. Because if you do a formative just to see if your summative protocol works fine. Then the risk is that when you reach the point of a summative, without any information beforehand. That summative is not a summative anymore, it's a formative, as Denise said. And… and this is as even cost… as a cost, a summative is more expensive than formative for… a number of reasons. It's… it's much more expensive. So you… if you go directly to that, and you go directly into doing a summative, what you're actually going to do is a very expensive formative.

to then have to redo the summative again. So, even cost, the cost-effective bit is, is not… it doesn't make, much sense. And we cannot treat formative as summative. This is… more of, a very, very strong suggestion we give, and of course, you cannot treat summative as formative. That is actually not only our suggestion, that is what actually the regulations say. Summative are strict, summative rules, summative are way to do it, a way to make them. If you treat a summative, like, formative. It's not going to be a good summative, and you will still have to repeat it, and still have to… to pay double. So,

Confusing these two evaluations is dangerous for many reasons, including A cost reason. But mainly, a safe reason. The device is not going to be as safe as you thought. So, one of, of, let's say, the, maybe the differences that are the… and we can go to the next. To the next slide, if you want, Denise. So… It's study fidelity. And this is a very important part, I don't know, Denise, do you want to chat about it? And then I can add something?

So I think, as we're looking at both formatives as well as summatives, we really need to think about what is the appropriate study fidelity for our overall study goals. And so, Study Fidelity can, take the shape of, you know, at least these 5 different categories. Physical, that's really what does the device look, feel, behave like. Compared to the real product. Does it have a similar weight? Is it made with similar materials? Does it have a similar surface, a resistance form factor? So, if you're testing out, say, a syringe or an injection pen. And the force that's required to push the plunger down is not representative of what the final product will be. It's a really rough

Prototype, then you wouldn't necessarily want to get information about risks related to applying enough force to push the plunger down, in that holder type. Instead, you might use that prototype to look at, kind of, where do they think they're supposed to push, to apply the injection? Or what do you… what do they think the dosing means? So, as you're looking at what you're going to evaluate, make sure that whatever you're evaluating Has the right physical appearance, if that is something that you want to include as a goal for your study. Next, functional. Does the device perform as it would in actual use? So…

If you're testing out, say, a generator with different user interface outputs and screens and alerts, do they pop up when they're supposed to? Is the alert interactive? does… if they push a button, will it proceed to the next step? So, those are all things that you would consider as you're designing something. If you want to look at, kind of, the general workflow, does it make sense? Can the user, kind of, follow from one thing to the next? And so, making sure that your… your prototypes Can work within the use environment that you're evaluating. Next is environmental. Does the setting reflect real-world conditions? Real-world conditions can include things like vibrations, noise,

constrained area… how many people, like, density of a population of people in the room. So, for instance, if you're trying to determine how a user can interact with, say, an emergency ventilator, and it's going to be used in an ambulance, then you want to have some studies at some point along the way that look at, kind of, can they do that in a constrained Environment? Can they do it with more noise, more vibration, in an emergency-type situation, as opposed to just evaluating that always On the bench, or in a nice conference room without any environmental factors. Or if you want to evaluate someone's ability to use something when it's very, very cold or very, very hot, those are some things, that might impact the use of a device, and so we want to make sure that going into the study, we understand, are there environmental factors that we really need to be considering?

Denise, can I…

just…

Please, jump in, yes.

I want to jump in on this environment, and then I see there are some questions, but I want to jump in on the environmental part, because I want to make an example that recently happened. I was speaking to a client, and the client said, my device is going to be used in… it's going to be a home use. And I said, that's great, where? And they said, home. And I said, yeah, where at home? And… and they said, well. we thought about it, well, in the bathroom. And bathroom is not the same as kitchen, and is not the same as living room, because how much space do you have in your living room to use something? How much space do you have in your bathroom to use something? And then, if you move that to public.

Public bathrooms are not as clean as your own bathroom, so all the part on where can… where can I… leave my device while I'm… I'm not using it, or before I use it. Is there a cleanliness part that we have to take into account? Is that a risk? It may not be a risk if your device, you just use it at your place, and in a living room. it may be a big risk if you're using in a public bathroom. So, environmental fidelity and understanding exactly what are the safety characteristics of your environment. It's super important. And sometimes you have to be as precise as possible, sometimes you don't, depending on the device, sometimes you absolutely do.

Thanks, David. I think that was a great example of where understanding the situation is really key to designing the most effective study and usability program, really. So I think that's why we always go back to understanding the use specification. Who are the users that are going to use the product? who are the patients that are going to be having this product either applied to them or used with? What is the use environment that it's going to be used in? Because getting a COVID test in a car is different than getting a COVID test in a clinical office. And so those are things that need to be considered as you're

thinking about usability as you're thinking about risks, as you're thinking about how you're going to evaluate your product in formative studies to improve your user interface, and ultimately how you're going to evaluate it in summative. And so those are things that are kind of key, foundational Building blocks for a usability program, and they matter, and there's a reason why we set those, at the beginning of a project, and refine them along the way as we learn things. So, the next, the next one would be procedural. So, if you are trying to get more of an idea of how a workflow will work for a patient.

administering, say, injection to themselves, then it's important to understand, kind of, what are the steps for doing that injection, and making sure that the preceding steps, you know, leading up to injection are also part of that process. Allowing them to use the product more naturally, and so that you can identify if there are use issues that come up as they're using the product independently. So I think this is the one thing that's, I see a lot of with manufacturers, where they will come in and they'll want to prescribe to the user Exactly the steps they're supposed to be taking. And what that does is it basically erases your ability to see what the natural tendencies are of the user for how they might use this product, and how they might use it wrong. And so, making sure that you're thinking through what are kind of the procedural expectations that I need to be evaluating for my study goal.

And then finally, looking at participants. Do the participants represent the actual intended users? And that's not just in age, or not just in gender. It's basically their skills, their experience, their health literacy. If it's a product for the general population, we have people who have an 8th grade education, all the way through highly trained surgeons that are using products at home. So really understanding what are some of those worst-case scenarios that you need to be thinking about to make sure that your product can be used safely. And for formatives, one of the things that comes up is that the FDA says, yeah, it's okay for you to use employees at a manufacturer.

Well, I think we need to really think about that carefully. If you're having engineers use a product that's going to be used by the general population, then really is that representative of an actual user's experience with the product? Probably. Only very, very, very small percentage of the population have that level of education. So… make sure that you're kind of smart about it. If you want to have, say, your clinical team evaluate a product that's used in the OR, and that clinical team has experience in the OR, maybe they were an OR tech at some point in their life, or a nurse. then I think that that's representative, and they can be an appropriate surrogate for those end users, but

just evaluate each case independently to make sure that you're making smart decisions on who should be using the product and does it make sense? If it's a home-use product, maybe you go out and you talk to people in your accounting department, in your shipping department, in your clinical, your engineering, so you get kind of a broad sample of people so that you're representing more of the general public, as opposed to just a highly trained, highly educated user. So, as you're looking through and designing your formative evaluations, really, it's key to think about what is your overall goal, and then what are the parts of that study fidelity that are most important for you to meet your goal without producing data that may

you know, lead you in a direction that's incorrect. So, as you're thinking through that, you don't have to design all of these perfectly in your formative studies. You can look at each one independently, and maybe dial in higher fidelity for one of the features that you find will most impact the overall success of your study. And success meaning that you identify use issues, you identify use errors, you identify shortcomings of your product, as opposed to saying success is, yeah, my product performed great. That's actually not what we want in formative studies.

That's great.

Take a pause here, David, if you… or Davide, if you have any other comments or questions.

Yeah, I do, but I think we have questions, from the audience, so.

8.

And probably take them.

Absolutely.

We… I'm from Finopress, and we do our own formative and summative testing. And my experience with a lot of formative testing is, well, it's, it's a lot of work, and it costs a lot of money, so you tend to, Make your number of users and participants Not that big, because that only raises the amount of work. But then there's a danger that, if you want to spread your participants in your cohorts that you're, you're having, and in different, in male and female. There's a danger that there's only a sample of NS1 in a cohort. And that will, if there's a different outcome than expected, it will have a large influence on your total outcome.

And I guess the buffalo reaction is to treat it as an outlier and disregard it and go on, but that's also very dangerous. So my question to you is, how do you handle that, that you have such a low number of samples? And I have another question on that, but can you please answer this one first?

So… The… it's true that usually formatives, have lower number of participants, because they are considered. smaller studies. It is also true that, again, Depending on how you design that formative, and if you have a specific aspect you want to test with specific participant, it's Absolutely okay to just limit. that formative to that specific aspect. Let me… let me be more precise on that. If you have a device that has 5 use scenarios, and you think, in this use scenario, or for those specific group of tasks. I may see big differences between, for example, women and men, because of, I don't know, handling, or strength, or, I don't know, whatever…

reason that that could be. You can limit, and you do a study specifically to that specific group of tasks, or that specific scenario, where your actual objective is to understand that difference, and if that difference is actually an added risk. So, the good thing about formative is you can really you know, use your fantasy on how to get that. And when I say use your fantasy, I say. If you want to understand something, just design something for that specific thing, because it's going to be cheaper in terms of, well, if I get 20 people for 10 minutes, it's going to be cheaper than getting 5 people for 2 hours.

Especially if you are taking users from outside, that you have to incentivize in some way. So… The idea is always… Your advice is to, to…

Have a special test for that outlier to see if it's valid or not.

It's difficult to say, because it depends on the outlier, what do you mean by outlier? Like, if it's something that happens. And it's… it's not foreseeable. That is one… one of… of the good way of saying it. When… when you, this is, more regulatory path. So, when you get your risks, you always have to think about foreseeable risks. So, you know, if you're talking about a surgical device. you don't think about, what about if someone for the normal public use it? Well, that's not foreseeable. I wouldn't, in my life, in my normal life, get I don't know, a surgical device and start using it. That's… that's not foreseeable. So, if that is… the outlier means that that interaction is not foreseeable in real life, then I would completely disregard it.

Now, if that is foreseeable, then it is not an outlier anymore. And now, how that has to be included into the study, of course, is something that has to be understood. How do you understand it? With formatives.

Well, and I think that… I think also it's important to understand that one formative doesn't mean that you're done, right? If you find something that appears to be an outlier, you've done a root cause analysis and determined that it's not study artifact, it's not some, you know. artificial situation. For instance, maybe you have a surgical tool, and you tested it in 4 men and 1 woman, and the one woman had trouble using it, and then I would say that if you dive deeper, you might determine that it's due to her hand size, or due to her ability to apply force. And so really looking into what was the root cause of those issues and problems that were found can help you understand if they are outliers or not. And if you're not sure, then

Recruit 3 more women, 5 more women, to be involved in evaluating your product, so that you can understand, is it truly, you know, an anomaly, or is this something that is, you know, bigger that we missed, and that we need to really hone in on to optimize the design?

We only have Davide for 5 more minutes, he needs to leave a little early, I want to make sure that he gets to make all of his major points. Are there anything in the following slides that you're gonna die if you don't get to say?

Not really. I think the main thing, and I would really want to have a chance to keep answering questions, is one important part of these five dimensions that I want to say is usually the last one, participants, is the one that we are less flexible on. As Denise said, you know, you wouldn't get someone that is completely out of your user group, because the data you're going to get are not actually valid. I wouldn't test myself for a surgical endoscope. why? I mean, I don't have any idea how it works, so why would you test myself? So, that is always, it's something that we are more strict, but all the others.

the level of how much you want that to be exactly… the fidelity of those has to be flexible during formative. Not summative, but has to be flexible during informative, 100%.

There was another question, from Rama. Rama, do you want to come online?

Yes, thank you so much. First of all, thank you guys for hosting this. I've been in MedTech. For 9 years now, across, physical hardware medical devices, active medical devices, standalone software as medical device. And it's really interesting, but I've only come across usability, human factors, with the giant corporations. You know, typically the smaller. SAMD developers or the smaller companies typically don't tend to do, formative, summative evaluations. So I'm kind of interested to see what your viewpoint is on… is usability… we know that it's referenced in the ISO 1345, but, you know, I've seen from the field that frequently manufacturers will use post-market surveillance, and things like complaints to justify not doing summative and informative evaluation or usability at all.

And, I'm kind of wondering what your guys' viewpoint on that is, and then secondly, for SAMD medical devices, where you don't necessarily have safety risks, or risks to injury because of use. Do you guys still see that there is a demand for usability from these kinds of, developers for SAMDI products?

Can I, can I go, Denise? Because I just have a couple of minutes, and then… So… Very, very true. Rama, very true. formatives have been, in the past, it's changing now, and I'm going to say why, something that usually big companies do. Why? Because they have more money. Guys, let's put it there. They have… a bigger budget to do things, so they can afford to do 5, 6, 7 formatives. And usually, they have a human factors group with them that probably push for that. So, because, of course, they don't throw budget just because. But they have someone inside that strategize, and from the… building of the SOP, they say, you know, formatives. They say, from the building of the plan at the beginning, they say we are going to perform formatives. Now.

Sometimes we get startups coming to us. With, without a human factor person presence, and saying, okay, we have to do usability, is a regulatory requirement. what do we have to do? And in that case, well, we have to kind of guide them. And the idea is that I don't know what I have to do. I have to do formatives. Again, if we treat formatives as summative, no… very few startups can… can actually afford that. Because running for summative for a product. I mean, that's a lot of money. But what I usually say, and I say it very clear to some of my clients, especially smaller ones, I say, look, formatives allow you to uncover risks.

that… and two types of risks. One risk is the one that you're going then to see in the summative, and you have to repeat the summative. Meaning a lot of money that you may not be able to afford. But even if you can, there is another risk that is way bigger. That is, you don't uncover them, or don't discover them in the summative. You discover them when the device is on the market. And… How many small companies, a startup can afford the repo? I mean, big companies sometimes can, because… because they have many products in the market, and they… Small companies not always can afford the recall. Not always can afford to have to redo a human factor process with a device already in the market by completely stopping to, you know, sell or use that product. So…

It is important… am I saying that if you don't do format, if you're going to have a record? Absolutely not. But there is the risk. to that. So why have it? And why… And we use our fantasy to adapt the formative we want to do with the budget you have. So, if you're a startup, then, of course, we are not going to say, okay, we're going to do 20 formatives, and every time there is a small change, we are going to do a formative. No, we are going to do Smart. process to get all the formatives, in place and get all the risk in place. And on, then, the part of the data that is, actually. it is part of the regulation, you have to use data to justify a lot of things. Not only not performing some part of the HF process, but also

again, what we spoke about before, getting where did you get these risks? And you say, well, I have data on devices that are very similar to mine, and I can say for the past 20 years, there are no… no risks. So… there won't be with mine. So data used a lot. Can those data avoid part of the HF process? Yes. Justification has to be big. It has to be huge. And usually, that justification doesn't allow you to not perform the simulative. It's, it's, it's harder. I mean. Again, depend a lot on what you're doing, and if the changes or the difference between your device and what is in the market. That is the main, the main thing. And as the software.

There is no physical risk in terms of you cannot injure yourself using a computer. But if it is a medical device, and again, like all medical device, we would do a risk assessment. And not being able to understand Information that a software is giving you, and with that, doing or making a mistake in terms of medical decision, because you don't understand what the software is saying, it is a risk. So the software is saying something, you don't understand it, you make a medical decision based on that. And you pose harm to the patient. That harm is directly caused from the software not being designed correctly, and as a physical medical device, that has to be mitigated, and that mitigation has to be

Has to be then evaluated.

in the HF process.

I hope I answered your questions.

Yes, yes, absolutely. Because, you know, there are a lot of devices, SAMD devices, that border on general wellness, and sometimes they're general wellness, sometimes they're Class 1. Would it be correct to say that if there are no safety risks or injury possible, then usability should focus on performance? You know, perf, security. Any kind of claims made about the device, as opposed to focusing specifically on safety? Is that something that.

I believe it's fair to say, I mean, if there is no critical task, if there are no risks with your device, and you can prove that. Absolutely, I mean, I don't think there is, the need to test critical tasks if there are any. I would still include usability for what you said, performance and intention for use, and even, you know, for possible Then, metamorphosis of that device, or that, software to a medical device in the future. Why not? I'm very sorry, thank you very much, Sean and everyone. I'm really sorry I have to leave you all, but I'm available for any questions afterwards, so feel free to contact and write in the chat, or whatever you want, and of course, Denise is

It's one of the… you know, most knowledgeable person I know in the human factor side, so, she will be able to answer double of the questions I would be able to answer, so, you are in good hands. Thank you very much, everyone, again.

Yeah, I mean, I think… I think to expand on what Davideh said. really, usability isn't just about safety, it's about how well, you know, how does the customer feel when they're using the product? Is it intuitive? Can they, you know, do the steps that they need to in order to use the product for the intended goal? And ultimately, if you don't figure out that information before you release the product, it will come later, and you will have a much larger usability dataset that comes from real-world use, and real-world problems that happen when your product is not usable. So…

And just one last… one last comment to Rama. I have personally seen small startups all the way to big companies recently get asked by the agency for human factors. It's been on the spotlight, and so while it's okay to provide justification, obviously go through the Q-Sub process with the agency to kind of vet those. assumptions really clearly with them, and I think that's the safest bet for companies that have a hard time supporting any multiple formative studies.

Agreed. Yeah, and make smart decisions based on data. So… So I think the other thing that is important is that as you're looking at formatives, early stage, all the way through the final pre-summative. the requirements and the goals of those are much different. So, as you're looking at early stage, you may have less user training, you may look at the ability of the user to follow something intuitively to make sure your product promote safe use, but when you get closer to the validation, pre-validation, even before that, you may want to do an evaluation of the training program to make sure that it's supporting safe use before you jump into that summative

Or validation testing. Similar with the test environment. You may be able to get away with something that's really low fidelity in some of those different areas of the study fidelity, or in specifically the use environment, and still gather meaningful data when you're in those early stages, but when you get closer to that summative, you're going to want to have a more representative use environment to make sure you're really not missing anything, the influence of the use environment on the safe use of that product. Similarly, with patient model, you could have no model all the way through representative at the very end. And study articles, the same thing. So, I… I have a case study I can share quickly, or if there's any other questions,

I think that'll be about all the information we can share today, so let me know, Sean, if there's other questions. Otherwise, I'll jump to the quick case study.

I actually have a question for Denise. Sorry, Sean. Can I ask?

I think, Denise, would it be great to clarify here? I think.

there's this misnomer of, like, thinking about usability only about an interface, right? And I know you guys have have tackled this a little bit throughout the conversation. I think it's really good to emphasize it's not just an interface somebody touches with, right? It's… there's a lot of different touchpoints, and can you just elaborate a little bit more to bring clarification?

Sure. Yeah, so a user interface is actually anything that an end user sees, hears, feels, and so that's not only the hardware and the software part of it, but it's the labeling, it's the instructions for use that are part of the labeling, it's the on-package labeling, it's the training program. All of those components are considered the user interface. And so, they should all be evaluated, early on, maybe separately, but then later on as a full package, in order to ensure that you can safely and effectively use your products. Thank you for… for bringing that question up, Angelina. I think that was great.

So, here's a very specific question that was submitted in the chat. It said, I'd love your thoughts on navigating the realism paradox. For example, when testing highly intimate products, a male at home STI swab or home fertility collection kit, in reality, the patient is completely alone in a private state. But in a formative study, we need to observe their physical technique, handling. How do we balance high-fidelity realism with participation modesty? Also, what are the biggest regulatory or IRB considerations regarding data collection, like video recording during these sensitive Simulations. Wow.

That's it.

That's quite a question, and a very relevant question. We get asked a lot for women's health, as well as men's health products. So, I think I'll answer on kind of a high level, and then I would invite, and I don't see who submitted that. because I'm not looking at my chat as, frequently as Sean and Victoria, but, I'll answer on a high level, and then if you would like to set up a quick call at some point. next week, I'm happy to jump in more to talk more about specifics, considering your stage of the product, what it's used for, and what markets you're going to be looking at, and what clinical data you have, and kind of the methodology that you're using for evaluating that.

So, at a high level, we work with a lot of companies that have more intimate products that are more sensitive, and we can do everything from, like, focus groups and interviews to help understand things, to actually usability testing. Just because they use a product on themselves doesn't mean you have to have them use the product on themselves for every formative test. You can, a lot of times, get them to use the product certain ways themselves, but then have a model available for them to show you how they would use it on themselves instead of using it on themselves. Now, there are shortcomings there, obviously, but each study, we would weigh, kind of, what are the goals that… and what we're trying to accomplish with what is possible, and how can we effectively gather this information

While balancing the sensitivities of the research itself. And then, obviously, anything that's more sensitive, definitely IRB review is necessary. We do have an IRB associated with our company that we regularly work with and, are in collaboration with, asking questions about, you know. Are there some considerations here that we need to make sure are in place to… Make participants feel safe and, and comfortable while performing, or while participating in a study.

That's awesome. So we only have a few minutes left, and, I want to make sure, so if we don't get to all the questions today, we'll forward them to the subject matter experts for. for follow-up. Again, I'd like to remind the audience that we are doing a session, next week on human factors for legacy devices, with, Angelina, will be a speaker, as well as, two speakers from Claramed. That's gonna be a great session. I encourage you to join that. I'll drop the link in, again. And the last thing that I want to say is, I don't know how many people have heard about the, cyber attack on… on Stryker, that happened in the last…

36 hours, but it was… it was pretty big. We're gonna be doing a session on, connected devices, next week, and we have incorporated into that, a session on, cyber warfare and preventing cyber attacks against. devices, and I… I think that this is a very timely and important topic, given the… state that we're in. And… let's see… We do not do certificates of attendance for the Friday calls, just for the weekly calls, just to answer the question.

And… Lastly, what do I have to say? If you would like, to speak with anybody from Clarimed, have, a session, please, ask in the chat, just write, contact me, And, you can, you know, work, speak directly to an expert to try to figure out your or issues. What am I forgetting? Hmm.

I mean, I think I was gonna say, we're happy to provide, like, a 30-minute free consultation if you had more specific questions about your product or the strategy, or kind of understanding.

what?

Support you might need.

That's what I meant.

Moving forward, step. And thanks for your participation. It's always great to have people asking questions and providing input as well.

Wonderful. Well, I'd like to thank everybody for their time. Next week, we are going to have, Ivan and Antonella from MedBoard are gonna come on and talk about, regulatory intelligence, regulatory intelligence platform. It's gonna be a lively discussion about post-market surveillance and clinical, and a number of ways, that AI is speeding up the regulatory process. So, I encourage you to join that. There will be, you know, the usual announcement put out. Shortly. Thanks. Thank you, Denise, and thank you…

you can't…

every day in… Thanks.

It was a pleasure. Thank you.

Bye-bye.