The 7 Deadly Sins of Test Method Validation

Hi everyone, welcome. I'm excited about today's call with Simon Foggere, from… Sifo Medical, he's gonna talk to us about, test method validation, the seven deadly sins. Having a little bit of technical difficulty at the moment, so please bear with me.

Hi, Maria, how are you? And also make me co-host.

Working on it.

Maria E. Donawa, MD: Yes, hi.

Happy Friday, everybody.

Friday, yeah?

Sorry. Being a little incompetent this morning. It's too busy. Chatting, where is… Where's the screen?

Go up on your… You can see on the top of your screen. Where's the webinar? On your computer.

Yeah, I seem to have messed something up on my computer. And I don't know what.

Maybe you should start. So…

Okay, while I'm trying to get started, or see my screen, I am going to, ask Simon to tell us a little bit about, his childhood, and, Did he dream of, being a test method validation expert when he grew up, or what did he want to be when he grew up? So then…

Very good question, thanks for that, Sean. And, in fact, I did not dream about… I take the bottom line away, I did not dream about becoming a test method validation expert. I don't know whether I am one. But I dreamed about, becoming a musician. I'm, by heart, a drummer. Drumming or music has been a big part of my life, all my life. And, yeah, this is what I do to… still to this day, and this is also the reason why this very webinar is called The Seven Deadly Sins. I had, one of my early… one of my first bands was a… yeah, I would say from a rather heavy genre, and it was called The Seven Deadly Sins, or Seven Deadly Sins, so that's why I think… thought it's a good idea to also name those,

issues that I'm about to show you regarding test method validation, the seven daily sins of test method validation. So that closes the loop, at least on the name, on the title of this, today's webinar.

That's awesome. And I want to let you know that I found my screen, so we're… we're back in business. I need to promote, Francesco, but while I'm promoting Francesco to co-host, I would like… to, ask you to look at my latest post on LinkedIn, which is for next week's speaker, who is, going to be talking about, what ChatGPT, won't tell you or can't tell you about, Medical device sterilization. And it's a, it's a funny post. There's been a lot of backlash against,

AI-generated content recently, and LinkedIn is changing their policy, so, If you're interested in learning more about that, ask me no… ask me more. In the meantime, do we have any, test method validation experts or people with a lot of experience with it on the call? Please put that in the chat. If, and, Those of you who are new to the call, we, you can be promoted to speaker, or to panelist, Or you can, add questions, Using the Q&A or the chat function, either one. Probably just the chat function is easiest. And,

And even if you're not a, test method validation person, just write in, you know, where you're dialing in from, just so that you can have something fun to do if you're not looking for my post on LinkedIn. So, without any further ado, I'm gonna let Simon get started. We got a lot of people joining.

Awesome. Thank you so much, Sean, and thank you for giving me this opportunity. Hello, and welcome, everybody. This is the 7 Deadly Sins of test method validation. A guide to avoiding TMV-related FDA warning letters. Let me just tell you who's talking to you. My name is Simon Filger, I'm the CEO and founder of Cypher Medical. I'm some 15 years in the medical device industry. I've led some 50 projects in… validation projects around the world, from process validation, test method validation, all sorts of validation. I'm also a trainer to the Tufsuit Academy, where I teach test method validation, process validation, statistics in medical devices, and so on.

And, the most important part, I'm based in the western part of Austria. very close to the Italian border, so that's why I spend a lot of time in, sorry, very close to the Italian border, and I spend a lot of time in Italy, in fact, because it only takes me 20 minutes to a very good pizza place, and my girlfriend is also from Italy, so there's a lot of reasons to go to Italy. And for you guys who joined a little bit later, the title might be confusing, and the title is actually very easy to explain, because I am a musician, all my life, basically, I'm a very proud drummer, and one of the first bands that I've ever played in was called Seven Deadly Sins, and this is why we are talking about the 7 Deadly Sins of test method validation today.

You are free to ask any question anytime. I think Sean is gonna handle that, and I don't want to… yeah, take away the important part. So, like I mentioned already, Cyphomedical, I want to give you a very brief introduction of what Cyphomedical is and what we are doing. We are a consultancy company. focusing on quality management, and specifically on test method validation, equipment qualification, process validation, and statistics and risk analysis. So we try to close the loop whenever it comes to manufacturing, when it comes to equipment qualification, test method, and so on, and the risk management part is very central when it comes to

justification of sample sizes. That's… why we are also focusing on risk management, and we even have developed what we call AIRA. IRA, which is an AI-assisted risk analysis, and this basically does all the heavy lifting for you when it comes to creating your risk analysis, and we have AI do it for you within a couple of hours compared to your normal, I don't know, probably hundreds of hours. We also have online trainings, online and TMV guides. We call them guides because we guide you step-by-step through the execution of a test method specifically for sterile packaging. Yeah, and then we also have a lot of free content available. We have a YouTube channel where we explain multiple different things related to those topics. We have very… a lot of blog posts online, and also on LinkedIn, and on our website, and free resources.

And the fourth column to the very right. It also explains why I spend quite a significant time in Asia every year. We… Take care of suppliers, we develop suppliers, we carry out audits, and we… you can even source components through us, injection-molded components, cables, whatever it may be, we are here to help. And today, we take out, or we zoom in into a very specific topic, test method validation, because test method validation is, in fact, a very critical part. Here you have a bunch of companies that we have worked with over the years. They trust us, and we are still working with them, so I guess they are somewhat satisfied.

And I want to make this session as practical as possible for you. You will get a lot of free stuff during this webinar. I will hopefully give you a lot of information that you can use right on Monday. It's Friday, I understand there is the weekend in between, you wanna… take your hands off any test method that you are working on, but four things I want to give you, which is, I guess, very helpful. So, I try to give you a framework of how to perform or execute test method validations, a practical path to redemption, because we are talking about the sins of test method validation, and I want you to prevent, to basically make the same mistakes.

We do this by the example of real FDA warning letters. And apologies already here, because I still quote the old references from the QSR, because all the warning letters that we have here as an example quote the old references, but basically, the story is still very similar. And then, you also get free templates and a self-assessment checklist. Just, drop something into the chat, and then we will send it within 24 hours, at least that's what we try to do. Good, how the next hour unfolds is very clear, I guess, so we have the introduction, I already explained to you who I am.

I will also explain why test method validation is so important, and why it really matters. Then we will focus on the FDA core expectations. Then, the 7 deadly sins, with, real FDA warning letters. three strategic methods of how to come up with your robust TMV, and towards the end, we will have a short question, or a question and answer session. And you can also get in contact with me through our website or LinkedIn, whatever you prefer. So, who is this webinar actually for? it's a very broad audience, I would say. So, medical device engineer, whether you are an engineer in the development, manufacturing, validation, verification, wherever you are, you are most likely testing a lot, and

whatever the result is that you are creating needs to be valid. So, this is very interesting, or should be interesting for you. Quality managers. Because you most likely, are the ones that are defending those test method validations, and depending on the size of the company, you maybe even have to perform them. So, better… it's good that you know how to perform them. CEOs and MedTech founders, it's important to know what's really required from a manufacturing and also, of course, design control perspective. And, of course, suppliers to the MedTech industry. If you want to become a supplier to big corporations, it's essential that you know what they are expecting from you in order to be interesting for them.

And last but not least, consultants. So, whenever you are a consultant within the manufacturing space, or probably design control space, and you are carrying out design verification tests, or some sort of process validation, it's also very relevant to know how test method validation has to be performed. And then, who is this webinar not for? I guess this is pretty clear, because everyone here has to do with medical devices, or with the regulation, or with the requirements around it, so I guess it's… for us here, it's very relevant for everyone here. And then, I want to start off with 3 questions.

If you can answer those into the chat, that would be amazing. I don't see the chat right now, but I will definitely have a look. while you are, answering those questions. And the first question is a rather simple one, is test method validation a regulatory requirement in MedTech? So, if you can answer this in the chat box, that would be great. Then the, I mean… We will probably get to the answers of those questions along the webinar, but what I definitely want to emphasize here is that test method revision is a requirement. It may be difficult to pinpoint where the actual requirement comes from, but based on the webinar title already, and based on the fact that I brought you a couple of warning letters, this already

guess is very obvious that it is a requirement. Then, the probably more interesting question is, is your management of the same opinion? So, do they really think that this is something that they have to do, or you have to take care of? Because what I see a lot is that management still says, we don't want to do this, this is so much effort, we would rather wait until the auditor finds something, or… yeah, they wait until the bad things happen, basically. That is also why we have prepared a slide deck for you that you can share your management so that they are aware of where this requirement is actually coming from, and what happens when they don't, or when you don't perform test method validation.

And… The also very interesting question is, are your test methods validated? And I don't mean some test methods that you can show whenever an auditor shows up, but I mean every test method that you are carrying out that is making a quality decision. Not the ones only that you produce in-house, but also your suppliers, because suppliers are your responsibility, and those test methods also have to be validated, of course. Good, then… Why is this even an important topic? Well, a lot of the FDA warning letters, we have a number of 25 to 30% of the findings have to do something with, test method validation. And this is not only from a

process control perspective, process validation perspective. This is also in design control. Whenever we test something, and whenever we create, or come up with a quality decision based on a test method, or a visual inspection, or whatever it may be, then this method has to be robust, has to be valid, and has to be validated. And I can tell you from the experience that I see, yeah, the number should even be higher. What happens if we do not validate our test methods? I mean, failures in test method validation are not a minor infraction. In fact, this can lead to real big issues. We have here recalls.

We have market delays whenever… or patient… I mean, this should be the very focus of everything, patient safety, and we want to make sure that neither of those three things happen. So that's why I would highly recommend looking look into your test method validation. Are they robust? Maybe, are they even here? And here is the FDA, or are the requirements from the FDA, and I apologize again, because I still quote the 820.72 and so on, because all the warning letters that we are going to look at later on are basically quoting, those. But we have, 4 here, or rather three, and one is for statistics, but we have A20.72 inspection and measuring test equipment, then we have the A20.75 process validation.

Statistical techniques with 250 and automated, processes, software validation, basically. And we, of course, have very similar in ISO 13485. Good. Now, the main event, the core of this webinar are the seven deadly sins. So, we have the assumption, hindsight, illusion, arbitrariness, subjectivity, idealism, and stasis, and we will look into each of them now. The first, and probably the most obvious, is a missing or incomplete test method validation. So, the issue is actually clear. You don't have a test method validation, or you have probably a calibrated device, and think that this calibration certificate, or whatever it may be, is your test method validation. I have to wake you up, unfortunately, because this is…

not sufficient. So, this, in fact, is… we don't even have a piece of document that we can look at, or we cannot even say what has to be improved, because there is nothing that we can talk about, right? And the way we have always done it this way is probably not something that helps us in this instance. So, and we already have a couple of warning letters here. you did not conduct a validation of the inspection method outlined in Section 612 of the seal inspection procedure according to 820.72. So, here we go, we already have the first warning letter, and it talks about a seal inspection. So now, one or the other might argue, okay, seal inspection, we perform ASTMF88, we perform ASTM

1929, whatever the number is, whatever the ASTM is. you think because you are performing according to a standard, you don't have to validate this test method. And this is exactly wrong. Because if you read ISO 11607-2, section 443, then it's very obvious that it says even those test methods have to be validated. Just because it's a published standard doesn't mean that it's validated in your lab, and you still have to prove it. And, in fact, this is where we can help with our test method validation guides, because they are definite… they are especially focusing on sterile packaging, all those ASTM standards that I just mentioned.

So how… what is the path of redemption here when you have missing or incomplete test method validation? Well, I guess this is also very obvious. You have to come up with an inventory list. You have to understand what are the test methods that you are using along the entire value chain of your manufacturing? All the methods that you are using, maybe even in incoming inspection, all the methods that your supplier is using, the methods that you are using in design verification. You have to come up With all the test methods that create quality decisions. Or that create the basis where you make quality decisions on. And then, basically, come up with an entire list, and then prioritize them, because you… we only have so much resources, so we cannot perform all and every test method validation at the same time. That's why we have to come up with a priority list. And then.

the work has to be done, and then we just work our way through the entire list, although it's super painful, but this is what we have to do. There are some tricks that we can maybe use here. We can… bracket some test methods together so that the number comes down, but… Eventually, we have to perform test method validation. There is no other way around. Sin number two, the sin of hindsight. No predefined acceptance criteria. And when I talk about acceptance criteria, we always have to keep two… at least two acceptance criteria in mind. We have the acceptance criteria for the test itself. So when we talk about the ASTM F88 seal strength test, for instance, then the acceptance criteria for the test itself is, for instance, 1.2 Newtons per 15 millimeter. This is one of the accept…

acceptance criteria that I'm talking about, the acceptance criteria of the test itself. And then we also have the acceptance criteria for the test method validation, which decides whether the test method validation passes or not. So we have always to keep two things in mind. And it's maybe very obvious, or at least it should be very obvious, that we have the acceptance criteria defined. prior to executing the test. I have… I mean, we already have it here. If the result is not what it's supposed to be, then we make the specification that it fits the result. So, but it's actually the other way around. I mean, I'm talking to a lot of experts here, that's something I'm pretty sure I don't have to emphasize, but…

I'm working in China, and I've seen very interesting things everywhere, basically. So, the sin of hindsight, no predefined acceptance criteria. And I know this is very difficult, especially when we talk about visual inspection, for instance. When is a scratch a scratch, and when is a scratch acceptable, and when is it not acceptable? This has to be defined, because when we then… execute test method validation on a visual inspection, for instance, then we have to show that all of our operators are able to make the same judgment on the same samples. And that's why Acceptance criteria have to be clearly defined.

Diamond, may I disagree that it is a sin?

What would you call it, then?

Well, if you start making a product, usually in the beginning, you, from some process, you don't know what the, what the, the pass or fill criteria are, because you still… you say, you know, let's say you have an injection molding for something, and you don't really know up front if it's gonna be where the, the, Let's say it this way, if you want to… you have a part that is made for you, and you want to have a criteria whether or not to accept it or not, you want to have a certain level that you can give the manufacturer to work in. And say, above this level, we have the impression that something is wrong with the process.

But you… a lot of times, you don't know… don't know that up front. Like, I've seen a lot of technical drawings where there are tolerances in there that are ridiculous. Either you go to a much too expensive process, or there are much to lose, and then things won't fit together again, so I don't think that… You always have to have predefined acceptance criteria. You can, you can find them on the way.

Yeah, this is, I have a somewhat of a different opinion here, because.

Oh, we may disagree.

No, no, no, this is fine, and I want to elaborate why I have somewhat of a different opinion here. So, what you are referring to, at least what I understand, is what I would call the process development phase, test method development phase, design, product development phase. This, where you are not sure about what you actually have and what your acceptance criteria are. This is, and I fully… this… I fully agree with you, you have this situation. But this is in a situation where you are still in a development phase. And when I talk about the predefined acceptance criteria is when we actually perform the formal validation, the formal test methodology.

But it also happened during production. Like, we had a device, and it had an embedded computer, and it had to be tested. And we thought that if it took too much current, then it was a failure. But then, all the tests before they were done in a production environment that was 20 centigrades, and then there was a heat wave, and it was about 30 centigrades, and almost all failed. So, the Pavlov reaction was, okay, we haven't had the temperature in, let's raise the amount of current the device is allowed to take. But if you do that, then in a lower temperature, you would pass a product that should fail. So, it's… you… you have predefined acceptance criteria, but during production, you can encounter situations where you have to adjust it a little bit.

Yeah, but I don't adjust them right away. I mean, there is a change control in between, there is root… there is a million…

We need more ideas.

just go ahead and change any acceptance criteria. What I am talking… and I've, again, I really agree with you that the acceptance criteria are sometimes… you have to come up with them along the way, but When I say this statement, this is really referring to the formal validation run, basically.

So, yeah, maybe we can take that conversation offline, and keep moving forward with the… presentation.

Okay. Good. Then, the sin of hindsight, I mean, this is, let me just finish up the sin of hindsight of how to… the path to redemption, how to get out is… have the protocols approved up front. And this is when we really approve the acceptance criteria. The acceptance criteria from the test method validation, but also from the actual test method, okay? Quantify both layers. What is the acceptance criteria for the test itself, and what is the acceptance criteria for the test method validation, okay? And this has to be justified, not guessed, not because this is what we had last time. I want to know why a gauge R&R smaller or equal to 10% is acceptable for you, because this, again, has to be tied to your risk management. Whatever you are using as acceptance criteria has to be justified. Okay, sin number 3, the sin of illusion, confusing calibration with validation. I mentioned this at the beginning already, some companies think whenever they have a

calibration or calibration certificate, that's what they need. Or maybe they have even a somewhat of an IQ of somewhat of a device. Then they think, this is what we need. No, unfortunately not. This is only a part of what you need. You probably have an IQ that you have to perform. It, of course, depends on the size of the equipment, but you have to perform an IQ, and this needs calibration, and whatever it else needs. And then, on top of the IQ, then we sit, we place the test method validation. So here, calibration is not Everything that we need when it comes to, Test method validation and measuring equipment, obviously.

operators documented results as a pass or fail, rather than documenting the measurement result. So this goes towards, they have a test method that is actually a variable test, but they only, note pass or fail, which Yeah, I never recommend variable… I never recommend attribute test methods. Whenever possible, please use variable data, because you can do way more with variable data. You can calculate averages, you can calculate standard deviations, and based on this data, you can even decrease the sample size a little, and I know that some companies are really looking down, looking to get sample sizes down.

So, I think I have to be a little bit quicker, it's only SIN number 3, and it's already 530. So, what is the path to redemption when it comes to calibration? Well, mandate TMVs, okay? Make it an SOP. Have an SOP in place that describes What and when you have to do, when you have to, when you have a test. you have to distinguish, is it an attribute data test, or is it variable data? Is it destructive? Is it non-destructive? How do you treat visual inspections? How do you make sure that the environment of visual inspection is according to standards? Because is there sufficient lighting? This should all be explained in your SOP for test method validation. And then quantify the variability. Make HR&R assessments, assess the repeatability, the reproducibility, make sure that operators are measuring the same across operators, but also within operators. And then, step three, set clear limits. We had this one already.

In the previous step with acceptance criteria, set clear limits depending on what test method validation you are executing, variable or attribute, whatever, and then tie this to your risk management. at this page, at this, point in time, I want to give you the first, goodie, so to speak, just write us deck in the chat box, and we will send you an entire executive summary, why test method validation is required, and what happens if you don't perform that. Some managements are more difficult to convince, but this slide deck should definitely help you. This is in PowerPoint, so that you can fit in your logo, whatever you need, but this should definitely help you to convince the management to, yeah, make better decisions when it comes to test method validation.

Then, the sin of arbitrariness, the statistical vacuum. This is very, very critical and what I see all the time. Everyone uses GHR&R, and everyone and his mother uses GHR&R AIAG standards. Everyone uses smaller or equal to 10%, between 10 and 30%, it might be acceptable, but no one actually knows why they are using those. They just use the AIAG, Automotive Industry Action Group, Reference Manual 4. And they have those values in there, and that's what they say. But how do they justify that this is acceptable for them? It might be a different story when it comes to attribute testing. We want to show why we, or how certain we can be when we test our visual inspections, and why we use certain confidence and reliability levels.

Here is another example from a warning letter. Failure to provide statistically supported evidence when sampling sample sizes were used. without documented rational tie to risk, confidence, and reliability. So this is nothing that I'm pulling out of my fingers. This is something that the FDA really, yeah, they… They have you on your… And so on. So this is something that you really have to take care of. So, how do you do that? Risk-based rational. What does that mean? You have to basically make sure that within your risk management, you can tie, on, for instance, hazard… hazardous situation, harm logic.

you can tie down the confidence and reliability level for three things. For the design, because it can go south in design, for process and usability. And those confidence and reliability levels have to be tied to your risk management in order to come up with a rational for sample size. For, if someone is interested in how to do that, we have had a webinar a month ago. This is still accessible on our website, just go to www.sypho-medical.com and log in to the free internal area, and there is an hour… one and a half hours long webinar on how to do that exactly. And I think that should explain a little bit, at least.

Good. Then, the sin of subjectivity. Unvalidated visual and manual methods. This is what I already mentioned, and this is what I hear a lot of the times. not only related to test methods, but especially also to process validation. So, unvalidated visual methods. When… and I am 100% sure every one of us has some sort of visual inspection along their manufacturing lines. And we have to make sure that those visual inspections are validated. We have to prove that no matter who is assessing the part gets to the same result and can tell a good from a bad part and the other way around. And the read and understand SOP logic is definitely not sufficient here. I've had a very interesting audit very recently, where the auditor asked for the visual acuity of the operators who perform visual inspection. So that means

We not only have to qualify the environment that we have sufficient lighting, we have the correct distance from whatever microscope that we are using, or the bare eye, or whatever, to the object that we are assessing. But we also have to show that the operator who is assessing or performing this test has sufficient eyesight. Imagine you have to distinguish between green and blue, and all of a sudden you figure out your operator cannot even distinguish between those two colors. So that's something that's very, very critical, and also regulated in standards. We have to make sure that visual inspections are validated, and there is an entire book about it on how to take care of this in terms of environmental

Qualification and, operator qualification here. Yeah, validate the inspector, not just the inspection, we have to show that also the training is sufficient. That's why we perform an attribute agreement analysis. This is what basically is a gauge R&R for attribute data, where we have multiple operators assess multiple parts. And, within this batch of parts is good and bad parts, and operators have to show that they are able to find those bad parts, and all of the bad parts, not only some. And, because when we don't find a bad part, what happens? Well, the bad part goes to the market, and obviously there could be harm on the market. So, attribute agreement analysis. Controlled conditions is what I already mentioned regarding the lightning background.

And so on, and the operator itself. And, last but not least, have robust defect kits. Have bad samples ready. Maybe they are even certified, or calibrated, or whatever you need to do. But train the operators on those defect kits. The sin of idealism, neglecting worst-case conditions. I have the, what I call the micrometer story. So, we performed a gauge R&R with a micrometer gauge. And then, we gathered all the data, and then when we plugged in the data into Minitab, we assessed it, then we figured out that one operator was… had lower results than all the others. So, what is the reason here, right? I mean.

What can you possibly… what can possibly go wrong with such a simple device as a micrometer gauge? Well, we had the two operators sit next to each other and perform the test again, so that we can observe them, whatever they do different. So, and one operator, and I can show you this real quick, when you have the micrometer gauge, there is a torque limiter, basically. And you are supposed to use the torque limiter whenever you close the micrometer gauge. However, we had people not use the torque, but, closed the micrometer gauge at the very end, and that's not torque limited. That means you are closing the micrometer gauge way more than all the others. That was also the reason why this operator had,

Lower results than all the others. And what I want to show you here is that you have to think about everything. And worst-case conditions could also mean that you have also your best operator, but also the one that just finished their training, and make sure that they are as good as possible. as the other one. We have to take those into consideration as well. And here is the finding. The test method validation does not support the method is capable of producing repeatable and reproducible parts. This is exactly what the R&R, engage RNR, stands for. Repeatable and reproducible. So that means It doesn't really matter who's measuring, we get the same results, and we get a result that we can rely on.

Good. how can you… what is the path to redemption here? Validate against your real factory and not the lad that you wish you had. So really use your parts and use your people to perform those tests. We can have a worst-case matrix, we can test at limits, and we can also bracket together families. This is what I mentioned at the very beginning already, when you have your inventory list, you don't have to really do Every little test, there are maybe options to bracket those together. Good. The sin of stasis, failing to revalidate after change. This is something that, I mean, should also, again, be very obvious, but treating validation as a one-time checkbox activity doesn't really work. A TMV done in 2021

And then there is multiple of changes. We have new equipment, we have new material, we have entire new procedure. Whatever has changed, we have to make sure that the test method remains valid. We have to assess the change that we are about to make on how it affects the test method validation. And then we eventually have to revalidate the test. So make sure that your change procedure also take test methods into consideration. Yeah, here, the FDA warning letter, failure to review and evaluate process changes and deviations and perform revalidation when changes or process deviations occur under A20.75. And here are multiple, warning letters that we found with, this… with the same issue.

Okay, The path to redemption, again, integrate into your change control procedure. Make sure that whenever you have a change request or a change that you are executing, that you assess the impact of that change on your test method. Risk-based evaluation, what is the thing that changed? And maybe even use periodic review if this is something that you want to do. So… the three things that I think are very, very important when it comes to test method validation. First. most important, build a culture of statistical rigor. Everything in medical devices. We have We test a lot, we have justifications, but whenever you ask for a justification, why you come up with such and such acceptance criteria or sample sizes, this is where

most of the answers become very thin. So make sure that you can really drill down to the very justification for your statistical decisions. Validate the system, not just the box. That's what I mean when I say we have to make sure that, for instance, in visual inspections, we make sure that the operator is qualified, has sufficient eyesight, have them perform visual acuity tests, make sure we have sufficient lighting when we perform visual inspections, make sure that we have clear acceptance criteria, so there is not only one device that is performing the test, it's an entire system with all its surroundings and all the people involved. And then, third, treat TMV as a life cycle. Things change all the time, it's fine, we just need to make sure that we assess it accordingly.

And this is the second free giveaway that I want to share with you. Just also type checklist into the chat box, and we will send you an entire TMV checklist where you can assess your own test method validations, whether they are compliant. To, yeah, the regulations, basically. Good. Here are some client feedbacks, based on what Cyphomedical is normally doing. I told you already, we do a lot of supply chain, related tasks, supplier development, sourcing, and so on, and this is, a client of us, of ours, where we perform audits for them. another client who used our test method validation guides, TMB guides, and they were then able to perform all seal-related tests, be it the seal strength, be it the dye penetration and whatnot, visual inspection, and so on, and they used our test method validation guides

Where I teach them step-by-step what they have to do. They get all the templates. In Word file, they just basically have to fill out the red fields, what I call them, and there is a video explanation to every step that they need to do, how to plug it in into MinitHub, how to do the analysis, etc, etc. And, for Seniors Cabi, they called me one day, last… two years ago, and they wanted to have an on-site training, which was very nice, because, I got to fly to the Caribbean, and got to play a little bit of drums there in the Caribbean, while I was talking about test method validation, and I guess, there are worth

things than playing drums in the Caribbean and talking about test method validation. Good. That having said, thank you so much for your attention. Whenever you have a question, now is the time. If you're not brave enough to raise it here, reach out to me on www.sypho-medical.com, email, LinkedIn, YouTube, our blog posts, whatever it is, I'm always here to help, and yeah, that's it from my side. Thank you so much.

Thank you, Simon, for a wonderful presentation. I, for one, really appreciated all the interactive moments that you created in the event, and the, and how well you prepared. For this. I'm sure the… the other people would appreciate it, as well. Do we have questions, from the audience at all? Is there anybody that would like to come off mute and, Let's see, while I sort my way through, is there… Someone asks, is there also scrutiny on the validation method for the test methods? Not sure what that means.

There's one hand up, I think… yeah, already unmute. Bohammer, I think you only have to speak.

Oh, sorry, I was looking at Lurk, who also, seems to have a hand up. Just some… Yeah, nuances, perhaps, in… in… sorry. with, we… when we're looking at the ISO 17025 about, laboratory acceptance, they seem to have… to talk about… they seem to talk about the… criteria for using a standard method in different terms. They have verification as one term that you need always to do to verify your implementation of a test method, and then there's validation, which seems to be applying more to

doing a new method. So it's also… validation has this aspect of you are now checking that this procedure is actually capable of doing what you want it to do. And you don't necessarily need to do that part when you are verifying that you have implemented a test method correctly. In my own QMS, I have, I just call everything validation test to keep. To keep our engineers, with a little simpler language, but, but, yeah, just…

You, you have?

We have that difference in…

You are correct, and this is something that I was not very precise about, I fully agree. It basically comes down to when we think about what validation actually means, right? The most obvious thing is when we talk about design validation, where we try to prove that the design that we came up with serves the user needs that we defined. And here, we also want to make sure that the test method is capable of, for instance, measuring seal strength. And when we follow an ASTM F88, the actual validation part has been performed by the ASTM standard itself, so we don't have to validate it, we don't have to show that it meets the intended purpose of the test method. But that doesn't mean that we don't have to do anything. But we still have to make sure that we correct it

implement, and that we, and that, forever, for instance, whenever we do all the calculations, the first 10%, the last 10%, and whatever, that this is correct, and that all, everybody, who performs the validation, or, sorry, performs the test gets more or less the same result. And that includes sample preparation. And this is where I see a lot of the issues happen. So when you really follow the ASDM every single time, then there will be no differences, but… try to cut a sample with 50mm width if you don't have a proper sample cutter, because I've seen companies, they cut the samples, they take a ruler, make the first cut, and then they measure 15mm to the other side and make the second cut. And if you have multiple people do that, there will be a lot of variation. And this is something that you definitely find out if you perform the verification. This is technically no test method validation.

But it's a verification step, and this is what Volkhammer tried to distinguish, and he's absolutely right. I was a bit dirty on that one, I'm sorry for it.

Okay, my turn.

Yes? Go ahead.

Okay, Simon, despite the little thing that I disagree on, I think this was an excellent presentation. We're currently in the process of validating all our test methods, but I'm a little bit scared of one thing, that is that, like, for one process, we need to define a mean and a standard deviation, but we have a limited sample size, and I'm also afraid that even if it's something that occurs one in a million, that that one in a million occurs in the 100 test samples we have. So, it will… might be an outlier, but it will stretch our mean or our standard deviation hugely, and we don't have the big pockets of a big company to increase the sample sizes. So, what would be your advice on these kind of things? Because we have to establish a set point for our test method.

Yeah. That's a very interesting topic, and I get this question a lot, because everyone obviously doesn't have enough samples, for whatever reason. I just, very recently, and that means 2 weeks ago, wrote a blog post about, I think, 5 different strategies of how to deal with this problem. Just to name a few. You can have worst case, you have some sort of worst case where you test way higher loads, for instance, or you have even finite element analysis done prior to testing, so that where you can at least figure out a little bit of a ballpark. There are a couple of other methods, and I wrote a blog post about it. I'm happy to send you the link where I go deeper on how to solve this issue.

Okay, well, in this case, it's a destructive test, so you have your samples and you throw them away after testing.

S… Yeah, that… I hope… what do you see now? Sorry?

I see a complete.

Sorry, sorry, I am a little bit confused now. How can I stop sharing?

No problem, we can interact in this way also.

Okay, now, I was about to show you the methods, but I think it's a bit difficult to pull it up. And nevertheless, I will get in contact with you, or can you send me an email, and then we can have a chat about it, and I think I can definitely help you.

I'll reach out on LinkedIn, if that's okay.

Yeah, that's fine.

Okay, so I told people to just write blog in the chat, and that you'll send them the link to the blog. And then you two can, discuss at length, I encourage you to get in touch. I also encourage people to follow Simon, and to, vote for him, or nominate him as, For the MLV100 at mlvx.ai. slash MLV100, which I'll talk about in a minute. But in the meantime, there were some more questions. There were, some questions in the chat. Chris, I don't know if you want to come off mute, to say anything or not. I know that you're in a related business, but, and you might have some questions or comments.

And then, going through the… Q&A… There was an interesting question, what if there is no material change? Do you still need to validate the test, e.g. a lab-developed test? If I don't.

So, that means you have… you're following the AST… I take the example of the ASTM… you're following the ASTM, you make sure that you follow it exactly, because that's also something that I have seen a lot. People think they are following the ASTM, and they are actually not. So that's why we also have an assessment form, especially for all those different ASTM-related packaging-related test methods, where you can assess step-by-step if you are really following the standard. So that's the first thing. And if there is no material change from the ASTM to what you are performing, then you still have to perform the verification test that we previously talked. You have to show that it works in your lab. That means you take a couple of operators, you take a couple of samples, and run it multiple times and show that the repeatability and reproducibility is acceptable, just like it's supposed to be.

Fantastic.

Hey, Sean, just to answer your question, I know when I was doing… I was leading system testing and performance testing for a huge system. for, a very large medical device company I work for. I don't like naming companies, because it always comes back to haunt me, but… what I liked was they had quality systems set up to do all the testing, that process, like HP Quality Center, stuff like that. But every test that we ran. Especially for a system test. every piece of equipment, or even if it was a virtual side, I had to go in and document everything associated, like the pump, serial number, et cetera, et cetera. So everything that I was testing, I had to document exactly what was being tested, and the state

So here's where we're starting, here's the state of everything, it's all running, etc, per the instructions, right, per the regulations, and then we would run the test. And my worry was, if it passed. Because I always ran it twice, and then I ran it again, because something would hiccup along the way, and then an engineer would go in and tweak something, say, oh, okay, now it should pass. and then run it, and we'll be done. I was like. No, I have to know what you did, I have to document it, add it back to the test to make sure it's been accounted for. Because the last thing… I don't want to be the guy that has to sit in the room when the FDA comes in and audits.

Because my name is on that test, and they're gonna say, why did you not… why did you have, you know, put this off, or neglect, that was the word I heard, why did you neglect to update all your… your test parameters, and like he's saying, the test method. had to be validated, and it was very clear on verification and on validation. Like, if we did intended use validation on the tools we used, same thing. Engineers always want to tweak something. Oh, okay, now you're going to get the numbers you like. Let's just run it again. It's like. no, now that you did that, I have to go rewrite that part of the test. And they just looked at me like I was crazy, but I never had anything come back on me, so I… Simon, I'm 100% behind you. I just… I can't answer towards other non-software testing

But I can't imagine it's any different. I don't know if that helps, but…

No, the logic is the same, right? And this is what, why… equipment qualification, or IQ, the IQ part is so important, just it is important for process validation, or in general, it is also especially important when we talk about test method validation. So, we only start validating, and I'm talking about the formula. I don't say that you cannot touch your test equipment before you have a TMV done, that's not what I'm saying. I'm saying is a formal testing, before we start formal testing, we have to have an IQ in place and a TMV in place. And I even say all the time that whenever we talk about… and actually, you always have to talk

TMV and process validation together, because all the data that you are generating in your process validation is data that you want to have to rely on, or you have to rely on, and you want to trust the data, so better perform maybe the IQ, and then TMV, and then OQ, because the data that you're generating in your OQ already has to be valid, too.

And here's one more reason to do all that. you… let's say you build this medical device, or a bicycle, or anything, and it's shipped, and then the customer comes back and says, I have a problem, right? Wouldn't it be nice to know you documented everything up front, so you can follow the breadcrumbs and go right back to whatever you missed, or whatever test was not quite didn't have quite the coverage you wanted, so that you can go back and say, okay, that test missed this, now we can fix it, and you can be done with that problem. That can be a low-level Kappa, come and go, everyone's happy. But if you didn't, and they have to wait weeks, months, because you have to do all this research, because it wasn't documented, what an expensive nightmare. So, Simon, I'm 100% behind you.

We used to call it our no-surprise policy.

I like that, the no surprise policy. So here's a question. What is the extent of the revalidation after changes? Do you guide yourself based on an impact analysis, or do you validate the entire test method? I'm not sure if that…

this is not one solution fits all. You always have to assess what was the actual change, right? And… In the very… worst case, so to speak, I mean, then run the entire TMV again. We are not talking about week… most of the time, we're not talking about weeks here, right? It's a… Maybe one or two days exercise, I don't know how long it takes, but if you are in doubt, then, okay, perform the entire TMV, unless you can really pinpoint what has changed, and you can maybe assess this very change, and how it affects the test method. This is based on assessment. I cannot give one answer that fits all, but this is the idea behind it.

So there are a lot of us on the call, like me, who are… not really intimately familiar with the subject like Luke and Chris are. And there are also, potentially, consultants or, startup owners or other, people working in regulatory that have a glancing familiarity with This, requirement, and they know it's a requirement, you know. Bottom line this for us, you know? How much time, how much money, how… what… what happens if we don't do it, you know?

So there is always Murphy's Law, right? what can go wrong will go wrong, and most of our clients, they say an auditor has never raised that question. And I say, you are the luckiest person in the room, because they haven't yet raised this question. And this is what you have to keep in mind. The very next auditor, or the very next question from the same auditor, can already ask… can be this very question that you don't want to hear. But what you definitely want to be is you want to be prepared. Because if you're not prepared, then this is the situation that you find yourself in, and you don't want to be in this situation. So…

as everything, no one has to do a process validation if no one would ever ask. And people behave like test method validation is a new thing, or even process validation is a new thing. I was born in 87, 1987. This is when the process validation first became a requirement, so this requirement is as old as I am. This is not an MDR thing, this is not a QMSR, nothing with all that. It's… Been there for a very, very long time, and… I'm quite surprised how patient investigators and auditors are with still accepting companies not having process validation in place, not having test method, not having any of this in place. So…

What I would recommend is get it done. and make sure that you, the painful part is if you don't know what you do, right? But as soon as you know how to perform it, then it becomes easy, and you just have to do it, so… go online at Cyphermedical… cipher-medical.com, purchase that test method validation training, and you are ready to execute all your TMVs. This is what I say.

Thank you so much. I really appreciated everything that you shared today. And your time, and the audience for sharing their time with us. I think this was a fantastic presentation with a tremendous amount of take-home value, I hope that everybody will connect with Simon on LinkedIn, follow him, and his posts, and his company, and his blog, and all of that sort of thing. And I encourage you to reach out to me with any questions, the, reminder, we're entering the final week for the MLV100, voting nominations. You can nominate anybody you want. To nominate Simon, you just, copy his LinkedIn URL and, drop it into, this, link right here, which is ML…

VX.ai, sorry, I didn't have this queued up, slash, MLB100. I think that should do it each… TTPS… Slut.

I forget, vote Irish.

See if that link works. Yep, so what you see if you click on the link that I just put in the chat is cast your vote. It's gonna show, the… Top people, then there's a section to vote or nominate, and then you're gonna see recently nominated by the community, at the bottom, so the people who are leading in the number of new nominations. you know, still plenty of time to, get in there and get counted. I encourage you to do that and to share it, with… with others. Again, looking forward to next week's topic on… Medical device sterilization. Thank you all, and have a great holiday weekend.

Thanks, Sean. Appreciate it. Nice job, Simon.

The blog is already in the chat box, and also follow us on YouTube. Thank you so much, and have a great weekend.

Thank you.

Nope.

Bye, all.