How to Become a Lead Auditor for Software as a Medical Device (SaMD) — Masterclass with Rob Packard

Hello, everyone, and welcome to today's webinar. My name is Sean Smith from MedTech Leading Voice. Today's webinar is How to Become a Lead Auditor for Software as a Medical Device, a masterclass with Rob Packard. president of the Medical Device Academy. I want to, begin with a few housekeeping announcements while we're getting set up. Hey, Rob, are the slides gonna be available?

Yes, I can make the slides available.

Okay, send me the slide deck, and I will, send that in the link, to the attendees and the non-attendees tomorrow. Certificate of attendance will be available upon request. Please write your name and email… your name that you would like to appear on the certificate, and your email address in the chat, and we'll make sure that you get a certificate of attendance for today's call if you're here. There will be a brief poll during the call, and a post-webinar survey. Please share your feedback. It's important to us. We're also going to ask you to participate a few times during this event by dropping things into the chat. You can start by dropping in where you're dialing in from, love to know. Please use the Q&A function to ask speaker questions.

And please, use the chat function to ask me questions, like, will this be recorded? Yes, this will be recorded, the recording will be made available to everyone tomorrow, and in a few days, it will be on YouTube. While people are still joining, I have a couple… of announcements. Those of you who missed the MedTech Leading Voice Updates, 15 super useful MedTech questions and answers. Today, I posted it on LinkedIn, and I'm gonna drop that into the chat so that you can show some love. Please show some love. Then, this week, tomorrow… MDR and IVDR transition. This is a webinar with 3 notified bodies talking about, guess what? The MDR revision that just dropped yesterday from the European Commission. This is a fantastic webinar and your opportunity

to hear directly, from subject matter experts. And then on Friday, honing your med tech message, how to be memorable in 2 minutes. with, Tino Chao from Giant Shoulders. This is perfect for anybody who, is less than memorable. That's it for my announcements. Rob, I'm gonna turn it over to you. Let's go ahead and get this party started.

Okay, thank you for having me. You had a little bit of an agenda, that you had suggested talking about software as a medical device and how to audit it, but I thought maybe the best place to start was, how do you become an auditor? What do you think?

Absolutely. So… Yeah, that's great.

For people that haven't met me, I worked for a company called BSI. They're one of the largest medical device auditing organizations. And in order to work for them as an auditor, I had to first be an auditor for 5 years minimum at a medical device company. And I was a lead auditor, I trained other auditors, I ran the entire quality system at 3 different companies. And then BSI hired me. And that's when my training began. I thought I was, like, I'm an auditor, like, yeah, you're a newbie. So, I had to go through 20 audits. being observed by somebody else before they'd let me even work on my own. And then I had to be acting lead auditor for another 15 audits.

doing the opening meetings, doing the closing meetings, scheduling the audits, the agenda, the report, everything was my responsibility, but somebody looked over my shoulder and critiqued everything I did for 15 more audits. So 9 months later, I was finally an official lead auditor for BSI. And… They start you out day one, they're like, yeah, none of the training that you've had is through an accredited program, so we're gonna have to send you back to school to learn how to audit from scratch. So I'm sitting there in the class. And the person that's teaching it, he's never actually done the auditing before. So it's a BSI class to be a BSI auditor, and the BSI instructor has never actually been an auditor for a notified body, so he was saying all kinds of things that were wrong, so I kind of

took over parts of the class. Stood up in the front of the room, like, no, this is how it really works, this is the forms we use. So the following week, I became an instructor. So that was how I started, but it… you have to… you have to actually be trained in a formal class to be an auditor if you're doing professional auditing. But if you're… if you're working for a company auditing suppliers or doing internal audits, you can take an online course. And you have to have somebody else, after you finish the course, watch what you're doing and give you feedback so they can correct the things that you do wrong and help you improve.

And just like driving a car, this is the analogy I always like to do, like. you know, I can't watch you once, and now you're okay to drive on your own, but we're gonna have to do a few trips around the block before I'm even letting you get out on the main road. So, that's how it is with auditing. You gotta start out small in practice and make mistakes. And one of the cool things about today, not today as in December 17th, but this generation, we now have Zoom, we have Telecommunications and webcams for everything on all kinds of devices, so you can record everything, and everybody's comfortable with recording now.

before, it was like, you want to record me? Forget that now. I'm like, yeah, go ahead, transcribe it, record it, save it. So now you can actually take a recording of your opening meetings, your closing meetings, your interviews of people, and you can share it with an expert remotely, and they can say, oh. This is… this is what you're doing, Try this instead. And you can watch your own self, and you're like, you think you interview really well, but then you cringe when you have to watch yourself, and you're like, oh, I didn't realize I do that. So you learn so much by watching your own self and having somebody else that's an expert critique you. That's how you become a competent auditor. So you're an auditor when you take the class and you pass the exam.

But you're an auditor in training. You're allowed to get behind the wheel, but not without somebody tagging along in the shotgun seat, and you need somebody to critique you and get you better, and that's how it's done. It's watch the other person, but it's so much easier with a video to watch. Because you can't really listen to the person live answering the questions, and watch the auditor asking the questions and taking notes, and think about, are they doing it correctly, and what can I help them improve on? And you can't really stop the whole audit and say, time out, I gotta give him some advice.

now we can do it offline, and we say, okay, let's watch the video. It's like… like college, college teams watching the game tapes before they go practice against… or compete against the competitor. They can watch those game tapes and say, oh, that's what they do. That's why they're always… get… get those fourth downs in. I need to do this. So, it really helps to watch your videos. So, the course, you can just take an online course, take an exam, that gets you your certificate. Now you're an auditor, but you're an auditor in training. You still need somebody to watch you and help you improve. And most companies say, well, you're gonna have 3 observed audits, and then you're all good on your own.

It took me something like… 40, 50 audits to get to the level where I felt like I was good. I've had over 1,000, so now I'm an expert. Somewhere between 50 and 1,000 is when you're an expert. I don't know where that is, it's probably different for everybody, but Most people out there do one a year, and they're like. Why do I hate auditing so much? Well, because you've done it once a year. It's like your taxes. Taxes are really painful when you have to start from scratch every year and read the book, and it changes every year. If you audit every single week, you become a master. So somewhere in between those two is where you want to be.

Okay. So, had a question. To audit for QMSR, do auditors need to be trained and certified? And so…

You have to be trained in anything. So, you want to write up a Kappa? Where's your training? So that… that goes without saying. Certified is different. So, if you take our course, and most courses for that matter. They will normally have a certificate at the end. There is another type of certificate called an accredited certificate. And that's only available from serv… From courses that have been accredited. So you pay about a quarter million dollars to have your course material accredited by an accreditation body, and then you can't change it without paying another quarter million. So people wonder why it doesn't get better?

there's a reason. If they change it too much, it doesn't meet the accreditation requirements anymore, they have to resubmit. So they make little edits, but they don't overhaul the course. Whereas when you take a course like ours, we can… we don't have it accredited, so we can make changes all the time. So I literally… this is the fifth time I've created this content, and we completely redid the whole entire slide deck. We had new people record it, added in new stuff, added in new standards. And we can keep one. updating it, so we actually offer the updates for free. Other courses, you take it now, you want to get an update 3 years from now, pay for it all over again, take the whole course again, and you don't get to keep any of it. I mean, you might get your handouts.

But that's about it. Our course, you keep all the videos, you keep all the quizzes, you have a place to store your training records, and you get all the free updates, and it's forever.

So, would this course be suitable for somebody who's never done an internal audit?

Yes, we specifically are training it at the level, like, you've never done this before. So what is an audit? What is software? What is a medical device? We even have things in there that people don't think about as, like. you're gonna go to a supplier, where should you make your hotel arrangement? Where… what flight should you take? That… we've got that stuff in there. What, like, what… how to do basic logistics, because there are people out there that don't travel. And all of a sudden, you're telling them, not only do I want you to audit in a place you've never been, but I want you to travel, and you don't travel.

Some people…

haven't left Vermont in their whole entire lives. Me, I've moved over 50 times. Literally moved from a home to a new home, permanently, over 50 times And I've traveled all over the world, so you mean me? Just… Blindfold me and drop me out of a plane, and I'll be fine!

So what about the people that are between 40 and 1,000? Audit, honeycomb.

That are between 40 and 1,000? The people that have done a hand… like, 2 handfuls. If you've done 10 audits in your life, and those could be 10 single process audits. You get almost the same value of doing a single process audit as you do doing a whole blown audit. So when you're a trainee, I recommend Doing single process audits, because there's less for you to think about. There's fewer clauses to worry about, there's fewer people to interview, it's shorter. So, you get to practice all the mechanisms of the audit. You get to practice an agenda for each single process audit. You practice an opening meeting for each one, a closing meeting. You practice your note-taking, your interviewing skills, and your report writing for each individual process.

So all the steps that you do, you practice no matter whether it's 1 process or 20 processes, so practice small. And get feedback on it, and then practice another one small. And once you get through 10 of them, once you've done 10, now you're at the point where you should be able to do the opening and closing meeting with a checklist in front of you, so you don't forget things. But you should be able to go right through the checklist comfortably in an opening and closing. You should be able to, by now, have an agenda that's pretty solid, and you don't have to make too many changes to it. And then, last but not least, you should have

Enough familiarity with the report that you can write a report where it doesn't take you as long as it took your first term paper.

You know, that…

Matthew… Matthew and I do audits together all the time, and we're kind of in a race, can we finish the report before we have the closing? Whereas some audits. it's like a month afterwards you're looking at the audit report. And people want their audit report as quickly as they can, so they can take action on it. So, getting that done as close to the audit closing meeting as you can is a real goal for every auditor. And you can't do that if you're learning how to write a report. And what's in an audit report? It's not like other… it's not like how to write an email. You have to show what you covered, what you looked at, what you looked for, what was conforming, what wasn't.

you have to document all that in the report, and there's a template, usually, that you're not familiar with. So until you've done about 10 of these. You're not gonna be really good at it.

So, are we gonna see slides today? Are we going…

I do have some slides, This… this slide deck here, it's… I have many of them that I can use, but the one I'm gonna show you… was designed specifically for software as a medical device, because what I end up with, more than half of our clients They have an introductory call with us, and they're like, we're a software developer. And we don't know anything about medical. But we've developed a medical device. It's software as a medical device. And we want you to help us get on the market. So, I have to explain to them what's required for software, and I have to tell them what's required for a quality system, and I have to explain what the FDA process is. All three.

And so, a lot of what I would explain to somebody about auditing is in this slide deck. So, one of the things that I have at the beginning for people, is… Just what is the definition of software as a medical device? or what is a… not what is the definition of software as a medical device, but what is the difference between software as a medical device and software? Software people say, the software needs to work. I don't care about the documentation, I just need it to work. In that they actually say that in Agile. The Agile manifesto actually says we emphasize working software over documentation.

Whereas, when you're dealing with software as a medical device, you can have it work as well as anybody can make it work, but if you don't have the documentation, I can't even submit it to the FDA. They have the form validated, so if I don't have something attached at every attachment point. It won't even upload. It'll say, sorry, there's an error, it isn't complete. So, try again. So I need the documentation. They're like, well, we put it all in one document. I'm like, that's great, but I need to put it in 10 different places, so how would you like me to cut it up? And they didn't, of course, didn't organize it that way. I'm like, well, you have to organize it that way.

So that's just one little piece. So I say to people, if you're developing software, it needs to work, but you also need documentation to prove it works. So, one of the… one of the things I've been trying to teach people is, in your Scrum team meetings, so a lot of people are using Scrum teams. Your backlog should include the documentation. You do not have to review and approve it, and bless it and make it holy. You just need to have it as a draft. Just like the software. The software, you iterate multiple times, and we keep on improving it. We have a minimum viable product, and we finally have a final version. We deliver the customer.

Same thing for the documentation. You can iterate many, many times. and deliver those pieces as you go and edit them. They don't have to be final, approved versions until you submit to the FDA. And even the FDA may make you make some changes, like labeling. So that's one of the things I try to teach them.

So here's a question. Can you speak on auditing software as medical devices that don't reference any standards for cybersecurity? I've seen software as medical device developers claiming compliance only to ISO 1345, ISO 14971, and IEC 62304.

So you can have software That doesn't claim, compliance with any standard. So, I don't even have to claim compliance with 62304. The FDA actually has a place, it's actually in the software documentation section. It's either, like, the fifth or sixth requirement for add attachment. And I can actually show you where that is. There we go. It's right here, here's a screen capture of it. Attachment number 7. You have two choices. You either attach a certification statement saying you comply with 62304, or you attach all your procedures explaining how you meet all the requirements that are in 62304.

And so you don't have to comply with 62304. You can invent a new way to document software. Go ahead. But it's gonna meet all the requirements that are in 62304, whether you comply with it or not. So, good luck! Why would you bother creating a new method when you can just copy somebody else's? So, and same thing goes for the cybersecurity. There's another section of 12 requirements for cybersecurity. And that's on the next slide. And you don't have to comply with TIR57, you don't have to comply with ISO 27001, you don't have to comply with any of the cybersecurity things, except for the FDA guidance, if you're going to go to the FDA.

Somebody put in there 82304-1. So, none of these… all those standards, they're voluntary standards. You can always come up with your own version, but it's gonna have to have the same Things addressed somewhere in your plan, and they're going to require your procedures to be presented, and they're going to review them, and they're going to want training records, and they just put you through the wringer because you decide to do it different. So, not a good strategy, but it's a viable strategy. I had a company yesterday that software as a medical device, and they said, I don't see any value whatsoever in doing the pen testing on our product. It's just testing for the sake of testing. It will absolutely learn nothing.

And my response was. From pen testing, we have seen zero companies, zero, a total of zero, come up with no vulnerabilities. All of them have vulnerabilities. Jose? he has… it's, like, less than 1% of companies get through. I interviewed another pen testing company earlier this week, on Monday. Less than 1%. The less than 1% is with a little asterisk. They wouldn't let us do our normal job. They restricted us. They wouldn't let us take apart the medical device if it was physical, if it was software, they wouldn't let us test it in the real environment. They restricted how we attacked it. So, like, as long as you wear these handcuffs and have both hands are behind your back and you're blindfolded.

You can do the pen testing. So, it's not… it's not reasonable to expect your software is secure. Everybody's software is… it definitely has vulnerabilities, definitely will have flaws in the pen test. You will have things you will have to correct, and don't even bother arguing with the pen testing's grading of your findings, because the FDA would be tougher. Every single one, the people are like, I disagree, they said it, this is a critical finding, we think it's not, I'm like, and the FDA will think it's more than critical, so why are you wasting your breath? Just fix it.

Interesting. I'm gonna go ahead and share the poll while we're at it. If I may, you can just keep talking.

I actually had a question in here about, what were the differences in, QMSR? So, what did you have to change to go from QSR to QMSR? And I thought of a fairly concise way to answer that. I heard earlier this week, This consultant we hired, he said we have to change everything in our quality system. I'm like, no, you don't have to change everything. There's actually very little you have to change, because the FDA wrote 1345 at the same time they wrote the QSR. And when they revised it in 2016, Kim Troutman actually made it closer to the QS… or made the QSR closer to 1345, so they would be prepared for this eventual

migration over to 1345. So, it was planned, and it's purposely very similar. But here are the differences. There are 5 big differences, and then I will list 5 minor points under one of the items. So, the big differences are, number one. We have the… actually, it's four. So, number one, we have… Manual. The manual will change. Number two, you need post-market surveillance. There's no requirement in the QSR for post-market surveillance. they have requirements if you are a PMA Class 3 device, but they don't have requirements for Class 1 and Class 2. That will no longer be true. Data analysis. They have statistical techniques in the QSR, they have data analysis in the QMSR.

they could be very similar, but they don't mean exactly the same thing. So, there are more requirements in data analysis than there are in statistical techniques. And then the last one is Kappa was corrective and preventive action, now you have two separate clauses, one that's corrective and one that's preventive. So those are the big differences. Within the manual. you're going to need a pro… you… first of all, you need a manual. You didn't have to have a manual under the QSR. Now you do. You need a process interaction diagram in that manual. You need to identify the hierarchy of your quality system documentation, which could be a pyramid with the manual and policy at the top, and then procedures, and then records.

You need to identify what's applicable and what's excluded. There's only one thing you're allowed to exclude. You need documented procedures or reference to them in the manual. The best practice is to actually have this information is found, or these requirements are met by this procedure, and you put the document number in there. What a lot of people do is they copy and paste in the standard. That's a waste of words. Just say it's in the procedure. If there isn't a procedure for it, then put it in the manual. But if it's in the procedure, you don't have to do it twice. So that's a waste of words that a lot of people do. And then the last thing is, you're gonna have some significant changes to definitions.

because… the definitions that are in the QSR are not the same definitions that are in 1345, and in fact. we're actually now following the 9000 vocabulary standard for ISO 9000 instead of following 9000… or instead of following the QSR. So, we do have quite a few different definitions. Among those definitions that are kind of important. DHF is no longer a thing. It goes away. There is no more DHF. Instead, you have Clause 7.3.10 that is the history of all your records of the design process. And so it's the same thing, just we don't have that name or acronym anymore. I give it a decade or two before anybody actually changes and calls it something different, but…

That's where we're at. And then the other one is the DMR. The DMR, or Device Master Record. That is the same thing as a technical file. Which is the same thing as Clause 4.2.3, which is Medical Device File. So that's your current version of the product, with any changes you've made since you closed the design project. So those are the key differences. It's a manual, PMS, data analysis, preventive action, and the manual has specific changes. And of course, everybody will need to be trained on that new QMSR as an auditor, as well as everybody in a company.

There are a number of questions in the Q&A if you want to look at them, or if you want to just keep going, it's up to you.

I'd love to look at them. Questions are always better than whenever I come up on my own. I got them open here. So, what is pen testing? So, that's an abbreviation for, or nickname for penetration testing. We have red teams that are attacking the software. And we have blue teams that are defending. And the penetration testing is you hire, Some hackers that are… that are ethical hackers to hack into your software on purpose and then write a report about how they did it and what they found. And any way that they found to hack into your software, they give you a report and say, this is a vulnerability, this is a vulnerability, this is a vulnerability. And if people don't think they can hack into your software, you're sadly mistaken. They can hack into firmware. All you have to do is take off your firmware, put on their own firmware.

People do that. And that's why they want you to actually look at software in a medical device just as much, to actually… the FDA says, well, if you have a port, if you have any kind of a software port at all, a USB port or anything that can connect to the outside world, you have to provide all this testing. But a good cybersecurity firm will actually take the unit apart and look for, can I… can I pull off an EEPROM and put on my own and make it work? And when they do that, they can modify your code, they can hijack your code, they can steal patient-protected information. And… they're gonna not stop there. They're gonna dig into, you were supposed to anonymize all the data, did you? Nine times out of 10, what we see is the company didn't display the patient protected information.

they just… But they didn't also… they also didn't truly anonymize it. It's still there, it's just not displaying on the screen, so when the hacker gets in, and they hijack your system, and they start downloading data, they discover all these things… all these things you said you did that you didn't do right. You were supposed to close this port, you left it open. So they'll test every single port. They'll actually… they actually sometimes will connect to a hospital to simulate what it would be like if you were connected to a hospital, and see if they can access the hospital system from your software. So that's another thing they'll check, and they're successful 100% of the time in finding something wrong that they can hijack on your system.

So that's what pen testing is.

And, it isn't all that expensive. People think it's, like, this huge cost. Of all the software development you do, it's the cheapest thing you're gonna do. It's actually the cheapest. The most expensive quote I've seen was around $30,000. I saw one this week for $4,000. I'm not saying everything's gonna be that price, but… I've seen other things, like usability, that's $400,000. So it's not the most expensive thing you're gonna do, it's one of the cheapest. I can even do biocompatibility testing for that. Other things we had here for exclusively SAMD products, is there a list of countries where I can leverage the U.S. FDA approvals? That would be a really cool list to create.

and have it be, like, publicly available. But I don't know of one anywhere. There are major countries that have their own process, like Europe has its own, Canada has its own, Brazil, Australia, EU, all these countries have… that are big, they'll have their own process. there are countries that'll have expedited routes, so if you have approval in the U.S, you can get into Australia faster if you have U.S. approval and MDSAP. MDSAP is a type of Quality System Certificate. So if you have those two things, it's a very fast track into Australia. If I remember correctly, it's less than 30 days. Same thing for Canada. If you have Canadian approval in MDSAP, really fast into Australia. So certain countries, it's faster if you have approval in one of the big countries like U.S.

Then there are countries like Trinidad and Tobago. If you have U.S, Just send money and sign a form. So… And I'm really not over-exaggerating that, it's really that easy. So if you have approval in a major country like U.S, they make it very easy on you. But if you want to go to Singapore, they're like, you have to have ISO 1345 certification. It's not enough to be FDA registered. you also have to have a certificate. And so they want you to be ISO 1345 certified, and you have to provide that. But that's also true in the state of California. California says it's not enough. You either have to be inspected by us before you launch your product, or you have to be ISO certified and provide a copy of a certificate.

So, it's a great question, I just, unfortunately, it doesn't exist. But all the major countries. the best you're gonna get is expedited, and the countries where you're gonna get the best deals are countries like Australia. would be a great country to go after, because they give you leverage. Saudi Arabia is another one, where it's pretty easy once you have U.S. I think UAE is another one that's like that. Korea's another one. So a lot of these countries, it's just easier if you already have US, but there isn't one, and it's definitely not specific to SAMD, it's just devices in general.

So, one person asks, I'm a BSI-certified auditor to 1345-2016 with 6 years of experience conducting quarterly internal audits. What additional certification is needed specifically for software as a medical device? So, needed, it could be interpreted as required, or it could be interpreted as unable to perform the audit effectively.

Well, the…

Take your pick.

There's two different possible answers here. One is if they… if you're doing the audits for BSI, they actually have two kind of requirements. One is you have to actually have worked on software before on a design team to audit it. So if you haven't, you can't be qualified to look at software. The other way… The other way that they qualify those people is they have them take exams. So they've gone to… it used to be you just gave a resume, and maybe some references, and said, I have software experience. So my experience back in the 70s hacking computers when it was, DOS counted. Not that that is all that useful today.

But it counted for BSI, and they would let me do software audits. I did good software audits, but… I wasn't a competent programmer by any means. Now… Now they actually have you take quizzes, so you have to know Scrum, you have to know Agile, you have to know 62304, you need to know something about cybersecurity. There are quizzes, Anybody that's half-competent in software could probably pass it without studying. But, But there's a lot of people that want to qualify for software because it's so popular, and they haven't done any actual software programming. So they're at a disadvantage. Our company, we have… You know, 50 software companies a year that we help

Review their entire technical documentation, and review all their documentation for cybersecurity in software. and have them make revisions to it. Sometimes we get involved in design reviews, and then we help them prepare and submit it to the FDA, so we're… we're doing this all the time, so we're really close to what the process is. for these companies, and a lot of times, we actually can give them advice on how to improve it, but if you want… you're a BSI auditor, and you want to migrate over to software auditing. You're gonna have to have previous work experience in software. And you're going to need to pass an exam, and you're going to have to show some objective evidence that you've been trained on 62304, and probably the cybersecurity standards as well.

beyond that, I know they have a list. But they also have some options in there. So you… and you actually have to put together, sort of, here's my… here's my resume and qualifications for why I should be able to give them this code. And that's what it is. They give you a code and say, okay, you get this MD code for… for software. And then they have another one for sterilization. I have it for tons of different codes. because I had worked for a decade in pharma, I had done a stint in automotive, a stint in a chemical plant, I had done programming, I had actually run my own company, so you do more variety, you learn more things. So I get lots of codes.

But, In companies where you're an internal auditor, you don't have codes. They don't have that formal certification process of saying you're competent in this code. They don't have codes. So, why are you qualified to audit to audit software, you gotta come up with some objective evidence, like, I took this course on this date, or I self… trained on this by reading the standard, and having some books on the shelf over here on how to code, and how to do, code reviews, and how… what the most common mistakes are in C-sharp, and I read these articles, and I go to webinars that Jose does, and I, I read, so you, you basically make your case for why you're qualified.

But I get asked all the time, like, so where's your training on 1345-2016? I'm like, well, that was 16 years after I started working in medical devices, so I was a little senior by that point. I was the person doing the training, so the day it was released, I downloaded the standard. Updated my slide deck and gave a training to a group of auditors on how to audit the 1345-2016. So, I did one of the first trainings in the world on 2016, the day it was released. So that was my training, but I have so many training records for that now, you know, like, I can say, well, well, who is your boss? Who's your boss's boss? Well, I train both of them.

So, if we sign up for your course, then show us what we're gonna learn. Show us, like… the syllabus, and, like, you know, walk us through that. And then also, I understand that you have an offer for, like, getting the first 3 parts free.

So we'll go right to the end slides here. Can you see my screen?

Yeah.

Okay, so… We've had a whole bunch of people that, pre-ordered the course, and some people have said, I'm interested. So, the people that just said were interested, we decided to create a sneak peek for them. So, these three, with an asterisk, one's an introduction to auditing, one's 1345, Stage 1 and Stage 2, And that's 3 hours of training right there. And then roles and responsibilities of the people involved in auditing. So, you know, like, what is a guide? What is an observer? What is a team leader? What is a subject matter expert? And we talk about how they… how their roles might be different in different situations. Every single one of those quizzes, or…

videos, trainings, and the quizzes associated with them. Every single one of these has a quiz. Those are free if you just fill in a form on our website. And, if you don't… if you're so lazy that you don't even want to fill in that, all you have to do is put in the chat, sneak. The word sneak. S-N-E-A-K, And we'll take care of registering you, we'll fill in the form for you, and make it available for you. We make everything available right now in Monday.com, which is a project management software, so you can track all your progress. And you can even set your own timeline, you can store your training records there, and it has all the hyperlinks for the course modules.

As well as the quizzes. For those of you that say, actually, this was pretty good, I think I would like to be a supplier auditor, which would be the most basic type of audit you're gonna do. Because you don't have to look at the whole quality system, you just have to treat them as a department in your company doing whatever that supplier does. So if the supplier's a sterilizer, all you have to do is look at sterilization aspects. If they're a packager, you look at packaging. If they do assembly, you do assembly. But you don't have to do everything if you're auditing a supplier. So we have 15 lessons there.

And this actually goes on for 3 pages. Oop, went too far. So, there are 15 lessons, we tell you which ones, and this will be up on the website on Friday, but If you just want to see what this is all about and what it's like. All you have to do is type in sneak in the chat, and we'll hook you up with these 3 for free. Then if people want to get the supplier routing, that's 900. That'll give you 15 of the lessons, the ones that are in green in this column. And then if you want to be an internal auditor, that lets you do the whole quality system for your company, as a team member of an audit team, that would be this column. And then if you want to actually lead audits, and train other auditors, and do the reports, and plan the whole audit program.

And you want all the softwares and medical device stuff that we have. all of the software as a medical device stuff is in this last one. So if you're in a company that doesn't have any software as a medical device, and you just want to audit a couple processes as a team member, the middle column would be for you. But if you want to be a lead auditor, and you want to do this as a job, and you want to learn software, that's this one. And then for those people out there that say, I just want the software as a medical device stuff, I already am an auditor, I don't need anything else, it's 100 bucks a less, and you pick which ones you want to buy.

So, if I'm uncertain, is there somebody I can talk to about this? Because I really feel like, oh, Linds.

So, at the bottom here, we have Lindsay Walker and Becca Taylor. Those are the two people in our company that handle those kinds of things. On our website. We have links for Contact Us, and you can schedule and all that, but to make it easy for you, if you want to talk to somebody, either Lindsey or… probably Lindsey would be the best person, just put talk in the chat. And if you want both, just write sneak and talk. And I'll set up a… I'll set up a meeting for you with Lindsay, so you can talk to her, and she can answer some of your questions. She can even write a proposal and send it to you. We had somebody this week asking, I have two auditors I want to train in our company.

can I get a discount? So we came up with a discount plan for if you have more than one auditor in your company for corporate plans, so that does exist.

Do you have a special Sean discount?

Well, after we're done here, I'll give you anything you want. You want to become an auditor, Sean?

I don't know, I might.

Okay. Well, I can hook you up. All I gotta do is put Sneaky in there, and we'll be all taken care of.

How do we audit for adequacy in AI-based production tools such as an automated software testing of the software as a medical device? What elements would a third-party audit… what elements would third-party auditors look for in CSV slash… CSA Records. I don't even know what those are.

Yeah, I'm not sure what CSV and CSA are. There are so many acronyms, it's hard to pull it out of context what it is.

Yeah, okay. Well, you know, I guess.

system validation. So I think what we're talking about there is software as a medical device, we're talking about software tools for the quality system.

Hmm.

And if that's what they mean… so there's a different standard for that that you should use. The one for software, so everybody talks about 62304, but there's actually a more… useful standard from a risk management standpoint called IEC 80002-1 that tells you how to apply risk management to software. But then they have a separate one that has a similar number, 8002… Dash 2 for software that's used for the quality system. And then there, in the very back of that standard, they actually give you examples. How to audit a… or how to validate a spreadsheet, how to validate a molding machine, how to validate production equipment. So, any kind of quality systems software, they give you examples of how to do those software tool validations.

We also have a procedure for that. There's also other guidance out there, but one of the things I will point out, do not follow GAMP 5 unless you are a pharma company or a biotech company. GAMP 5, is… twice as long. In terms of the length of the standard, and it'll be at least 4 times as much work. And it's not required by the medical device industry. The medical device industry allows you to use a risk-based approach. And the pharmaceutical and biotech industries say, follow GAMP5, GAMP5, you shall do what's in here. Medical device, risk-based approach, so the level of rigor that we would put into a spreadsheet isn't even remotely close to how much work we would put in to something that's used for, let's say, releasing batch records and releasing product.

So we have a very different level of scrutiny that would be on something simple that's used as a tool versus something that's actually controlling our entire quality system. So, it depends on the risk. But, if you… have somebody that's good at writing protocols and executing them, it doesn't have to cost tens of thousands of dollars. When it… when it costs tens of thousands of dollars or hundreds of thousands of dollars to audit software quality systems, it's probably because the company customized it. Usually, those companies have templates for validation. If they don't, you should probably shop elsewhere.

But when you start customizing it, now we have to do a lot more work, because we gotta cover all the customization. But a good cloud-based EQMS system probably already comes with, out-of-the-box A validation package that you just have to execute. Somebody asked her, what's the website for our training? So, it's MedicalDeviceacademy.com, And if you type into the search at the top, Lead Auditor Training. So that's on it, and even on the home page, there's a hyperlink. I think it's in the second paragraph. It says, Lead Auto Training, you click there, and it takes you right to the page.

And, we can make that available to all the people that are… just type in Sneak, and we'll… we'll give you that link.

That's great. there are a number of questions where people are like, well, what if I don't need all the auditor training, just the software as a medical device? Or another one that asked, can I choose only to have software as a medical device training?

Yeah, that was the point of having the…

$100 or less in a la carte. Right. That's what I…

specifically why I did it, because I knew there'd be people there that either don't want to be an auditor, but want to learn about software as a medical device, so you're going to get… you can choose this one right here on Amy TIR57, that's cybersecurity. On the page above, we have 62304, we have one on usability, we have one on Agile, Scrum, and Waterfall, we have one on software tools for auditing, not exactly the same thing, but so don't… if you're looking for software as a medical device or software tool validation. that's not what you're looking for. This is, like, how to use webcams, and how do you use tablets, and how to use, like.

this tool here.

And how do you use…

Google Drive and stuff like that. How do you… electronically sign things. So that's software tools for auditing. If you want software as a medical device, it's 62304, number 20, And then on the previous page, number 30 for cybersecurity. Those are the two most valuable ones. There's the… the Scrum… Agile Scrum and Waterfall, that's useful. Every software as a medical device needs usability. the FDA is forcing everything through, Europeans are forcing everything through, so you need to do… have a usability engineering file, and you'll need training on that. We actually have a whole entire course just on that, but this is a lesson on it.

We have this video here, it's fairly short, medical device versus software, what the differences are. And then… Is there anything else? I would say those are the key ones that you want.

So…

We certainly have other stuff, like, we have one on MDSAP for people that don't have anything on MDSAP. Hello!

So another person asked, as a recent graduate in medical device regulatory affairs who wants to get into the software as a medical device space, what are the key skills or experiences that are needed? I'm curious to hear your answer. Where would you direct a recent.

You… you absolutely want to… so a lot of people say, go work for a big company, get experience. No, no, no, no. Go work for a small company, because they don't have enough resources. And they will let you learn anything you want, and let you do more than you can handle. That's how I've learned over the years. I've worked for a lot of startup companies, and I like that environment. You learn, you learn, you learn, and they're small environments. And if there's something you don't know how to do and you want to learn, you say, I don't know how to do this, I didn't learn this in school, I learned everything but this, can we hire this consultant to give us a training on it?

And we… and these trainings, like, if you bought the 62304 training, and you're not buying the whole entire system, you just want 62304 training, buy the 62304 training, you have… it's a YouTube video, it's a private video, but play the video for the whole entire company. Now you all have seen the 62304 training. So that's a great way to get training for a small company, is just buy one of the modules that you want to get training on, and we have… we have hundreds and hundreds of videos on all kinds of topics, and we're always, like, somebody will suggest something, like, oh, that's a really good idea, let's do that.

That's how some of these came about. Here's one for you. Do you also include information on AI-related information for the auditor training? Is BSI… has BSI now needed for AI-enabled software as a medical device?

So, we don't have it as part of this yet. That's one of the things that's on our backlog list that we decided it's not going to go in this version, it's going to go in the improvements for next year. And if you buy the lead auditor course, you get all the improvements for free.

But.

If I'm looking at this… set here. I had something in here specifically for… Software as a medical device. and AI. I thought I did. Sign History Files, cybersecurity… Procedures… Well, it'd be easier… it'd probably be more efficient if I just explained. So, I actually did a training on this earlier this week, I think it was Monday. Somebody wanted to understand… they were a software developer, they had a product that worked, but they didn't know anything about medical devices. And it was AI, and they wanted to know, how do we get to the point where this is there?

Ai technology, you're going to have to do a clinical study, so that's one of the things you're going to have to do. There are… there are very, very few examples where we don't have clinical data for AI. Because you're comparing ground truth with a gold standard method against what your AI algorithm does. And so you have, like, a radiologist looking at an image, and AI software looking at an image, so you have… they count The radiologist more highly as the gold standard than the computer, so you have to get a lot of data to prove that the computer's as good. So clinical data is one of the things you're going to have to learn.

Second thing you're gonna have to learn is datasets. You're gonna have to understand the difference between a data set that is for training the AI algorithm, and your data set that is for validating your data… your software. Those two datasets have to be similar, but there can be no overlap. So if you draw a VIN diagram, there can be no circle overlapping. They must be separate. There can be no samples in this dataset that overlap with this data set. They have to be separate, but they have to be similar in terms of The inclusion criteria, the exclusion criteria, what they were being diagnosed for, the type of imaging, whatever it was. They have to be similar data sets, and you have to explain that in detail.

And the FDA requires, in the software description and in the device description, how you generated that AI algorithm. So what software tools did you use? What engine did you use to create that? And if you want to revise and update it, what a lot of people want to… they do is they want a ladder, they want to… I'm gonna… validate, but then once I've validated, I'm gonna use the data from my validation to retrain the tool and do it again. I'm like, - can't do that, because now they're overlapping. You just trained it with validation data. So you have to… you have to have a set of data you keep as your validation set, or you have to go out there and get a whole new set, and you can't use any of the algorithm that you developed before

If you're going to do that approach. So, much better to say, this is my validation data set, and I'm not gonna touch that. And every single… if I want to improve the algorithm, I gotta go out and get more data. to reprogram the algorithm, not more data to validate. You already have a validation data set, stick with that data set. Don't change that data set. Just add more data to the training. So that's one of the mistakes we've seen a lot of companies do. They just want to keep on laddering up, laddering up, and you can't do that. But they have something called a… Predetermined Change Control Plan, or PCCP. That's the plan on how you're going to update your algorithm.

And the FDA used to say no. They'd ask for a plan, and then they would refuse it. They've started approving plans now, but only a handful of them. and only at the time of clearance. So you get your 510K, or you get your Genova, that's when they approve your PCCP, or after it, they'll approve your PCEP, but they will not approve a PCCP before 5 can clearance for Jenovo. They'll give you feedback on it, but they really don't even want to discuss it. They want to wait and concentrate on, let's get you on the market first. Once we decide you're good enough to be on the market, then we can talk about how you're gonna change it.

Which makes sense, but that is their approach. There is a guidance, it's not very useful. I mean, it tells you what they want, it tells you what controls they want in place, it talks about the things I mentioned, but if… there's no way it's going to keep up with the state of the art for how we develop algorithms, and how we develop AI software. And… people are really coming up with some very novel ways now of validating their datasets. Somebody came to me yesterday, like, well, we want to have something that's more like a real person, so we won't… we don't want every single validation to be done the same way. We want it to be done the way the company… the doctors do it in the UK, and the way they do it in France, and the way they do it in Brazil, and the way they do it in Australia. We want to look at the way everybody does it.

so we see… have more diversity in our data set, and that makes the head spin at the FDA, like. no, we… it's all gonna be the same. I'm like, why? And it's not a diverse data set, it's not realistic. It's very narrow. FDA doesn't know how to deal with that yet. So, AI algorithms, we don't have a module yet. But definitely on the… on the list for we will have it next year early. We'll also have one for 27,001. Those two are already on the list.

That's fantastic. We only have a couple minutes left. I want to make sure that everybody knows, what to do. If you would like to receive 3 free lessons, write sneak in the, chat, and…

Sneak?

Send that to you. If you'd like to talk to somebody, because you're not sure, because this is confusing. If you want the slides, put slides in there. If you want your certificate of attendance, put certificate… your name and email addresses, you would like it to appear. And… What else? Join us tomorrow for the MDR-IVDR transition, at 10 a.m. Eastern Time.

I would like to point out here, I… we didn't even get to talk about this, but… A lot of people are wondering, like, what causes don't apply if you're in software as a medical device for 1345. I actually made a list. I haven't seen anybody do that before, but I actually think that's really helpful. So it's in the slide deck, so you just ask for it. And, that's on slide… 17. And then, there are a couple of clauses that are tricky. So, like, incoming inspection. A lot of people say, well, I buy software off the shelf, do I have to inspect it? No, you can say right in this… in your manual, we don't do…

verification of purchased product on software. Instead, we handle that as part of the design control process. We verify the off-the-shelf software in the soup by doing validation of that as part of our design. So that's how that's done. Or if it's a quality system tool, then it would be through the tool for QMS software validation. So that's how you handle that particular one, but here's what… these were a couple of things that I thought might be nuances that people might need to finesse by making a note in their In their manual, like, we don't have non-conforming product. It's something I've heard a lot of people in software say. I'm like, yeah, you do. What are software bugs?

What are vulnerabilities? They're like, but we already have a procedure for that. And I'm like, so that's your nonconforming material procedure. So just make sure that you've figured out how to make it meet those requirements of 8.3. So, some of these you have to finesse, and it takes a little coaching on how to do it, but a lot of people struggle with those particular ones, so I put the finessing requirements on 18, and I put the clauses that don't apply on slide 17. And if you have software in a medical device, ignore that, because all of them apply.

Well, Rob, thank you for your time today. I thank everybody who attended for their time today. I hope you… Inspirational, and you've decided that you want to become a lead auditor, software as a medical device, or any other kind of auditor, and that you will take advantage of the, courses and opportunities, presented today. And… that's it from me. Any… any closing words, Rob?

I don't have any closing words. I'm just really excited that we had people, showing up for this, and there was awesome questions, so please send me the questions so I can, answer them by email to people.

Great. Okay, well, thank you all. You know how to get ahold of him, medicaldeviceacademy.com, and me, MedTechLeadingvoice.com. Cheers, everyone.

Cheers!

Happy Holidays!