Lead Auditors: You Need Them, but Can You Guarantee They Really Add Value?

Okay, if you guys don't want to be promoted to panelists, just decline. I'm waiting for people to join, I just… Feel free to be on camera, off-camera, it's up to you. Welcome.

What?

Okay, I'm good now.

I… you scared the hell out of me.

No, because he said, like, because I was not a panelist. So that's why I couldn't.

Okay, it's 11.

were surrounded.

Good morning, John.

Hey, good morning.

Good, I just got back from Auckland.

Oh, did you have a good trip?

Had an excellent trip. And some vacation.

Bonjourn.

In Giorgio. Comistario.

Venegade, tuta busto.

Mine. Oh… I am waiting for Rob.

BS. I.e.

Okay, well, while we're waiting for Rob to get here, let's… Take a moment… there he is. To.

Whoa!

They're up.

There he is.

Let's take a moment to, talk about what's coming up, in the next two weeks. So, next week, we're gonna have, integrating human factors in, your QMS. With a speaker from Claremid, who, will be talking about the QMSR requirements, ISO 1345, you know, alignment by the FDA. And the fact that Human Factors is really going to be integrated throughout the QMS. So what we're going to do is we're going to do a collaborative workshop, and I'm hoping everyone will participate. Where we're gonna build the deck together, rather than having the speaker go through the deck. We're gonna build the outline of the deck, and we're gonna walk through filling in all the parts. So basically, identifying all the places where human factors shows up.

Throughout the… throughout the life cycle, and the QMS, and 13485, and QMSR, and tie it all together into a nice, neat package. How's that sound, everyone?

Looks like fun.

We gonna show up for that? Okay, that is number one, and then the following week, we are going to, have a session, with Tino Chao, who I met recently at the… MedTech Symposium in Philadelphia, who's going to be talking about honing your MedTech message, how to be memorable in 2 minutes. And his basic assertion is that it's hard to be memorable these days, You know, particularly when you go to a lot of pitches. Anybody who's been to a pitchfest has listened to… 10 pitches in a row, can't remember the first 8 by the time they get to the end. So…

how do you become memorable, either as a founder pitching, or as a consultant trying to stand out in a crowded… crowded marketplace? So, this is really something for everyone, and I encourage everybody to come on the 18th. I don't see any reason to have a Friday in Focus call on December 25th. I think that would be really strange. And same for New Year's. I think that what we'll do is we'll start after the New Year, if that's okay with everyone. Any, any objections, please, you know, start screaming and yelling. Hi, Carrie, welcome. I encourage the rest of you to join on camera, but if you don't want… if you want to be invisible, remain invisible. Francesca will promote you. Feel free to just turn it down. Let's…

get started, I want to introduce Rob Packard. Rob Packard is the president of Medical Device Academy, and has been educating the, MedTech market on, FDA and all sorts of things, related to getting your medical device approved, has worked with hundreds of startups and small to medium-sized businesses, and I'm gonna let him introduce himself and do a better job than I'm doing right at the moment. So, Rob, thank you for coming. What do you have for us today?

Today we're going to talk a little bit about auditing. And I know everybody thinks auditing is so exciting, but I will do my best to show you the light of why it's so exciting. But, as far as a little bit about me, for those of you that haven't seen me, and I don't think there's anybody out there that hasn't, I know Leo's seen me forever, My background's, I started out in medical device in 2000, but before that, I worked in the pharmaceutical industry in R&D. And, when I started out in the medical device industry, I didn't start in quality, I didn't start in regulatory. I actually was working in operations, and then I became CEO of my own medical device company.

And I know this will shock Leo, but I actually had an electromedical device.

I knew that, Rob.

Yep.

But a few times.

So, I… the last thing I did in my entire medical device career was picked up quality and regulatory, and I did it grudgingly, and when I say grudgingly, I was this close to quitting or getting fired, and I wasn't sure which way the lunch meeting was gonna go. And, I, I… calmed myself down, had a couple extra sips of the iced tea, and I said. You know? This might be a really good consulting business, because nobody likes doing paperwork. And maybe quality and regulatory would be the place to be if I could do the paperwork better than everybody else, and it's always going to get more complicated.

The paperwork's gonna get thicker and thicker and thicker, and everybody's gonna wanna outsource it. Now, if I were to think about it today. and I listened to other people, they would tell me, you know, AI's gonna replace you, we're just gonna do all the quality and regulatory paperwork with AI. I think that's extremely unlikely, since we haven't even been able to figure out how to get AI to write a standard yet, but… If we can get AI to write all the reports for us, I'll jump on that bandwagon when they can prove it to me. But I… In the auditing world. the day that I actually became a professional auditor, I remember sitting at the table of my… it was actually in my office at a medical device company.

And, I, I was considering taking a job with BSI to become a full-time auditor, and I said, you know. Do I really want to do this? And I said, this is… this is gonna be my pathway into consulting, should I do this? And I couldn't make up my mind, and the phone rang, and it was somebody asking me to do a big consulting project, and they needed an auditor that could do it. I was like, well, I guess the decision is made. And I walked down the hall and handed in my resignation. Quit my job and became a full-time auditor. Now, I teach people how to audit, which I love to do. But… The one thing that I think has changed dramatically since that day in 2009

Now, I actually love auditing. I don't love traveling 7 days a week, which is what I did for BSI. But, I love auditing, because I get to go and meet other people, and I get to learn about their quality system and their product, and then I get to help… I get to help them improve it. And that's the fundamental difference between my mindset in 2009, when I looked at it as. This is just a consulting business for me. to now, I actually really enjoy doing this. And, the… how you get to that mindset can be a lot faster than it was for me. if you learn some of the things that I've learned over time, up front, and there was nobody there to explain those to me.

Should I keep on talking, or do we have any questions?

Well, what is the qualification to be an auditor?

To be an auditor, You have to have training on… the… whatever you're auditing against. So there's… you're auditing to a standard, or a contract, some requirement that's written down. And you're making sure that the company, or the operation or the process is compliant with whatever that documented requirement is. So it's really a… just a verification of, are you doing this? So you need training on that area, at least to the point where you understand the jargon and can ask intelligent questions. You don't necessarily have to be a subject matter expert in that area, but you have to know enough at least to ask the questions and read the records and understand them.

I love to learn, so I always went a little bit further to try to learn more about it than most auditors would. And I think that makes a real difference in the quality of questions you ask. When you really want to learn about something and understand it, you ask smart questions. that really show your thinking. When you are just going through the motions, you say, can I see a record? Can I see a document? You don't really care what document or record you made. Just throw some paper in front of me, and I'll write something about it, and I'll get my report done, and you'll be happy, and I'll be happy, and I'll leave.

But that's… that's not really auditing. Auditing, you really have to understand what's important. and what kind of problems they are having, if any. Maybe everything is going swimmingly, and there's nothing that you need to fix. But at least you have to ask enough questions so you understand that. And I don't think, auditors that are sort of going through the motions get that far. And the thing we like to describe is checklist audits. I have a checklist, it tells me to ask this question. I asked that question, you gave me an answer, I wrote it down, I turned it into a report. Anybody can do that.

But that's… that's really just going through the motions. If you are really adding value as an auditor. you're asking smart questions instead of dumb questions. And the dumbest question I ever heard… I was at a facility, and I… I mean, it's been so many years now, it was a company that makes tampons, so we'll give you that much information. And the other auditor walked… working with me. tells me, and he's the… he's the lead auditor, I'm the team member, so he's the boss, my job is to do anything in my power to help him get the audit done on time and correctly. So he said, you know, I'm really concerned they don't have enough traceability of raw materials here. They don't know what raw material lot they used, they don't know what machine they ran it on, they don't know any of this information, and they can't show it to me. I want you to go out there and figure out how they do this, because I don't think they're doing it well enough.

And I looked at him, I don't know what the look was, but it was just, like, complete incredibility. what do you mean? I'm like, this facility has robots! How would the robots know where to go? And he didn't understand that, like, you have to have traceability If a robot knows where to go, but the traceability tag is for the robot, not for you. So everything they had was digital, everything was computerized and automated, and he didn't understand it because he couldn't read it, because he didn't read robot. So these little RFID tags everywhere were just messing up his mind, he didn't understand. I'm like.

Aren't you a computer programmer? Shouldn't you understand this? So it was a really rough week, and at the end of the week, the client said, could you please make sure he never comes back, and please only let them send you back? So, that's not how you want to be as an auditor. You don't want to ask the question, like. How do you know how to get there when the robo… when… and not notice the robots walking around?

So, what is the difference between a lead auditor and an auditor?

So… The lead auditor is responsible for planning the audit. They're responsible for the agenda, they're responsible for the opening meeting, the closing meeting, and the final report. So any decisions about what's nonconforming and what's conforming in the final report, those are the final decision of the auditor, not the team members that help them. And if you have a small company and a short audit, you might not have a team. It might be me, myself, and I doing the audit. So then you're both the team member and the lead auditor, and you do everything. But if you're trying to get an audit done faster, you might have another team member. And because some of these audits, particularly the MDSAP audits, Medical Device Single Audit Program audits.

Those are so long, they're almost always two auditors. And typically, 3 days each.

And so you, you have…

It really creates a challenge, because we have to have one team of people to help this auditor, and another team of people to help this auditor. And a lot of times, to save money on travel, one of the team members is off-site. Usually the team member is, and the lead is on-site. So that creates a real challenge, because the team's like, I gotta keep one Zoom or Teams session open for this person, and keep on scanning documents and uploading them for them, and then I gotta answer all these other questions for this other person that wants them in paper form. And, you… you might not have enough people in a small company to support two people.

So if that's what's coming your way for certification. you need to kind of simulate that in your own internal audits. So, even though you might be a small company and might get away with just one auditor, it might be a really good idea to practice That experience and prepare for it by saying, hey, we not only want you to come in here and give us an audit report, but we want you to give us the whole experience so we know what it's going to be like To have a certification body payer come in and do the audit, so we know how to support both people at the same time. And so that's part of what auditors sometimes do, is they actually simulate what it would be like to have the FDA there, or simulate a certification body audit.

just because that way, your team is better prepared, they know what to expect. And then at the end of each day, not during in the middle of the audit, the end of each day, you can have a debrief and say, you know, this worked really well, this worked atrociously, never do that again. And you can do some consulting as an internal auditor or a supplier auditor, but the third-party auditors and the FDA inspectors, they're not allowed to do any consulting of any kind. They can't even write opportunities for improvement. So, that's considered too much like consulting. So, they can literally only tell you what you've done wrong, or that you've done things right, but they can't tell you how to improve.

And so, certification body audits are almost absolutely useless. To a company from a value-add standpoint. And I think, unfortunately, that's where everybody gets their impression of audits, is because the person that is required by law to do it isn't allowed to give you any advice. But when you hire somebody to come in to do an independent… First-party or second-party audit, an internal audit or a supplier audit. If that person doesn't add value, kick them out of your building. Hire somebody else. And ask for references. Say, what kind of value did he bring to your company? How did he help? How did she help?

Were they personable? Were they boring? Were they funny? Were they blunt and very, you know, this is what you need to fix, this is what's wrong? Or were they afraid to make any waves? And what are their strengths? Does this person really know a lot about sterilization? Do they know a lot about software? Do they know a lot about electrical engineering? What are their expertise? Because nobody's an expert in everything. So, you might need more than one. You might want to rotate, like, this year we're gonna have this person, next year we want a different perspective, so you get a different person. Not just because we didn't like their perspective, but because if we have the same person looking at it every year, eventually they're going to get a little stale and start looking at the same things the same way.

Whereas, if I haven't been there for 3-4 years and I come back, I see all kinds of new things, and my perspective is, wow, you really improved things over here. But what happened over here? You stopped doing this thing that was really good! So, that's the kind of feedback you're giving as an auditor. When you're hired for a first-party internal audit, or a second-party supplier audit, those third-party audits that are required by law, all you're getting is a piece of paper that says you passed. And you don't technically pass, you either have things that are conforming, so you don't have to write extra paperwork, or you have non-conformities that say you have to write extra paperwork.

Ultimately, at the end of the day, everybody has to write more paperwork, so don't get all upset because you've got a nonconformity. It's an opportunity to practice your Kappa writing skills. This may be the way to look at it.

Okay, so you've convinced me I want to be an auditor. Now, how much money am I going to make as an auditor?

An auditor Any, any consultant. Has a range that is typical for what they do. There are some that make very little, and there are some that make very… a lot. I'm able to charge $350 an hour. Very few auditors charge that much money. I'm… when anybody hears that rate, they usually… I usually hear a pause, or a gulp. Sometimes they never call back. But I don't want to be the cheapest consultant on the planet. I want to be the best. So that's part of what I do. I try to be the best, but if… When we hire new consultants, and we train them.

And we, we teach them how to audit. We typically start people out at $200 an hour after they're ready to be on their own, and they're… they can be independent, and they work up to $275, and we have one auditor, Matthew Walker, he's now at $350. But he's been doing this since 2017. And I don't know anybody that writes or reads more than he does, except myself. And he just… he's so smart, and such a great writer. But, he'll sit there with a client, and they'll ask a question. And I look at him like. How do you remember all that stuff? He remembers the guidance documents. He… it's one of the most valuable things that I learned working for BSI.

I had handcuffs on. I'm not allowed to give advice. But I want to be value-added. So how do I do that as a third-party auditor? This is something nobody else does, but I did it. And I said, you know, there might be a guidance document, they can help you with that. It might have this number. And if you look on this page in this section, you might find the answer to that. And I actually did that while MHRA, which is the, UK, competent authority, they were actually observing me, and the guy smiled, and he's like, that is a clever approach. You're not allowed to give advice, but you didn't give advice. You pointed them to the guidance document, right where they needed to go. That's perfectly legitimate.

And the team leader, I was only the team member, the team leader just breathed this huge sigh of relief, saying, I thought you were gonna kill us. He was like, nope! That's how to do it. You're allowed to point somebody to a guidance document as a third-party auditor. So Matthew, he's… I've taught him over time, I was like. you can help them sometimes just by pointing them to the right place to look up the information. If you spoon-feed every answer to somebody, that's not always the best solution, because they don't always learn. But if you make them look it up. They might actually learn something because they had to struggle a little bit to get the information.

And sometimes, what I would do, if somebody asked me a question at the beginning of the audit, and I gave them a standard, the next day when I came back, I'd say, so what did you find when you looked that up? And they'd look at me like, I was supposed to do that for homework? I'm like, you asked the question, I didn't bring it up, you did. It was important enough to ask, but you didn't think it was important enough to look it up? Should we audit more in that area? He's like, no, no, no, I'll look it up, I'll have the answer tomorrow. I'm like, there we go! So I… I push people a little bit, sometimes, like.

I'm not gonna spoon-feed you! At $350 an hour, I'm gonna make you do a little bit of work. My trainer, he can't give me a pill and make me in better shape. He makes me do… planks and push-ups and lift heavy weights. So that's how you're gonna learn working at a company. You're gonna have to look stuff up, try it, see what works, try it again, see what works, have another auditor come back, challenge you. See if you can get better, but it's… there's no free ride here. There's… I tell my kids all the time. There's no easy way to get better at anything. You have to work. They don't like that answer. I'm like, how did you get good at playing video games? They don't like that answer. It's like, you practiced. Did you pick up the video game and instantly you were a savant and could play it well?

You know, even the savants of this world, they only knew how to play chopsticks in the beginning.

So, I want to take a moment to ask if anybody in the audience has any questions, but before I ask them if they have any questions for you, I have a few points of clarification. One, has anybody in the audience ever been an FDA auditor, an MDSAP auditor, or worked for a notified body as an auditor? Or worked independently as an auditor. Shove hands, or just blurt it out, or, All right, so we have some experienced auditors in… in the audience, and… You know, I… I guess the word audit gets used rather generically to cover all of those categories.

And, that's… that's kind of confusing. An official FDA audit versus an independent audit that you hired to come in and give you advice about some part of your system, you know, and then you're talking about auditors and lead auditors, right? And then you're also talking about, Things like… Being an auditor for a software as a medical device.

Yes.

Does… Any… and any number of other things. So specializations, and qualifications… now… this is the part that I was getting to earlier. Is there any credential that I need to call myself an auditor?

Yes, you need to have taken a course That's a lead auditor course. There are… there are companies that offer it. As a on-site course, you go and take that course. It's usually several days long, sometimes a whole week. There are also a few online on-demand courses. That's what we're building. Another company that offers, that is Stata Matrix. There are organizations that offer an online version, like AMI offers one, we created that for AMI, but that's… you have to start at the time we start on Monday morning and finish at the time we start up on Wednesday. And you'd have people on the West Coast, we'd be starting at 9am, they'd be getting up at 6 AM, they didn't particularly like that time schedule, and they had to do it the week that it was offered.

So, the on-demand, self-paced ones are nicer, but the downside of them is you don't have a live instructor answering your questions. The other thing is there's always the potential, somebody buys that course. And it just sits in there. Their computer is somewhere and collects computer dust. So you actually have to open up the course, and watch it, and take it, and learn from it. So, like I said about the guidance documents, you gotta actually go read those guidance documents if you want to learn something. But, you have to take a course, but that's part one. There is a part two. The part 2 is…

just like when you're driving a car, you went to driver's ed class, you took the exam, you passed the exam, you got your certificate, you passed driver's ed. But now you have to go out there and drive a car on the road. Some people have driver's ed classes where they take you out on the road and drive around with you. But you still have to pass a driver's test, a practical driver's test at the DMV. For auditing, we don't really have the opportunity to say, who would like to volunteer to have their company audited by a whole bunch of trainees? I haven't found anybody that's willing to do that yet.

But… Since COVID, we have this really cool thing that's happened. Everybody's learned how to use Zoom now, and everybody's comfortable with Zoom, and it can be Zoom, it can be Teams, anything else. But now, people are also pretty okay with recording it, or transcribing it. So now, I can have a trainee go do an audit. and I can have them record it, and I can watch it, and they can watch it. And you'd be amazed what you'll learn from watching yourself on a video. We record all kinds of things, but we tend to let them collect that computer dust. But if you actually go back and watch your own videos, which I have to do all the time with this YouTube channel.

It's a cringe moment every single day I hit play on my video, like, oh, ow! It's physically painful to watch yourself talk, and everybody has that reaction, because you know you can do better. But that's how you get better. You watch yourself and say, okay, don't get overwhelmed with all the things you did wrong. Pick one thing to improve, and work on that next time. So what we tell trainees, and I did this to Matthew, and he loves doing single process audits, because that's how I trained him. I said, instead of planning a whole big, gigantic, full-quality system audit, pick a process. Pick the process that's doing the worst, needs the most help, just had a CAPA and needs an effectiveness shift. Pick one process, and do an audit of that.

and record it, and watch yourself. And then if you want help, just send me access to the video, and I'll give you help. I don't have to be there. I don't have to give you feedback real time. That's how we used to do it. We used to give people feedback real time. We'd be listening to the oddity, we'd be listening to them, so we're not listening to either one very well. And then we're supposed to give you advice, and we're not supposed to interrupt the audit process. That's almost impossible! But now we can record it. So do a remote audit, record the single process audit, watch yourself, learn what you did wrong.

And then try to improve one thing, and just keep repeating that process, one process at a time. Yeah, it's a bunch of small, single-process audits, but it's so easy to learn when you can only focus on one thing at a time, and let's say it's a 2-hour audit. That's 2 hours of your life, you gotta concentrate and try to improve one thing. That's practice. That's the definition of practice. You get feedback, you try to improve something, you get more feedback, you try to improve something. But if you don't have that constructive feedback. You don't improve, and if you don't concentrate on trying to do one thing better, it's not practice.

So that's fundamentally what you have to do. You have to take a course, get a certificate after you pass the exam, and our exam is 100 question multiple choice. And then the last piece is, now you're gonna have to actually do some audits. But you're gonna have to have somebody else that knows what they're doing critique you and fix your mistakes. So you can improve. So… you do an agenda. Here's the process of auditing. I assign… let's say I assign Sean to do the audit. I say, Sean, you're gonna audit the CAPA process. This is the person you gotta contact, go schedule that audit. So, Sean calls that person, gets some information, gets some records, writes an agenda.

Sean, give me your agenda, let me see what you did. I correct some things in the agenda, so next time, when you make an agenda, try this.

Improve this.

The next thing Sean does, he does the opening meeting with that person. I have him record the opening meeting. I give him feedback on his opening meeting and tell him to do some things that are better next time. I tell him what he did well, here's what you need to improve. Then he interviews the person. I watched the interview. I said, well, you missed an opportunity here to ask a really good question. We call that an audit trail. There's a trail that you could have followed, you missed it. You missed the marker. And another thing we do is we say, you know, you forgot to select that record. Those records, those are… those are… if you don't ask the person for the record, if you don't ask for, give me a log, give me a record, if you don't come up with a sampling plan.

You're not gonna have enough information to really get a sense of whether the process works or not. So there's an art to asking for records, there's an art to sampling, there's an art to interviewing the person. Sometimes it's just being silent makes the person uncomfortable. And then they talk more. Sometimes it's putting the person at ease. Sometimes it's being 6'6 tall and realizing that the short Asian woman will talk more if you sit down. When you stand hovering over them from 6'6", you scare the crap out of them. So, one of the things that I do is try to put the person at ease, and I work on my interviewing.

Now that I've gotten all that information, now I get to write it up as a report. I look at their draft report, I give them feedback on their draft report, and yes, I critique their grammar, their spelling, their formatting, their font. I critique everything! And they're like, why? Why? Why? And I'm like… You want to make money, make something that looks attractive, make it presentable, make it clear and easy to read, write factual statements. Don't tell me how you feel. I don't care how you feel. Give me facts. Tell me how to improve. Add some value. So, that's part of what's involved in giving feedback. It's not just, yes, you did this. So, I even see people that are critiquing, they use a checklist too. Yep, he did that, yep, he did that, yep, he did that.

You passed, because you did everything you were supposed to. I have no suggestion for improvement. That is the most useless coaching that you'll ever get. You go to a coach and they don't tell you anything to improve, that's useless. What did I do wrong, Coach? You did everything you were supposed to do. Sorta. And that's… that's the audit process. You… Agenda, you critique. Opening meeting, you critique. interviewing skills you critique. I did forget the closing meeting. So you have a closing meeting, you critique that, and then the audit report, you critique that. So each of the major things that they do.

You critique it, give them feedback, and they do that over and over again. How many times, if you're working for a notified body, they actually have a requirement? I think it was either 15 or 20 audits that I had to do being observed by another person to be allowed to audit on my own, and then another 15 or 20 before I was allowed to lead the audit.

We had a question about a notified… working for a notified body, from one of the people in the QA. She wrote, I think Romania, Let's see… sorry. I am an auditor for ISO 1345 from 2014. I work with LRQA. Unfortunately, in Romania, we don't have any notified body certification. Under what conditions could I work with a notified body? How do you get a job with a notified body as an auditor?

You apply. Like most jobs, they have an application process. You need to keep track of every single audit you've ever done. That may seem insane, but they actually ask. And they will go back and verify some of those audits, just to make sure it's all truthful. They're looking for not only that you have the experience, the minimum for BSI was 5 years of actual design experience in a medical device. And also, during that 5 years, you had to have quality and regulatory experience, and you had to have auditing experience. I had design, I had operations experience, I had auditing experience, I was quality, I was regulatory, and it was…

more than… it was just over 5 years for medical device, but then I also had my pharma experience, which they gave me some credit for, and I had an engineering background, so I had relevant education. I had relevant work experience, and I had the number of years.

But the first thing they said to me is.

All your training is self-training, or, like, courses that aren't accredited. You're gonna need to take an accredited course, so… If somebody wants to go work for a notified body, my course will teach them how to audit, but they're gonna have to retake an accredited course in order to have the official certificate for working for a notified body. They will be bored out of their mind in that class, but they have to do it, just like I did. I sat there in the class, and the person that was instructing, he wasn't even qualified as a notified body auditor, and a lot of the information he presented was incorrect, but it was an accredited course.

And I just, like, halfway through the course, I was like, do you want me to cover that part? Because that's wrong. He's like, oh, you work as an artified body? Yes, I'm working for BSI, and I have to take this course, but you're getting it all wrong. I'm like, so I actually took the procedures we had, and I'm like, here's the procedure, this is what we fill in, this is the rules, this is where it says it. Okay, you're not allowed in any of my classes. That is, that is, that is… Yeah, the next week, I was teaching the class.

You're right. We have another question. Companies tend to place more trust in auditors who are certified or affiliated with recognized firms. How can an independent auditor increase their credibility and perceived value? You just mentioned one way, which is to take the accredited course. Obviously. What else?

Writing down what you've done is key. It's… like, when you take this class right now, this… it's a… it's a… it's an online meeting, but you can record this on your resume as continuing education. Now, not necessarily ASQ and RAPS will consider that to be a valid course for their continuing education credits, because Sean isn't giving us a quiz, or I'm not giving you a quiz. But if I gave you a quiz, and I gave what the curriculum was. You'd get continuing education credits for it.

Oh, maybe I should start doing that.

You make it very popularity with that decision, but… But that's how you get your continuing education credits. You have to have, this is what I took, this is when I took it, this is what the agenda or curriculum was, and here I… there was an actual quiz at the end, and they gave me a certificate. So I attended something yesterday that you put on, on breakthrough devices in Europe. you're gonna give me a certificate for that? I gave you your name. If you gave me a quiz, no question I could get some continuing education credits for that. I personally am not going to work for a notified body. I teach the Notify Body Auditors.

Hmm.

I've taught BSI, I've taught DECRA, I taught TSC in, In Turkey, I've taught SGS, so I've taught 4 different notified bodies. Leo and his partner, they teach the FDA. Like, some of us, that's what we do. We teach so much, and we're subject matter experts in an area, we teach the experts. But… I still have BSI auditors saying, where are your credentials? Where did you take training on 1345-2016? I see the 2003, but I don't see the 2016. And I've actually got the record, I'm like. here is the day that the standard 2016 dropped, I was at this McDonald's, I downloaded this standard, I read it from cover to cover, I updated my slide deck, and later on that day, I taught a 2016 Lead Auditor course.

And since then, I've taught the following lead auditor courses, and I've done the following audits. And for… my list is ridiculous, but I have to actually… defend myself all the time against the people I used to train, showing them, yes, I have the credential. So you never get out of this. You always have to keep your resume up to date and say, I have this, and like, it's 40 pages long! We're gonna have to get a hand truck to bring it in.

Can I have a… can I make a comment on that? So, there was a… an audit of one of my clients, and they also… the auditor says, well, anyone can fabricate a resume, and therefore it doesn't, like, it didn't hold up to muster, which was silly. But… so that was one part of it. So I do have certification, et cetera, et cetera, and qualified, you know, I don't even know why certification body does the… main ones, but… My question, actually, is… On that same topic. So we have standards, we have lead auditor training on them. Now, what about… all of the regulatory requirements that are embedded in ISO 13485, so Canada, US, Japan, Australia.

I'm having a difficult time trying to justify that I have experience in all of those, because there is no certification that I'm aware of. to be able to back all of those regions. I have EUMDR, there's an actual course on that, I took that, got another training to do lead auditor for that.

But when it comes to all these regions.

It's… hard, so I would love to hear how to deal with that.

You… you gotta be really… Detailed in the… and you have to put specific information in there about what your training was and what it consisted of. Even when you do self-training, They like to see exams that are graded by another person that has credentials. But, ultimately, The trainer… comes under scrutiny as well. And so I'd say, well, okay, BSI. Who trained you? Oh, Greg Jones did. Okay, where did Greg Jones get his training? I don't know. Exactly my point. Somebody is the top trainer, the first person that ever offered the training on that, they're the top. Where did they get their training for? They self-trained.

So, if… it has to be valid to do self-training, but you have to document it in a way, and… The date you did it, what reference standards you used, how long you took, and… what you did with that. Like, you can read it and forget it, but did you use it? Have you… have you been through multiple audits of that topic and been successful? So when I start pulling out my audit history, like, okay, he's been through more MDSAP audits than I have, he might know what he's talking about. And that's what you end up with, is it's piling on the… on the boilerplate, I have all this experience. But when you're new.

One of the things that can help you is, Documenting what you did, what reference standards you used, what dates you did it, and then asking somebody that you know somebody in this group that knows that topic, ask them, could you send me a quiz? And I'll take it, and you grade me on it. And you laugh, but that's exactly how they did it in 1994 when 9001 came out. There were no people out there that were trained as ISO auditors, so how do you become qualified? How do you become independent? they had to cross-audit for each other. So up in the New York State area, they had a whole bunch of auditors that got together as a team, and they would audit for each other, because there were no independent auditors. There were only, like, 5 or 6 of them, so they each audited each other's company, and that's how they first did

independent auditing, was they just knew other companies nearby that NAD9001, and they would audit each other's companies to achieve that. And we can do the same exact thing. We can give a quiz to somebody… it doesn't have to be long, a 10-question quiz. BSI's quizzes aren't any longer that they do internally. And don't make easy questions, make hard questions, make them look stuff up. have an actual record of it, and so we've started putting everything on Gravity Forms on our website, and it gives the… here's what you answered, here's the correct answer, it allows you to print it out, and then we'll send you a certificate. And now you have your training certificate.

But that's how you have to do it, and it's… there's not a magic wand that all of a sudden you're certified. Unless you take an accredited course, pay a couple of grand, and get a piece of paper, and they… they can't even do that for… For the Canadian regulations, for a long time, there were only two instructors in all of Health Canada that were allowed to do it, and you had to be sponsored by a notified body to go take it. And you had to wait in line to get on that list. And if you failed it, you paid for it yourself to go the second time.

It sounds a little…

lucky enough to be able to do that many, many, many moons ago, when I worked for a notified body.

It sounds a little bit like being a marketing consultant. In… in the sense that, you know. there… there is education, there are qualifications, but at the same time, anybody can say, I'm a… marketing consultant, or I'm a business coach, or I'm a… So then, it's the same… standard, well, like, how do I know? How… you know, where does your credibility come from? And… and how do you demonstrate that? And so it… it comes down to, a topic that we're going to cover on the 18th, which is honing… honing your MedTech message, and how to be memorable.

Because I think that, you know, for everything that… all the examples that Rob has given, what is the thing that's going to convince the people that you know what you're doing? You know, that… that you're qualified to come in and audit them, and what… what's gonna drive that over the, Over, over the, over the finish line. why are they going to pick you over the next auditor? So, the question that I have for you is growth areas. What areas do you see growth, software as a medical device, obviously, but, you know, by standard, or by, you know, MDSAP, or by, you know…

There's one that comes to mind immediately, and Matthew predicted it, so I give him all the credit for this. ISO 27001. And I'm sure Leo knows that, and maybe a few of the other of you know what this is. It's a security standard. It's not specific to medical device at all. But the Europeans are calling it state-of-the-art. And as soon as they start calling it state-of-the-art, it's gonna be on everybody's technical file, as we comply with this standard. And how are you gonna show you comply with this standard? The easiest way is to have a certificate. So I'm convinced that it's just a matter of time before Europe pushes the envelope far enough that people start getting the certificate to 27001.

And I think that'll be a game changer for security.

We have a BSI audit for one on December 22nd, certifying one.

There you go! But it's a baby… it's a baby 13485. It's just a… it's a… Yup.

The lead auditor…

And in fact, if I had gone through the trouble of preparing for 27001 a few months ago, I probably wouldn't have had a virus infect my computer, and have a whole bunch of my credit cards get compromised.

You need an RFID wallet. They're $9.99 on Amazon.

I have them, that wasn't the problem, but it's another story. But, yeah, I mean, you're never, you're never impenetrable for security. But… there are things I would have done differently, and I realized this… I mean, I have a cybersecurity team. Matthew is a cyber forensics graduate now, and so I have the resources, I have the knowledge. I just got lazy, like we all sometimes do, I mean… You know, does everybody lock their car door in Vermont? No, they don't, because it's a dirt road off a dirt road off a dirt road. Good luck finding my car! But if I lived in New York City. The locking the door would not be enough. I'd need, like, 3 other methods of locking up my car. And they'd probably still get vandalized.

So, you have to have security, and it's not just for medical device, it's for your whole company, you need security. And so I see that being a big area. What other areas do I think that are gonna be huge in the future? I think that clinical studies Are going to become the norm. Because all… everybody's doing a lot with big data now. I've gone from 10% of my clients having software to more than 50% of my clients having software. A lot of them are doing software as a medical device, using big data. When you start using that big data and developing your own algorithms, you have to validate the validity of your outcomes against clinical data.

So you have to have, here's my clinical data set to build it, and here's my clinical data set to validate it, so you need to understand what the rules are for clinical studies, and I see that being another area. the FDA has auditors that go out and look at the quality system, but then they also have, for pre-market approval inspections, they have a clinical team that goes out and validates or verifies that the company is following all the steps that they're supposed to do for a clinical study in 21 CFR812. And for ISO, it's ISO 14155, I believe. So those are the standards that people need to start learning if they're getting into

Validation of clinical data. or validation of, algorithms that are based on this clinical data, and so I see that being another big area for auditing. I don't know anybody except a handful of ex-FDA people that audit clinical data. But CRAs could do it, CRCs could do it. Carrie, are you the person that can do it? There we go! Now I know somebody! So, that's a really powerful skill.

What about combination products? Susan's on the call. I would think that combination products would be particularly tricky, since you're at pharma and… Device, and they notoriously don't get along, or speak the same language, or, cooperate with each other in any way, except for to put the drug in the device.

Yeah, they are definitely hard. I actually built a lead auditor course for a pharmaceutical company teaching them how to audit drug-device combinations. So I've done that. I worked for a decade in pharma, so I had the skills to do it. But it's definitely harder. What a lot of people don't realize is that even though the FDA inspectors will audit you to 820, And they'll look at your quality system being compliant with 820, the regulations encourage you to follow 9001, they don't require it, but they… they're all written for 9001. And they actually have, in 210 and 211, specific requirements in there

for the same things that you'll find in 820. And it will be 1345 in February, but for example, we found some nuances in labeling. So I could create an entire course on how to audit labeling, and if you added pharma labeling to it, it'd be 3 times as long.

Because all the specific requirements for the pharmaceutical labeling that are slightly different.

So when I come to a pharma company, and they're doing combination products, like, well, here's what the device side's gonna audit you to, and here's what the drug side's gonna audit you to, and they're like, well, which one do you follow? I'm like, yes. And they love that answer. Yes, you have to follow both of them. And you have to show evidence that you've audited to both, and you have to be compliant with both. Like, why couldn't they just write one that does both? Like… Yes, in a perfect world, that will happen. In my lifetime, not at all.

I can't seem to get my hand to raise, so I didn't know if I could make a comment.

Yeah, go ahead.

Yeah, sorry. So… when it comes to the labeling, you also have to be very careful, because jurisdiction by jurisdiction, the expectations around that labeling are different, and in the United States, under the streamlined approach, there's also exceptions built into the labeling regulations. Like for UDI, and the NDA, you know, when you're… you know, if you look at those exceptions, and they just came out this year with an updated guidance on it to try to make it clearer for industry. But, they don't always make you do all the things that are in the drug regs if you have a device-led combination product. And if you have a drug-led combination product.

You, if it's a single entity combination product, you don't necessarily have to have all of the device information, you need to have… for single entity, you're gonna have the drug information with the applicable device information built into your drug labeling. So it's not… just be careful about over-complicating it, which sometimes I think people do. And then it gets even more complicated when you go to Europe, because in Europe. Right? You're, you, you know this because you're from Notified Body, that notified body approach is quite different than it is in the U.S, and if it's a medicinal-led combination product, but it has a device constituent part to it.

then the EMA and the National Competent Authorities don't want to see any of the device information on that label outside. So, it's… it gets, you know, I'd just be really careful with that one. Anything about labeling, as you said it, I was like, be careful.

Join us in January for a special with Susan Needle leading on combination products.

There you go.

And there's a difference between what you teach people. And what's applicable to the product that they have. oftentimes, whatever they have is… there's an exception. There's always something that's an exception. Especially the Class 1 devices that are sold in massive volume. I can't believe how many exceptions are in the Class 1, like, oh, you have a Class 1? Okay, that's gonna cost you extra, like, why extra? It's a Class 1, like, because there's so many exceptions, it's gonna take me longer to look them all up.

So, Tiana had a question that, I don't think we answered. I was on the first part of the call, so I apologize if I missed this, but Rob, what do you think about audit… about the audit model and task list? Like it? Hate it? I'm not…

Task list?

The audit model coupled.

You're talking about the MDSAP audit approach?

I guess so.

So, yeah, okay. So, MDSAP, I actually… I like the concept of the seven chapters. I like the concept of the way… because they basically took what the FDA was doing and added to it some others that I thought were really important. And I liked the concept of having it all harmonized. What I disagree with is the way the notified bodies have executed it, and I know they didn't have any say in it. The FDA said, this is what we want, and we'll take our ball and bat and go home if you don't play our way, so they did it the FDA's way. But… The way it's evolved to is they have so many country-specific questions that they have to document the answer to, that they don't have any time but to ask the question in the checklist, write down the answer, and generate a report.

They do look at a few records, but they've reduced the amount of records they sample. They've increased the duration of the audit, and they ask a bunch of useless questions.

Hmm.

When you… If you're… first trying to get MDSAP certified, and you haven't had it before, you kind of have to do that approach to the auditing, but I've learned that it's really hard to do it in 5 days. It's extremely hard, and you can't get distracted, and you can't go off on tangents. You have to stay on task and do your audit, and get it done. And it will take you all 5 days. And it won't be fun for anybody, and it won't seem to add a lot of value, except you're gonna get your MDSAP when you're done, because I'm gonna make sure there's nothing missing. That's your job, is to make sure nothing's missing so you get certified.

But your second internal audit on MDSAP, don't do that again. You've already done it. All those questions have been asked, you shouldn't ask the same exact questions that you asked in your internal audit, and they were asked again in the NDSAP certification audit. Now ask new questions, so you can follow that 7-chapter framework, but don't go through every single country-specific question again. If they modified it so it's now version 10 instead of version 9, look at what's different between 9 and 10, And ask those questions, because they're new. Just like you would a quality system, have you updated your procedure? No, we haven't changed that in 4 years. Well, let me read that again, because I'm sure it'll be really entertaining. No! Write down there's been no change to the procedure in 4 years. Maybe the observation is, maybe it should change, because it's been so long.

But if there's been no change, you don't have to read it again. And people do. They, like, give me the procedure, give me the log, and I'm like, why did you spend 2 hours on design controls looking at design changes when they had no design changes? And the person says, because the agenda had 2 hours for it. I'm like, sometimes you have to use your head and realize if there's no records, I'm gonna write down there were no records to audit, and I'm gonna shorten that duration, I'm gonna spend more time on things that are important. So, what's… what's changed? What's new, what's important? Not what the agenda said.

Okay. Not what the checklist said. On that note, who wants to become an auditor? Show of hands? No? Rob, I understand that you're gonna be offering a new course that you're developing?

Yes.

Real soon, tell us what it is, tell us how to join.

On our website, MedicalDeviceacademy.com, if you just type in Lead Auditor. There's a lead auditor training course there. The actual slug is lead-auditor-training. It's 32 lessons. It's all pre-recorded, all self-paced, But you also get all the updates that we do in the future, for free, so as new standards come out, we'll update the videos, update the quizzes, so you get those… you don't have to pay again. When somebody says you have to take a refresher because it's been so many years, there's your opportunity, just take it again. You get the free certificates. We're making all those certificates automated through Gravity Forms.

And… This course is a little different in that we put a lot of content in there just for software. So it covers all the basic stuff that you would see for most medical devices. But it has, 62304 training in there, it has 62366 training in there for usability, it has, AMI TIR57 in there. It, of course, has the auditing standard, which is 19,011. So we put a lot of technical standards in there. It's 14971 is in there. We've got, 1345, of course, so you get a lot of technical standards that are covered in detail with quizzes. And then we go in depth.

And we even have some role-playing videos where you see, like, here's an auditee, here's an auditor, what did the person do right, what did the person do wrong? So we can actually demonstrate good and bad practices for interviewing. How to write the… the nonconformities, how to grade the nonconformities. So every little step of the audit. We even covered things like logistics, like. When you're scheduling an audit to go see a supplier, maybe you should take into account that it's 3-hour drive from the airport that you picked. And build that into your schedule. Otherwise, you'll do like I did, and end up getting to the hotel around 3.30 in the morning, and they won't even let you in, and you gotta go audit tomorrow morning at 8 AM, and you're gonna be practically unconscious.

Okay, well, we're out of time. I want to thank everybody for joining. I want to remind everybody that next week, we're going to be focused on human factors, since it's Human Factors Month. I'd like you to all show some love to the article that I posted today on human factors. Where I interviewed 8 human factors experts, to help pull that together. I think that, human factors is an important topic, and I hope that you'll all Join us for that next week. Rob referred to usability standard, and then the following week, we're gonna be, how to make your MedTech message memorable. And that's gonna round it out for the year. I'd like to encourage those of you who are on the call that would like to be speakers in January, February, March.

to get on my calendar, to ping me, to let me know, hey, Sean, I have something that I'd like to talk about, you know, I'm… friendly enough, and, you know, generally try to be responsive. If you, have any other questions, please feel to reach out to me. Or Rob, my final question for Rob is, Rob, as I sign up for a course, and I run into questions, can I email somebody and get an answer?

Yes, we've actually delivered it using Monday.com. And every single person on my team will have access to the individual Monday project for the student. So you can put questions right in there, or you can email us, and we get the notifications, and we can help you out, and you can even schedule appointments with me if you have any problems or questions. In the beginning, with the first, you know, dozen or so people going through this, I'll probably actually want to… I'll be calling you and saying, hey, what did you think of that? I see you finished that, what did you think of it? all this material is new. I went back and I took everything that we had from four different courses, and I was like, what did I like, what did I not like? And we're trying to re-record everything.

Okay. Well, that's great. So, we're gonna jump. Stephanie, I'll email you, directly, and, you know, if we didn't get to your question today, I'm gonna forward all the questions to Rob, and maybe he'll follow up with you, okay? Banks.

Bye-bye, everybody! Thank you.