The MedTech Audit Before the Audit

Hey, good morning, everyone. Welcome to Friday in Focus. Today's guest is Danny Crew, who is going to be talking to us about audits. I'm excited for that. While we're waiting for people to join, I'd like to make a few announcements about what's coming up, if I can get show, we have… next Friday, I'll start with next Friday, Etienne Nichols, some of you may have heard of Etienne, and, he's gonna… he's gonna be talking about marketing, and, one of my favorite subjects. But… You know, Greenlight Guru does a great job of marketing. I was one of the few companies that I see doing marketing really well, and so…

That's gonna be great. This one here, next week, next Wednesday, is gonna be super interesting. We added another speaker, Matteo Moso, from, Donowa, who is the person who actually spends the time visiting the sites. And So, if, you know, I wrote an article about it yesterday. It's like, when we moved to Pennsylvania, and we didn't know where to find contractors and how to, you know, get things done in our new state, imagine trying to run a clinical trial In Europe from the United States, and how are you gonna find a CRO when all the websites say, we're great, you should hire us, here's some testimonials?

we've got a thousand years of experience, you know, and so these are, you know, topics that are near and dear to me, but, you know, talk to the people who are doing the work, and this is your chance to talk with Daniela and Matteo, who will be running your clinical trial. And, you know, vetting… vetting a CRO is hard to do. Particularly from publicly available information. So that's gonna be worth attending, and they're gonna be talking about everything that goes wrong. And Show, and then the, the business case for quality. This is really, an interesting thing. The emphasis from the FDA on quality is, increasing, and they're talking about having a culture of quality. Well, what does that mean?

Michelle and Tiana Benson are gonna, enlighten us. And help us understand what exactly the FDA is talking about, what they expect, and more importantly, how they measure whether or not you have a culture of quality, so it's not just because we said so in our mission statement. Couple things. I did a post this morning. I'd love for you guys an article on MedTech Leading Voice about this new, 9… 9-day thing in California. If you're in California, anybody in California, say California, Telephone!

California.

Yeah, there we go. They're doing, this 9-day roadshow, and it's free, and it's interesting, and, we'll put a link in for that. At the same time, I'm gonna be, doing something really cool with Project MedTech. We're gonna be live creating a sort of living document at the conference, where we're gonna be interviewing people, we're going to be taking photos, we're gonna be doing all this stuff, and we're gonna compile it all into a yearbook after the fact, right? So, people are going to have lots of ways to contribute to this thing. And it's really, you know, it's a visibility issue, right? So how do you take what happens at a conference and give it a life after the fact?

Raps is coming up. Team PRRC. I'm planning on attending both. I might be attending, the MedTech Strategist Innovation Summit in San Diego in November. it, you know, that my plan is to want to go. And then, you remember a few weeks ago, we had… PHMTI was here. I'm going to be speaking at that. So those are the upcoming events. So, anybody that's new to the Friday in Focus call, we meet every Friday. It's completely free to attend, there's absolutely no charge, and we encourage participation. If you'd like to be elevated to panelists, please raise your hand electronically, or message Francesco, and he will elevate you to be a panelist. We encourage

not disruption, but, lively engagement among the panelists. Questions. We don't want this to be a one-way lecture. What we want to do is to create an environment here where, yes, we're handing out information, but we're also exchanging information. And I think it's one of the few places where that happens online. We do have a private membership group where I'm just gonna say you get lots of special attention if you join. It's only $3.99 a year. If you'd like details, write membership in the chat, and I'll be more than happy. To follow up with you, and have a conversation and see if it's right for you, see if it makes sense for you, or for your company, or, you know, something like that.

Again, lots of special attention, lots more speaking opportunities. So, with that said, Danny, did I cover everything? Did… oh, oh, I didn't ask. Is anybody here from Canada to hear Danny? Please write Canada in the, In the chat. I'm gonna drop links in for these events, so if you see a bunch of links from me, you know why.

I just sent them all. They're all in the chat.

Okay, thank you.

Okay. Do I should share now?

Sure.

Alright, try to share. Okay, share now. Okay, everything's visible, everything's good. Okay, well, I appreciate… thank you so much, Sean, for inviting me to be able to share some information about auditing. Auditing is my passion. It's what I've been doing for over 30 years. I, have had, very good experience, very good coaching and training in auditing. I worked with BSI for a while, as an external resort, so let me just move my slides, if I can just… Okay.

So, I guess, Sean, you're gonna start with the polls?

Sure. So, to encourage you guys to participate as actively as possible, just go ahead and drop a letter in here, or write what you do, so that we can understand who's on the call today, and Thank you, and You know, we do have a lot of consultants and service providers, and what I like to point out is that the easiest way to get new business is when another consultant walks you in the door. And so, keep that in mind. If you need an external auditor, Danny, I'm sure, will be happy to talk to you. And I'm gonna leave it at that. Looks like we have a lot of people, in,

regulatory consultants and other areas, A and B, yeah, just great. Keep the responses coming, and that helps us understand who's here, But, yeah, absolutely. We got the whole gamut going. Danny, back to you.

Okay, so a little bit about myself. I'm the founder and president of Docuses Corporation. It's been around since 1994. I was an external resource, as I mentioned before, for BSI. I still, for medical device audits. I worked for them for 2004 to 2019. I did most of the audits in Canada, but there were several in the United States that I was doing as well. My company… The purpose of my company is to help clients comply with requirements, and I specifically work in medical devices, but I also do work in aerospace. I'm also an AS auditor as well. But I've worked with many clients with regards to ISO 1345, MDSAP, EUMDR,

Also, software as a medical device. My niche is Canada, so when I have things going on where, you know, issues with FDA, I always go somewhere else, and I've partnered with some people who are even on this call. I've sent some of my clients to other people, because that's not my expertise. I try to stay within my expertise. But it's not just me alone. I have several people working for me. One of them used to work for Health Canada for 5 years. He was… he was a subcontract, Class III evaluator for Health Canada, so he's my regulatory expert. And he has PhD in biomedical engineering. And the second person who works for me is my QMS expert. She also has PhD, and she works as a fractional quality manager for several of our clients, primarily in Canada.

One of the things that happened about a year and a half ago was that I had one of my clients who has software as a medical device, and they said that they wanted to go for ISO 42001. I have a little bit of background with 27000, which is, not similar, but it's in the same group, and So I took… I took a lead assessor course on 42,000, and I helped them by doing the internal audit for 42,000. It was quite an interesting experience, and this was one of the first companies, U.S. companies, to be certified to 42,001. Besides that. For the last 9 years, I've been a lecturer. I created a course at McGill University in Biomedical engineering, the master's Translational Engineering, and I teach that every Thursday.

during the fall semester, and I also invite, experts, like maybe those who are on the call, to provide some, some lectures on different topics. So I've had people talking about 60601, risk management, biocompatibility, and… because I'm not the expert in it, but I've invited I've invited people that I know, my contacts, and that has made the course A much better course, and something that there is a waiting list to get in. Okay. So… Audits… for me, the audit's process can be divided into three. What you do to prepare? What do you do before the opening meeting?

Second part is, how do you actually run the audit? And the third part is, how do you get the information, or the report, or the results of the audit to your auditee? To… to your clients. And… if you do have any audit situations that you want to… you want us to make comments about, or you have any comments, we can always… you can always put it in the chat, and… and I guess Sean will… Well, let me know if there's anything to talk about. So, before the opening meeting, For me, preparation is the most important part of doing an internal audit.

And… It's… if… it could make or break an audit. So, what are the things that I want to know before the internal audit? I want to know about their previous internal and certification audits. What am I looking there? I'm looking for the weaknesses, I'm looking for the audit trails, I'm looking for what was audited before, what you know, what… what were the results? Were there any… Issues that were repeating. And it gives me an idea of where I want to focus. I also look at, if they have multiple products, which products weren't covered. If they have, you know, if they have different production areas, I want to focus on the areas where maybe there were more complaints. It's, it's…

It gives me an idea of where to focus my audit. I'm also looking at information outside the quality system, so I will go on to the different sites, the FDA sites, Health Canada sites, looking at also the MOD database, and looking at what are the weaknesses. Also, looking at their website. What are the claims that they're making on their website? Sometimes. The scope of your audit is different than what's on their website, so it's just… giving me some more information. I'm trying to simulate what I learned as what auditors do during an audit. I want to simulate what What they will get when they get a third-party auditor.

I also want to review briefly the quality manual and key procedures. I want to know I don't want to read it, you know, I will read procedures or the manual during the audit, but I want to already have a good idea of what… of how their process works. And by… by reading through and maybe taking notes in terms of areas where I think there may be issues or things that I would want to explore further. I'm… you know, it's… I think it's quite important. And the last thing is the management review. So the management review is supposed to give you a photograph of how the quality system's working. Gives you the health of the system. You see what are the complaints, what are issues that happen in audits, you know, if there are any resource issues. You get an idea of how the company's working.

And… and what are the… what are the challenges the company has, and what are the strengths of the company as well. So, it's… it's really important for me to get all this information before I start asking any questions. So, as I mentioned before. not everybody looks at the regulatory requirements, or the regulatory clearances, and the approvals, and any adverse events. It's something that… what I've seen in other companies, and I've seen this because, you know, sometimes I get called in to work at a company, and they had some issues at their external audit, and they said, your internal audit process is not working, or they decided that they wanted to make a change. And what I see sometimes is people doing checkbox

Checklist audits, checkbox, and… Some of the times, it's by people internal to the company, and the problem is, when you're internal. Sometimes you don't… You know, it sometimes creates some friction with your colleagues if you're doing an audit on your colleagues. And sometimes, you know, there's something called soft grading, meaning, oh, okay, if you fix this, I won't report it, or even though it could be a systemic problem. So… I'm not saying that happens all the time, but I see it happen quite often, and also as… because I'm also a third-party auditor, I see it often as well, where it's basically… it looks like they didn't spend very much time doing the audit, there's not much…

Objective evidence, and it's… it's, it's really… it's really not, I don't think they serve the audit profession very well. Okay. So, as we mentioned, Management Review is a map. And… before you go into an audit, if they have had issues with resources in terms of training or competence, or if you look at their Kappas, and you look at the root causes of their Kappas, and what does the root cause say? It says, well, we're we don't have enough resources, or we don't have time, or a lack of training, or something. That kind of points out that there are some weaknesses in the quality system, and something that…

I may want to go a little bit deeper in. Looking at the quality objectives, well, are they really… quality objectives? Are they KPIs? Are they… Are they giving you an idea of what the health of the company is about? And it's not something that I would tell them, this is not a good quality objective. It's up to them to decide what objectives they want to have, but I just… the companies are really serious, really have very good objectives, very good KPIs, and… You know, I… I… from seeing what their quality objectives are, I can see… how much of an investment are they making into their quality system? Are they really using their quality system to improve their company?

And there are some companies that are just going through the process, and they're just… just trying to conform, and it's just a necessary evil. And there are some companies that are using this for improvement. And… Those are the companies that I usually do a lot of work with. Looking at complaints. it gives me an idea of what products are… are… have issues, and… and by doing that, I may want to review the design development files for the comp… for the issues, for the… for the, products that have issues, or do anything further, you know, go into further audit trails in terms of production of those products, and looking at how they've handled complaints.

Supplier performance. You know, sometimes… companies treat suppliers all the same. Even though the risks related to their suppliers and the controls related. have to be different because of the complexity and because of the risk involved in terms of the suppliers. So, looking at how they handle suppliers, if they've had multiple nonconformances and issues with their suppliers. Well, what additional controls are they putting in place? Are they doing audits, supplier audits? It doesn't mean that they have to, but when there are critical suppliers, it's usually a good idea to… because they could have a big impact on your product.

And recurring problems. So, looking at the… if you're looking at the history of Kappas, and you see the same things popping up. You know, it's not just a Kappa pro… it's a systemic problem. It's something that we really have to… to, to address and see if there's something that we could point out. So the purpose of the audit for me is to help the customer. to help my customer. It's not there to load up on nonconformities. I don't usually get bonuses for issuing more nonconformities, but it's to point out things that maybe… Are not so obvious, or maybe because they are…

You know, they're involved in the day-to-day, and they don't see, sometimes, the big picture. So, I guess we're… At the point where we'll talk about the live poll, before an internal audit, do you check information outside the quality system? So please provide your answer in the chat.

Yeah, so last one, we had 12 people answer. This one, I'd like to get 20 people to answer, so please get your little fingers typing. Thank you. Luke, you're the fastest on the keyboard. And we see them coming in. I'm going to tally them, so leave this up for a minute. And, feel free to talk amongst yourselves. Just kidding. And, no, Yeah, so I won't, hold you up. Any longer, Danny.

Okay, great. So how do we run the audit? So, the audit could be run either you know, in the last few years, I've been doing many more, software as a medical device audits, and these audits are handled usually remotely. And… But in the past, I've been always going on-site. But remote audits… present a lot of challenges, and maybe we can discuss, or maybe that's something that you can also share your experience of what… how… Remote audits become a little bit more challenging in some respects. So, going on. How do I run the internal audit?

So… Most of the audits that I do are MDSAP audits, because Canada requires, if you have a medical device, a Class 2, 3, or 4, you want to sell your device, you have to have a MDSAP audit, so I usually do MDSAP audits. I also do audits just for ISO 1345. for contract manufacturers. And one of the things that maybe I do that's unique, that I don't see too many other people do, is that because I have an aerospace background, I sometimes go to contract manufacturers, and I'm able to do both AS9100 and ISO 1345 audits. at the same time for a contract manufacturer. And that's something that I haven't heard too many people do, and it's something that my clients

say that they… they save in terms of time, and also in terms of cost to get that done, because they're able to, you know, when you're looking at a management review for… for 1345 and AS9100, there may be differences, but you only need to look at it once. Or the Kappa system as well, and other things. So… so it's, it's… it's… it's something that may be a little bit unique that I offer. But before the audit. In order to have a successful audit, I have to provide them with an agenda. Why? First of all, for them to know the availability of personnel. Not everybody will be available at all times. I send them

after I see their quality manual, and I see the issues that they've had, I have a good idea of how long I want to spend on each item. It doesn't mean that it's going to have to be exact, but I build the agenda. And I want them to confirm, and sometimes they ask to make changes because of availability, which is fine. You want to limit the amount of changes that will happen once you show up at the audit. Because then that could cause some, some issues where you may not have enough time to cover the area that you want. So, send it out in advance as much as possible, usually about 2 weeks. One week before, it's usually good enough.

The opening meeting is just an introduction. Explain the scope, explain the objectives, criteria, logistics. Making sure that there's a guide for If it's… if it's a remote audit, you know, that will take a break every 5… every… every hour for 5 minutes, so that we can all stand up. And, And, sometimes… sometimes I forget about the time and just ramble on, and… and it's… and it's about… Could be about 2 hours, but anyhow, it's important so that everybody knows what the rules are and what we're going to be doing. But really important for me, and I guess for… all auditors to audit senior management. Why? You want to get a level of commitment. You can't say that you audited Section 5 of the quality stand… of ISO 1345 if you haven't

really spoke to senior management, and it doesn't have to take an hour. Usually, it's between… it's usually about 15 minutes. Getting a sense of what their participation level How much they know about the quality system, what's important to them, how is their company doing, how do they do in terms of commitment, communication, what are their customers telling them about… about… about their company. It's getting a higher level view, but getting a view that, really, you want to be able to go talk to the senior management at the end of the audit and say, hey, yeah, you got a great company, or we spoke to some people and they told us about this and this and this. And it gives them some feedback in terms of how…

their system's working, and it develops a certain relationship. Some people, some senior management don't want to get involved. Because… They're afraid to get involved, and others… really participate. So the ones that really get involved, you can see that their systems are… Sometimes better, and they have They make decisions faster because the senior management is involved. Follow the trail, so… Audit trails are really important. As he said, when I'm looking at the management review before I go, you know, I have an idea of… I have a sense of how the company's doing. And… you know, I will use the trails, or looking at, internal… looking at what

problems happen during the internal audit, the previous internal audit, whether they've taken Kappa and whether it's effective or not. It's… I look at customer complaints and try to tie it back to design development, or to production, or inspection, or test. You know, those are the type of trails that we try to do. And… it… It usually works out well, because if they've done their job, they're very happy to show me how they've corrected things, how they've done training, how they've… how they've resolved issues, and it really shows good engagement and a good… and a good, quality system. It's not there for show. It's not there at a check-the-box type of quality system.

The other thing that we do is, we don't want any surprises. You know, I'm there… for the customer, I'm not… for a third party, and I like to tell them about what's happening, and of potential nonconformities. I will not tell them it's a nonconformity. I would say it's a potential nonconformity, because maybe I didn't talk to the right person. Maybe… maybe… there was some miscommunication. Maybe there's some additional information that was not available, but this gives them at least a chance to confirm, yes, and I'm always looking for agreement when it comes to nonconformities.

So Danny, we have a few questions and comments, people complimenting you on the thing. I want to say that the slides will be made available after the fact, as will the replay, and all of our replays are available. on, on YouTube, very easy to find, MedTech Leading Voice Exchange. Or something like that. Fra, if you could put the link to that in the YouTube replays. So how do MD… How to do an MDSAP audit? Should we follow only 1345 or MDSAP guidance document? Should be followed thoroughly? Documents to generate? Documents to generate for MDSAP Audit question mark.

Okay, well, you can do it either way you want, but the way that I do it is I follow the MDSEP Audit Approach document. Which has the different chapters, like management, purchasing, design. And, production service provision. And I basically try to simulate what an MDSAP auditor does. Now, that's me, because I do this on a regular basis. I, you know, every week I do audits. So it's… it's very familiar to me. I under… I know all the tasks. I've been involved with, some clients where they've been audited by MDSAP auditors. And I think the biggest benefit for my client is if I simulate it, and I try as much as possible to reproduce what

their third-party auditor can do. And not everybody can do that. It takes a lot of… Practice and experience, but that's really… That's rea- for me, that's really the best thing for my client, and that's what my clients ask me to do.

Okay, thank you. Another person says, I have run remote audits as an auditor. not for medical, but, ISO 17025, which I have no idea what that is, and I find there tends to be more of a barrier between the auditor and the auditees when remote, which takes a larger effort to bridge than when in person. What do you think about in-person audits versus remote audits? How do you feel about that?

Well, there's obviously an advantage to be on-site. But if you're… if you're auditing something where they're going to be showing you their screen all the time, and you're dealing with a remote company where nobody's in one one space, and everybody is spread out, then a remote audit can be done. I mean, there's no benefit to be on-site, but if you're auditing, like, calibration systems, like 17025, or you're auditing, aerospace, and production activities, well, it's obviously better to be on-site. It's recommended to be on-site. So what have happened in some cases is where I've been on-site for one day and covering the administrative aspects, like CAPO, or management review, or training, where it could be easily displayed on the screen, I would be remote.

But, I agree, there… You know, the human contact, you know, the, it's easy to mute somebody when, when, when you're, When you're remote. But it's much easier to deal with companies when you're not remote. But for something like software as a medical device, or some companies that actually operate remotely. It's… doing it by Zoom is, is, is, is, is good, and it works well. You're, you're… Okay, go ahead.

You can go on.

Okay, so we're looking at tracing requirement N10, so this is… Especially when we're talking about the lifecycle process for IEC 62304, we want to see design development… we want to see software development plan, we want to see… all the different software design documents, SRS, software requirements specification, all the documents, and we want to see the progression. what I usually do. is I usually select a requirement, so it's a requirement that starts usually in marketing requirement specification. I want to follow it through the SRS, through the design development, looking at if they're using Agile, looking at sprints, looking at

at how they did the sprints, any sprint reviews, going through testing, so usually testing is part of the sprint, going through all that activities. And I want to follow through from there all the way till the time that it's been tested and validated and released. And that is usually a good exercise. Obviously, you don't have a lot of time to look at all the processes. you know, all the activities that they're doing, so going through it horizontally, starting from the requirement and going through the end, and then looking at their matrix. They usually have a traceability matrix, and being able to find

you know, make sure that their traceability matrix is complete. And sometimes what we do is we say, finally, we find things at the end Sometimes validation, sometimes there's stuff missing, or, you know, there's some loose ends, and that's something that usually is beneficial for us to tell the, you know, the people who are in design development. you know, maybe they have to, firm up some areas. I'm not there… I'm not a coder, I'm not a software developer, I'm just looking at the system and looking at how… And their documentation, and how their system works, and hopefully bringing some value in terms of, of, what, what we can do to improve their system.

And the thing… the funny thing is, usually, when we start, software developers Where the people who are involved in this are the most reluctant to talk to me. The most reluctant. And at the end, they say, is that it? You know, I was, you know, I was enjoying it. And… And, you know, if you're working from a checklist. you're going to be a robot to them, you know? It's going to… it's going to create issues in terms of… but if you… if you seem to have an idea of how software development works. And you're asking pertinent questions, and you're asking them to show you how they did this or that, or their cybersecurity controls, or other things, you know, they see that this is an intelligent conversation, and actually, that's what an audit should be.

It should be a conversation. It shouldn't feel like it's them versus us, and that's really the goal that I try to… To, that's really my goal, is to make it a conversation. Okay. So, just recently, in, I believe it was August 3rd, MDSAP came up with a REV11, There weren't so many big changes. The biggest change, at least in my view, and I didn't really… you know, update all my documentation yet for it, because it just came out a little more than a week ago, is the device cybersecurity. So, there are so many you know, it's something that's changing all the time, and I believe in February 2026, the FDA came up with new requirements, or guidance documents, and…

And this is an area where I believe I'm going to have to start learning a little bit more about it to add more value to my clients. But the cybersecurity requirements, I'm lucky that I'm… I have a lot of very good clients at Software as Medical Advice, and they basically And especially one that has already got their 510K cleared very recently. I think it was in March or April, and it's… and they showed me all the cybersecurity controls that they have put in place, and that they said that they went through their… their… their 510K, and the FDA just had a few comments, but no big issues with that. So, that was really…

very impressive, and I was able to see their system. So, I learned something from looking at their system, and I think that that's an area that Everybody is going to have to learn a little bit more, especially auditors, because this is… this is somewhat new. Okay. So… We're going to go to our next live poll. So, Sean? You're muted. Okay, well, I'll introduce it.

I'm sure…

I'm here, sorry, I was just sending out the last poll results. So we got 30 people to respond to the last poll, and I only asked for 20, so you overshot by 10. I'd like to keep the momentum going. Which part of the internal audit takes the most time in your organization? A, preparing it, preparing and scheduling it. B, running it, C, writing the report, D, closing the corrective actions. E. all of it roughly equally. F, I advise on audits rather than run them. So if you kindly put your responses in there, I've finally figured out how to tally them correctly. I have to take screenshots of the chat. Thank you, Zoom, for that. So, yeah, I'd love to hear what… which part of the internal audit takes the most time in your organization.

And if you don't have anything to do with internal audits, just say hello. Hi, Sean. Danny, back to you.

Alright, thank you. So the last part, of the auditing process is the report, and that's the deliverable. And one of the things that I was kind of amazed, because I work with lots of different companies, and some of the companies I work with are pharma-based companies. And I apologize if I'm insulting anybody in the form I feel, it's no intent. But what I found their process, especially, was that they did very thorough audits, mostly checklist-type audits. GMP-type audits. But… The audit results, or the audit report, have to go through so many reviews. And it took so long that it was about 2 months after, before they even started to work on their kappas.

And I found that crazy. You know, the whole idea is you have a momentum during the audit. People see that they have to… they may have areas for improvement. You want to give them that information so that they can work on it as soon as possible. It's… it's… it kind of defeats the purpose of the audit when you have to wait for so long to respond to CAPUS, and to… and then they give them lots of time to respond to CAPUS. Well, sometimes it takes, you know, everybody's busy in an organization, it takes a little bit of time, but I think that… This is an area where…

at least the bigger the organization, the more time it takes. At least that's what I found. And what I do is usually the next day. I give them a report. Now, how do I do that? You know, I don't… you know, usually if an audit is lasting 3 days. you know, After each day, I would try to update my report. So that the last day comes, I'm not… I'm not updating it for the 3 days that I did the audit. And it's quite… You know, I've been able to do it. And, it's a lot of work, but… It's something that I believe adds value to the client, is appreciated by the client a lot, because… because… because when they are able to… to act on something, and they may have

seen other audits where they've had to wait days and months or weeks to… to get their report. It's… it's… it's… It's nice to see them appreciate that you're giving them the information in a timely manner, and that they're able to start working on the areas that they need to work on. Because it's important. Because otherwise, I wouldn't issue an onconformity if it's… it's not important. I mean, there are some minor issues, so they can deal with that, but… But… It's… defeats the purpose if they're… if they're not, if they're not being addressed in a quick manner. So, in this… in the report, there's… there's… there's standards, the auditing standards, I think it's 19,011, if I'm not mistaken.

It talks about what should go in a report, or what is recommended to go in a report. So, these are the type of things that we would add to a report, the scope, criteria, standards. I mean, these reports Are not just evaluated by the company, but they're evaluated by the third party. or Biden, or right now. They could be evaluated by the FDA or any regulatory agency as well. And… they… you know, what the auditors, the third-party auditors, are looking for is audit trails. So, what I've seen in some reports is people… people having, you know, looking at purchasing and looking at one PO, looking at design development, okay, one big project is okay. But there… but… but if you're doing a thorough audit, you may want to look at more than that. Looking at

you know, Looking at the objective evidence and seeing that the… the… that the auditor really did a thorough audit, and… and wasn't just looking in one area. I mean, the idea of the audit is to look at one… the MDSAP audit is look at one area, and go in depth. But it doesn't limit you from looking at other things. And it's… it's really… Important. And I think the audit report is kind of… Less worth, you know, there's… there's less value in an audit report where it just gives the nonconformities, and… and that's it. And I've seen some in the past.

I think the audit report should give another auditor or another, or the next auditor, an idea of what I looked at. And… and that's what I learned when I was working at BSI, is that, you know, you don't just do it for the customer, you do it also for the next auditor, you do it for the… for the company, for their system, so that… They can look at the audit report and see the story, see what happened, and get an idea to see what are the improvements that have happened since then. The nonconformity. So, as I was trained by BSI, it's usually a three-part nonconformity, statement of nonconformity, the requirements.

and the evidence. If you don't have a requirement, and you don't have evidence, then you don't have an unconformity. And I've seen so many nonconformities from Other auditors, where it sounded like an opinion. And sometimes the client pushes back, and sometimes the client doesn't. And it kind of leaves them in a… Difficult situation, because they don't really know how to resolve an issue where it's because the auditor saw this in a different company, and he said that this is a requirement. So, executive summary, so sometimes management doesn't want to read through the whole report. They could…

Look at the executive summary, and that there shouldn't be anything different from what you've discussed at the closing meeting. That's really important. Because… You know, at the closing… because at the closing meeting, you're basically telling them. you know, this is the story. I mean, there could be situations… Where you may have to add something, where you're reviewing things and you see something additional, but this is something that you would tell them before you would put them in… put in the report. Don't… The worst thing to do is to provide surprises to your client. Okay, so we want to know.

Yeah, we want to know. So, 25 people responded to the last poll. And screenshot this one so I can actually include the answers with the letters. But, how long after the closing meeting do you normally receive the report? This is… this is a fun one. One to two days, within a week. 2 to 4 weeks, longer, or it varies. I have seen one of our reports not applicable in my role, or again, hi Sean, I got four high Sean's last time. Maybe I can get some more high Sean's this time. I like to feel. special, too. Not just Danny. Okay, I see a hand raised.

Okay.

Don't know if… she wants to be promoted, I will ask. But yeah, any questions, now would be a great time to hop in. And ask out loud if you're a panelist.

Hey, Danny. This is Chris. You were talking earlier about going through the root cause analysis. How often did you run into companies where their risk analysis wasn't fully fleshed out, and you would go down to the root cause of reported defects and realize, boy, they should have expanded more on risk analysis? They would have caught some of these more difficult, critical defects that are hard to replicate. Did that… how often did that come up?

It happens, You know, you have to be lucky to be able to… lucky as an auditor to be able to… be able to find that sometimes, but it's… it's… it's… it's… it's happened, and… and it's… You know, you look at their faces, and some of them… feel embarrassed, or some of them, their faces light up when they say, oh, really? That's really how it's supposed to work. And, you know, they don't see the relationship between risk management, root cause, and post-market surveillance, and… And it's… usually the quality manager is usually the one that's happy. Because… because sometimes they're running against the brick wall, sometimes they're not.

Alright?

They don't have an easy job, and… My job is to make their job easier by giving them Giving them an idea of what could be improved.

That's awesome. Thank you.

Thank you.

Yeah, feel free to keep going, I'm gonna tally up the book results.

So, bring the findings that keeps coming back. So, what are the things that you see in your quality systems that, Or as… if you are an internal auditor, what are the things that you see that comes back? Like, Kappa issues, always, you know, Kappa system issues with regards to training? Or a lack of training. Share with me what, what you feel that, Something that you would like to discuss?

So I'm sitting here multitasking, trying to tally results. I'm not going to be able to tally the results and run my mouth at the same time. Danny, thank you so much, for pulling all this together. I encourage people to drop their questions in now, or just come off the mute. I don't know if, Michelle has any comments from, the U.S. perspective on this, how this differs from, U.S.

Thank you.

audits.

Michelle just found out that her back bedroom, the entire wall is, wet, and the pipe bust in the ceiling, so that's where I've been during this whole presentation. However, I was interested, I was so interested when you started that, I made, my, my, Junior employee who's learning to audit and, you know, getting his audit credentials and whatnot. I asked him to join, so if you want to put him on the spot, we can learn what, what Nate learned. And he's the other Michelle here, because I gave him my link. So… Nate, you need to take yourself off mute and video.

I'll try to promote him. Fra, can you help me? Sorry. technically challenged.

What happened?

You need to promote Nate, who's the other Michelle.

Yeah, I'm doing it right now.

Preetam had a question. PreTam will be rejoining as a panelist. Again, if you'd like to rejoin the call as a panelist, you can be promoted simply by raising your hand. And, pree Tom, you would like to… Yes, hi, welcome.

Pree (PRITAM): Thank you guys, you can, you can call me Pree. I interestingly have, almost a similar background like Danny. Pree (PRITAM): I've got my bachelor's and master's in aerospace engineering, but I ended up working in medical device industries, and I also worked at TUV and SGS, which are competitors of BSI. Pree (PRITAM): Yeah, I come with a lot of, you know, similar sort of experience and background here. One of the things I wanted to bring to people's attention is, how when you write a finding. Pree (PRITAM): Clients, you know, either hastily provide their response or hastily do corruptive actions and preventative actions.

Pree (PRITAM): That are very limited to the finding itself, and don't really expand the scope and take a look at the, you know, 30,000-foot level view. Pree (PRITAM): To kind of go through the, you know, inherent process issues, rather than just, like, one item, right? Pree (PRITAM): And, when you go to the following year of audit, right, it could be surveillance or recertification audit, you kind of see that, you know, like, the issue is a systemic issue now, instead of just, like, a single outlier. And you end up… you end up writing another finding. Sometimes these minor findings, you know, get escalated to major, right? And, if, again.

Pree (PRITAM): they are reoccurring, they ended up… they ended up also requiring cancellation of certification, too. So I've seen that in a couple instances, where Pree (PRITAM): one of the things I highly recommend clients is to not hastily provide their, you know, Kappa resolutions, but actually take a deep dive and, like, a 30,000 foot level view on understanding if there's truly a Pree (PRITAM): A single event or a systemic issue to actually, Pree (PRITAM): you know, address this for a ones and all. Well, for a one and all. Once and all. Yeah.

Yeah, I see that. Sometimes, their client's just doing correction and not doing corrective action. And, and the root cause analysis is just blaming it on resources or training or human error. And, it… it's… their systems don't work, and And you have to… it's difficult if you work in the company to challenge that, but, you know, I leave the company right after the audit, and I challenge them sometimes, and well, whenever I see it, I challenge them. I say, well, it doesn't look like your systems are working. And, think you have to go spend a little bit more time and, and get, you know, get through the real root cause and find a way to improve, to deal with the situation. You don't just write it off and say, okay, you know… and… and that happens

quite frequently, where we see that, you know, if you just look at their capital log. and you see that… you normally see a trend, and that's where you know that you can go there, and you may… you may be… you may find another issue. And it's not that… as the auditor, I'm there to look… to tally up more nonconformities. It's usually more work for me, and I don't get paid extra for that. It's just to show them where the issues are, so… So it's, I agree with you, and, you know, what happens is aerospace You know, I'm an AS auditor as well, I work with others, and it gives you a different perspective, but it also gives you

You know, and I think it's… it's a benefit for me to do… to have this aerospace background, because I can go… Through an audit, and through… certain areas. And looking at it as a different perspective, different areas where the AS standard kind of puts more emphasis on whereas the medical device… so I'll give you an example, KPIs. So if you don't meet a KPI in aerospace. You can get a major nonconformity. Doesn't happen the same with regards to… to… to medical devices. So… I'm a bit of a stickler in that area, and I try to, you know, aerospace, they want to see performance results. You have to be 100% on time and 100% quality.

medical devices, That's not really… That's not really emphasized. But it's something that is important for these companies to know about. But thank you for the question, and the comment.

Danny, just one more point about just your slide here, the finding that keeps coming back. I think is going to shift under QMSR, like, in terms of what is the FDA citing, and there are two things that, companies that have only been audited by FDA are very under-prepared for, and that's risk-based approach. That goes throughout your quality management system, and then if they find that all the links that are supposed to be there between different systems are not there, or they're weak, or broken, whatever, then they will say, you also have a problem with your culture of quality. Because a culture of quality wouldn't already know that something was broken in their system, because they wouldn't be getting feedback and data from each other.

So, I think those are going to be the two in the future that… that are gonna, knock industry, you know, on its knees to… to figure out what… what does those things mean, because they sound kind of, etheral, you know? Something's kind of out there, you know? So…

I think that'll be interesting. You'll have a webinar coming up on that.

As a matter of fact. As a matter of fact.

In fact, Michelle is gonna be doing that, coming up. Fra, can you drop Michelle's link in? So we're doing that in, what, 2 weeks? Is that… Right about 2 weeks.

But I'd love to have everybody, and I'd love all these stories to come out in the context of what I'm going to go into with the culture of quality and the risk-based approach, so… Yeah, bring your… bring your… Opinions and commentary.

I agree with you, Michelle. Although I don't see too many companies that just have… just had QSRs, because in Canada, mostly companies have ISO 1345, but I did see some supplier quality systems that… some suppliers from the U.S, And I looked at their quality system, it didn't look very robust. And they were just… Exactly. They're basically just pieces of paper that they… look at maybe once or twice a year, but they don't really follow them, and it's very… It's very disheartening to see that when you see the… how much… You know, what the auditors are expecting, and what… they would have… They wouldn't be able to pass an audit, for sure.

Exactly. Yep, that's a great point.

Okay, give me 2 more seconds on the link. Sorry.

But, so, yeah, Danny, the people in the United States have only been auctioned by the United States. They've gotten really lazy, and, like, compliance is, like, they want to do just what it takes to not be in trouble, but the second that you move into okay, now let's make improvements. They keep knocking you back down, and then that's where your culture of quality gets contentious. You know, between, how regulatory and quality people get labeled inside a company and industry, and how all those are constantly fighting engineering and design change and manufacturing changes and etc. So…

Okay, so it's noon. I'd like to remind everybody that we have, a clinical event coming up next week. Then we're gonna have Etienne is the speaker next week. From Greenlight Guru, but he's gonna be talking about marketing, and what's shifted, and, what you need to be thinking about. It's gonna be a great conversation, because we're gonna go back and forth. And then, the following week is Michelle's quality presentation, on Wednesday, so I put that link in the chat. I'd like to thank Danny for his time and preparation and thought that he gave to this. Please show some love in the, in the… with the little clappy things, or the, you know, you know what to do.

You're making me smile.

If you would, if you would like to, talk to Danny, please reach out to him on LinkedIn, or, And we will be also sharing the slide deck, so that has his contact information on it. And I want to thank all of you for your time and participation. I hope you liked the polls today. We had lots of people participate in the polls. I want more interactivity within this group. I want more people coming off of lurker mode and in… and joining us as panelists. I think this is really one of the unique places, where… where something like this actually happens, and you have an opportunity to, introduce yourself. Thank you, Pree.

For introducing yourself today. I know that it takes some courage, within a senior group like this to raise your hand and be like, hey, I know something too. But if you know something, why not introduce yourself? Right? And, so, again, thank you to everyone, especially Danny, for your time and attention today.

Okay, thank you, Sean. It was… it was fun.

Thanks, everyone. We'll see you soon.