AI System in Quality & Regulatory Affairs

Start. Hey, Michelle.

Sean, good morning.

Good morning, how are you?

Okay.

Sean.

Can you, make me as a co-host?

Sure. All good. So we're a minute early. While we're waiting to get started. I'm gonna be posting a number of announcements. Let's see, what was I working on? Show some love. Here's the first thing. So… now, I'm gonna hold that one. I'm gonna go with the conference reminder. There is an international med tech safety conference,

in Boston, June 2nd through the 5th. And I'm putting the link in. for that. And… Let's see… Sue… I hope that you'll go. I hope that you'll share the information with your colleagues. I hope that if you're in the Boston area, particularly, you'll think about it. Hey, Andre! Daniel, welcome. The next thing that I want to share is Andre, who's here. Say hello, Andre.

Hey, how are ya?

has… He's, like, in this contest, right? And, actually, I need to hit everyone. And… So… what did you do? You build a thing, and it's the coolest thing made in New Jersey contest, right?

Yeah, we designed a… it's actually a medical product that's going to be going before the FDA, but we introduced a veterinary version, which is doing extremely well for non-invasive treatment of musculoskeletal conditions in dogs and horses. I actually presented it on our Friday calls about a year ago when we were in the process of developing it. So now the veterinary version's out there, and somebody saw it, and they nominated it for the contest called The Coolest Thing Made in New Jersey. And so we're trying to, you know, we're trying to get the votes on it, and that's… I asked Sean if he could vote for it, and then he said, yeah, and he says, I'll tell people on Friday to vote for it, too, so…

Yeah.

So it'd be really not… and you can vote 5 times a day. The voting ends on… at the end of the month. I voted probably 15 times so far, and you can just hit the vote button over and over again. So please, everyone, everyone on this call, I want everyone on this call to click the link that I put in there, And to vote for Andre's thing as the coolest thing in New Jersey, particularly if you love dogs and horses the way that I do.

I mean, if anybody, takes a look at some of the testimonial videos, you know, they see these dogs that have not been able to walk because of hip dysplasia, and after a treatment, they get up and start walking around, so it's pretty dramatic results.

That's awesome. I love that. And then they, the last thing is, to… Take a look. I went a little nuts this week, for the Connected Device webinar and created all of this content, that, would hate to see go to waste, so I'm gonna put the link to that in here as well. It was a really great session, Particularly for people who are concerned about their connected devices in this heightened security environment, but also Yeah, just generally, that was with Mark Omo. Now, that we have a critical mass of people joining the call, I want to introduce today's speaker.

James Pink. I met James, last year at the conference… at the above-referenced conference, the MedTech Safety Conference. I went to his session, on risk, and it was… it was great. He's a really captivating speaker, which is fun. He's also talkative, and, has a great accent. So, without further ado, I'm going to introduce James, and please tell us, when you were a small child, what made you want to get involved with medical devices?

Well, that's a really quick and simple one. My, my sister was born blind, and… Obviously, she was diagnosed with astigmatism, and I used to… Look at, her struggles. And think I don't want her to struggle. And so, being a farmer. sort of a son of a farmer, it sort of became very evident that there was going to be some sort of biological solution, but there needed to be a mechanical one too, so I was always fascinated with sort of that transition between human and component, and thus, sort of, Really wanted to strive towards having some level of impact with people, patients, and in particular.

you know, ensuring that I could at least enable some of these devices to get into Into the people's arms, hands, eyes, ears, noses, hearts, and whatever else it is. So that was the first stimulus, and it was from a very young age.

That's awesome. Now, We're gonna skip ahead 30 years, and we're gonna go to… Q-a-r-a, AI, And what we've seen happening in our industry over the last, you know, 2 to 3 years in particular, and the adoption of AI systems, and… and controlled risk. And so, that is the topic for the day. I think this is going to be really interesting, but I encourage everyone that wants to to come off lurker mode. Raise your hand if you want to be promoted to speaker. And… The, and yeah, let's get started. Tell us why we're here.

Okay, so… Well, I think we're here because… I don't know if you've had the same level of anxiety that I've been having over the last couple of years about AI systems are gonna… Obviously, take away all our jobs, you know, ultimately. I've sat there as a quality, safety, and regulatory person, thinking, what the hell do I bring to the table once these really intelligent systems that can almost, sort of, resolve Fermat's last theorem to however many nth degrees, and You know, and now we've got agentic systems, and systems that are actually, you know, not just physically intelligent, that they can actually act on that intelligence as well, which is a fascinating thing, so…

I think one of the things I've been involved in, firstly, is that through my career, I've worked with Incredibly good medical physicists and mathematicians and simulation engineers who, you know, we're talking over 40 years, have really refined algorithms in A variety of areas, cardiovascular, radiology, imaging, you know. And so we've seen the impacts of science and mathematics and physics, very, very positively. It's almost like a UFO landed, and we all of a sudden became amazingly brilliant at X-rays and some of the technologies that we can get our hands on, but… I think what I've seen is that there's this almost imminent

If you want, Frontier, new Frontier. And that new frontier is the sort of the federation of intelligence. You know, intelligence has always been really akin to our own knowledge, experience, skills, capabilities, and of course, you know, the things that we set out to be and to be become, whereas now, you can simply Query a large language model, and… be able to articulate with, you know, that fields of chemistry, science, you know, you name it, and in a way that we've never been able to do before. And also, we can pair it back to an understandable level, so we can take ourselves back to kindergarten, and then

Take us through to grow an adult in terms of beginning to understand concepts, theories, models. Now, of course, that has left us with a huge gap in In our worth, it's left us with a huge gap in terms of you know, what do we do? I think we were just discussing earlier, Sean, that I don't see it as being, okay, well, then the robots will do everything, and we'll go off and make pizza. I see it more of. Hold on a moment. We, as humans. we have this new tool, or tooling, at our fingertips, which simply is a translation, a speed of light thing, where you look at it and think, where it took us hundreds of data points, where we had to have Excel sheets after Excel sheets, and we had to do lots of computation, lots of mathematics, lots of equations, lots of, sort of, filtering and trying to fit information.

We… we've now… that's gone. You know, it is more than obvious that we're seeing that Data can be reviewed at lightning speed. We can see that patterns can be uncovered that us as mere mortals, you know, can't… can't easily see without doing all the hard yards, you know, from You think about it, I… I just started work, in 1994, so 32 years ago, and that work was predominantly administrative. You know, there was a… there was a YZ network. I'm sure the Americans amongst you would have been a little bit more advanced, but the Brits here, we were sat with, like, one computer in an organization that employed over a thousand people, you know, and…

You sort of look at that, and you think, wow, you know, that computational power is now there, but we still did stuff, we experimented, we made things, we designed things, we used things, we… you know, there's still that physical thing of, we're going to get ill. We're gonna need things, you know, we need a roof over our heads, we've got the Maslow laws still acting upon us, and nothing changes. And as a result of that, you think, okay, so there must be something in terms of my profession that I can begin to do to make sure that, ultimately, the thing that drives me is that we have something that is safe, effective, and quality. You know, it… we used to say quality was the…

Totality of features and characteristics of an entity that bear on its ability to meet stated, implied, or latent needs, or we said fitness for purpose. And… I think that we can look at AI systems and go, right, we've got a company, it's designing, manufacturing, supplying, and monitoring for the next, you know, the next phase of its life cycle. I'm a quality or regulatory or safety person. And we've got all these systems and signals available to us, so I now bear responsibility for ensuring that the quality outputs are produced. And if you look at AI systems, one of the things I've noticed already

is… An amazing amount of hubris from product managers and software engineers and data scientists and, you know, everything. And I don't know about you, but most of the people who are, sort of, managing the implementation of any AI systems within an organization are cyber people. I'm like, what the hell does that have to do with processes? We are processes. Quality people build processes. Input, activity, output, measures, monitors, requirements, you know, resources, skills. We build that model. And then we make it go input, activity, output, and then we build the systems. So we hook and connect those systems together. So, for example, a design control process can interact and operate within a risk management process, which can interact and operate within a

design transfer process, which can interact and operate within a scale process, which can interact and operate within a manufacturing process, which can interact and operate with a post-market surveillance process. So why the hell is… why is IT involved in this? Why is my cyber person telling me that these AI systems can be deployed? Because we own the processors. We design, we help design them, that's what our worth is, whether it's regulatory-based processes, safety-based processes, or quality assurance-based processes. So I've been sort of sitting back and thinking, wow, I've been operating in data science for 34 years.

I've seen the evolution of how that has been integrated into hardware, such as imaging modalities and the likes. But now what I'm seeing is that this is now finding itself, not beyond software, but into information systems, where we make decisions based on these AI systems. And, quite frankly, as quality and regulatory people. We have to take back control. of how AI systems are deployed, why they're deployed. Because if you think about it, when you were asked, can you remember when Six Sigma was everything, wasn't it? GE was all over the place. Six Sigma was a thing. Well, who were the people who pioneered that? It wasn't manufacturing, it wasn't design.

It was operational excellence through quality people, because quality people knew statistical process control, they knew how to… the theory of variation, we studied under Schuart, under Deming, we studied under Duran, under Oakland, all these people, these gurus, who understand about processes, input, activity, output, variation. All we're doing with AI systems is we are baking in wonderful things, autonomous, semi-autonomous, assistive systems. Into those processes, which will now take that input. And actually, the activity goes to warp speed. And the output. forms something.

Now, at the moment, it can just form data, it can form information. But what I've discovered from… I've been involved in this now for 7 years. I've run multi-multi-million dollar experiments, I've had… access to over 100 data scientists, machine learning operations persons, and I have looked at the jobs to be done in quality regulatory and safety. Over a 7-year period, over 200 experiments. I've conducted, and we've deployed, so I'm not talking out my backside. I've actually seen the things that you name it, everything from, can we actually get to write a technical file? Can we get AI to review one?

Can we actually understand how a standard can be translated into engineering requirements, testing requirements, and requirements that go onto a medical device label? You name anything that you think in terms of a use case, and I've run experiments from proof of principle through proof of concept. and in some instances, into detailed design and integration of these systems. And we've looked at how well those systems, AI systems, operate. What is the best fit of the AI system itself. So, what we're talking about today is, how can a quality regulatory or safety person Start to take that control.

sort of grapple it from the hands of the IT department and the cyber teams who are just basically saying, oh, it can't give us a weapon of mass destruction. Well, no shit, Sherlock, that's not difficult, is it? You know, we don't want that, but what we want to do is, if we say you're going to get a sales order. And it's now going to analyze all of the sentiment within that sales information and tell us what we should be designing next. I want it to be right. You know, I don't want it to drift. I don't want it to make up some bullshit. I want it to be right. And I want it to do the things that I need it to do. And importantly, as a process person.

I also wanted to have some discipline about how that happens, and I want it to deliver knowledge and insight that I can take action on. So, the course that Sean is sort of referring to is an AI systems course, where I basically take normal quality regulatory and safety people through a journey of how we can prioritize opportunities, like a Lean and a Six Sigma project, where they used to go, oh, where's the… what was it, the low-hanging fruit? Where can we… where can we go in this… this huge, nebulous organization of processes, and where can we have an impact that can deliver some value? So we look at that and we go, well, let's just have a think about the capability of AI systems, from pattern recognition.

to processing information. to… you know, prediction systems, you know, whatever it is, probabilistic systems, they call them. But let's try and work out how we could fit an AI system within these array of processes and systems that we have, often defined under a quality management system, but not always. How can we do that? How can we take a little bit of control? How can we spearhead these, and not fall foul of the snake oil salespeople who are hanging about in the Wild West right now. that have the most wonderful-looking bottles, the most amazing stalls and counters, you know, but are basically full of shit. So how do we actually do, you know, the job of a quality assurance person, a regulatory person, or a safety person?

to… Distinguish the wheat from the chaff. How do we best fit AI systems into our processes and our quality systems? And how do we address this concept of something that you might not know is something called symbolic, which is a rules-based quality management system. Into a sub-symbolic one, which is a predictive and environmental scanning one that reacts within the context of the rules. But it's basically like an autonomous vehicle, you know, driving around a British country lane, and a horse is just about to come out, and the dog runs across the road. You know, it's basically, okay, we know we should be traveling at the right speed.

But now things are out-hitting us, so how do we create this symbolic quality management system And operate a sub-symbolic one, where the concept for me, and this is the most exciting bit as a quality person, was… and it only came to me at, like, 3 o'clock in the morning yesterday. was that… hold on… My role is about orchestrating preventive and predictive quality. and adaptive. These are terms that we've never heard. We've heard inspection, quality control, quality assurance, total quality management. Six Sigma, Lean, all this, but we've never heard of predictive and adaptive quality. And we can put that into a predictive and adaptive safety, predictive and adaptive regulatory. And that is a really exciting thing, because with AI systems… sorry, Sean, go on.

So, I just want to take a moment to take the temperature of the audience and ask, you know, in thinking about sort of where we are with the adoption of technology in the medical device industry for, you know, quality systems, then, you know, how many of you are reasonably worried for your jobs? How many of you are excited about the changes that are coming? And how many of you are just anxious about the whole thing? And you can, you know, either use emojis or words or whatever you want to, to put it into the chat, and, you know, pick a word that describes how you're feeling about all this. anywhere from, you know, trepidatious to, you know, whatever you want. You know.

like that, MJ.

I can… and put the word pizza if you think we're gonna be all making pizza, in the near future, or… mowing lawns.

I'm with Faison, I'm so… I have gone from complete… what the hell am I gonna do? What worth? How can I impact? Because I was a really terrible clinical engineer. Awful. I'm an engineer, but I was an awful clinical engineer. So my aim of, you know, being able to sort of rectify problems with my sister, and her eyesight, and, you know, that ambition of being a doctor, the ambition of being a, you know, somebody who's physically going to do that, it wasn't there. I wasn't good enough. You know, but then, in terms of quality regulatory and safety, I'm… I found my niche. But then, all of a sudden, you see that all these AI systems, surprisingly, can do everything. They can even write, you know, all your design requirements, they can do all of your dossier work, they can… they can analyze all your complaints, they can do everything that I've spent 3 decades doing.

And you sit there and you think, holy crap, I am out of a job here, but more importantly, I am… I am lost. You know, I, I, I, I… I had the, you know, the ability to do all sorts of things, but I made the the very specific choice that I was going to be a master of my trade. And now my trade seems to be lost. You know, and then all of a sudden you think, well, hold on a minute. Literally, this is the most exciting opportunity, because we don't have to do all the… bullshit stuff that we used to do. Can you, you know, like, can any of you from statistical process control backgrounds, how many of you really enjoyed your tally charts, your control charts?

You know, you have to be a certain individual to really enjoy that. And then how many of you loved going back to your Murdoch and Barnes tables, and sort of cross-referencing all of the orthogonal work and everything, and then starting to work out whether or not you've done the statistics correctly? we were all really happy when Minitap came along, or other software systems that helped us with the statistics, but now that just goes. And it's like, well, why were you doing all of that? Well, I was trying to find variation in the critical to quality feature. Why? Because I was trying to prevent this.

from going wrong. Why? Because I was trying to avoid defects and cost the company and cost to the patient. And now you're like, why? Because that's who I am. So you're just basically getting from the what I'm doing to who I am, like that. At war. I speak.

Yeah, can I make just a couple of comments? One is… I… I remember when… We have computers coming into the workplace for the first time. And the comment that was made over and over again is, now that we have computers, people are gonna lose their jobs. Because computers are going to replace people, things are going to be so easy, so people are going to be basically finishing the work day in 2 hours, and people will have nothing else to do, because everything's going to be expedited. And that was the biggest lie. technological eye that we have ever been told, right? And I think it's gonna be the same with AI. What's gonna happen is that AI is actually going to create more work for everybody, because

There's going to be an acceleration of knowledge, understanding. You know, in so many respects. So, so we're gonna have to be catching up with that. And that's what the demand is going to be, so… I think that's the second technological lie, is just incubating right now, and then one day, 20 years from now, something else is gonna come up, and then we're gonna have the third, you know, technological lie coming up. That's my first comment. My second comment is. And we talk about AI in quality systems, but in reality. this is not just… you know, I think sometimes we think that quality systems is just like this little territory, right? This little function.

Yeah.

But in reality, we're going to have AI in maintenance and calibration, we're going to have AI in manufacturing, we're going to have AI in engineering. we're gonna have AI in process and product development activities, and you mentioned regulatory, so… so I… in a certain way, I think saying that, you know, AI is all about AI in quality systems. it's also, you know, maybe in a certain way, a misconception. So those are my two comments.

Yeah. No, it's… it's… totally agree. You know, this is… the way I approached it was when… we started to think about automation, and good automation manufacturing practices. You know, automation was going to take over, IT was going to take over. We've got… we go through these industrial, sort of, you know, continuums. And then just discovered that it just makes things faster, but the whole intent is for us to make better, safer products that are more effective for patients, you know, in the medical device field, you know, that's what we're trying to do. Or in the IBD field, we're trying to do that. In the pharmaceutical field, we're trying to do that.

And you're quite right, it's ubiquitous. But that's what I'm thinking about in terms of a quality management system, is that… and this is the big difference, Daniel, is that a lot of people think a quality management system is a compliance management system. I'm a purist. I'm a shoe art Deming person. You know, the Japanese, sort of, you know, progression was not through somebody saying you need more procedures and you need more forms. It was, you need to discover variation and not accept it. You know, only control it and understand it. And you need to understand inputs and outputs and activity and systems in order to be able to deliver

what you consider is fitness for purpose, or any other definition of quality that you want. So… and you're quite right, one of the big areas was always predictive, or… you know, maintenance. That was a huge impact, wasn't it? But that was 15 years ago. 20 years ago, predictive maintenance was really coming in. It's called it condition monitoring. You know, what can we monitor to increase the reliability of a system? So I think that that's the… that the real issue is… is, as I'm describing here in the… in this course, is that your quality systems, yes, your compliance systems might be AI-generated, but

as you rightly say, SAP and all of these MRP systems and, you know, maintenance systems and PLM, product lifecycle management systems, they're all getting layered in with AI now. And that's something that you just need to begin to think about, is, well, what's it doing? You know, and is it now, as a purist, moving us away from… Real quality, and that's where quality regulatory and safety people really do have a worth. But more importantly, play a humongous part in orchestrating these systems to make sure that they continue to deliver what ultimately we're trying to deliver, which of course, from an agency theory perspective, is always shareholder value, which means effectiveness

ethics, trust, you know, safety, blah blah blah factors.

I want to make sure that you stay on track with your slides, so I'm going to push you a little bit. Otherwise there's no way you're going to get through your slides.

I'm fine, we're all right. And I think, in terms of the course, I'm… The sort of the three-day course is to help us get some tool sets, you know. I'm not gonna… I'm not here to start giving you a book of prompts, you know, that someone in quality can use to do a Kappa process. you know, that's… just no interest in that whatsoever. Prompt engineering is an interesting… snake oil, sort of, ingredient book. You know, what we really want to be seeing is how can you characterize a process, how can you identify an opportunity for an AI system to improve that process? How can you distinguish what type of AI system could be

optimized and incorporated. And we also teach you about how you can then run experiments. You can work with the providers to make sure not only are they transparent, that is, that they explain how the AI system works, on what data it's been trained, if it's a trained system, on how How it works from both deterministic and inference-based measures. So that is, okay, where do you get the rules, and also where do you get the prediction? And how do you get that, and what happens next? So that we can begin to challenge this, but more importantly, not just sit there as the sort of, you know, as we typically are called the sort of the anti-marketing department, or whatever, or the anti-product department, where we're holding up

you know, the systems, but where we're going, right, let's embrace this. AI systems is about computation, it's about, you know, speed. It's about accuracy in terms of the data. So, where do we want to be increasing that? You know, and I always think in three dimensions, and the three dimensions that I teach are basically, right, okay, this AI system. or these processes with AI systems, how do we get the value from this? So, do we remove the number… how do we remove the number of steps or operations? How do we increase the efficiency, and how do we actually increase the accuracy? So, can we actually be better

through the AI system at predictive accuracy, or just mathematical accuracy, so that we can… if you take that into the context of a complaint system, for example, you know, how do we remove the number of steps of processing the complaint? how do we actually then remove the, you know, create the efficiency, you know, the cost itself of handling all of that information? And then how do we get better accuracy? And we can basically say. Reactive accuracy, so actual statistics, information that's delivered for us to say these are trends, emerging new trends, or predictive accuracy. So that we can actually say, well, how can we increase the level of prediction?

You know, it's… so, again. the FDA uses something called the Gamma Poisson shrinker methodology statistics that are used, but now with all of this new data, there are more accurate, predictive markers that would help us act on a complaint much faster. So. the session, Sean, really, that we go through is not prompt engineering. It's not, woe is me, we're all going to die. It's not, oh, you're the quality department, so therefore you ought to check how good these systems are and use a checklist. Nothing like that at all. It is like we are back in the world of operational excellence, and we are saying, how do we take this company's organizations, processes, and systems, how do we work with teams to select AI systems so that we can actually

give a, you know, give a… an appropriate selection and integration of AI systems so that we can deliver on an end goal. And this is a core thing that I teach in this, which is basically the discipline of understanding data, and how that translates to information, how that information becomes knowledge, and how that knowledge becomes a decision. So, we start to walk through… whether or not there is quality data, so I don't know if we talk about the rules of data quality, we talk about how, you know, that data can be considered as a quality data, and then we talk about how it's translated into information.

And then from that information, we talk about how, through knowledge and rules and models and confidence, we can actually now become… we can translate that into formal knowledge that is now actionable. And then we talk about how the AI systems deliver a decision, or do you deliver the decision? And lots of people talk about this human in the loop. as if it's something like the AI does a load of things, then the human checks that it's all alright, and hey presto, everything's good. Again, total bullshit. You know, we've just outsourced data, information, knowledge. and all your… you don't get the decision anymore, because it's already happened. So we basically say, where does the human

Interact within those four stages, and how. So, if you say that an AI system is now delivering a pattern that is telling you that if you do not act, you are going to have to recall 500 batches of your reagent, for example, someone has to take that decision. Now, do you want to take the decision as soon as the AI system delivers it? Or do you want to gain confidence that the knowledge Is actually being appropriately arrived at, which requires something called explainability. So now you need to uncover how did that knowledge arrive. Now, if you don't know how the information drove the knowledge, you need to have explainability again as to how did you get the… how did you get from the data to the information to the knowledge, to the decision.

And… We'll talk about the levels of AI systems and autonomy. But ultimately, based upon how autonomous an AI system is. Is on how quick, though, it passes through that 1, 2, 3, 4 layer. And as a quality regulatory and safety person, you need to be looking at that and thinking. is the risk proportionate to this walk speed activity? And if you now use a Gentic AI, Which is basically a series of bots that are doing the job for you, right up to the executable action. And then does the action. You know, you go to sleep at night, and you could be waking up with 300 medical device reports issued out to the FDA.

You know, already done. already registered, already gone into the systems. So you have to be very cognizant of what is happening. Another thing we teach about is the fact that these quality system providers, the EQMS providers, are layering in so much AI system work in there to help you take away from the data information to become knowledge and decision. But of course. the data might have been really poorly designed. Old database theory, really bad root cause codes, you know, I've got 10 cause codes, I've got 15 failure codes. So, already, AI systems don't need that. They can work on what we call fuzzy logic.

They can work on things you don't see. just through text and information, and inference. So, you might be getting information and knowledge That you're taking decisions on. And it's a crock of shit. He's all wrong. So, already, you're not optimizing the AI system to do the job properly. You're not releasing the data in the way it should be. You're creating what we call taxonomies. As opposed to… Ontologies. And knowing the difference between how that data is arranged is really, really important. And again, we go through that sort of sequence, and we say, what AI system is it? We get you to go through the data, information, knowledge, and decision.

And that helps create what type of integration, verification, validation activity you will deliver.

Hey, James, I think you touched on something that's very important. When I look at the four steps that you're presenting there, for me, I think that before I get to the fourth step, which I would call probably the fifth step, it's really to introduce a concept that you just mentioned. Which is a verification validation. Step, because the decision you… the first three steps that you have there are all generated by the AI, versus the steps 3, 4, and 5th, which would be verification and validation. And then decision, those would be human-based. So, to me, once we gain access to that information, we still have to do some form of verification or validation. Is this the right information?

Is it complete? Right? Is it apropos to the decision that I have to make, so… So I'm wondering if you would consider, actually. that fifth step added to the process. Just a question.

Yeah, so, this is the interesting thing about the hubris. So, regulators are all for that. They're like, right, let's have a fifth thing where it's verification and validation. I'm like, screw that! you need to build quality into those steps. So, what we find is. the fifth step is, oh, we've given the human in the loop a checklist, we've produced it, hundreds of people have put together that you go through this 100-check thing, and then we human in the loop it, because that's perfect, and we'll sandbox it, we'll let you play in the sandbox, et cetera, et cetera. But no one goes, hold on a minute.

Number one, data. What was the quality assurance regimen deployed over the data? Number two, information. What was the quality assurance built into the information? Number three, what was the quality assurance in the knowledge? Because if you take me all the way through that. with this black box, which many of the AI systems do, my human in the loop is simply just transferring legal responsibility for your bullshit. You know, and that's… that's the reality we're facing right now, with this amazing amount of hubris and naivety around quality. You know, software quality is not about verification and validation, it is about software quality engineering.

Writing good code. ensuring that that code works and is acting appropriately, looking at the logic and the mathematical expressions, and choosing the right ones. The same as product qualities. Otherwise, what we're doing is we're going back to the old inspection processes. Of quality control, but we're just saying that's a human now, so we let all the system do everything it wants to do, we get to this fifth dimension of verification and validation, and transfer all of the liability Away from the system to this poor human individual, who, after about a year, has got no context anymore. Then there's lots of papers in radiology about this context drift.

That, you know, when you're a radiologist practicing at the top of your game using an AI system, it's great. But all the decisions, all that 1, 2, 3, 4, keep doing it, keep doing it, keep doing it, and you get this latency of… contemporaneous thought and knowledge, it goes. And so you sit there and you think, I might as well just sign and say, yeah, I'm the guy who misdiagnosed your tumor, and I'm going to take that to the medical council. You know.

So let's take that example, right? Let's take the radiology example, and let's go a little bit deeper on that. Somebody asked, you know, share some examples of how AI was used to write, critique, improve, design, input, or out… input or out requirements. You know, so, radiology, you know, we go with that example, or any other example that you want. You know, how do… how do we make this real, then?

So… If you make it real, I mean, ultimately, what we can look at is exactly what are being used at the moment, so we go through… you know, these AI systems that are either assisting you in quality. So, if you were thinking about a design input, for example. then we now have AI systems that understand the symbolic requirements, so the rules. So ISO 134857.3, we need to have design control, and then we move into 7.3 point, you know. to… we need to have design inputs, okay? And design inputs need to be unambiguous, not conflict within each other, testable, you know, all that type of stuff. And so we can create, in our own systems, a symbolic rulebook

So, when we do actually write a design requirement, an engineering requirement, a design input. We can actually have some assistive means that understands that symbolic rules, and every… in each sentence, it will say, well, actually, there's nothing in there that makes it testable. Oh, actually, this is now conflicting with another requirement that you've written. Oh, no, this is ambiguous, based upon… You know, a… Some form of language. modeling, you know, so ultimately you can have that assistive bit, but in some of the other systems, we'll actually go, stop, we've done a full literature review of every single FDA guidance for this type of thing.

We've extracted it, we've done an inference engine, and we say that these are the requirements, and we've looked at your design, in terms of mechanical, electrical, software, etc, and we recommend that these are the inputs. For that particular thing, in order to comply with 60601, part 2, part whatever, you know, or in order to comply with XYZ. Or you might have just something that's semi-autonomous. So now what it's doing is it's saying, we've written all of that, we've incorporated it into the design plan, it's now going straight off into the engineers for them to go and build. We just need you to, human in the loop, click it and say, yeah, you're alright with it.

Or we've got autonomous or agentic, which, if this is a software system, it goes through and the code's written. You know? And you go, well, what requirements are it? Well, I can go back and take you to what requirements there were, but it's already done, and it works. You know, you just… you tell us. So, ultimately. Current deployment in regulatory environments are mostly assistive, which is great. But we as quality regulatory safety people need to become very sensitive to when these systems are moving us away from the data, information, knowledge, decision, and now are moving into action. You know, and we need to understand what's going on in there, and whether or not we've appropriately selected the right ones. Now, some other examples in QARA that we've seen, lots of risk management.

Lots in design control, lots in regulatory submissions. I've just seen an AI system that says. You write your intended purpose. We'll tell you the classification of your software, and we already have got 9 tenths of your entire submission done. And you're like, could you imagine being an entrepreneur, where they've got to pay $150,000 for a regulatory or software person, and they're being told that they can now do that? The problem that you've got is… the one input that they ask for is, can you tell us what your risk class is? And then we'll write your risk analysis. Now, if you know software, any of you have done software development in medical devices, you know that the only way you get a software safety risk classification

is to do the risk management work first, so you can begin to understand how the software can impact it. But strangely enough, many of these systems are actually just probabilistically looking at your intended purpose, and probabilistically defining what software safety class it might be. And then just builds huge assumptions. So the entire organization is devoid of understanding What a risk a software presents to a person, a patient, the environment, the property. the… you know, it's completely defunct. So these AI systems have now removed the knowledge. So you go, stop there. You can't… a good product manager's, yeah, said, oh, the pain point, the job to be done is to remove all this paperwork.

And you're like, it's not paperwork. It's safety, it's safety communication, it's quality assurance communication, it's not paperwork.

Exactly. Martin King said to me when I first started looking into all this, he said, what MDR is asking people to do is to understand their devices. Right? And, you know, people… complain that MDR is too rigorous, that the requirements are too stringent, that it's too expensive, that it's too this, that it's too that. But he said all they're asking them to do is, you know, is your device safe and effective? Does it perform as it's expected to? And is it delivering the required, or the intended benefit to the intended user? And from that perspective, we get lost in the weeds, and we get into annex this, and blah blah blah, and MDG, or, you know, whatever guidance on this. And, you know, and… but what…

what they're being asked to do is to know their devices. And as quality and regulatory professionals, you know, it is my belief that that's… that's the core of this, and you can't hand that off to an agent. An expected… To do the job.

Yeah, totally agree. And that, I think we mentioned that in the safety conference as well, is that… your engineers become engineers, not clinical engineers. And this humongous distinction between… the differences are, is that we've worked with patients, we've seen pain, we've seen the impacts of interventions, we understand, through the development life cycle, what has and can go wrong. We understand, then, with working with quality personnel, what systems and processes we can put in place to identify and react to any potential defect or vulnerability in our system, and also to predict it. So again, this is where you get into the orchestration, and you think, wow, wouldn't that be great?

most of the AI systems at the moment are basically doing my literature review for post-market surveillance. Some of them are translating that into reports around clinical evaluation. Excellent! But now what you've just done is take the information, the knowledge, and the decision-making layer away from people who understand why. That is important. And have the overall concept and knowledge of the risk management work that was done, or the safety work that was done. And thus, again, we talked then about, well. What is the benefit of an AI system in post-markets events? Test it. Have a look at your existing data that you've got.

Have a look at recalls, field safety, deaths, serious injuries that have occurred. Put it all on a timeline. and then give the raw data to an AI system. Right? And then give it certain parameters to try and predict failure within that time bound. And what you will see is the predictive power We used to call it Wisdom of crowns, but it's actually a multi-factor attribute analysis using regression. But ultimately, what it will do is identify, and I've done… I've run this experiment. It was like, right, hold on, we took the manufacturer's information, we saw this point on the 3rd of July 2016, this event happened where this patient died.

And it was predictable 4 years earlier. The predicted markers were available 4 years earlier. let it sink in, because it's real. It's no bullshit. The facts are all there. So, you think of the predictive power that AI systems can have, but they are so contingent on data. information. Knowledge. decision. action. And you build that up, and you build in layers of quality, you're building layers of, as Daniel was saying, verification and validation at each step. You don't do it at the end. Because it becomes a black box that gets a sandbox, and as I keep saying to everyone, the only good thing I've ever seen in sandboxes is what my cat produces, you know, ultimately. So you have to bring it all back down and start talking sense around

what can we do? The one thing that we've missed out, Sean, as well, is this symbolic and sub-symbolic thing. How many of you would appreciate a continuous audit? So that a system at any point, can tell you that a process is now drifting. So imagine, right now, as we are speaking, in our medical device organisations, people are doing work. Somebody's writing a report, somebody's taking a phone call, somebody's in a service area, someone's swinging a spanner. somebody's doing something. Now, imagine right now, if you could just look at that like you would in a nuclear power station. And go, hold on, there's a little warning there that we're drifting away from the process.

Wouldn't that be amazing? Wouldn't that be something that we could do? Rather than going and doing internal audits, wouldn't that be just amazing? So we can actually use AI systems to help on the symbolic layer. And therefore, then, if we know that that's under control, and we've… quality assured the AI system in those processes that are symbolic and rule-based. What can we flip our attention to? We can move away from the rearview mirror. into those positive, predictive activities that stop having a mountain of corrective and preventive actions, mountains of investigations and deviations. We're now in a situation where we can move into pattern recognition.

And one of the things that I've posited around that is that in rearview mirror. we produce things called critical to quality, critical to safety, and critical to process. If any of you have worked in operations, you'll know what that means. Critical to quality means that we've got an attribute, a variable, that if we don't control within expectations, it will cause a defect. Critical to safety, that defect will cause a harm that is unacceptable. critical to the next process, that defect will affect the next process. So one thing that we now want to begin to think about is what data points are critical to prediction?

And that's a wonderful… a totally new area in quality management and quality assurance. We are moving away from very physical… Monetary measurement around attributes that are kind of static and retrospective. into prospective predictors. And that's it, that's… that's a… Think about that in the context of not just our world, but in the context of health. In the context of environment, Everything. A phenomenal opportunity exists there. So, we talk about that as well, Sean. And we also give you the tools to be able to understand what is quality assurance. And what are the…

actual measures of an AI system's performance, from talking around things like sensitivity, positive predictive values or precision, negative predictive values, what we talk about, calibration and drift, but we do it in a non-technical, sort of mathematical, probabilistic way. We do it in a way of, how could that affect these processes? So, when you get to the decision-making. How could it affect the knowledge? How could it affect the information? And then you guys look at that and go, right, well, if I deploy a AI system that is now writing my design history file, compiling it. What impact could that now have if it doesn't have the appropriate sensitivity, precision, negative predictive value, and calibration and drift?

And what impacts could that do? And as you can imagine, it can, you know, it becomes like compound interest, you know, in some of these… think about how many actions there are, how many events, how many decisions, how many outputs. We can soon end up with, you know, the proverbial hitting the fan very, very hard. So…

On that note, with the proverbial hitting the fan, we have 5 minutes left. I want to make sure that, if people want the slides, they should write slides in the chat, and maybe he'll share the slides with you, that's up to him.

Yeah.

And, give you a chance to say, you know, okay, I've created this course. The course is taking place at this time, and if you would like to find out more about the course, then write course in the, in the chat as well, because, I want people to understand what it is, That… that they would be… Attending… you know, signing up for, because I think it's… you know, that's the reason why I reached out to him in the first place, was because I was like, oh my god, he's developed this whole course around this stuff that, you know, many of us are grappling with trying to understand, how is this going to fit into our world?

You know? And… So I'm gonna shut up.

Yeah, no, no, that, I mean, the cover, the course… It's about quality and regulatory people who look at their processes, are… seeing an array of either integrated AI systems in their EQMS, in the systems that they already use in terms of software that are part of quality, quality assurance. And really give you the tools to… Understand, scrutinize, but moving away from scrutiny enabled. So actually become the sort of the Six Sigma black belt, if you wish, in terms of saying, look, we'll grapple this off the cyber team now, okay, it's safe, we know, we get it. You know, the information, confidentiality, integrity, availability is all there, but now let's give it to the real process experts.

the quality people, you know, the regulatory people who know the symbolic and sub-symbolic requirements for good processes. And now let's have a look at how we understand and prioritize, where we should deploy AI systems, can scrutinize AI systems that are in deployment. Give us the tools to be able to, run experiments. And how you would run an experiment using your own data. getting a knowledge about how you QA those different steps. So, how do you QA your data pipeline? How do you QA your information? How do you QA your knowledge? How do you QA your decisions? And how do we actually build quality into all of that? And of course, then we start to talk about your own skill sets, because they have to change.

If your background's more around inspection and testing, quality control, quality assurance, if it's more around compiling common technical dossiers, you're going to have to move away from that and move into how do you orchestrate prediction systems? You know, so what do you need to understand? You now need to understand probabilistic theory. not to the data science level, but to a level you understand it, and you can articulate it. Your literacy in AI cannot be about prompt engineering and LLMs. You know, it has to go much, much more Into how we actually enable operational excellence. within a quality system using AI systems.

I think that's an excellent summary. We have one… exactly one minute left. I'd like to thank everybody for their time and attention, today, and, remind you that next week, Praya Bhutani, the CEO of, RegDesk, is gonna be here talking about regulatory intelligence. This is only the start, managing impact standards and submissions, workflow in one… in… work in one flow. So, It's a… She, should be giving a provider presentation, a demo, and talking about, the… these… these challenges, in… in day-to-day regulatory work, I encourage you to tune in for that. In the meantime, please check out the links that I put in there for,

stuff that was done this week, and James? Always refreshing. Always candid. No sugarcoating. I, for one, appreciate that. And I hope everyone had a good time today, and we'll tune in again next week. Thank you.