Putting AI to work for RA QA Arie Henkin, CEO and Co-founder (OneComplAI)

I don't think anybody wishes to become a quality and regulatory professional. And I wouldn't wish that onto anybody, really. You really need to want to do this in life. Once you get a little bit into your career and you have some understanding of where the role is, and how it fits into the overall organization.

I fell in love with quality and regulatory. I'm actually a mechanical engineer, by training or education.

My first role out of school was in manufacturing quality, and then I slowly drifted into the dark, dark world of quality and regulatory, clinical reimbursement, etc. And then I fell in love with process. I fell in love with helping companies get to market. I fell in love with the… the angle of innovation that you see when you're a regulatory professional, which I think is a much more holistic angle than you would see if you're just a, you know, if you're a mechanical engineer, or if you're a software developer, etc. So it's, … I have a slide on… a little bit on myself in the presentation as well, so we'll get into a little bit more of what I've done, but that really is what attracts me, is, …

getting as many products to market, as many innovative products to market as possible. And that's where AI comes in as well for me.

Sorry, getting a message. I'm gonna… Francesca's gonna be promoting people in a minute, but I need to promote him first. … Show… … From there. Tell us what you've built, why you built it, and how you think that it's gonna make a difference in people's day-to-day lives, as regulatory professionals, and moreover, what an ideal outcome for this call would be. ….

Yeah, so I think some of this will get answered as we dive into, the, the actual presentation. I have… so, in terms of what I built, or what we're building, I should say, what we're building is a platform that aims to accelerate Innovation in the medical device industry, and we're doing that by attempting or looking to reduce compliance burden. So not the role of compliance, but the burden that is associated with compliance processes. So these are things like duplicated processes, these are things like having to have, day zero consultants that generate documentation for you, and really leveraging some Gen AI methods, to do this.

The platform itself right now, the way to think about it is there's three pillars to it. First pillar is the regulatory search component, where we help companies figure out what their path to market is. Second component is the commercialization, or the launch component, so helping a company navigate what regulations and standards are applicable, how do you comply with them, how do you generate SOPs, DHF, etc, that's needed in order to go to market. And then, sort of actioning those, those SOPs with the help of, experts. And then lastly is you're compiling your actual submission and then scaling from there. So analyzing what's happening in the market, analyzing what are functionalities or features that your competitors are working on.

And then figuring out what are some low-hanging fruit from a geographies standpoint that you might want to consider, and then what are some functions that. Might serve as easy predicates for you to look at and to consider as you're looking to expand your product indications.

So, my understanding is that you built this because you needed it.

Back in the day, I built this in a SQL database, because I needed it. And we all know how user-friendly SQL is. So the idea of looking through predicate mapping, what happened in the past, what predicated on the particular device that you're looking at, how do you create a testing strategy based on that, how do you massage the functionality and the indications, etc. That is essentially, what I built, and then I managed it using a bunch of spreadsheets. And we're moving away from that, or at least generative AI allows us or gives us the tools to be able to generate some of that information automatically, and then rely on the expertise of people like me and people that are on this call, so quality and regulatory professionals, to figure out, well, what now, right? So how do you actually take all those insights and coalesce them into a strategy for the organization?

And getting back to the idea of, like, what outcome, like, how can, people on this call engage with you about this product? And… … Maybe, you know… introduce it to their clients. We have a lot of consultants on the call. And this is an opportunity for them to get a first glimpse. of it.

Yeah, so in my presentation, I'll share a few examples of where the device or the software could currently be used, and where we're heading. Some of the pitfalls of AI-native solutions versus generic AI solutions. And then lastly, I'm always open to talk about the way that our role as quality and regulatory professionals is evolving, and what role generative AI has in this evolution. And then how do we, make it work for us, rather than… you know, ending up in a place where a founder decides that they can click a button and generate an entire design history file or entire quality management system out of a ChatGPT prompt.

And then eventually hand it over to us and say, hey, you know, I'm ready for… I'm ready to go to market, right? Look at this great output that I was provided. So we're trying to avoid that reality. We're trying to avoid that world of things, and instead create tools that are tailored To work with quality and regulatory professionals in, ensuring that we can, Get a safe and effective product out there.

Great. So, starting to promote everyone to panelists, if you want to stay incognito, feel free. Please keep your microphones muted, and why don't we kick off with What you want to present today?

Sounds good. Could I have, … Presentation ability or rights?

Do you not have that?

Nope.

Oh, ….

Oh, maybe I do. Sorry, it's hidden.

I'm gonna make you a co-host. Make a co-host.

I think I got it.

Good. Great, thank you.

Alright, can you see my screen?

Yes.

Perfect. So really, as I mentioned up top, I want to talk through how this tooling is going to evolve with us. And I want to really give the group here some actionable insights, some things, some areas where you could use LLMs, ChatGPT, etc. for your purposes and your team's purposes. And then we can dive into some more complicated, or some more complex, examples of where this world is going, and how it might evolve, and how we might evolve with it. So feel free to… I'll be monitoring the chat, or at least I'll try to monitor the chat. There's a few cases where, I have a little interaction built in.

And I… I just… I'd love to, to see what people think as we go through this, otherwise I'm sort of lecturing at you, and I… and I really don't want that to be the case. Okay, maybe I'll make a controversial statement, maybe it's not so controversial. I think about 70% of your company is affected by compliance requirements. Is it, is it 60%? Is it 80%? I don't know. I don't think it really matters. I think what's important is that if you do a mapping, let's say, of 1345, or of the MDR, You can see the speaker notes. Hold on, am I sharing the wrong screen? Oh, I am. Hold on.

Hold on….

Go to present mode.

How about now? Better?

Much better, yes.

Perfect. Sorry about that.

No worries.

Okay, so whether it's 60% or 80%, I don't think it really matters. If you do the mapping with 13485 or with MDR, I think you'll convince yourself that it's something like 70%. The real question, I think, or an interesting question might be how much of your budget is that, right? So, whether that number is, direct or indirect, meaning is it something that you're paying for, as a… for your EQMS being a direct cost, or something that you're paying for as a submission to the FDA? Or your notified body, what have you, or if it's an indirect cost, meaning It's a cost that is the cost of inefficiency, or the cost of process burden, or whatever. You know, I don't know. Is it 10? Is it 40? Is it 70, is it something else? I'm not sure. I think what might be interesting for us

is to figure out what's the effect of AI on this number. And that number might change, right? That number, might be, higher for, let's say, a startup that, is looking to commercialize, and so Basically, your entire budget is taken up by a clinical trial, or by your desire to submit to the FDA, and if we look at a PMA, we know that that's not a cheap process. Versus a more established, larger company, where perhaps you've driven that cost further down into manufacturing, and the design and development component, or the burden during design and development is a lot lower. Alright, so who am I? I gave you a little bit of a background already, on me.

But, at a high level, so I have 3 projects on the go. I'm chief clinical officer for a company called Ignition Neurotechnology. It's a neurosurgical planning company, software-based. I'm a founder and principal consultant for a company called Mithra Medical Consulting. This is my own consultancy where I primarily help startups Seek or look for the path to commercialization? How do they navigate? Quality regulatory, compliance clinical requirements. And then, obviously, the theme of this presentation, or the template for this presentation, is from OneComply, where I'm co-founder and CEO, and OneComply really looks to try and implement some of these generative AI methods that we're going to be talking about, so that they're working for quality and regulatory professionals.

I spent quite a bit of time in this world, 15 plus years. I've worked on quite a few different products, mechanical, software, electrical, optical, IVDs, etc. Worked in a number of startups, spent some time as head of a certification body for both Global MDSAP and their 1345 program, and really. what drives me, and I alluded this to… I alluded to this in my conversation with Sean right up front, is setting up business systems. And really figuring out how do we… how do we utilize those business systems to get as many innovative products to market as possible. So, the types of inefficiencies that we're talking about here, I've… I had a chance to live them from all three angles, in terms of the auditing world, the consultant world, and then the manufacturer world. And so, I'm extremely passionate and looking forward to explore this, or to open this Pandora's box with all of you.

Okay. So what is it that we're actually trying to solve in this, in this session? I don't know that we're actually going to be able to solve it, however, I think we should at least try to define what the problem is. So… To start defining the problem, I think we need to start with the definition of what compliance means, right? State, process of meeting quality and regulatory requirements so that products are safe, effective, and legally marketable. And obviously, we know that this matters, we can't… get a product on market or stay in market without some compliance processes. And when I say compliance, I mean things like submitting a 510 to FDA, maintaining an ISO 1345 certified QMS, conducting PMF, all those sort of things that we have to do as quality and regulatory professionals.

If there is the concept of compliance, then there might also be the concept of compliance burden, right? This is the overhead, inefficiencies, duplicated efforts, outdated processes, outdated tools, etc. All the stuff that we gotta do in order to try and achieve a state of compliance. Some of it is not burden, right? Some of it is just required work, but a portion of that required work is probably, there's probably some hidden burden in there. Why it matters? Well, we already talked about it, right? If 70% of your company is affected by compliance, and if a… that transfers to some amount of budget, then there's utility in looking at it, right? Back in the day, we used to call this cost of quality, very similar type of analogy, or type of principle.

So, some examples of this are things like multiple teams writing… rewriting and writing the same documentation, the same SOPs, searching FDA databases manually, copy-pasting, language, going through multiple approval workflows for the same stuff, etc. I'm sure you can find examples for that from your own world as well. So the core of it is… Does AI help? Or is it part of the problem, right? Is AI actually going to help us on this journey that we're embarking on to reduce some of this compliance burden, or is it actually going to increase the burden that we feel as quality and regulatory professionals, just like the example that I alluded to earlier on, right? And unfortunately, and for those of you who are consultants, I'm sure you've seen this already.

this problem of a founder, especially new graduates out of university that have fantastic research, fantastic ideas, and they come out of it saying, hey, ChatGPT put together a regulatory strategy for me, and, you know, I think I'm ready. That ends up in huge inefficiencies, and huge duplication of effort, potentially. Okay. Let's baseline on some terminology here, and I apologize for those of you that have been steeped in this world a little bit more, and so this might be a repeat, but I think it's useful to make sure that we're all in this… in this evolving world, that we're all talking about the same…

The same stuff, or are understanding the terms in the same way. So our large language model is an AI that's trained on a massive dataset to generate and summarize natural language. You can think about it as an engine for a car. Gpt, Generative Pre-Trained Transformer. that's the architecture, right? That's the way that the LLM is actually used, and the way that a tool like ChatGPT predicts or tries to figure out what the next word is, and there are a variety of different GPTs out there. That's basically another engine, another connective tissue, another thing that drives this overall car that we're talking about.

This is a prompt. The prompt is the instruction that's given to an AI model. The instruction is sort of the… the prompt is sort of the interface, right? That's your steering wheel. That's how you're trying to navigate or figure out where this car is going. Last slide for terminology, but hopefully everything is making sense now. Please stop me or raise your hand if it doesn't. Rag, Retrieval Augmented Generation. Think of this as, so you're combining a database search with, some document, with some documents or some information in order to create as specific an answer as possible. For the particular topic that you're looking at. So think about it as a map, basically, for where it is that you're going, and how you're navigating that landscape, right? The various, information that you're pulling together.

A chatbot is the conversational interface that allows users to interact with an LLM, and probably 90% of what we do today with AI-related workflows is through a chatbot, fortunately or unfortunately, right? So, that, that, the little window when you open ChatGPT or Gemini that says, hey, ask me anything. That, at its core, is essentially just a chatbot, and is the only way right now of interacting within those tools. That's essentially your cockpit, right? The chatbot is the way where you as a user are sitting down and are figuring out how are you going to work with this, all this background stuff that's happening, the LLM, the neural networks, the RAG frameworks.

Okay, something that's AI native, that's a tool that's built specifically for a domain or a workflow, so it's not just general AI that's bolted on. General AI is, so… in Google Docs or in Sheets, you have this, like, colorful circle on the top right corner that allows you… sometimes it bugs us, right? It sort of says, hey, can I help you? So that thing is more of a general AI interface. You click on it, you're able to ask questions about that specific document. Something that's AI-native is really a tool that integrates AI and allows you to… with the user, and allows you to work concurrently on a particular problem.

And it's not just specific to a particular document that you're looking at. Essentially, it's… the analogy here is something like an SUV or a Jeep, so a particular type of vehicle that you would use for a particular type of terrain. And lastly, Agentic AI. So Agentic AI is the evolution that we're, that we're staring at today, and that's AI systems that can plan, that can remember, that can act autonomously. And they can sort of navigate a series of steps on their own, and they can pull much more difficult to find, or much more difficult to synthesize information together from a variety of different sources. The vehicle example here is something like an autonomous car, right? A Tesla, or a Honda.

Whatever other cars have autonomous ability, so, you know, it's able to plan a route and then actually follow that route. So hopefully all of that makes sense. In terms of baselining terminology. Okay, so… as a field, we've gone through quite a few, different evolutions in our life, right? So, I am old enough to have lived with my own stamps. I've had, you know, my rejected and accepted, stamps that I have stamped on documentation. I've lived through hard copy signatures, chasing people around for, hey, I need your signature on this document, and then I need to file it away. In a filing cabinet, when I used to work at the certification body, we had an entire basement, essentially, dedicated to archival. And I'm sure some of you that are working in some larger companies might have the same thing still.

And then obviously all of that got upended once computers got introduced, and I… I don't know that anybody else is… is looking back fondly at the world of, you know, printing a bunch of… printing your FDA submission, putting it in a bunch of… boxes and shipping that over to the FDA manually, or for that matter, burning a CD or a flash drive and sending that over, now that we have the electronic ability to submit. So things have changed, but things have largely or generally improved, I would say, as well. Have gotten easier, but more complicated in some other ways. The kicker with this is that…

the world of AI, even though we've only been in it for, let's say, 3, 4 years, even then, it's already evolving pretty significantly, right? So, a prompt, like, something like CREATA has an analysis for an MRI medical imaging viewer. about, I would say, maybe a year ago, the only ability, or the only thing that I would be able to do is go to our… my chatbot interface with ChatGPT, enter that in, and then look at the response. But we're moving towards a world where Agentic AI allows a much more contextual, rich answer to that question, right? Where the AI actually goes out, or the agents go out and pull existing templates from your QMS, review context, clarify questions, identify hazards using literature, current literature, using databases, align on risk acceptance criteria, etc.

So, they're able to provide a much richer answer, to this. prompt of creating a hazard analysis. So we're moving from this world of reactive, chatbots that just generate answers, sometimes blind, or essentially blindly, to your questions. to a world that's more proactive, where the user and the AI are orchestrating actions towards some sort of outcome, and some actions are orchestrated by the AI, and some actions are orchestrated by the user, some decisions are orchestrated by the user, by the user, I should say, towards some sort of common outcome. So, you know, coming back to our original question of does AI help, or is it part of the problem.

Think… There's a point to remember here, right? Which is, when we're talking about compliance. And when we're talking about the burden of compliance, we're not actually talking about safety and efficacy. I'll remind everybody that compliance is a state or process of documenting, of following checklist, submitting paperwork, following the regulations, etc. Safety and efficacy is the outcome, right? That is the thing that all of us are driving towards as quality and regulatory professionals, and that… those are things that are… that's an outcome that's achieved through what it is that we do, our culture, our knowledge, our effort, our systems, etc. And in fact.

You can look at safety and efficacy in a way that you don't necessarily look at from a compliance standpoint, i.e, it's much more important to look at safety and efficacy from a submission standpoint, meaning, is that snapshot in time of my product demonstrating Safety and efficacy to a regulator, as an example. You can then abstract that into safety and efficacy of the actual product on a continuous basis, right? So, do I have enough hooks in my PMS process, in my non-conformance process, in order to lead to a more safe and effective product? Do we talk about risk management reviews at the product level?

how ingrained is that in our culture and in the way that people actually behave when they're working on our device or on our product? And we can abstract that again to look at it at the company level, right? So, are we running a company that has a chance of producing a safe and effective product? And that, again, comes back to culture, comes back to the way that we are running our management reviews, the way that we're running our general document management systems, are we incorporating risk into it, how are we thinking about all these problems? And the kicker with this is that AI can essentially be incorporated into any one of these, levels of abstraction.

And the reality… is that it's already happening. So, there's an MIT report that came out about a week ago, or two weeks ago at this point. that… made the claim that there's this shadow AI economy, where 90% of our employees are already using AI tools unofficially. And there's various ways that we can incorporate, or talk about this, right, in terms of data security, or IP, or any one of those worlds. What is more interesting for me to talk about in quality and regulatory context is where can generic AI help versus agentic AI versus AI-native tools, and how do we connect that back this point of, you know, your team's already using AI, how do we connect that back to the outcome, right? To safe and effective products, to the culture that we actually have?

In developing devices that are ultimately going to save lives. So… AI is already here. … And it's disrupting quite a few different… Industries. Or functions. So, in the software development world, you might have heard of cursor. That is one of the first true AI-native solutions where they're, the software developer and The Agentic AI solution is essentially working in tandem in order to lead to faster code generation, easier, cheaper, whatever. On the design end, you have solutions like Canva, not quite AI-native, but helps a lot in generating, graphics, visualizations, etc. On the law end, you have, quite a robust legal,

AI framework. Genie AI is one of them, there's quite a few other ones. So, … there, you're using it for contract generation, for comparison of different contracts, creating your initial drafts, analyzing for pitfalls, stuff like that. And then. In general, or in life, basically, some people on this call might have heard of vibe coding, which is the idea that there is a world where you should be able to, as a user, have an idea for some sort of app or web app. that you want to generate, whether it's mobile or a web app, go to a website, put in a natural language prompt, and then the generative AI methodology methods in the background, just create this work for you, both the front end and the back end. And that's happening now.

On the quality and reg front, … There are a few tools that are available out there, obviously OneComply being one of them, there's a few other ones, and we know that FDA is also demoing their… or not demoing, but using their own in-house developed tooling, called ELSA, and they… they finished a pilot in June, and I think they're expanding it, or looking into expanding it now, helping with… with review of submissions and review of, type of the… type of documentation that is coming through. Okay, so that leads us to our first interaction. So, you know, let's use AI the same way that FDA uses AI, essentially.

And ask ourselves, how would we check the compliance of an SOP against AI regulations? We know that there's quite a few AI regulations out there today. We're going to pick the EU AI Act, because that's a big one, and, is, I think quite difficult to actually figure out where it is that you fit in and what's missing. Okay, so I'll give you a vision in three parts, right? There's what you would do today, there's what you would do tomorrow, and there's what you would do in the not-so-distant future, potentially. So today. the way that I would approach this problem, right, is I would look at the scope of the EU AI Act, I would look at the scope of

internal SOPs that I might have, I'd create some Excel spreadsheets, I would extract text that's relevant to my particular application, and I would run a comparison of that text. I'd figure out, essentially, what are the deltas, both in terms of documentation and in terms of missing processes. Hopefully, everybody agrees that's a fairly standard way of running a gap analysis. What you might do tomorrow is look at the external scope, or create an external scope, create an internal scope, and then run a prompt. through something like ChatGPT or Gemini, and then analyze and review the outcome. So I actually have a sample outcome to share with you, where I ran this exercise.

You can see it on the screen now. So I fed it the EU AI Act, I fed it, NAIML SOP, Then I came up with a number of top gaps, took it about 4 minutes to think through this. Is this… is this a complete picture of all the possible gaps? Probably not. Is this a good place to start? Probably yes, but it still requires you to actually review and take a look at how this maps to your reality on the ground. And the claim that I'm making here is that in the not-so-distant future. You won't even have to scope things. if… your EQMS, or your document management system, or whatever you're using to handle your documentation, has access to everything that you're, … that you've created, then you should be able to just prompt, and then analyze and review. You should be able to just ask the question, and then the solution itself will give you the answer.

A more interesting question might be, do we even need to prompt? Are we heading towards a world where… you have an AI agent that's constantly running in the background that's looking at new regulations, new standards that may be applicable to your particular device or your company. And then automatically it generates a gap assessment, or at least it prompts you to say, hey, there's a new version of X, do you want to do a gap analysis against your current procedures? And so. We're, I think very quickly approaching this world of continuous compliance, or continuous compliance state, at least on paperwork.

And I'll come back to this idea of, at least on paperwork, which I think is one of the biggest pitfalls with this type of tooling and this type of world that we're heading towards. Okay. So let's… let's go a level deeper, right? So we just use ChatGPT in order to do a very generic or high-level review. And it identified some issues. pretty quickly, right? Again, maybe the same as a junior engineer or something along those lines. I'm not quite sure. It would take some analysis to figure out. But it's clearly not all sunshine and lollipops, right? From that same MIT report, about $30 to $40 billion has been spent in enterprise investment into generative AI, and 95% of organizations are really seeing no return at all.

So… What do we do about that? And does it mean that we stop? caring or stop using AI? Well, we know that we're not going to, right? Because 90% of people are already using it. So the question is, how do we make that better? This 95% number? There's a few success factors that came out of that same source. said that companies or users in companies generally expect tools that are being implemented to learn and adapt to workflow. So we come back to this idea of native AI, right, something that can change and improve based on the context of the organization. As the organization evolves, as the product evolves, the actual workflows themselves end up evolving and changing. If you're adding

biocompatibility, for example, with a new product, you expect whatever it is that you're implementing at the enterprise level to be flexible enough that now there's a biocompatibility component In that workflow. Deep customization to process, right? It's not good enough if you have a, you know, if you have a three-step approval workflow for documentation. there's a reason why you implemented it this way, and it's not good enough for a company to come in and say, well, you know, with us, you're just… you get one step, and too bad, so sad. The expectation is that you're able to deeply customize the way that a solution is being integrated into the company. And then lastly is this point around integration, right?

People don't want this colorful, circular button on the top right of an application that allows you to ask some questions through a chatbot interface about a particular document. They want something that is directly integrated with the business, that is directly integrated with the organization. Not something that's bolted on as an afterthought. And so, really, that's the pitfall of ChatGPT and some of what we've been talking about in terms of this chatbot interface, right? There is no concept of workflow. It has no memory, or no, well, has some memory, but no real memory of your organization, right? It doesn't remember that you have 3 approvals, because at some point, an FDA inspector came in and said, hey.

you know, you have an issue with your approval workflow, and that was your corrective action for that particular finding, right? It has no idea. It doesn't know what your audit trails are, right? So it doesn't know Or you can't retrieve who made what change when, and where that particular prompt changed, and who changed it, and what led it down that particular path of thinking. And it has no context. It has no context for your organization, or no context beyond what you feed it initially. Now, can you evolve that or change it on a document-by-document basis? Probably. The question is, well, you might, but is the rest of the organization going to do the same thing? I think that's really where, the rubber meets the road with this.

All right, so let's go through another example. So we'll get deeper here into ways of working with ChatGPT, which is what we have today. And by the way, I'm agnostic to what it is that you're using, whether it's Gemini or ChatGPT, it doesn't really matter to me. The same principles apply regardless. Okay, so let's say we want to audit our Kappa SOP for weak spots. I will suggest to you that there is a bad way and a better way of prompting for an answer to that question. Just out of interest, in the chat window. Could somebody put what is the bad prompt for this? How would you ask this in a bad way of ChatGPT to audit your Kappa ISOP for weak spots?

Would that be audit my Kappa SOP for weak spots?

That's right. Pretty, pretty simple. That was an easy one, right?

So, the problem with this, right?

Yeah, is my capital process good? That's right. So it's generic, right? It's vague. It's lacking some information, and we all inherently feel, whether you've been trained on good prompting practices, which is an actual field, engineering best practices for prompting, whether you've been trained on that or not, you have a sense that this is not a good way of actually prompting it for a result. And so I'll give you the recipe, right? We love recipes. In this field. And so those are the elements of a good prompt. What you need to remember are those five things that are in color there. You need to give it a persona, you need to give it an objective, you need to give it an audience, you need to give it some context, and you need to set some boundaries.

Okay, so with that, does anybody want to attempt Just quickly, a better prompt in this.

Sure, I'll go, I'm doing that. I'm doing one right now. For my internal audit, I'm in ChatGDP. It's procedures I've already audited, so I decided to use ChatGDP as 100%, just to see what happened. And I created a project, I loaded the regulations, and said, here are the regulations that are applicable to this company for the purpose of this project. And then I specifically, now, I ask it to please provide one to three questions, potential nonconformances, opportunities for improvements. And also include any, positive feedback that they would have, and suggest records that I should request. And therefore, then I would output it and review it, and it's about 70% accurate. I have to just kind of massage it a little bit, but that's what I've been using as my prompts

And… writing it to the audience, the oddity.

Nice! That's great. That's a lot better than what I had here, just as a short example. The only thing that I didn't get in your prompt is, did you cover the role component, the persona component?

Oh, well, actually, my chat GDP, I call him Chappie. He has my persona already. I have a.

Beautiful.

So, you know, I didn't have to say, here's my persona. I… what I do is I've instructed my chat GDP to write at a 7th grade level at all times, because most people can't absorb or consume anything at that, and then with a lot of my posts, it's gotten to know my personality, so it automatically assumes who I am. I didn't have to say who I was.

Fantastic. Great. … Good. Okay. So, that, that, I think, absolutely should lead to some useful results. Again, would I trust it blindly? No. And hopefully everybody is getting the sense, from these examples that this is not something that we trust blindly, as experts in this field. It's a tool. Let's go through another example, that hopefully makes it even more concrete. Say we want to create a risk management plan. For a company, or for a particular product. And again, there's a bad prompt here, right? I'm not gonna belabor the point, a bad prompt is create a risk management plan. Horrible prompt will give you a bunch of nonsense.

There is a better prompt here. that utilizes all of these different principles that we talked about. I think I only have… 15 minutes or so, so I'm not gonna do the exercise of getting you to write it, however, you should think through how you would actually prompt ChatGPT to do this. This is what I came up with, right? So… act as a particular something, you're aware of other things, and you want to create a document that's easily understandable by other members, so that's the audience part. I want you to create a risk management plan for a particular product, I want you to consider some context, and I'm gonna set some boundaries, which is you have to follow the format for a risk management plan from 14971.

Okay. So, I do actually have the result of this. Hopefully you can see my screen still? Yes, you can. Perfect. So I don't have 14971 memorized, I'm sure some of you do, but generally, 14971 risk management plan requires scope, responsibilities, requirements for review, risk acceptability, overall residual risk, verification of implementation, and effectiveness, and then some considerations around production and post-production. And if we look at the general risk management plan that ChatGPT came up with, it's not bad. You know, there's a scope, there's references, there's some responsibilities, there's the activities, there's a risk acceptability criteria section, hazards and sources, risk control options, verification of risk controls, and evaluation of residual risk, as well as production and post-production activities. So, at first glance.

It's pretty good, right? Again, as a founder or whatever, I'd basically file this in my documentation, in my risk management file, and I would say, done. As quality and regulatory professionals, we look at this and we say, hmm… there's probably quite a… there's a few problems here, right? Device life cycle, right? It doesn't really know anything about our life cycle, so it just… it's defining some things that may or may not be applicable. In terms of the scope, is this the scope that is actually relevant to your product? Maybe you have some CT-MRI merge functionality that makes this not as applicable.

or that you want to add. From a reference standpoint, right, where are the internal SOP references? From an activity standpoint, it misses, really, the point of verification of implementation and verification of effectiveness. Risk acceptability, this is where we get… we start getting a little bit more, even more unhinged, right? So… there really isn't the idea of an acceptability matrix. What does it mean for a risk to be acceptable? What does it mean to reduce it as far as possible, etc, right? So you can see how we start breaking, breaking apart as we dive deeper into this. So… With all of this, I'm not saying that ChatGPT is stupid, or is a stupid way to start. What I'm saying here is that…

templates, or a starting point for conversation, is essentially a commodity now. Anybody can go to ChatGPT and start that conversation and get, some template. And so, the answer to where do I start is very simple. there isn't an easy answer to, where do I go from here, or how do I make sure that that is correct, or how do I… how do I know that I'm done? Right? So, realistically, Hopefully everybody on the call agrees with this. You know, this could probably cut your, documentation cycle time. in some percentage, right? In half, 30%, whatever it is, in terms of just starting with a sample. But it doesn't get you to, to the, to the end.

Result. All right, let's go a level deeper. So we talked about, overall review, we talked about… document generation. We're gonna go specifically into the regulatory world here, predicate searches. So, for this example, I need to give you some background. So, deep brain stimulation, basic idea is you implant a lead. The lead has an electric field, stimulates particular areas in the brain. That stimulation reduces your instances of sensual tremors, or Parkinson's, or… other disease states that are related to this sort of thing. Obviously, it depends on where you implant the lead. The important part is that there's three components. There's preoperative planning, there's intraoperative guidance that happens, and then there's post-operative programming that occurs.

So let's say we're developing a device that merges CT with MRI. We then extract some leads from that CT, and then we display those leads in context of the brain structures that we get from the MRI. This is not a regulatory lecture, so I'm not gonna go through what… how do you get to substantial equivalents for a device like this, etc. The important part is that everybody should understand that there is some process that we follow, that is mandated by FDA, where we look at similar devices that are out there, and we try to prove that the devices have the same intended use, they have similar technological characteristics.

The degree that they don't raise different questions of safety and effectiveness, and the methods of testing are, acceptable. Okay, so how do we go about searching for this similar device? In the past. we would do something like this, right? We would Google for some similar devices, and I have no idea what people did before Googling. I guess you'd go to some, like, workshops, or you'd talk to people, or you'd look through documentation in hard copy, or go to the library. I'm not sure, but you'd Google for similar devices. Then you use the FDA database to find some devices of interest. And for those of you that have used the database, you know how fun that is.

And then you would extract or download the summary for every single one of those devices, and you would review the intended use, and the predicate device, and you'd figure in the testing strategy, and you'd figure out how close that is. And you curate a spreadsheet, essentially, with options. K. How do you do that today? Well… You can start with ChatGPT instead of Google, or OneCompler for that matter, to find some devices of interest. And it… it… Both solutions will actually generate a list of candidate devices. Then from there. You could use a tool like OneComply to understand what the predicate landscape looks like. So this is an actual screenshot from the tool. It's available free to use. You can take a look at your particular

candidate predicate, you could take a look at other devices that it predicated on, you can take a look at devices that predicated on it, take a look at the number of days to a decision, what the major functionality was, etc. And then you can export a list of options. So you get similar results. From both something like OneCompliant ChatGPT, definitely easier and better than just Googling around. But… ChatGPT lacks workflow, lacks understanding of your company, lacks some of the context, … Essentially, that you don't give it, or you don't pre-prompt it, through projects or some other means. In the not-so-distant futures, there are some obvious limitations to all this.

Right? Do I even need a predicate, right? What class is it that we're talking about? How do I test to it? What are the features that can move me to different risk classes? Etc? And that's… Basically, where we're heading with OneComply is the intention is to feed company and product profile into this application. The software prompts you then for other functionality or other features that might be of interest, and then we generate a regulatory strategy. That gives you an idea of what a Class 1, Class 2, and Class 3 pathways, obviously I'm talking about FDA here, look like for your particular company and your particular product. And that takes into account or resolves this issue of context, specific information that isn't available

Via other means. Okay. So we talked a little bit about the future, and we'll talk about the call to action here. I think it's safe to say, based on everything that we're seeing here, that our roles are evolving. the way… that compliance is achieved, or the way that we go about compliance processes is changing. The way that we're going to get to a safe and effective Submission, product, or company. is going to remain the same, right? Because at the end of the day, it entirely depends on what it is that we do as quality professionals, the culture that we impart on our organizations, the knowledge that we have, so on and so forth.

The tools that we use are not going to be what leads to a safe and effective product. They're only the thing that helps us on this journey on this compliance journey. So, today, those tools that we're using for compliance look like templates and checklists, some generic document management tools, generic authoring tools like Word and Google Docs, etc. In 6 months to a year, I'll suggest to you that there's going to be a lot more AI-generated outputs that we're going to end up supervising and reviewing, and it's going to be a lot more around validating proposed directions, where we actually look at that proposed direction or that proposed decision, and we get to decide whether that's what we want to do or not.

And so, where it leads us is, in 3 to 5 years, we're acting more as benefit-risk officers than anything else, right? We're applying our knowledge and understanding, most importantly, we're applying our empathy and our understanding of what… how an organization, functions and what it takes to execute on some of those proposed paths. We're integrating a lot more cross-functionally, we're weighing, this innovation versus compliance drag piece, and we're really acting as the conscience of an organization. And for many of us, what I wrote here is essentially the way that we're functioning today. Only we're changing the tools that are in our toolkit, to help us a lot to reduce some of this initial friction that might exist in creating a document, or in answering the question of where do I start, or do you have a template?

So my closing challenge to you. Few statements first, right? Native AI tooling, I think, will become commonplace in organizations, whether in quality, regulatory, med tech, pharma, etc. Companies that are not ready are going to fall behind, unfortunately. Just the cost and time savings are pretty material for this not to happen. So when it comes to, is AI going to help us, or is it going to be part of the problem? I'll give you the classic regulatory answer, which is, it depends. it really depends on what it is, or how is it that we end up working with these tools, right? Are we going to be hands-off, and we're going to just allow

you know, generative AI to happen to us, and… You know, not really navigate, or not really help navigate where the solution is going? Or are we going to mold it? to something that, is actually going to be useful to us and is going to lead to a much better result. And so, 3 things that you could do this week. Try ChatGPT for self-audit. Audit some SOPs that you might have. It's… it's… obviously, I'm not suggesting put in IP… into ChatGPT, that's a whole different session, but you can strip some of that information and really do a self-audit. Play around with purpose-built tools, right? Try OneComply, Google some other tools available out there, find your device, find some predicates, see how that helps, or it doesn't.

And then start reshaping your role, right? Really think through your role in the organization, and if your role entirely revolves around creating documentation, really think through where else you might have value. End? reach out to me. I… I love this topic. I love the world of quality and regulatory. Feel free to set up a Calendly session with me, even if it's just 15 minutes. Just… just… touch base, and let's discuss where this world is heading, and… and head there together. So, appreciate your time. Thanks so much, everyone.

Thank you, Ari. … I have to say that… That was genuinely inspiring. For me, and … I have not seen, somebody walk through the steps. of, … using AI as it is, and where we're going with it, and how it could be applied Particularly in regulatory and quality work, at that level of detail, usually I see a lot of hand waving. AI is amazing. It's gonna spin on its head and spit nickels. And, you know, the greatest thing since sliced bread. I also found it interesting about the, …

Shadow economy, growing up around it, and the security implications and the privacy implications of the data, and the idea that people are taking proprietary data and putting it into public view, inadvertently, probably. Like, hey, they're asking a question inside of, I don't know, large OEM, nameless. OEM, and they're asking an innocent enough question, but they're actually sharing proprietary data. You know, and that gets even worse in pharma.

Oh, yeah.

To say the least, because, you know, that's super protected information, and you have these people using these tools, haphazardly, so not having, guidelines within your company for AI usage, and the… what information can be put into AI, and which information cannot. I think that we're in a super interesting period, and then, ….

If there's interest around follow-on sessions for data security considerations, or for IP considerations, and how you might be able to Guardrailed at an organization, … Happy to maybe structure something along those lines.

I am going to ask everyone for the last time to complete the survey, or the last time this week to complete the survey. … the survey will be live until the 27th, and I'd also like those of you who attended today's session to put in a plug on my recent post about the session, that this was a fantastic session, and that people should take the survey. And that, we need your participation. And so, let's, … help, … help each other. Again, I've… I've asked for feedback. Does anybody have any comments or questions Yes, Paul.

Great presentation, I think it was fantastic. I have a question about your thoughts, ideas, and do you have an offering for an enterprise class AI… framework, platform, Because a lot of what you were talking about works at an individual level. And I didn't see automations, I didn't see agentic AI business automation processes, things like that. Is that a separate offering? Because everything you talked about, I think, is very relevant for consultants to help them with their process, like, I remember Leo and I, every time I spoke to him for 6 weeks, he's like, I'm working on these slides, they're taking me 4 hours a slide, because he was doing the, you know, perfect ChatGPT use case, which was referencing the old version of the regs and the new changes coming up, right? And that's his genius on the 60601 space.

I use the chat GPT.

But when you go to an organization, what are your thoughts on that? Because I've come from that space, and I think that's where the market is. Really, there's a huge need, I'm just curious.

Yeah, so that's the direction that we're heading. So, throughout this presentation, you would have seen some… some hints as to what's gonna happen within, you know, the next year or so. At the moment, we are focused on helping individuals, so through this regulatory search component, in the background, and I'd love to show those interested in a smaller setting. What it is that we're working on, but we are working on some enterprise tooling.

Who's the we? Tell us, what's your background? So, so what….

One Comply was founded in November of last year, and it was founded by myself and a co-founder out of Boston. He's a developer of MedTech, working at an IBD company. And so he's lived this generation of AI solutions and creation of, software and hardware products in this world. So we're both extremely passionate and from this field, with this background, and that's what we decided to work on, is how do we, like, accelerate? How do we use native AI? How do we use agentic AI in our world, purpose-built for our people, right? Purpose-built for med tech, rather than trying to use generic document management stuff. So, reach out, we'd love to show you what we have.

Schedule a call already, thank you. Awesome. Sounds good. ….

So, this session, first of all, I had a question earlier, will your slides be available? I would say, go ahead and put your email in, Ari, and tell people if they want your slides, they should email you. That's the best way to get them. Secondly is, this session will be will be on YouTube, on the replay channel, hopefully later today, early tomorrow. Sorry, I just lost my screen because I'm pressing too many buttons and talking at the same time, and I'm an idiot. Stephanie, you had a question?

Yeah, I had a question. I just recently used ChatGPT just to find out how current the information was, and I just asked about a specific regulatory topic. And I got really outdated information that was, like, 2 or 3 weeks old. And then I asked it, are you really sure this is the most current information? I still got the outdated… information that was 2 or 3 weeks old. Is there… it… are the people that are… behind ShopGPD doing this on purpose, that you buy the paid version. Or what's going on there. And the other thing that I noticed recently was I used to get really good links to some of the comments that were made, and some of the statements that were made, and now it seemed to be linked to consulting groups and stuff, and

So maybe the updated, regulatory information, all the links were… went out to consulting groups. Sort of… Asking you, what is going on right now with… that particular AI tool. Thanks!

Yeah, so… I'm not an employee of OpenAI, so I can't… I can't quite speak to what their business model is and whether they're deciding to direct people to specific links for one reason or another. In terms of the outdated information, that's always going to be true, because there's always a lag time between the training component or the actual observation. or generation of content, and what's coming up next. The best way to deal with that, that I've found anyway, is, if you have a source, so let's say it's a new MDCG document or something along those lines that hasn't been. index yet. Just feed that source directly into ChatGPT, and then, ask questions then on that source. And so force it, essentially, to read that particular document. I'm using read very loosely here. …

To read the doc and figure out, whatever the answer is to your particular question. In terms of referencing consulting companies, etc, it's really… the current querying mechanism on the web. I don't know that they have a great way of differentiating between … authoritative sources versus not. I know that that's something that they've been working on, but I… I'm sure there's more work to be done there, which is, again, some of the issues with using generic Tools for this sort of thing.

about 2 weeks ago, there was an article published on TLDR AI about how, the 50% about a recent change with ChatGPT, and that 50% of the answers are now coming from Wikipedia or Reddit. But when I probed a little bit deeper into regulatory equality, it said, yeah, that's not really true. That's more general knowledge. We're moving people to Wikipedia and Reddit as the primary source because we're able to rely on that, but when you get into a niche area, it still goes to blogs, authoritative websites. And so, to answer Stephanie's question, that's why they're surfacing information from consultants.

Primarily because those are the only people producing content that's publicly available about those niche topics. Warning, Ronald Bowman said a week ago that he had written an answer on his own blog, he wasn't certain about it, he put it into ChatGPT, and it referenced him as the author of that comment, and so he took it down, because he was the only verifiable source that ChatGPT could find. So, you are kind of on… you know, thin ice in this area, where you're looking for authority on a particular area. This is why I emphasize the importance of experts and expertise, and servicing that expertise, and creating

answer engine authority rather than search engine authority. And so, structuring… answers that people are going to be looking for, on your website, either in a blog or in an FAQ, where you're responding to, Stephanie, for example, they're looking for information about Switzerland, you know, creating content around stuff that only you know. You know, and so what we're gonna do is we're gonna take this webinar, we're gonna transcribe the webinar, and we're gonna make publicly available in small sections the questions that Ari was actually answering. So, if you take a general statement, and you say, yes, but what question was he answering? Well, the question he was answering was how useful is…

an LLM for me as a regulatory affairs professional. The question that he was answering was, well, how do I make it work for me? The questions that he was answering were extremely helpful, but they weren't set up as Q&A. So the ability to translate raw commentary, raw expert knowledge, and your ability to translate your expert knowledge, I recommend you just pick up your phone, you hit the record button, and you just talk for hours. And then you take that information, and you put it into ChatGPT, and you say, structure an FAQ around my knowledge base. I am an expert. These are the things that I know about that nobody else does.

This is the information that I can give the world that is unique to me. Take your… knowledge, and start downloading it, categorizing it, putting it into question and answer format, and making it surfaceable for the world. That is what the future looks like. from my perspective, for experts, make your… take your expertise, bottle it, can it, and sell it. That's my soapbox.

Great point, Sean. And great presentation, Ari. Thank you.

Yeah, and thanks for the 61 people who are still here on the call at nearly 10 after. We're gonna go ahead and wrap it up. Ari, I don't know if you put your email address in the chat.

Yep, it's in the chat. The Calendly to book a 15-minute catch-up is also there, so… I see some questions that we're just not gonna get to, but I'm happy to, ….

I'll send you.

through those in the.

pile. It sounds good.

Yeah, perfect. I can reach out directly then.

Okay, I want to thank everybody. I thought today's session was incredibly helpful. Our speaker for next week, has an eye infection. He recently had surgery, so I'm scrambling to replace next week's speaker, but I think I have somebody on the hook. However… if… I don't, and one of you has a suggestion. There is a possibility that, you know, we have an opening. So, with that in mind, I wish everybody a happy weekend, I'll be working all weekend, and I hope you're not. Thank you.

Okay, guys.

Everyone, take care.

Bye.

Thanks, Ari, thanks, Sean. I'm sorry.

We are a long time.

Yeah, good to see you again.

Happy Holidays!

Bye.

Thank you.