CREATE on AI for Regulatory Affairs + Live from Rome with Team PRRC

Yes.

Oh, that's strange. Right? Better. So…

Morning!

Greetings.

Hi, Sean, I still had to register. If I click on the link, nothing happens. Strange.

I don't know what link you're using. The one that comes…

I actually got an invite for the first time.

Yeah, you're supposed to get an invite.

I wasn't getting them before, I got one this week, so…

Huh. Well, I'll look at the list and see if Luke is on it, and see if…

I still have to register when I hit the link. I think the problem is if you hit the join button, you have to register, but if you hit the link, you shouldn't, so I'm gonna try that next week.

Alright.

at the link, but… Anyway, you're in Rome now?

I'm in Rome.

Lovely, isn't it?

the event, running on fumes, so if I stop making sense today, don't wonder why. Any less sense than usual at any rate. And… I'm promoting Francesco.

I'm already a panelist, I think.

Oh, you are? Yeah, I have to make you a co-host. make co-host. Confirm. Right. Now you're a co-host. Now you can do anything that I can do. Consulting.

Where are you located in Rome, Sean? part.

Aor?

Hey, or…

I don't know, kind of.

Aurora is the, is the business, neighborhood of Rome. Let's see that. Where's all the big companies are, and all this stuff.

Okay.

Yeah.

And if you're a tourist like me, you go to Trast Avera or something.

Yes.

I'm dropping everything this morning.

It's strange.

That shows that gravity still works.

Interesting.

It's almost then, Sean, when they're presenting, just mute yourself, because I can hear a lot of background.

Yeah, we got background music in the lobby here. I'm already checked out of the hotel, so… You have?

11 right now. You can go.

I see Martin is here. Is Dina joining us?

Can you hear me now?

Yes.

Perfect. Let me pull up my PowerPoint as well, it might be handy.

So, while we're getting started, I just have a couple announcements. We have a minute left. So, I'm obviously here in Rome, so I'm going to try to stay on mute as much as possible, because I don't have a private place to do this call from. But I'm also going to give an update from Team PRRC. Those who haven't seen my posts, from Team PRRC have, posted an article in two posts. about what's happening here. Lots of excitement about, the proposed changes on MDR coming from, the EU Commission in mid-December. Lots of anticipation around that, and more news from TMPRC. In the meantime, I'm evaluating two new webinar platforms.

And would very much like to not be on Zoom webinar ever again. As long as I live, it's the worst platform that I can imagine. Many of you have had trouble logging in, receiving emails. Some people that have subscribed… unsubscribed from the list are never able to attend another webinar ever again. Apparently, they're blocked. So, you know, there's… there's all kinds of issues with the platform, so please be patient, with me and with the, the technology changes. It's not my favorite thing to do, but it's something that I'm working through. And more exciting news to come. In the meantime, I would like to introduce today's speakers. I met Martin and Dina at… from Create AI, at RAPS. They were… they had these nifty bags,

What did it say on the bags? Cute.

We had two sides, do I have a bag here? Can't see them right now, but, one side said, FedEx me the floppy, I will, I will, I will fax later, and then it said, like an obituary, regulatory AI, regulatory without AI, 1974 to 2025. And the other one, which is more, more or less our all-time favorite, also from the Brussels wraps, was, Still doing, RA without AI. Cute. That was our slogan. By the way, Dina has trouble joining Zoom, so your words were indeed very, very accurate there, Sean. And you yourself are mute now, I think.

I will send her a link now.

Oh, now she says she's here, so…

Okay, good. Francesco, please promote Leo and Dina.

Yes.

Martin, if you want to start, you can go ahead.

Sure, I will, share my screen. Here we go. Yes, please? Fancy feature. Well, now you see my head in there. Okay. I hope you're not too distracted by my face over my slides. If you can't read, I'll just move. Okay. So, what we call regulatory augmented intelligence. I'd like to share a few slides. More interestingly, perhaps, I'd like to show you the tool, and Dina, as an experienced regulatory consultant, has some real-life examples for us as well. So, who is Create? Create is the company founded by Dina in 2020, in the midst of a pandemic. She thought, you know what, I'm going to start a regulatory consultancy company, based on the fact that with good strategy, you can get your customers ahead of the curve.

And then, in 2022, we encountered AI. So this is a history of what we call regulatory augmented intelligence. It's a very short history, as you can see, it's only a few years. Because in 2021, dinosaurs roamed the Earth. There was some AI, but we didn't see it yet. It was in research labs, and maybe on your iPhone classifying your pictures, but nobody really knew about it yet. And that changed, actually, in 2022, when ChatGPT came out, and some visionaries, Dina among them, thought, wait a minute, this is something that can process text and can combine things and analyze things. This might be really good for regulatory affairs.

That led to, 2023, when ChatMDR was released, followed by Chet IVDR, Chet FDA, and the Chat AI Act. And that could have been the end of the story, right? So everybody had a perfect RAI tool. But actually, as with many AI tools, we also found some resistance. The year 2024 was marked by people kicking the tires of the tool and putting some exotic example in and saying, see, it doesn't do as well as I do. Whew, my job is safe. And that is changing, and that has changed. Actually, in this year, what we've seen is more of a position of, okay, AI is good, we can use it, we can use it to our benefit.

But how? And this is a typical cycle for a technology that is known as the Gartner hype cycle. You might be familiar with it, and it goes something like this. There's a technology trigger. something happens, 3D printing, AI comes out, blockchain, which leads to a peak of inflated expectations. Everybody thinks, oh my gosh, this is going to save the world. And then, unfortunately, usually it's not as great as first seen, so everybody is disappointed. And then, slowly, the real applications come through the slope of enlightenment to the plateau of productivity. So this was where it was invented. Then, here at the peak of expectations, people said, okay, AI is now going to write my technical dossiers, it's gonna do my audits, fantastic.

That turned out to be not that easy yet, at least, so we're not quite there. And where we want to get to, of course, is the plateau of productivity, that these tools actually do work for us. So where is everyone now, today, on the curve? They're analog people. They still miss those facts from the talk bags, they are not adopting, they will not adopt until everybody is on. Then, at the peak of inflated expectations are CEOs. You hear sometimes in the news, CEOs saying, we fired our entire customer support staff because we have AI. almost everyone else is in the trough of disillusionment, right? They have Microsoft Copilot, their employer said, okay, the CEO said, go, be more productive, here's Copilot. And he said, but, but, doesn't work for me, it doesn't do anything.

I think nobody's really on the plateau of productivity yet, with the exception of a few use cases. Maybe software coding is getting there, making funny pictures for PowerPoints is a good application, but that's not for regulatory, I don't think anyone's really there yet. So where are you? Because you're listening to me, I know that you are here. You are on the slope of enlightenment. You have done the co-pilot thing, and you're now wondering, okay, how do I get productivity? How do I come to that plateau? And that plateau, that slope, sorry, is your competitive advantage. Because if you think about it, in 5 years from now, AI will be in everything, and all the devices we're wearing, all the software we're using, and it's gonna work pretty well.

Even the fax users will grudgingly adopt or retire, maybe, and, you know, otherwise they'll just go away. The big difference, I say 5 years, and that's actually conservative, some say 3. Those 5 years is very fast compared to how fast it was for the smartphone to be adopted, or email, that took 3 to 5 times as long. So, in the next 5 years, the pioneers. will experiment. They will find out what works well. You know, find the things that work, and come back to the things that don't work yet. This means they will get productivity and quality gains before everyone else. And right now, what you can do is find a need in your department that will create stickiness, something that is easy to adopt and works, that really brings benefit, so people are not skeptical like the dinosaur, saying, you know, this won't work for me.

So, coming to the tool, ChatMDR, and Chat IVDR, and so on, and so on. What makes it special? What sets it apart? First of all, the information it retrieves and combines it with your queries. So it takes information from the regulations, be it FDA, MDR, IPDR, guidance documents. That information is curated by humans. So it's not scraped off the internet, including, you know, Reddit forums and things. It is really detailed and up-to-date information. What also is very important for us is that data is kept private. So the data people put in, whether it's their chats, their copy-paste things, details about their products, files that they upload.

That is not used to improve the product or train the models, not with us, and not with our AI vendors either. We've opted out of all of those. Your data is your data. And what this leads to is, in a safe environment, a high level of factual accuracy. I will never say the tool is perfect, please be the human in the loop and check, but much higher accuracy because of all this knowledge integration than, let's say, a ChatGPT or a Gemini. So, like I said, there are four tools at the moment, MDR, IVDR, FDA for devices and IVDs, and the EU AI Act. And each of these tools has their own distinct data set to really keep it clear when you have some, for instance, some of the European regulations. When you talk about Article 10, it's very clear which Article 10 that is.

Hallucinations are, or that's the phenomenon when an AI starts to make things up, right? So when you ask it a question, and it doesn't have enough information, or the model is not strong enough, but it wants to give you an answer, because that's how it was instructed, that's called hallucination. Now, how does ChatMDR minimize that? It actually adds a lot of context, so articles from the regulation, MDCG guidances, TeamMB documents, the Blue Guide, you name it. It puts the relevant pieces, not everything, the relevant pieces of that with your query. Then it provides references, so you can click on the reference, and I'll show you that later in the demo, and you can go in very easily and check, okay, is this actually what this article says? Is this correct?

And then you can get reliable and verifiable answers. So this is what it looks like. It's a regulatory sidekick, really. It's the virtual junior colleague that you never had. It's laser-focused on regulatory. When you ask it for a recipe of pumpkin pie, it'll respectfully decline. It's always up to date, right? If you're like me, you're always a little bit behind reading. This thing keeps up to date. There's actually a date on the screen at the bottom here. I don't know if you can see my mouse pointer here, you can see the date. Two weeks… sorry, one month ago, it was last updated. It's always available. Some companies like to give regulatory people some spare time. This thing is always on, so if the product manager wants to know something 11pm on a Friday night, that's fine, the tool is there.

Okay, before we go to the demo, one word about file uploads. We have a special enterprise version, where also files can be integrated, so not just text and chats, but also files, Word, Excel, and PDF. And then the users can ask and collaborate questions about those files. So, think of a technical dossier or a standard operating procedure. You can put that in the tool and ask questions about it. Now, to do this, we added an extra layer of security in the Enterprise version, which is really a separate server in the Amazon Cloud. Now, as someone pointed out to me yesterday, Amazon didn't have a very good week this week, but our services fortunately, kept on running.

But in the Amazon cloud is where all our servers are anyway, and for the enterprise, you get a separate server with a single sign-on connection to your organization to really keep your data extra safe on top of the already high data security of the normal version. So, in a nutshell, regulation trained AI with a lot of contextual intelligence, very high level of security, and that enables us to do document integration as well with proprietary information that is, of course, of very high value if you're a manufacturer or a notified body, or a regulator. So what are some real-world applications that people use? For instance, change impact assessment. You know this situation, perhaps, where the R&D department says, oh, we're making a small bug fix, and while we're at it, we also change the computer.

Is that a minor change? Is that a significant change? What's the regulatory impact of that? They could actually use the tool to figure this out before they do it and confront you as a regulatory person with this conclusion. If you are in a conformity assessment, you can, of course, create answers, but also, if you have formulated an answer yourself, have the tool be the devil's advocate and challenge that a little bit. Claim validation is a big one, where marketing claims can be checked against evidence, documents. Vigilance reporting, let's say you're active in 20 countries, something happens in the market, where do you have to report, by which time?

And, very important, I think, is education. So learning, onboarding of new employees, getting people up to speed, and growing new regulatory people. So, what are some others say? Here are some of our friends and collaborators. So, Basil, his view is that AI is a way for organizations to get the most out of their people, right? So they're highly qualified people, but there are not that many of them, and this is an augmentation tool. Eric has a slightly different view. He has experience in cases where human bias… human bias can come from experience, like, oh, we've seen this before, it's always like this.

But it's always a… it can also be a risk when, oh, this is probably a case like that, and it's not, then you might go down the wrong path. He says AI is objective, it can actually eliminate this bias by being a judge, more or less. So, let me go to the demo. I will,

Before we go to the demo, I'd like to open it up to other people to comment. Basil and Eric both have very valid opinions. What is everybody else's opinion? Anybody would like to join in, and share their idea about how AI can augment or detract from Human experience, human expertise, Anyone?

I have one question.

So, you say you always use the latest, data?

Correct.

But then… How do you prevent it from being a system like garbage in, never out? So, you have outdated data in there. Who decides that the data is outdated? That it's… that there are new insights? what I see is when I use AI tools. Sometimes you get responses from an outdated… and it's based on an outdated version of Python, or it's based on an outdated idea of how you should do things. But it was so common in the past that there's so much data on it that it keeps bringing it up. What is your approach to circumvent that?

Oh, that's a really good question. So, the data we put in are things like guidances and new versions of a regulation when someone adds Article 10A or something. So that's curated and responsible is, is Dina herself as an expert. And this information, this up-to-date information, is provided to the model first. So you have a question, you have these 30 pages of text that was on my slide earlier of context information which is up-to-date. So, it won't have to go and find the outdated information that was in its training dataset that you have when you. have a general purpose model, which went over the internet like a hungry hippo and ate all the data that's out there. So, the outdated information is in the AI model, right? It consumed the entire internet, more or less, but it will prioritize the current information.

Now, what we haven't had to do, but what is also a possibility, should it occur that it would go for outdated information, is we can add instructions as well, up front. Saying, look, don't read that anymore, because that's outdated. MDD is an example, for example. You know, if you ask now MDR questions, then it will ignore all the MDD references.

But, if I may add, Martin, so we train the data on… so we have the, the data that we train the model on is, our, is the data that we gave it to it, and that's data that, we… I review. And also, if there's some outdated information, or part of the MDCG is not applicable, we take that data out. So we pre-prompt our AI tool to look at the data that we train on first. And if it doesn't have that information, you have an option to browse the internet. But because we also give references where the data is coming from, you can always click and check where the data is coming from, and you can check it. As a human in the loop, you can check where the source of the data is from. And you would…

definitely in many cases that we test the data, see that these data are the data that we train the tool from… with, and I can show you more example.

So two people have their hands raised? I think, Faisal had his hand up first.

Yes, I would like to ask about a video I saw that had to do with ChatGPT, and I wonder if it applies here. This was a musician who claimed he had basically broken ChatGPT. What he had done was, he had given it an assignment. And asked it about factorial of 51, I believe. The reason he asked this question was because he had made a video, or somebody had made a video about a deck of cards, and how many combinations you can make of it. And this data really was only available on how to pronounce that, number, was only available on a YouTube video. And he was able to figure out that ChatGPT was not trained on the YouTube videos, it had other sources of data.

He asked it to pronounce the number. He said, give me the number, write it down. And it first tried to avoid answering, but then it tried to answer, and the answer was completely wrong. So, even though it could solve the problem, you could say, with the numbers. It didn't have data to pronounce, you know, the number itself, but it gave it a try anyway. And it failed in trying. So, would something like that be an issue in this system as well, and how would you address that?

But let me first, while I answer your question, let me just try it. It's dangerous, right? I've never done this before. Pronounce the factorial of… 51. Now, you all know this demo was not scripted, right? So I'm asking Jet FDA, but it doesn't really matter. It may refuse, we'll see what it does. So what you're explaining is, it's two things. First of all, it's an edge case. And secondly, it is something that is probably outside the training set, certainly outside our training set right now. So, that is the dangerous area of hallucinations. So hopefully now the model will come back and say, well, that's really not my area of expertise.

But otherwise, underneath is still the GPT-5 model, right? Which is, by many benchmarks, the smartest model in the world. But I'll be honest, it has its limitations, obviously. It is not a PhD level in everything. So, that could happen. Where our tool differs from a general purpose model is that it will, for regulatory questions. Yeah, in Dutch, we call this by the hunt. So it, it, it refused to do it. Let's see what it thought. In the meantime… Yeah, so it, it was coy about it, I think that's the English translation.

I will say, of course, there are ways to break the model. If you're going to find the edge cases, probably there is. if you use it within the intended purpose, I would almost say, then it will get the right information, and it will give you answers that are relevant to regulatory affairs. But it is, the underlying model is the same, right? So we don't train $500 million AI models. I think nobody can, except for maybe 2 or 3 companies in the world. So it is possible. In this case, I think we just got away with a sly answer. But, that is possible. You can break it if you have an edge case, I'm quite sure.

I found that video to be useful, and it kind of… it told me that we should probably constantly be aware of how these things actually work when we listen to the response and.

Yeah, yeah, I fully agree with that. I would call that AI literacy, actually.

Yeah.

Yeah, good point. More questions.

the… this is Susan Needle. The question or comment I had is… so I've been using a tool called, Rhizome, which is very, you know, similar kind of mindset, where it's… it's looking for… you know, different FDA warning letters, CRLs, and 483s, and guidance, and regulations. It looks into Europe. The big watch-out, I guess, I had was… unless you are approaching any of these, AI tools with scrutiny, you can really get yourself into trouble, because they do seek to please, right? They try to give you an answer, and the answers can sound really great. They can even give you references to documents. But you do have to take the time to actually go look at those references, because sometimes the data in the document referenced does not actually match up quite well.

to the conclusion that the AI model is drawing, so… I've seen that less frequently with ChatGBT5, But I've seen it in a number of AI models, where that's a… that's just a big watch out for, like, as long as whoever the human being doesn't fall into complacency with whatever the tool is that they're using, and they use it, but you have to have a level of expertise going into it, then I think you're great. If you go into it and you're a naive user. you know, straight out of school, oh, I just graduated, got my regulatory degree, and now I'm gonna write a submission, and I'm going to go use this tool. Watch out, because you might go down the wrong path.

No, I fully agree. I often compare AI, actually, to, indeed, a junior employee, and if you ask a junior employee a question, or you give an assignment, you know, make my 510K submission. And, then the employee says, you want to see it before it goes out, boss? And you say, no, it's fine. You wouldn't say that, right? And with AI, it's no different. So indeed, be the human in the loop. I think it's important. Just like with the junior employee, I mean, it's a good practice anyway in quality systems to do reviews and checks. Do apply that principle here. I can't agree more.

Martin, that exact thing happened to my team. Like, I've got a brand new employee right out of college. One of the things I tasked him with was, give us, you know, regulatory rundowns every week, you know, keep your finger on what's happening. And one week, he was like, well, it was a slow news week, so I just, like, looked at some random articles. I found this one that's talking about, and it started out, the special 510K, parentheses, SBD, And then the whole rest of the article, it referred to a special 510 as an SBD, and an SBD is a small business designation that doesn't have anything to do with Special 510K.

Because he would… because he was brand new. He would… how would he have known that that was totally wrong? And they had used GenAI to push, publish an article. So, yeah, it's very dangerous, to your point.

But you knew, right? You knew.

Exactly.

You could be… you could be the reviewer loop here. Yep.

The carbon base.

Okay.

cases that we actually see, less often with our tool, and I, I can, again, when we are showing you the demo of the tool, you can see it for yourself, as I, I also invite you to use the tool, but, so, Our tool is really trained on the core regulation. The data that it gets, initially, is on the regulation itself. It gives references to regulation itself. So, I have been using this tool for the past two years, and I really see that, that, common mistakes that you see with ChatGPT. I use ChatGPT as well, so I can see, compare it, and I see less often that common mistakes coming out.

And, we also tested this tool with, RAC exams, and we see that it actually passed the exam with a really high score. And, so… Yeah, so I just… all I can tell you is try our tool, and we are open to feedback, but I… from my experience, from how we prompt the data, because we prompt it also, and as part of the development, we see less of this, Incidents happening with our tools.

But I think that you nailed it there. I think it's a matter of trying it, and, you know, the tool is even free to try. You can sign up and go for it, doesn't cost you a thing. Find the use cases that work for you, get a feel for what is it capable of now, what is it not capable of yet. Indeed, let us know. We'd love to build it out and add capability. But it's not magic, right? I told you already in the PowerPoint, we're not selling perfection here. I'd love to, although it would be expensive. It would be very costly for you, perfection. But, right? So, it's a tool, it's an aid, but it's not.

I thought that the example, that you gave about the, rep's, practice exam was good. You know, they have a publicly available practice exam. I assume that's what you tested it on, not on the actual exam. And, you know, that is a good illustration of the idea of going to a core curriculum. looking at the information and being able to answer a specific set of questions within a realm of knowledge. This is not… for example, for Michelle's example, where she's like, tell me what happened in the regulatory world this week, your tool probably wouldn't help with that. I don't know what the answer to that is. It's not… it's not trained on… on the very latest data, and it's not…

oh, by default, go search the web. Rather, it's, you know, this is… this is not a substitute for ChatGPT. This is not, the same as ChatGPT. In fact, I'd like you to explain, you know, a little bit more, about how it differs, you know, that it… that… that it is… a subset of knowledge specifically trained to answer regulatory quality-related questions within… within… and then, more specifically, you know, how it can assist, a department Or one of… one of your clients, one of… one of, you know, the many consultants on this call's clients,

in, you know, short-cutting the… the way that… the way that I use tools like this, and they are tools, let's be clear, they're not magic solutions, as you said, they're tools. And the way that you use a tool is very important. You know, if you use a tool incorrectly, it doesn't work. And so what I would say is. How… Can the people on this call then share this information with their, with, with their clients, and work with you, and discover, you know, the value of the tool that you're presenting?

Yeah, well, yeah, I think there's more than one question in there. So I often compare ChatGPT or Gemini or Claude to a very smart person who knows a lot about a lot of things. But they're not specialized, so that's why we send humans to university to become really, really good at something. That is how you can compare. So our tool has been to Regulatory University. is a person coming from university perfect in all? Answers, always no. Humans nor AI. But, what you might have noticed when I typed a factorial 51 question is that, it actually took quite a while for the answer, for 3 words to appear. That's because it spends quite a lot of time thinking, so our settings and the amount of information we're putting into it are also different from a general purpose tool, that almost when you release the enter key, it starts talking and it starts spooing information. I mean, we'd love that performance, don't get me wrong.

But it is, it is actually, because of all the extra analysis that goes on in the background that it takes a few more seconds to do that. So, how would, how would I recommend that, that you, you, you try this? Play with it yourself. I think… play, I mean, explore, evaluate, and how… how we in Creative have been using it with our customers, to actually usually first We use the tool ourselves, and we get great answers. Dina can maybe share a story about a meeting she was in. In a few minutes, she had a great answer, which turned out to be correct. But then, at some point, as consultants, we also like to, you know, of course, elevate our customers and get them more self-sufficient, so that they don't have to come to us

with every little question. We'd love to build the hours, but it's, in the end, better for them also if they can be more self-sufficient for the simpler stuff, and not be dependent on an external party. Dina, you want to share the spare parts story? You are on mute, I believe.

So, I was on the call with a client, and they asked me about spare port in Europe. I know UMDR regulation. quite good, so, I could answer them, their questions, but then they said, well, we have also a device in the US, what about that? And then I, said, okay, they said, well, perhaps we can, discuss it in another meeting, whereas I said, oh, let me ask my tool. And when I asked the tool, it came up with also references to 21CFR, so I pulled it out. on the, so I can show you an example, for example, for that. put it on the screen, and they said, wow, how could you know so quickly the answer with the references as well? And then I told them, we have this tool, and that's actually how our chatbot,

became more commercial, because the customers actually asked, can we have access to this tool? Because we also want to have access to such references and such knowledge that you are using. So, another example I can share… do you see my screen?

I will stop sharing, one second.

Do you see my screen? So, I was on a call, and then the customer was mixing, rework and servicing. It's quite often I see this, they said, okay, we send the, product for… to the client, and then after that we do rework. And, well, as you know, rework is during the process, before you release the product, and servicing is after. And then I asked this question to the chat FDA, and it gave the answer, and if I, for example, click on this reference, you can see it goes directly to the servicing, and you can read it yourself, together with your client, or, Or another example I can give you is, is a… this is a RAPS question.

And, they… as a U.S. manufacturer, produces Class 1 medical device, the mark that is sold in Germany. The author's representative is located in the Netherlands. A technical medical device is sold outside the EU. An incident happened in Japan and leads to FSCA. So, now, what should the manufacturer do? The US-based manufacturer, and so, as you see, it refers to Article 87 of MDR, and also what should you do when, FSCA happened in the third country. And if you… if I click on the MDCG 2023, and it opens it, I opened it before. So, it refers to MDCG 22 and 3, question 18 specifically, and then I go to question 18, and it's, indeed, is FSC undertaken in third country, which is device, legally is on the EU market, and what you should do.

So, this is the level and references, and I, each time I'm using the tool, I'm also amazed myself how accurate it is.

That's a great example. We have two hands, one from Luke, one from Faiswell. I don't know if my hands are still up from before, and I didn't lower them, or… Is that a no.

No, it's a new question. Is your tool, Smart enough… to say I have a lot of documentation, and in one document it says, let's draw… let's print a line here on the device, and it's clearly visible on the… on the… on the drawings, and that would make it a measurement function. And… It was totally obvious from the drawing, but not in the small line. Would your tool… Suggested… Please watch out with this line, because it will have effect on how it's classified.

at the moment, the tool doesn't analyze pictures, so drawings would not be taken into account, so it depends all on how this line is specified. If then the text said, you know, requirement number 23.5, we draw a line, then… Well, depending on the context, but I don't expect people to say, wait, align that, could be a measuring function. But if you…

tricky with the. It's tricky, but those are the questions that you come up with when you analyze your current set of data.

Yep. Yep.

I wish…

No.

So, I wish we could use it on our current corpus on data, but… Our company is now 40 years old, and I thought, well, let's click… right-click on what do we have, can I put it into OneDrive to analyze? And it was counting over 40 gigabytes, and then still counting as well. Let's not transfer that to OneDrive.

No, one of the things that AI models do quite well is it's called semantic search, it's a little bit technical, but it means it doesn't actually have to go through all those 40 gigabytes to search. And that's also what context is. A semantic search is, if you have a keyword search, and you look for apple, you find apple, and apples, and applesauce, but you won't find strawberry. If you have a semantic search, and you look for fruit, you will also find Apple. So if you have a question about markings on a device and measurement, it will, in your 40GB, find that. It doesn't have to load the 40 gigabytes of data into memory.

So, I mean…

It's worth a shot to analyze that. AIs are relatively good at that sort of thing.

Yeah, what I tried to say is that, we have to use Copilot mandatory.

Hmm.

for leaking out business information. But I want to search from the corpus of existing data that's just on a network drive, and Copilot doesn't work that way.

No, unfortunately not. Maybe they'll get around to it. They have access to the AI models, but the co-pilot implementation is, yeah, it's probably good at other things. I haven't found them, but they're probably good at other things.

Martin, I see another… I'm sorry.

You said you can, You can, sign up for free.

Yeah, you can, yeah.

No, that's delicious.

or any of the other websites, and you can see…

Wide open.

And, and try it for yourself. By the way, you mentioned Copilot and information safety. I'll say it again, that's also what we do, right? So just like Copilot, we don't share your information, and we'll put that in writing if you want, so… No problem there.

And once you have access to one tool, you have access to all the other tools, so one account is enough. Martin, somebody's asking about prompting, so, maybe you can also show the prompt library quickly, if, Sean, if we have still time on it? Yep, okay.

Sean is nodding. Okay, let me take the screen back, share, I'll just… Put my face away, you don't… you've seen me enough by now. So, prompting is indeed a bit of a skill. So, of course, you can type or paste in whatever you want. Pro tip, by the way, AIs are very good at making prompts, so you can also ask it to make a good prompt for you. But, to help things along, we have actually a prompt library here. Let me get a new chat, and then a prompt library. So this is for a chat FDA, so it has American things in there. And there are some prompts that are tried and proven for these things, so device classification is a favorite of mine, and you can actually, use this as a template.

So, classify my, endoscope, I'm just gonna try this. with intended use, the viewing… the mucosa, I'm not a doctor. And key features, recording… on video and analysis with AI. Let me make it a bit… So if I now say OK, Then this goes in my prompt box, and I can send it off. And then, now, in the FDA context, it's still chat FDA, it will give you a walkthrough of the classification. So the prompt library has, and will probably extend that, where you can save your own prompts, and as templates.

So that's an aid for, not having to start with, like, this intimidating blank screen that says, what do you want to ask in the blinking cursor, you know, blink, blink. So it helps you along the way, and hopefully it's also educational in the sense that when you see a few of these prompts, you say, okay, that's how you should ask a question to an AI model.

I can also share another example of…

So what is… before you do that, Dina, what it's doing now, you see these little, the fine print, that is not to save you a trip to the optician to see if your eyesight's still working. This is the inner dialogue, the reasoning of the model. So it's, it's actually sort of talking to itself, and you can use this, first of all, now you can see where it's going, but you can also use it if it took a wrong turn afterwards. You can open that, as I just did with a factorial of 51, which I think someone put in the chat. I saw a paste of a very large number. You can kind of see where's it going, and what assumptions did it make, if any, and then if those were wrong, you can correct for that.

So it's, thinking hard about this one. Like I said, a lot of information is being processed. What were you going to say, Dina?

So, I wanted to share another example for junior employees, as well, that, I had a case, that the client, so I'm sure… I think I'm sharing, no?

Are you? I don't know, I still see my own screen, obviously.

I'm seeing executive summary generally in an electronic order.

Yeah, so this is, so this is, we are working with the incubator in the university, and the guy at the university came to me and says that it's my electronic prescription system is a medical device. So, normally, sit with him and explain, I just put it in our chatbot, I put an equal here. But, then it came… went through all the MDR and also the MDCGs, and it says. how, it's, if it's… in which cases it becomes a medical device, in which cases it's not a medical device, and it gives, also information on how you classify it, which really would fall under it. So, it's,

again, I think it's, pretty accurate. I read it with my experience, and I, so with all these references that you hear, even, you can see Article 2, Definition of medical device. You can click on it and check for yourself, and it's referenced to the actual regulation and the MDCGs, and nothing more.

So I like that example. My question for you, Dino, would be if you asked it the same question again 5 minutes later, would you get the same answer? Because I'm somewhere on the disillusionment curve, where, I've created some very extensive prompts that I use over and over again, and get wildly different answers, depending on you know, when I ask the question, or not wildly. I get… I get variations on the same animal. Where they've just completely reworded the whole thing. And I find that very frustrating that I'm not able to duplicate my work. So, does your tool manage that any better, or is it…

Well…

Same deal.

What I can tell you is that, for example, we have run the RAC exam 59 times with R2, so statistically valid number, and it always, when the question is clear, it always gives the same answer. So, the questions that they were clear, and there's no, reason to, you know, that it's not ambiguous for the tool. It gives 100% always the same answer. And, well, it gives different wording, but it still gives the correct answer. And, when there were cases in the… on the RAPS exam, I can also show you that example… that the questions were ambiguous, and that it can be interpreted

In different ways. And we saw that it's sometimes, like, it's a 60 person gave one answer, 40% gave another answer. But because it's justified why it gives this answer, you, as a human in loop, you can still check it and draw your conclusion and say, okay, based on this, I believe this is the right answer. So, yeah, it doesn't replicate exactly. But I am, I am confident that it gives you the correct answer. Based on our, exam, practice exam.

Yeah, so just to add to that, it's like you said, it gets to the same factual answer, but the formulation might be different, which is what Sean said he found annoying sometimes.

Yep.

Recently, actually really recently, I think the day before yesterday, our AI supplier published a white paper on how to get more consistency. Now, what we have to experiment with is you can put a certain seed number in, and that will actually fix the model in a certain way, but we want the answer to be the best way, so we don't know what the best seed is. So we'll actually have to experiment with that to create, let's say, verbal consistency. I think the content consistency is there, based on our validations, and the RAC exam is a beautiful example. But the verbal, it's still a little bit different sometimes, and that can be annoying. I had the perfect formulation, and now I have to find it again. Of course, there is a history function, you can go back and see it, but I see what you're saying.

So my classification example is also.

ready if anybody wants to see the classification of an endoscope with AI in it.

Yes, please. Also, Mr. Jose has a question that he put in the chat about whether or not you're essentially using a rag.

In a very short… if you… very high level, it's a rag with, fine-tuned weights and, prompting. enhancements. So, actually, every, every query that goes to the system is a two-step approach. So… It is… so RAG, for those who don't know it, by the way, it's called Retrieval Augmented Generation. I have an AI model, and I put a lot of relevant information on top, which is what I described earlier. So that is the core of it. That's how we started back in 2022. But, there has been, let's say, 2 years of enhancements in software development on top of that, so… It's a rag in the same way that a Ferrari is just a car.

Yeah, kind of my follow-up around that was just, if the RAG contains, presumably, a lot of the standards that are copyrighted, do you run into issues there where you're using copyrighted material for the augmented generation?

Well, it doesn't contain the standards, because then we would run into issues with copyrighted material.

Got it.

So, so you're able to pull from the public available things like guidance documents from FDA or the EUMDR.

But if you had questions, for example, on a specific standard, does it then just default back to the training model, or…

The way to use standards is, so I explained the enterprise model, where you can upload your own files. So if you're a manufacturer, or a notified body, or whatever, and you have purchased the appropriate license of the standard, I believe there is always this network license that means you can share it with your employees on a network drive, that license is appropriate for, use in our tool as well. So then you actually, just to use the word RAG again, you actually build your own RAG with the standards that you have the rights to use. So you can do that.

Got it. And when you upload it, does it take care of, like. conversion of PDF to, you know, whether it's Markdown or whatever.

It makes it in such a way, I won't go into detail because I put everyone to sleep, and I know that Sean wants to report from news from Rome, but it creates it into a format that is digestible by the AI model.

Got it. Thanks.

So, I would suggest, Sean, I would invite everybody on this call, just try it. If you see we are open for feedback, feel free to write us an email or connect on LinkedIn so we can answer the questions, but we don't say that our tool is perfect, but so far, it has been always impressing me, so…

That's personal, feedback that I have. And in case anybody wants to make an endoscope with AI, it would be a Class 2 device, but there are predicate devices available. I haven't even talked about the predicate device tool, the FDA classification, which you will find, I'll leave that as an exercise to the reader to try that one here. So, any endoscope manufacturers, rest assured you can do a 510K pathway, you don't have to go to PMA.

And you mentioned that, the people that are trying it can also upload documents? their own documents?

Well, the enterprise version allows for documents, but that's a server we have to set up, so, to make it really blunt, you have to pay for that. But what you can do if you want to run a trial, is you can paste, from Word or something, you can paste right here in your, context window, in the, sorry, in the text box. And you can include your own documents that way. That is not suitable if you want to do 40 gigabytes, then I would happily offer you an enterprise server, but for one document or a passage of a document, you can paste it in the chat. It will take quite a lot of information.

Wonderful. That's… that's great. Well, I want to thank you both for coming. I do want to spend a few minutes, sharing with people what happened here over the last two days in Rome, but I encourage you, to reach out to Dina and Martin with any questions. comments or concerns, feedback that would be helpful to them. you know, I am an AI enthusiast, and I appreciate you coming here and sharing what you're developing for the RAQA community. I think that, I look forward to the developments as they move forward, and, and, enjoyed meeting you both in Pittsburgh.

Yeah, same here, Sean, thank you.

It's expensive.

the opportunity.

Yeah, so I'm just gonna move on to my, program update. from, Team PRRC. Team, PRRC is the, persons responsible for regulatory compliance. It is an EU-based, association, primarily. There were a few people here, From… from the U.S, but it was almost… almost all people from… from Europe. There were 100 attendees, and there were a number of sessions, all of which were really interesting. One of the most interesting ones to me was, Richard Houlihan's Utimed update. Now, I'm, you know, sick to death of Utimed updates.

In the sense that, you know, nothing ever changes. They keep, you know, pushing out that it's not ready, it's not ready, it's not ready. However, one of the things that he talked about was, machine-to-machine uploads. So, apparently, updating your information in Utimed is kind of a pain in the butt, and, particularly if you have large numbers of UDIs to, introduce into the thing. And, Both his company and a couple other companies that I talked to here are now offering machine-to-machine uploads of information into the system, so if that's of any interest to anybody, we can certainly have somebody come on and share what they're up to at that.

The other thing was, is that, this year was in Rome, Italy, so we had, 5 Italian notified bodies here, which is, the most Italian notified bodies that I've ever seen in a room anywhere. And a lot of interesting perspectives. Perhaps the most notable perspective is that Italy is really the only country in the EU where there are fines for the PRRC. And they explain the rationale behind, creating a fine system. Apparently, there are people who have are acting as PRRC for dozens, even hundreds of companies, and they, may not take their responsibilities very seriously. It's not…

Go… not to say that anyone with dozens or hundreds of PRRC jobs is not taking them seriously, but, they're using it as sort of a stick. The incentive is, you know, to, to comply with the legislation, and the stick is, if you violate it, you can be fined between 20,000 and 120,000 euros. Depending on, I guess, the severity of the violations. But, you know, so, this evolving role. a person responsible for regulatory compliance, it sits between RAQA and management, and is really kind of a hall monitor, is my understanding of the position. And there are,

tasked with being a whistleblower, and the practical advice that I've taken away from this, Conference is that the person has to use a lot of judgment. Not every problem is fixable. Companies will have problems with their QMS. Companies will have problems with their production. Companies will have problems, and they don't all rise to the level of, this needs to be flagged and reported to the authorities. immediately. They need to address the situation with tact, and that a lot of times, the PRRC doesn't have the authority to actually implement any changes. So now, we've all been in positions where we had a lot of responsibility and no authority, I'm sure.

And, so that's the classic dilemma, where you're telling the owner of the company something that they don't want to hear. Aren't unable to hear, or unwilling to hear. And, And that… and that leaves these people in… in an awkward position, which leads to, liability and insurance. It's highly advisable that anybody that takes on a role like this, protects their self, from any liability, and insists that the person hiring them, insure them. take out insurance, that it indemnifies, the PRRC, from, any potential liability, because it… it does, it does exist within the law.

There were other, reports on clinical evaluation, clinical investigation. Regional updates, and I provided a number of those on, on my LinkedIn, so if you have an opportunity to check out the article or the post that I did while I was here, please take a look at them. As I mentioned earlier, we're gonna be, transitioning to a new platform away from Zoom. There are so many issues with the Zoom platform for presenting these, but one thing that I wanted to raise was. The idea, one of the platforms that I'm looking at only allows 10 people on camera at a time. Now, would all of you be terribly disappointed if you couldn't be all on camera at the same time?

I'm just, you know, thinking that 10 people on camera is probably enough during a group webinar, or whatever we're calling this, but, I'm…

Yeah, but then you're kind of taken away from the original concept of this Friday meeting. The entire Friday meeting isn't the data, it's the relationships you're building in this team.

Yes, and everybody would be able to raise their hand and come on camera and comment, it's just everyone would not be live all at the same time. Which is not necessarily a benefit when we're watching somebody eat or sleep. You know. So, I agree with you, Nellie. I do want to, Maintain the integrity of the… of the purpose. But I'm trying to find a platform that isn't quite as user-unfriendly as this one. Many of you have mentioned problems with the links. I think the answer to Luke's problem is not to use the join webinar link, which generally does not work in Zoom, but rather to copy the URL and paste it.

I don't really know why. they can't seem to make their own technology work, but they cannot seem to make their own technology work. That's important.

URL doesn't work either for me, so…

Oh.

I'm here.

Yeah, best… best way is to copy the meeting code and the password and, like, enter that directly into the Zoom thing. That's the only way I got in today.

Oh, good lord. Well, I would like to thank everybody for their time and for their contribution, and I'd like to thank our speakers again. Please connect with them and give their tool a try, and provide your feedback. I appreciate it. Does anybody want to add anything before we leave? Nellie, were you okay with my answer?

I was, I just wanted to make sure that we revisit the purpose of the meeting, just because I think it's important when we're doing changes to understand the… or how we… how we started.

Okay. Thank you for that. Great, everyone. I will see you next week. I will let you know who next week's speaker is, just as soon as I figure it out. Be back from Rome on Monday. Bye, everyone.

Thank you, Sean.

Thanks, guys.

shine.

Thank you, everyone.

Thank you, everyone. Bye.

Bye!

Bye-bye.