AI in Regulatory Intelligence, Clinical Evaluation & PMS

Thank you.

Hey everybody, welcome. I'm Sean. I'm glad you could join us today. Looking forward to an exciting, presentation, AI and Regulatory Intelligence Clinical Evaluation in PMS. Our guests today are from MedBoard and Antonella Lombardi, and even Perez Chamorro, and very happy to welcome them. I just saw them in Baltimore, at the Regulatory Intelligence Conference, and I just wanted to give them a moment to introduce themselves, but then ask them, you know, what they thought of the regulatory Intelligence Conference, and if anybody here on the call was at the Regulatory Intelligence Conference, please give a thumbs up or something like that in the…

in the… Deerhickey. So… MedBoard, hello, welcome, even. Antonella. Ciao.

Thank you so much for having us here.

Thank you so much, Sean, and thank you to everybody as well for joining.

So there were a lot of people in Baltimore, it seemed like about 200. For the… the conference is growing every year. It's really impressive. What did you do?

There were a lot of people, so I think these… these conferences that are growing but are still targeted, with targeted people and audience are great, so to having also more targeted discussions, right, on… on the role of RegI Intel in that case specifically, but yeah.

I have challenges seeing my, seeing the Zoom, so I can now see, faces. I can see Leo. So, Leo, hello, very nice to see you. So, yes, I think Multimore was a very good conference. I think things are becoming more mature, especially with regulatory intelligence, so the conversations are much more mature, are more focused on Specific goals and specific, processes, outcomes as well. And I think the conversations are, as well, more mature in terms of what the AI can do for you, and how to use AI within those processes.

Okay, so, yeah, just, just a couple notes, before we get started. We, had a busy week, we have a busy week coming up, so we did, human factors evidence in… for legacy devices on, on Tuesday with, like, 700 registrants. It was huge. And we have this great, human factors, checklist, it's like a six-page document that we built. So anybody wants that, just write checklist in the thing. I'm also gonna drop in some links to some replays and to some upcoming events. Most notably, next week, we have two… well, three webinars, really, including the Friday call.

And I'll drop notes about those into the, into the chat as well, so I don't spend too much time doing advertising. Let's see, and if anybody has, any, Questions, would like to, be a speaker, wants to figure out how to, you know. build their business, reach out to me, obviously. That's what I do. And… I think that's it. Let's go ahead and start the presentation.

I want to try to share a screen, if I can. Is that working now?

I can see it, but I can see the navigation down the side. If you go to View, and you do Slideshow, it should.

W. Ladies.

Yeah. Okay. Now you should be able to… now it looks perfect. Everybody see it okay? Yeah?

I cannot see the… one second…

the people.

Yes.

In the upper right, the view. There's a little… thing with… Oh, that's interesting.

Let me see…

That's side by side.

I think I'm gonna prefer.

No, no, it was… Second ago, it was perfect.

Oh, really?

I mean…

Where he had his image floating over the top.

Let me, let me, let me do a more rudimentary, because otherwise I cannot see you guys, so I cannot see the… the chat, and so on, and as well, if there are questions, or, you know, so I think that, we can use it like this. I know it's not perfect, but, you know, we can, crack into that.

Okay.

Thank you so much, everybody, for attending this, this, you know, webinar, this open discussion as well. So, you know, we're gonna share from our side, you know, how, how we automate, especially, like, compliance processes with literature, reg intel, vigilance. My name is Ivan, so I'm the CEO and founder of the company. With me is Antonella, the growth of director, and been helping a lot of companies implementing these processes. You know, during the… during the last years. Before, obviously, we start, I just want to say thank you to… for… to… to Sean for having us here, and as well, you know, to… to promote as well the discussion, because I think these discussions are very important.

For… for the industry. In order to understand, you know, what is possible, what is not possible, and so on. So, what I would like to cover, especially with the slides, is, you know, what is our experience, and as well, cover as well how actually it's working in practicality, and some nuances about AI, so what is working and what is not working. So, Sean was mentioning before about, Baltimore, conference, and there was a lot of, discussions about AI. a lot of discussion as well, how to perform those processes. So, for those that don't know much about us, just to give you a bit of background, so we're talking about this topic because a lot of companies use MedBoard, so this is not used by 10, 20 companies, so we have more than 150 companies. We have more than 99% retention over all these years. We are based in the UK.

Even if we operate globally. And as a person that, you know, myself, I work in regulatory inside of companies for more than 15 years, we obviously, we comply with certain certifications, regulations, and so on, and obviously, we take validation very seriously as well. I will cover a little bit how I got into this as well, because obviously, it's because I was exposed to these, processes in the past. But one of the key things, you know, for you to understand is we talk about this topic because this is what we do day after day, and this is our focus to help, you know, companies with with these issues.

Moving to the next slide is, why do we do this? So, this is… this is the key thing, and those that have been in the industry for more than 10 years, 15 years, probably they will remember when FDA was only releasing a guidance every month, or every quarter, and every time there was a new guidance, it was… it was like a celebration. Okay, let's… let's have that guidance. let's review it, let's see the impact, let's put the comments, and let's prepare a presentation. I would say, like, when I was working in big companies, we had time, you know, to review the guidance, create a presentation, explain to everybody what it meant, and plan with time, obviously, the actions. Life has changed massively, obviously, since those times. Back in the days, as an anecdote, my first role when I started my career in regulatory affairs, more than 15 years, I was working with a small consulting company with Neil Armstrong here in the UK,

Okay? And one of my tasks was to check websites. and to check all the updates on websites. So, this is the first time that I experienced the pain of what is… what it is to go through every single website and understanding what has changed, what hasn't changed, and all those specific impacts that you need to do. And back in the days, it was just collecting in a Word document, collecting in Excel, and 15 years after, some companies companies, and some people still do the same. Obviously, there are some advanced intensive light technology, so we will cover as well, you know, what it means in this.

But obviously, there are, like, two things that have changed, in my view, is… first is the increase of data and news within the organization that are available within the industry, and as well, the speed of those changes. So, the speed of those changes have changed massively. As well, their requirements, or the demand for having those requirements, that information, and to report this not only internally, but as well externally, that has changed as well. The level of expectations has changed, very much. And if we look at the pain points, the pain points are mostly, too, across all these processes that we're going to cover, especially for reg intel, vigilance.

literature, and so on, is manual work. I don't know how many of you in your CV has, as one of the tasks, check, manually, websites, or check, copy and paste from Excel, so check and paste from documents. I think no one of us, because obviously, as regulatory professionals, what we want to use is our brains, our capacity of, Decision-making to make decisions, but as well, obviously. to do different impact assessments. So, the first main point is manual work, especially for those tasks, they don't add a lot of value. So, obviously, it's not scalable as well, because obviously, if someone learns how to check manually websites, or check how to do those impact assessments manually.

Then, if that person gets replaced, or that person goes to another department, someone else has to create the same process and keep the same level, which it creates, obviously, a risk to the organization, but as well, you know, to the team. And as well, this becomes a person-centric knowledge, so it's not transferable very easily across those. And to be honest, this is very tedious, repeated, and boring. Because if you have done this specific task, as I've done in the past, the first time could be very exciting, the second one could be maybe a less exciting, but what you need to do this 10 times, 20 times, 30 times, there is nothing exciting about it.

what, as regulatory professionals, we appreciate is to actually review the information, draw conclusions, and obviously suggest actions, implementations, and so on. So again, there is a core reason as well, in terms of the… this is for helping people with the job satisfaction, with the core value in terms of what they do, you know, for a living. Now, the other, interesting pain point is isolated or disconnected processes and data. Means that if you use two different systems that they don't communicate to each other, still somebody has to move that, data from one place to another, and guess what? It will be manual. So, you will depend on a person doing a very manual task from one place to another. So, especially, like, having a lot of multiple solutions,

Especially when there was not a lot of, all-in-one, solutions. it created a lot of, you know, in terms of, like, vendor management, but as well, learning, training, so it just creates, obviously, more complexity to the, to the companies. And if we look in the… in terms of the… who is actually affected by this, is… it's across the board in any company, as well in consulting companies, and so on. So it's regulatory people, clinical people, R&D people, PMS, commercial people. and so on. And all of that represents a risk to the organization. So, from, you know, from the story from Medbor is we started all of this because, personally, I suffered those, and I thought, okay, there must be a better way to do this, because I'm spending most of my time actually checking things, manually.

And obviously, this was, 15 years ago, 10 years ago, we started, Medbor, 7 years ago, with these, processes. Now, what I want to, you know, as well to share with all of you is this was our vision from the beginning. So, we've been setting, as well this category in terms of all-in-one. to connect all the data, and as well to use AI and automations to eliminate any specific manual task between those. So… Our goal is to have an all-in-one compliance and intelligence automation in one single platform. We were the first one to have rake intel literature and PMS vigilance together. The first one, as well, adding a reference manager, the first one having a citation manager, and the first one that has everything connected into a product's informational rims.

So, what I'm saying for this is we are innovators creating a category and connecting everything together within a platform. Having said that, we have APIs, so if a company already uses other systems that they trust and is embedded, we can connect that. But what we wanted to create is an ecosystem where people can connect all those solutions without the need of those manual steps. Now, there are, like, four reasons why we do this. First is knowledge access. So, back in the days, when I was, as well, presenting to companies or to borrow directors, I used to, like, help them with strategies for the market. People used to ask, how do you know this? And the answer was much more simple than I spent 30 days researching, is I just needed, or I knew

which document, or which guidance, or where that information was. So, it was actually the information was out there. Obviously, since then, until now, technology and AI have changed, obviously, how you retrieve the information, and the speed of that information. The second thing is processes productivity. Everybody wants to do, obviously, the processes much faster, and more reliable, and more robust as well. Why? Because you reduce risk to the organization. And that's a very important part in the compliance, but as well as an organization, operationally speaking. The third thing is company governance. So, you want to ensure that the company operates very efficiently, ethically, as well aligned with the best interests of the shareholders.

And the fourth one is job satisfaction. And this is how I relate to this, because I suffer these pain points, right? So, if you automate, simplify, and connect, you know, and make easier the work and processes, obviously the job satisfaction of those professionals will be much better. And our goal is to automate, and as well to simplify, those tasks that are boring for the specific professionals, right? So, when people come to us and say, you know, I need AI to make decisions for me. And I ask, okay, how do you do this right now? So I copy and paste everything, I check the websites manually. But I want… I need an AI to make decisions for me. So, it's actually the opposite. You need, actually, AI or automations to automate those specific, like, collection of information, and then you need you with that context, because obviously it needs a lot of judgment, the regulatory area, to make those decisions. So, one of the key things as well in job satisfaction is.

It's very important where we use automation AI, so the professionals keep feeling, obviously relevant, valuable, and as well. You know, in terms of, like, satisfaction with their job. Beyond, obviously, this innovation and ambition, we have, obviously, a lot of, like, reg intel across 225 countries. We cover globally, multiple regions, regulatory areas, and so on. We cover as well in terms of company profiles, and so on. So we wanted to have a real, knowledge access, obviously with what is public, out there. Moving to the next one, so this is kind of like representing what I was mentioning before, is the advantages of having a single global platform. Again, we are offering this ecosystem, but this is the aim that every company should aim. Whether you take pieces from, you know, one vendor, another vendor, and then you put them all together, the future is a whole orchestration of

the different processes, and connecting all those processes together. And obviously, regulatory professionals, they are focused on making decisions and ensuring that those processes are efficient, but as well, they are operating well, and there is a continuous improvement as well for those processes. So… We focus, for example, on our side, we don't focus on consulting. We don't focus on advisory. We understand, and I understand from the years I've been in the industry, that making decisions in the medical device industry, in companies, is very, very difficult sometimes. Why? Because the context. You need to understand the context. What are the stakeholders? What are the specific issues? What is the past with our organization and the authority?

So, obviously. what we do all here is to focus on the technology, the AI, and as well in the global data. And when we talk about global data, it's the hard intelligence. So databases, data that come from public sources that you can easily verify. So that's, from our side, you know, that's our job. Obviously, there's an area that is, as well, soft intelligence, other areas that is the knowledge and experience of the professionals. And that's the big difference between one of you professionals with one of, maybe they have less experience. That's why people choose you, because at the end of the day, regulatory is about trust.

It's about ensuring to make the right decisions. So, moving to… for you to all and to understand, you know, what we offer, so we have an area in terms of, like, global data, and a lot of power-ups, so from, regulatory intelligence, compliance, RIMS, product information, but as well in clinical, literature reviews, reference managers, citation, market and PMS vigilance, as well, to monitor vigilance databases, a watch list, and everything is connected with a task manager. Why? Because one of the key challenges for companies is not detecting information, but implementing those actions actually after detecting that information. We do, as well, standards management, as well as standards monitoring. I know Leo is in the audience, so I just want to highlight that we cover standards. And then we cover as well, as I mentioned before, 225 countries, so anything that is

published out there, we actually, you know, cover. And one of the key advantages in our platform is real time. So, over all these seven years, we created a system that allow us to be real-time. So anything that is being published today, it will be on the platform here. And it will be curated as well. So it's not only about getting anything that is published by the authorities, it's already created within the day. in our platform. Now, I want to just move away from talking about us. I want to talk about a bit, you know, what is the importance of these processes and learnings as well for last years. So…

I just want to simplify as well, because we're going to focus on this webinar mostly in regulatory intelligence, clinical evaluation, and PMS. And at the end of the day, these processes are very, very similar. You need to collect some information, review the information, make some decisions, and update documentation, right? Or databases to show compliance, or to show that you have the data up to date. So. In terms of the clinical, evaluation, PMS, and regulatory, at the end, we have three clear principles, is what it is and what has changed. And this is one of the key things that has changed, basically, over the last 10, 15 years, is, obviously, what had changed, now it goes at much faster speeds, because obviously authorities and sources, they have better means to publish information much faster. So, all this automation AI, what it's actually creating is more people are able to publish information much faster.

The second thing is a core principle for regulatory intelligence, but as well in terms of, like, literature, PMS, what it means to us, what it means to the organization, what it means to you, what it means to your organization. So whether it's a new paper, whether it's a new regulation, whether it's a new adverse event, what it means to you, that's the judgment which is very important and remains within organizations, and Those are very key risk decisions that that cannot let, you know, be performed by external AI without any oversight, because that can determine any critical actions straight after.

And obviously, the third part is what to do, implement, and most important, as this is a regulatory webinar, to record it, because if you don't record it, obviously, it hasn't happened, so you do not have the means to actually show that evidence. So, these principles actually apply across regulatory intelligence, clinical evaluation, and PMS. Now. If we look into, you know, what is happening right now, maybe you feel identified a little bit, and then maybe that's you, maybe you have a few more colleagues in your team. But, obviously, the updates that are happening, I always use the analogy of a sea. The waves are not waiting for you. So, whether you are looking at the sea or not looking at the sea, the waves keep coming.

Whether you are in an audit, or whether you are traveling, or whether you are on really nice holidays. the waves will keep coming. And this is continuous across different type of, jurisdictions, and you multiply that across standards updates, regulatory updates, literature updates, vigilance updates, that creates, obviously, a pipeline with a lot of information. I don't know if many of you have experienced this increase of updates from, like, 10, 15 years ago. You can put it in the chat as well, if you feel that, obviously, things have Increase over… over the years. One thing that I want to cover is, as well, what is the process, right? And many times, and I don't have the animation, sorry, because SpongeBob, it was meant to appear after I said I click, okay, but because otherwise I cannot see the

the rest of the, the screens. So, many people think automation and AI does, like, you have a one click, and that's a very complex process, regulatory process, and it's completed end-to-end. I've seen many times as well, people saying, yes, I click one button, and then I have a CR that is completed end-to-end, and that's ready for a notification body. As a regulatory professional, I've been involved not only in… on the manufacturer side, but as well, helping authorities or different stakeholders at that level. That feels quite difficult, without any oversight. So, what is the solution? It's not magic. What the solution is obviously to break down processes and to see where we can use the specific automations and AI.

I don't know if any of you had this experience as well, like, people, like, clicking on… or having these expectations of. you click on one button, and you're gonna solve the problem. I have to say, it's very… it's very tempting, right? Especially when you have a big challenge and say, you know, I wish I could just go to chat GPT or any tool, and then I'm gonna click a button, and the problem is solved. But actually, the nature is actually much more difficult. So, what I would like… I like always to, and this is one of the central, you know, slides of this… of this presentation, I would like, in conferences and different webinars, to show how actually you can break down the process, and whether you're doing reg, intel, literature reviews, and vigilance, and we can dive in with, you know, any questions as well related to them, is you have a process, and the process is very, very similar.

As I said before, so you have, obviously, a place where you need to collect results, you need first to have a forms and a price file, so the first… before you start, you need to decide what is the flow chart, what is actually the forms that you're gonna use. What are the questions that you're gonna inquiry that new data? Then you, obviously, you collect the data, then you will have, obviously, duplicates, suggestions, you can do filtering. And then you will do any screening in Reg Intel as well, it's called triage. You want to identify as much as fast as possible which items are relevant to you, which items are not relevant to you, and focus your resources on the ones that are relevant to you.

And obviously, you will need to then read in detail the full text, all the PDFs, or, you know, the content of that specific news, and then you have different type of areas where, you can report, right? So you can do an extra report, award, you can have Prisma for literature. You can extract, obviously, tables, analytics, or you can transfer that data to specific compliance databases. So, you need to have, obviously, the list of requirements, the list of standards, the list of regulations, and so on. RIMS, or up-to-date information in your RIMS, and technical documentation. And between the full-text reporting and the up-to-date, obviously, there will be many tasks that you need to,

You need to complete. So, that's the process, and as you can see in the principles, if we look at the simplicity of the process, whether you are doing REC Intel, literature, vigilance, the process is the same. You have a systematic view to review new items and information, and obviously to appraise in a consistent way. Now, if we want to use AI supporting tools during the workflow, there are so many. So, for example, we offer more than 10 during this process, so they suggest keywords for highlighting, translations automatically, potential duplicates, screening, overviews as well, PICO extraction, you have a PDF chat mode, forms for, you know, answers, data extraction.

requirements extraction automatically from regulations and guidance, it saves you a lot of time. And we have, as well, AI writing assistance and AI for ask multiple PDFs, or to build, you know, different type of reports across different PDFs. So, as you can see, the approach that we take in here is not to, obviously, just one process, click on one button, and you don't have control, but to have, as well. Multiple tools that, depending on, you know, your risk appetite, depending on how you feel comfortable with one tool, one another, you use those tools that you feel more comfortable, or they switch better the task that you are performing as well.

So, what we always said, especially for these areas, is you need to have a defined process, because obviously you can digitalize them that way. Otherwise, if you don't have a defined process, it's quite difficult to do it. And one of the key things is to have intuitive user interface. So, in terms of the adoptions, what is very important is to work very hard on the user interface. And from our side, we always take very seriously Any feedback related to user interface, or any feedback that can enable us to improve the user interface. The user interface as well applies to AI, so AI, I will cover that later, but having explainable AI where you can actually understand where that information is coming from, and verify the information, that becomes very important, you know, for regulatory professions. And again, in that orchestration, that ecosystem that every company is building now with new technologies, AI, automation.

What you need to have is collaborative and integrated task manager, and as well, actions, because then you don't need to duplicate, you can visualize as well, or be transparent about the actions and the functions that have been created. Where… whether you use this for reg intel, literature reviews, or vigilance, again, this is the same principle. So, we're gonna cover, around this principle, in this presentation. Now, what everybody wants is, obviously, to have the full integration. So, the next slide for me Is our proposal on end-to-end compliance and intelligence ecosystem that we do within our organization, right?

So, there are, like, 4 main areas. So, first of all, obviously, collecting the information. You know, from our side, obviously, we cover a lot of data, but we understand as well that some data, it could be, through internal sources, or through any other source, especially depending on the country. where you have, obviously, other specific channels. So you need to customize and monitor those updates, you need to have a process, whether it's for literature, reg, intel, PMS, complaints, so it could be applied to many different processes, for doing the review and impact, and then you need to, obviously, act and implement any action following those.

And then you need to have, obviously, up-to-date and compliant, products information, rings, technical documentation, and compliance manager. In products information, I include already descriptions, UDI claims, so… because everything in that products description applies to any other process as well. So, if you are reviewing literature or rake intel, you want to know what are actually the intended views, or what are the specific claims for that And products. And you can see as well, I put the products and families in the… in the center. The reason is If you link any of those processes with the products, then you have full trustability over that product lifecycle across from the review, to the impact, to the up-to-date information. So you can just give exact data about specific products at any given time.

We don't cover eQMS, we understand that QMS is more looking inside, and then we connect through the QMS processes, through APIs. But our vision is a lot of companies, if not all, will move to this into 3-5 years, because what you want to have is an orchestrated ecosystem where you have control over the whole compliance and intelligence ecosystem. Especially for regulatory and clinical. Now, what I would like to cover now in a bit of more detail is understanding as well about automation. So, you know, my background, you know, for those that don't know, I'm a master's in physics, so actually, like, coding and all of this as well is relevant, you know, to me, AI, and so on. So, I always like to, especially for

using AI in regulatory, I like to get my inputs, or, you know, my insights related to how we can use it. And every time we talk about automations, I think we need to bear in mind that automations means the use of software without human intervention. And I always used some memes, you know, in terms of, like, ChatGPT and so on, when they said, hey, can you do this? And then, oops, I made a mistake. And then it goes, obviously, the opposite, and then everything goes well. Obviously. When you make a huge mistake, there is no going back, especially in regulatory and clinical. So, the definition of automation is super important within this context. And one of the key areas is we need, obviously, data quality, so we need clean energy structure. Data engineering, you can do a lot of automation with that.

It's very true, and this is an anecdote that I always say, that people always come to us saying, I need AI for this, I need AI for this, I need AI for this. And I always put the analogy that is, like, asking for… I need a plane, I need a plane, I need a plane to go to the grocery store, I need a plane to go to Asia, I need a plane to go to, you know, another city, or I need a, you know, a plane just to go downstairs, you know, to pick up the, you know, takeaway. So people are… going into this phase of asking AI for everything, and sometimes AI is not the answer for everything that you need. Simply, you may need just a button that just does a deterministic action to have that, that you need. So, for data science and AI, obviously a lot of machine learning and AI, and so on, it can be used.

So, one of the key things I always like to emphasize is, if we are talking about processes. So, the context I'm giving here is about processes, right? So, you can do, obviously, many different types of automations with or without AI. I keep always, like, software engineering, because it's very deterministic. You validate that input, that output. And it's gonna always do the same. But obviously, when we move into machine learning, more classical. Those that use more deterministic outputs, so obviously you can validate within that specific range, but as well, obviously, the models are more structured, numeric, and categorical, so you can do a lot with that in terms of the machine learning. But as we move to the GenAI, obviously, they are more, like, or they are more, structured in terms of, like, predicting the next token, and that's a big difference, they are stochastic. So, obviously, when they are stochastic or probabilistic.

human judgment and perception becomes very, very, very important, because then it can just give you an output that doesn't align with your context, and I always said, especially in regulatory. decisions, especially the critical decisions, require a high understanding and a high judgment to understand the situation and to propose the solution. So. This is a big differentiation, especially to make with AI in these processes, right? That not everything, or not all the gen AI can be used for all those processes, and the way we use AI can be actually different. So, it can be, just to get input, suggestions. The other could be to actually do a process end-to-end, so it depends, really, on what is your process. When people ask, I want to

you know, like, automate this process, the first question I always ask is, what are you trying to, automate? What is the… what is the input, and what is the output? Because that's going to define, basically, how you approach, the challenge. And then, what I would like to cover as well, especially in terms of the automations and AI conclusions, is if you use, like, software and machine learning automation, obviously everything should have a defined input and defined output, and that's the beginning of everything. So, what it's going to look like, that process, and which ones, like, does the required human intervention or be validated? For example, clicking a button and

do an export. You don't need a human to validate once that is validated. Obviously, you can connect the steps and data automatically when you can, but you can always do, actually, that automation. So, if you require a very high judgment decision making, you don't want to automate that step. You want to have some safeguards. And going back to the first slides, one of the key things when we break down these processes is to have different safeguards and verification points where the users can say, yes, okay, I'm good with all the suggestions for the screen. I'm good with all the extraction of data. And then you have verification points where the humans

Actually verify and take ownership over the output of that process. So, going back to this, obviously, not every process can be automated or digitized, especially the most creative, and so on, and it has to be ready to use in terms of reporting and KPIs. One of the key advantages that we have with automation is to have real-time KPIs and reporting, and this is a huge advantage for companies that are data-driven. So, if you can click a button. and tell, okay, this is my scope, all the regulations that apply to me. You can click a button and name straight away the five regulations that are high risk for organization.

that will give you a huge advantage, not only to you, not only as well for you to lead the change within organizations, but as well as a company, and as well, the perception from the top management could be different. One of the things, and I worked a lot with top management in my career. is speaking the same language as, you know, the board of directors, the upper management, and so on. And KPIs and metrics are normally the language that they understand much better. So if you have metrics that you can instantly tell them, okay, our reaction time to new updates in regulatory are 30 days, or 40 days.

that gives them some specific metric of what is the risk to the organization, right? So then, speaking the same language is very important, especially for upper management. And then I want to just to do a final remarks about the GenAI. that on my view, obviously, based on all my experience and all these years, as I mentioned before, I work a lot in regulatory. Obviously, I say that old enough that I work in all these processes, right? Because back in the days, if there are people that they've been working for a long time, the regulatory professional used to do the standards, the technical documentation, the literature, the vigilance, the biocompatibility as well.

So, I'm coming from, obviously, that time. And as well, I work a lot into global registration. So, like, Japan, China, you know, all the different type of countries, LATAM, obviously, US and EU. And I always, I'm always fascinated when people say that, okay, yes, I have an AI that can register products, in one click. You clicks across the globe. And I always said, you haven't been actually there, because the context of regulatory is as well very social, very political, and, you know, sometimes it's about communication. If you've been with, you know, colleagues from different, regions, it becomes a challenge sometimes, the communication, and understanding, you know, what the other party needs. So, especially for GenAI, my suggestion is to use that as an input rather than automation. What has changed massively over the last year is the speed of communication, the speed of the retrieving information. So, what it becomes more…

important, especially the regulatory professional, is analyzing or having the judgment whether this is good or bad information, and I can use for my specific task or context. My second advice is to use always enterprise-grade solutions. Very important. Where are you putting those prompts? Where are those going? In the prompts are more risk than any normal search, because in the prompt, you give a lot of context, you give a lot of information that it can be used to identify the issue, it could identify you, it could identify the company that is related to that. So, always ensure that confidentiality and privacy, you know, are a priority.

In my view, outputs needs to always be verified by the user, especially with GenAI. Having a verification, having an output, and then saying, yes, I agree with this, I'm gonna use this. So, explainable AI this is having a user interface where the GenAI systems you can verify the outputs with citations, highlighting, referencing, and sources. So, creating the systems where you can actually, get very quickly from where, which chunks of information these are, getting. Those chunks is super important, because we want to remain, especially in regulatory, in control, and we want to be, as well, responsible for the actions and the words that we share with different stakeholders.

And, you know, something I cover, and I cover a lot, I have a, a session in Baltimore about this and direct intel, it was about ethics. AI doesn't know what is right and what is wrong for you. And what is right and what is wrong, as well, it depends on the person, it depends on the culture, it depends of a group of professionals. So, this is a very important topic, where, obviously, the AI will not understand your full intent, the full context, and, you know, what the expert is trying to achieve for this. one of the things that we need to acknowledge, especially with regulatory, is we don't all… we don't all perceive risk in the same way. And this is very clear globally.

where COVID happened, right? All of them, you know, all of us, we had the same hazards or situation. But most of company, or most of countries, they put totally different risk controls. And that tells you, explains at a global level, a macro level. that, you know, ethics or risk are perceived completely different by, you know, different cultures, different countries, but as well, within the same country, there will always be discussions whether a measure, a risk control was good or not. If all the risk, morals, and ethics would have been perceived the same, all of us would have had the same risk control and the same measures. But obviously, this is the world that we live in, and, you know.

we are different, you know? So, you know, that's the, one of the key things. And one thing that I emphasize, you know, in my panel about ethics as well, it's very important always to do the right thing. And especially working in regulatory is, you know, pushing to do the right thing. And with AI, obviously, it will not understand whether, because sometimes what AI will try to do is to solve the problem in a in the most efficient way. And the most efficient way could be a shortcut that actually is not, more ethically or moral. So, obviously, it's not a magic wand, it's a tool, you know, that's how it's, especially at Gen AI, right? Obviously, there could be certain automations, but in my view, they need to be always, like, verified and validated. And that's why the user interface becomes very important for these type of solutions. And again.

for those that they work, you work on those in regulatory, you understand how difficult it is for AI in very complex project, because they don't call you for the simple task, they call you for the complex task. So, obviously, our perspective is we need to help more to help in the complex task, and To move away from the manual, you know, simple tasks for the professionals, so they have more time to make decisions, and as well to plan, or, you know, to have better, you know, planning for, for the, for the future. So just to finalize, I want to finalize before I, you know, I want to share, you know, a few things that we have with AI, how we automate things.

In my view, there are, like, four pillars that are super important in this process. So, the first one, the first one is the foundation is the data. You need organized and trusted data, always up-to-date, and obviously that's a challenge, but, you know, that's, for our side, is what we provide. But do you need any system that gives you always up-to-date information, trusted, that you can always verify it? the process, you need, obviously, systematic reviews, whether it's REC Intel, PMS, literature reviews, and so on. The key thing, in my opinion, is to divide and conquer, to divide the steps and conquer, and to keep it simple as well. And to use the tools that provide value, or, for example, in terms of the organizations, they may be… they may be happier to use any specific AI tools or not. This is a discussion we had as well into the reg intel, in terms of the AI governance. So there are, like, more companies bringing AI governance.

and telling the employees, obviously, which tools to use, they can use or not. Obviously, use the automations and artificial intelligence to accelerate you, to accelerate the processes, not to replace you. Obviously, there are big gains in terms of the automations, because you can detect information much faster, you can access information much faster, but obviously, there are some risks that, obviously, you require some verification, and you need to understand, you know, in terms of the Whether it's missing some kind of information and so on, so it always requires some research. In terms of hallucinization, it always depends, you know, because we can hallucinate ourselves as well if we don't recall or remember in our memory. You know, for example, if I don't remember my birthday.

and I retrieve a chunk in my memory that says it's in May, when it's in January, you may think I'm hallucinating, but my problem was in the retrieval of information in my brain. So a lot of hallucinations coming from retrieving the wrong information. And this is always what is important as well, always, to verify. And, in my view, no, because obviously I've been working in regulatory affairs for more than 15 years, and I have, obviously, key roles on those. But the regulatory professionals need to keep, as the decision makers, need to keep in control, and as well, the guardians of those processes. So, just having, obviously, the metrics and the control, but obviously, like, keeping

the regulatory professional as the important actor. So, as well, the process must work for the professionals, and not the other way around. So, remember that example I gave you? That some people were asking the AI to make decisions, but they were checking manually all the websites, you know, on the net. So, it should be the opposite. Let's use the machines and the AI to do that hard work, and the professionals to actually make the decisions. So, this is my… the end of my presentation. I wanted to show you few things that we do with AI, they are very cool, and then obviously we can take any questions after. One of the key things I see these processes of using AI is to identify faster the risks and opportunities. That's the main role of REGI Intel, you look at that, and as well to have faster compliance, or as well better compliance. So it's not only faster, but as well better compliance by having more access to, to information.

I hope you like the presentation, the slides. Obviously, welcome any questions. What I would like to have is, like, 7, 8 minutes. I show you some, you know, cool things that we do with, AI, and then we can take any questions, after. And now I need to learn how to share, right? So… If, I'm sharing this screen. I'm sharing my screen right now, right?

Perfect.

Okay, good. So our platform, I guess maybe I did this with the red, I don't know who did that. Maybe that was me. So this is Medbor. Obviously, our platform is huge. I'll just give you, like, a few things how we use AI, right? So, the first thing is we have a huge database with a lot of information. We have, research in terms of, like, search, so you can look, for example, if I want to know, like, a clinical evaluation, you know, for example. this is a deterministic search, so I always emphasize, if people want to find specific documents, the best way is to use a deterministic search. if you want to do a bit of research, what is better is to use, you know, like, some kind of, like, AI search and so on. So, I'm going to put an example, for example, what are the, for example, the registration steps for medical devices in Brazil for a Class 2. And it's going to retrieve, from all our databases and formulate an answer with the specific citations. Again, what is important from our side is to always give you the citation

Because you have the right to disagree with the AI. So the AI, obviously, may not be right in all the contexts, and so on. So you can see that we can provide answers and questions and follow-up, and so on. Other areas that I want to cover that is helping a lot, and this is very linked to the specific that we talk about, the scope, right? So, if I have a scope with all the regulatory areas, all the countries that we cover, everything comes in that dashboard, right? So it's already in here. You can create, as well your appraisal criteria with your specific questions. You can as well, add, you know, the specific context, you know, okay, this is, my company is this.

And then you can give, as well, more context to the AI. So, if you have a specific, for example, you know, guidance, so let me look for, for example, like, something that is more, a guidance, this one. So, the AI can be used… first, a lot of automations has happened here, and, you know, there is AI they use in the background, but it just helps you to have all the information at your fingertips. So, you have already your FDA guidance. You can click the AI to provide you a summary as well, with the AI assistant. You can ask follow-up questions, you know, does it cover Udi. So again, having the ability to retrieve information super fast, it will just allow you to retrieve information much faster. If you have created forms, for example, in our case, there's a systematic review, so you can decide whether it's relevant or relevant at full text. One of the cool functions that we have with AI, given the context.

he can decide whether it's… he can suggest, you know, like, backing back to our discussion, whether impacting your career. So in this case, I portray myself as an orthopedics company, and it's telling me that this is about weight loss, so it has nothing to do, so the suggestion is probably not. In order to comply as well, EU Act and many others, you can use the suggestion or not, so you don't have that obligation, and we always specify when the AI is used or not used. So, as well, you have, you know, all the questions, so it can just give you all the suggestions as well, based on that. If I go to, for example, the next one, you can create tasks and actions and so on based on this.

is, for example, this one is about PMS. I can as well do, very quickly the… the specific, summary, and I can answer these questions, right? So, this is, about PMS, so potentially this is gonna say, yes, because obviously, yeah. Because it's applying to any medical manufacturer, and this is where we've seen, as well, accelerating these processes to answer questions, you know, much faster that are linked to your organization. In this case, I put examples of my processes. So this can be put in the account, or it can be linked through an API. So it can tell you, likely, it's gonna impact this SOP, this SOP, and this SOP, because

mindset. Again, you can agree or disagree, but you have a starting point that can be used as well as a food for thought as well within organ… within the organization. We use, as well, different type of AIs as well. I don't know, we have in here… We don't have in here any translations, for translations, so you can use, as well. For any translations as well. So, automation of translations, that, you know, that's super fast. Obviously, that helps to understand, you know, what a specific regulation is about. You don't need to have a certified translation to understand whether it's impacting or impacting, or whether you need to speak with your colleagues in Japan. So this enables to have, obviously, faster, you know, communication and faster action.

As well, in… in here. One of the areas that we are working quite a lot as well, is what I was talking about metrics is obviously having flowcharts to decide how many items you receive, how many items you have done as well. Having automated metrics that, and this is through AI and other automations, that can tell you whether you are in compliance or not in compliance, and what is the end date, what is the date of enforcement. what are the specific applicability dates? That gives a massive advantage in terms of, like, communications, you know, across the teams. And then we have other AI tools that help you to

So let me see in here, so I'm gonna clear my filters in here and select the relevant. Two reports, as well, based on the inputs that you have provided in the answers to draft something around those, those, using those inputs. So again, what we always use is verification put, inputs to provide, you know, the next actions, you know, for that. So you have for introduction, so it will just summarize all the outputs that you have provided on those answers as well in here. So again, it's about accelerating. We have different type of, report options as well to use, in there. And then one of the AI tools that we are finding that people are liking very much is, for example, in their compliance manager, is to have, for example, if we look into the

Ndcg. is opening a specific guidance, and then we have tools to extract regulatory requirements. Sometimes this takes a lot of time, but if an AI can extract all the regulatory requirements, give you a citation, give you exactly where is that citation, and then put it in a tabular format. it just helps you massively in terms of speed. It's not replacing you in terms of, like, making decisions, but it just gives you the citation, the summary, and the specific requirements, and the citations where that information is found. One of the things as well we find, as well, very interesting is when you can enquiry multiple documents at the same time. So, for example, if I work in, in UDI, for content lenses, and I have these three documents, I can ask AI for these three documents at the same time. You know, where are…

Their requirements, highlighted. So, I can ask about those documents, go inside of each document. So, this is a very agentic process that looks into each document and summarizes the information, you know, across different papers. You can do this with not only regulatory documents, or goal as well is to use this to draft, you know, work instructions, SOPs based on specific requirements, and as well, you can use this for literature and references, so this is very effective as well to summarize papers across, different type of, areas. So, I would like, let's wait for the eye to start writing. I will show you very quickly as well, so you can just see how's the answer.

And the last thing I wanted to show you, no vigilance for literature reviews, we find this, again. highly effective, especially when you have a lot of articles, to use an AI screening that can learn from your reviews. So you have, you're likely to say relevant, not relevant to specific articles. It learns with you in that process, in that process, and it's gonna give you a suggestion based on that. As I mentioned before. One of the key things, you know, for us as well is the assistance, that it just extracts the information, you can chat. One of the things that is never talked is about the fatigue of professionals, right? If you need to review one article, it's fine, or one regulatory update is fine. When you need to review 10, you are a bit more tired. You need to review 20, you're a bit more tired, 30 more tired. If you have a tool that extracts the information

all the time. Obviously, this fast, you will be able to extract information much faster, and verify the information, you know, much, much faster. And if you have, for example, a different type of answers, for example, in this one, and then you have the full review in here, again, you can ask the AI to extract what are the limitations of the study. Where is this for the same device? And it's going to look for that specific information within the PDF. So, for example, is there any limitation in the study? I can use this suggestion, and now it's gonna give me the specific, citations, so I'm gonna wait for… for the answer. So if I click in the citations, it should take me exactly to the limitation. So it takes me exactly what it is, I can verify the information, agree, disagree, tweak things, and then obviously put that as an answer as well. So, for example, I put my device was not rocks, so it's identifying that this is Natrox, and it's taking me to exactly the paragraph

in there. So, this accelerates, nor replaces, obviously, professionals, but it makes the process, obviously, much faster and more in control, and as well reduces the fatigue of the professionals. So, in the same way as I show you for the for the report before, you can do as well in here a draft report of all the articles that have been evaluated, and so on. So, this is just a sample of, you know, all the AI, different type of AI tools that are integrated within the processes, because in our view, the most important thing is to have very well structured, very good architecture, orchestrated processes that communicate with

With… with each other. So, I know that, you know, it's, 2 minutes, you know, to the, you know, to the hour, but, you know, obviously happy to, to take any questions, and as I said, like, normally from. you know, provider, we do AI, a lot of innovation, and, you know, I see people in the audience as well that they've been helping us as well, you know, with inputs and suggestions over the last years. But as well, from a regulatory professional on my side, I've been working regulatory for more than 15 years, so… happy to take any questions. I don't know, Sean, if we have time, but, you know, obviously, I'm happy to, as well, to cover any question after, either Antonella or myself after this session.

I'm happy to hang out for a few minutes. If anybody else wants to drop, I understand. I thank you all for your time, today, and would like to, remind you that next week we're gonna have James Pink, who's gonna be… Talking about some amazing stuff that I don't remember offhand, but, so, just to let you know about that, and then I'll say, any questions for… Even. I believe there were… there was one in the chat that I saw. Which was just curious if the platform is AI-native, or just AI-enabled, powered, driven… okay, well… you know, I'm not sure what the difference is, but, yeah, good question.

You know, I mean… I would… I would say, you know, probably like most of the platforms, AI has been added in in the last few years since ChatGPT and the LLMs have really opened up and become a lot more accessible and, you know, that sort of thing. But, You know, this is an amazing, amazingly robust platform that you've built. You know, who… who primarily purchases this from you, and… Is… Do you have, like, a…

Yeah, manufacturers, regulatory departments, clinical departments, vigilance departments, and commercial departments as well. So, medical manufacturers, we have as well, you know, regulators that use the platform, we have as well, consulting companies that use the platform, so we want to have an inclusive platform where everybody usage, but as well, they can share the learnings across the industry, because I think one of the important things with adoption is to make this accessible, you know, across different stakeholders.

Okay, well, I know we have a lot of consultants on the call, so if, they want to reach out to you and see how you guys could work together, or collaborate, or, you know, they could use the platform, open up that channel, what should they do? Should they email you?

Yeah, yeah, we collaborate with a good number of trusted, you know, consulting or partners as well, you know, so happy to collaborate as well, so they can reach to Antonella or myself, you know, for… for that, and as well, you know, for medical manufacturers, we partner with them to improve their processes, to evaluate how is their process, whether it is in a good fit, you know, for adopting digitalization or technology. So, yeah, we're happy to provide input and to help, because at the end. What we believe is, if we can have better processes, obviously this is going to impact into patients, into organizations, and everything is going to be more in control, and the risks are going to be identified, but as well, the opportunity is much faster.

Well, I think we're about out of time. If anybody would like to be followed up with individually, please just write, you know, contact me in the chat, and I'll make sure that I pass along the text, the chat file. To our speakers, want to thank everybody again for their time, and I look forward to seeing you next week. Thank you.

Thank you so much for having us. Thank you, bye-bye.

Bye.