EU AI Act Countdown: Critical Actions Medical Device Companies Must Take Before 2027

Hello, and welcome, everybody, to EU AI. act countdown, and I am Sean Smith. Publisher of MedTech Leading Voice. The EU AI Act Countdown. Today, we're focusing on what medical device companies need to do between now and August 2027. Focusing on near-term milestones, notified body capacity, and the documentation and oversight that protect EU market access. I'd like to take a moment to introduce, today's speakers and panel. Start with Felicity Ellis. She is head of legal and compliance at the Notified Body Scarlet.

Cecile Vanderhaden of, Axon Lawyers, Attorney at Law, and… Emma… Oh, my ragage. Medical Device Senior Consultant with Accra Team. Sorry, Emma, I apologize again. I'm doing my best. At any rate, have a couple of housekeeping announcements before we get started. Number one. This webinar is being recorded, and the recording and slides will be sent to all registrants within… 24… Ours.

If you would like a certificate of attendance, please put your full name as you would like it to appear on the certificate, and your email in the chat. Use the QA button to submit questions to the panel, use the chat for comments, or other remarks to me or the panelists. We will answer as many questions as we can as time allows. There will also be a short poll midway through. the event, and there'll be a survey afterwards. Please complete the survey. Little… point, save the date, October 28th. Our team is going to be presenting a panel discussion on EIFU and labeling 2025 in Europe, not to be missed. And,

Yeah. I think that is it from me. I'm gonna go ahead and turn it over. Thank you.

Hi, everyone. So I have the honor to give you a brief introduction to the AIX countdown. And to get started with that, a really quick reminder. So the AI Act regulates both AI systems and AI models, and AI systems is where we need to look at for medical devices. And that definition includes rule-based systems, statistical estimators. It doesn't exclude logic-based tools, and it covers a lot more than machine learning. And I've assumed that you at least have a little bit of basis about the AI Act? So, what we'll do today is we will move through to a couple of high-level things you need to keep in mind before we go to the panel.

So, important, it may not look like an AI system to the development team, but if the system infers, classifies, filters, or predicts on the basis of input data and generates output, it is qualified as an AI system. And the key element is whether that goes beyond basic data storage, data processing, and handles somewhat complex relationships and patterns in the data. Ai actually is not just another tech regulation, it's also not another tick box exercise, which I know we would all have loved, but it's that it is a realignment of how The European Union looks at subjects as traceability of data, trust, and fundamental rights with respect to medical devices that use artificial intelligence.

And… it really redefines how you need to look at innovation in Europe. This is not something that's typically addressed when it comes to medical device AI systems, but core of the AI Act is Recital 1, which addresses fundamental rights, which includes gender difference, discrimination. right to health, and that is baked into the DNA of the EU AI Act, and as a consequence, that's also something that you need to keep in mind while you're looking at the AI Act. There is a really large overlap with MDR and IFDR, at least seemingly. And it also means that if you're looking at, for example, your clinical evidence strategy, but also your processes, you need to consider the AI Act and MDR IFDR.

and clinical evidence is going to remain the backbone of compliance, you just need to consider new AI-specific dimensions. So, there's a dual risk assessment now under the AI Act for medical devices that are also high-risk AI systems. If a medical device isn't… is or contains an AI system, either as a safety component, or the AI system itself is the product. And it's covered by MDR, or IVR. and it needs to undergo a third-party conformity assessment, then it is a high-risk AI system under the AI Act. And that means that under this specific classification rule. the AI systems that are either in-house developed devices, or Class 1… Class A IVDs, or Class 1 medical devices.

except for 1S, 1R, or 1M devices are excluded from this scope. They would, however, still be AI systems, so they still need to meet the requirements for that under the AI Act. but they're not medical… they're not high risk under Article 6. And it's Article 6-1. And it's important to say that so specifically, because there's also an extreme high-risk AI devices, and NXTRE lists several categories of AI systems. Such as, systems that are used for biometric identification or categorization. AI systems used for employment, personnel management, etc, but also AI systems used to determine the right of access to and the use of essential private or public services, including emergency services triage.

And a lot… and if the AI system is not a medical device, but still would fall… that falls under Attle 6-1, which falls into scope of this, it's still a high-risk AI system, and you still need to comply with the whole, high-risk regime of the AI Act. For this category, in-house developed devices, or Class 1 medical devices, Class A IVDs, that are AI systems, are not excluded, which is a category that's often overlooked. So, as I said, there's going to be… a large interplay with MDR and IFDR when we look at the AI Act. So MDR and IFDR really focus on the safety and performance of the medical device part, whereas the AI Act focuses on algorithmic trustworthiness.

There are very different definitions, there are overlapping definitions, there are very much different scopes, and yet, at the same time, there's a lot of overlap in practice. And that's mainly caused due to the fact that the AI Act explicitly says that in case of, legislate… of products that are regulated both under the AI Act and MDR, FDR, That's the provider of the AI system, which is the manufacturer in MDR and IFDR, will have the choice of integrating, where appropriate, the testing and reporting processes, information, and documentation into the documents and procedures under MDR or IFDR.

In theory, that sounds very practical. In practice, it leads to significant friction, because there really the regulations really regulate different things? So, you always need to be mindful of which regulation are you looking at for whatever you're doing, and how does it align once you're trying to integrate procedures or documents. So, the AI Act is a horizontal regulation, which means it applies to several product categories. And the same applies to MDR, and MDR and IVDR are vertical legislation. There's never no hierarchy between them. If we're looking at problems that still need to be solved, then…

Important things to consider are the obligations for market… obligations for market participants, so your whole supply chain, again, needs to be looked at. There will be a lot of need for cost correction during implementation, because a lot of the times the AI still says the European Commission will provide an implementing decision, or there needs to be standards, and those are just not yet in place. We will have concurring obligations. So, for example. reporting obligations under MDR, IVDR, and under the AI Act may, depending on what your member state is doing, need to take place at the same competent authority, but there will be two reports that need to be made in case of an incident, if both regulations are triggered.

And in a worst-case scenario, you might also need… might also need to do a tourist report at the supervisory authority under the GDPR, for example. So, that's something to be mindful of. And then there is the question of designation of notified bodies that we're going to discuss in this seminar a bit. as well as how the conformity assessment integration of the AIX conformity assessment into MDR and IVDR is going to take place. If we look at terminology, it all sounds extremely familiar, but you do need to be aware that terminology might sound the same, but has very different meaning. And for a good example of that is the post-market surveillance definition under MDR5DR, which I assume you're all aware of, but then under the

AI Act, we will have post-market monitoring, which will focus also on those fundamental rights I mentioned earlier, which is definitely not something that we're used to looking at under MDR and IFLDR. And there is a whole list of those, so this is just one example. So, as I said, MDR and IVDR and AI Act will apply concurrently, so both need to be satisfied for market access, but integration… but you can integrate the AI Act requirements into MDR IVR requirements. There will be friction points, and… Because of that, we are going to see a lot of joint guidance. look, for example, at the MDCG slash, collaboration with the AIBORC, guidance, the, AIBORC 2025-1 guidance, that is.

But also, we will see collaboration between agencies and supervisory authorities, where there will remain a significant risk of either those authorities contradicting themselves or providing overlapping guidance. Which will be something to be on the lookout for. So, in regards to timelines, early alignment during development of devices, in as far as the devices are not yet established and finalized and placed on the market, early alignment will be essential to avoid regulatory gaps. And that also means that you need to take a stance with regards to different definitions, such as manufacturer, provider, and how you're going to deal with such things in, for example, the instructions for use.

And as I said, standards are still absent, so you are working in a bit of a vacuum at the moment in your preparation for compliance with AIX. So, MDR and IFDR really establish, also, standards for clinical evaluation, safety, and performance of the devices. So, that requires robust data collection, for those medical devices. And then the AI Act also sets a data regime, but then it focuses on transparency, data quality, and risk management. And the focus really also lies on testing, training, and validation data. So, MDR, IVDR, AI8 also need to be seen as containing a lot of data requirements, but it's never the primary focus.

Which means that, on top of the integration with MDR and I for the after AI, I actually also need to think about how am I going to integrate this in my GDPR and other data, legislation strategies. So, there are also a lot of new obligations, some you might already have come across, some might be new. There's a high requirement for human oversight. There's an AI literacy requirement, which requires you that users of AI systems actually have some knowledge about how the system functions, and where it's, also its… Failures or limitations lie. There are transparency… additional transparency and information requirements.

Usually, that's going to be done mainly through the IFU, but it can be broader. There are accuracy and robustness requirements, as well as cybersecurity requirements, which go a lot further than what's required under MDR and IFDR, and also the research regime under the AI Act is going to be something you want to keep an eye on, because technically there is no research regime, but at the same time, there is the possibility of sandboxes, there is an IUO exception. And there is an exception for, pre-market research, except where there is pre… where there is, how do you call it, Testing under real-world conditions.

Which is typically something that we would do if we are trying to bring a device to market in the MDR and IFDR. So it's going to be very important to get guidance on this, and I would urge you to take, To keep a close eye on the developments in that respect. So then, before going over to the panel itself, it's good to take a quick look at the timelines. So, D-Day for medical devices that contain… that are also AI systems, or contain AI systems, is the 2nd of August 2027, when the high-risk regime for AI systems enters into… becomes applicable. But there is also a sort of conditional grace period.

for which the timeline starts applying from 2nd of August 2026. So, a year before even a high-risk regime becomes applicable. Which is going to be very interesting, and that's because of the following. So, the MECG guidance, which it is in collaboration with the AIBORT, assumes that, the regime for high-risk AI systems only applies from the 2nd of August 2027, even if the device is on the market prior to the 2nd of August 2026. That's relevant because of the following. Article 111-2 of the AI Act says that devices that have been… AI systems that have been placed on the market prior to the 2nd of August 2026

2026, do not need to comply with high-risk AI regime unless… They are subject to significant changes in their designs. be… after the 2nd of August, 2026. The guidance that we just saw from the AI bot says, well, you only need to be worried about significant design changes on or after the 2nd of August 2026. 2027, but that is not what the Act itself says. So the 2nd of August 2026 is the D-Day, if you want to be on the market. without having to comply with the high-risk regime of the AI Act. So that's going to be troublesome, because that's really, really soon.

We don't really know what significant design changes are. The only thing we have is a recital that says, well, it's similar… to a significant… a substantial modification. However, that's actually not identical. So this is going to be a lot of calls for discussion, I would expect. And with death? Over to the panel.

I'm back. So I'm gonna go ahead and launch the poll, and just a reminder that anybody that would like to have a certificate of attendance, please put their name as they would like to appear, along with their email address in the chat, and we will be sure to send that to you. Save the date, October 28th, for EIFU Instructions for Use and Labeling 2025 with Accra Team. And with that, I will shut up.

Okay, so while we are waiting for the pool to close, I'm gonna ask you, Felicity and Cecile, to share one to three key items you would like to emphasize as being pivotal to consider with this topic. Maybe, Cecile, you can go first?

Yes, of course. So, for me, I think two topics which are… essential to consider when you're looking at the AI Act are, architecture and literacy, and special… and… probably goes to briefly explain those. I just… I already mentioned AI literacy, which is a concept set forth in AI Act, but I think you also need to have literacy with respect to the act itself, so not just the products. And that may… the best example I can give you of that is the guidance. That I just discussed in relation to the Act. So you not only do need to know what your device does, you also need to know what the Act actually requires of you.

And architecture concerns the topic of the integration of the AI Act into the wider procedures and the wider legislatory framework. Because you can't look at the AI Act as a silo. You can't even look at it as a nice little package with just MDR and IFDR. You really need to look at it in the full scope of legislation that applies to the products.

Thank you, that was interesting. Felicity?

Yeah, I think my key point is just not to panic if you're manufacturing AI medical devices. I think often I say the word regulation, and manufacturers run screaming for the hills, and that's exactly what I'm really keen to avoid. I totally appreciate that there is a lot of regulatory change in new regulation that's coming into the EU at the moment relating to the data space, new technological products, the AI Act, AI regulation, in the EU, and to be fair for multinational manufacturers across the world at the moment. What I'd say is, be aware of the AI Act. It is designed to work, in tandem with the MDR, so there will obviously be an uplift, but I have some fear that manufacturers will start to build less innovative or less ambitious forms of their product.

to try and avoid regulatory complexity. And I think if there's any takeaway from here, it's not as big and as scary and as much of a change, as I think, the market sometimes likes to say it is.

Okay, thank you. So don't panic, you'll get there. Thank you. We'll just wait for the pool to close.

Oh, I thought I had closed it. Is it still on the screen?

Yeah, I didn't see it, I guess it's closed.

Okay, good.

Yeah, okay, so we can move on to the panel session. So, I will ask several questions to Felicity and Cecile, and then we'll see how far we can get. So, the first question goes to Felicity. What are the most critical milestones medical device companies should hit in Q1 and Q2 of 2025 to stay on track for August 2027 compliance, and what are the early warning signs that a company is falling behind?

Yeah, so I guess it's somewhat consistently with the earlier point I was making, if you're building an AI medical device at the moment, and you're doing so in compliance with the medical device regulation, and you've never heard of the AI Act, I don't think that means you're falling behind. I think there is a lot of movement that's happening at the EU level, and also at national implementation level at the moment. that it's, quite important to stay aware of and on top of. Cecile pointed out the MDCG guidance around the AI Act MDR interplay. I would suggest… I would have ordinarily said laminate that and put it up on your wall. These days, I'd probably say build it into the system prompts of whatever tooling that you're using, but that, to us, has been super helpful in terms of

helping to identify what the MDCG, and what the AI boards see as the material uplift or delta, and how the two regimes are intended to work together. To Cecile's point also around AI literacy, absolutely agree that there is a big difference between, having machine learning engineers on your team that are building your AI medical device, and having regulatory experts who truly understand how that tool works in the medical device regulatory framework, what we find is that it can take actually quite a long time to, hire for, find, recruit, and onboard people that meet that specific type of competency. And I think that's one thing that people should be really wary of in terms of having, enough time in advance of legislative timelines.

I think it's… in my mind, it's potentially 50-50 at the moment, given the developments that are happening at the EU and national member state level, as to whether the August 2027 deadline will be pushed out, but I don't think manufacturers should quote me on that and sit on their laurels. I think just stay on top of guidance, technical standards that are coming out, guidance coming out of MDCG, AI Board, yeah, it just pays to stay on top of developments.

Okay, thank you. Another question is, are there any steps that manufacturers should have already taken by now, you know, in order to stay on track with the timelines?

My… my view is get your AI medical device certified under MDR as quickly as possible. Cecile sort of mentioned the August 2026, August 2027, inconsistency across the Act and the guidance. whatever you can do to get your device on the market by August 2026, and build some form of, to the extent possible, future vision in terms of the changes that you'd like to make. But nothing's set in stone in terms of deadlines that have already passed that manufacturers need to have met, in my view.

Okay, thank you.

Sorry, sorry, I should qualify that briefly. I'm talking at it from specifically the notified body conformity assessment perspective. Cecile might jump in here.

Great to be in Texas.

prohibitive devices and those sorts of things. But from the notified body perspective, in terms of the conformity assessment process, if you're manufacturing an AI medical device, then you've got time.

Okay, thank you. Cecile, if you want to add anything? Feel free to add.

Yeah, Felicity already briefly touched on it. So we had two deadlines that we… that already passed, but those are not medical device specific. Those are general, you are using AI deadlines. there are some prohibited practices that you need to look at, which I do not necessarily expect you to be doing, because they're all fairly logical that you shouldn't use AI for those, that prohibition has entered into force, I think, February this year.

Also…

the obligation to make sure that everyone in your company that actually uses AI, regardless of whether that's to write a birthday card for their colleague or for their work, is, also AI literate, so has an understanding about pitfalls and the potential uses of AI use.

Okay, thank you, thank you. So, let's move on to the next question. We go through Cecile. For companies currently in MDR-IVDR transition, or with pending submissions. How should they prioritize AIAG compliance work alongside existing regulatory obligations without derailing current market access timelines?

Yeah, so the thing is that the legislation really was set up with similar concepts. So, what you want to try to do, if you're looking at the AI Act, is look, how does it integrate? How can I make my life as easy as possible? That means if you're looking at risk management. try to look at what do I need to do additionally, instead of thinking of, I need to change my full risk management system and procedures. Same goes for post-market monitoring. We don't have the specifics yet, but you can already take a look at what's generally required of you as a company, and try to integrate that

on top of what you're already doing, so that you're not going to set up an entirely secondary system for compliance. Three easement, as Felicity said, to work in tandem. And there will be struggles. I mean, as a lawyer, I see a lot of things that don't align. But I think also a lot of the issues can be addressed if you think very practically.

Yeah, that makes sense. Felicity and take to add, or we can move on?

Yeah, I agree with that. I think, I'd focus on getting your AIMD certified under MDR. I think we, as a software specialized notified body, we've been assessing AI systems incorporated into medical devices under the MDR, and when we do that, we look at, what are the specific risks that relate to the AI components of that device? What are the data inputs that we need to be wary of? Is that device actually safe and performant in light of its AI components? So I think if you're focusing on the safety and the performance of the device under the MDR, then I think it's important that you keep doing so. And then, as Cel said, where you can, just build on that gaps analysis to make sure you're protected under both regimes.

Okay, thank you. Let's move on to the next question, Felicity. A little bit about notified bodies and assessment capacity. So, the first question, what is the current reality of notified body capacity for the EU Act assessment?

I think the shortest answer to that, and the one that's probably the most controversial, is zero. And the reason for that is the infrastructure isn't yet in place for, notified body to be designated under the AI Act, so we're waiting for two primary things. One is the national implementation, and essentially what that requires is each member state has to run through its own legislative process. To make sure that there is a competent authority or notifying authority for notified bodies to apply to. That was due in August this year, and I think… don't quote me on this, but I think current status is about 25% of member states have met that deadline, and that might actually be quite a generous percentage.

So, there are a bit of delays there. The second thing that we're waiting for is guidance or a designation process out of the MDCG to say this is what you need to do, and the confidence requirements, for example, that you need to meet to be an AI-specifically designated notified body, for the purposes of assessing AI medical devices. Yeah, so we're willing, we're itching, we're doing all that we can to prep for those two things to come into place, and as soon as they are, we'll be acting very quickly. But at the moment, there aren't any notified bodies specifically designated under the AI Act to assist.

Okay, interesting. Then, on the same topic, should companies start engaging with multiple notified bodies now? And, what questions should they ask in terms of about the realistic assessment timelines?

Yeah, so taking those in turn, and for the purposes of the first question, assuming that, we do have designated notified bodies under the AI Act relatively eminently, if I was a manufacturer, I'd be trying to consolidate my processes under one notified body. I think that's, quite important in order to be able to take advantage of the dualities across the MDR and the AI Act, rather than having to go through two separate conformity assessments, one under each act, with two different notified bodies. you get the benefits of the kind of consolidation if you are able to pursue it with one notified body. The exception to that would be if

you're manufacturing a hardware device. At the moment, Scarlett would love to help you with your software and AI-specialized conformity assessment, but we can't assess the hardware components of your device if it's built within, sort of, the same or similar product. Likewise, if you're with a notified body that's hardware specialized but doesn't have the AI competency, then you might need to split things up a little bit. But if you can, it would make sense to stick with one.

Okay, interesting. Cecile, anything to add?

Yeah, I think I agree with Felicity. I think the most important thing is to start talking to your notified body. Hey, are you able to go to have the necessary resources to help me? Will you actually be going for AI designation? Which I would expect them to do, but it doesn't hurt to check.

Yeah.

No. Check whether they can already say. where you need to be… where you'll be in timelines, if once they receive designation, it might… might be interesting if they could answer that. I would definitely start talking with my existing notified body, A lot is unclear, as Felicity said. It's simply also… there's also the question of how the certificates are going to integrate, especially if the device is already on the market, in the NDR. by the time the 2027 deadline starts rolling. So, say the device comes on the market after the conditional grace period timeline of 26, But before the 2nd of August 2027, then you need to have some intermediary solution for the conformity assessment and the declaration of conformity.

So that's definitely something that notified bodies will certainly also be thinking about. Doesn't hurt to ask.

Yeah.

Definitely once we get closer to the deadline, because it might be too early right now, but it's something you want to know once there's more movement with regard to the member state designation.

Yeah, okay, so start with your current notified body, check if they are going to pursue AI assessment, and then take it from there. Okay, next question, Cecile. How are notified bodies preparing for the integrated MDR-IBDR AI Act assessment, and what new documentation or process changes should companies expect during these evaluations?

I think that would be one that Listy could answer better, perhaps?

Okay, yeah, sure.

Happy to pick that one up. We are doing everything we can to get more information at the moment, so we are watching closely what's happening at both the European level, and because we're a Dutch notified body also within the Netherlands. And we're meeting with as many officials as we can to get as much information as possible. The… some of the prep that we are starting to take is, as technical standards are rolled out, we're doing what we can to… to implement them into our assessment process as part of the, sort of, state-of-the-art assessment. We already have, significant or material AI competency, fortunately, but we are waiting for more specific parameters around what regulatory bodies expect to see.

In that process, because we need to make sure that we meet that in order to, obtain and maintain our designation. And… I mean, primarily, we're already assessing for, sort of, the safety and performance of, the AI components of medical devices, so… so what we're doing at the moment is, sort of, gaps analysis around what additional assessment may be required, under the AI Act. And yeah, sort of, again, I'd really encourage everybody to look at that MDCJ guidance document on the interplay. We're expecting, I think next month, we're expecting Round 2 to be released on that, so I think the intention is that it will be a live document.

And so that… we're watching that with… with close… closely weighted breath, because that's really helping to determine the gap analysis.

Okay, thank you.

All in the dark, really.

Yeah, helping to determine consistency in application, that's for sure.

Thank you. Well, either of you can pick the next question. What constitutes a substantial modification trigger under the Combined AI Act and MDR-IBDR framework, and how should companies redesign their change control processes to handle these dual requirements without paralyzing product development?

Do you want to go? Do I… shall I… shall I take it?

You can start it.

There is a definition in the AI Act about substantial modifications, which refers to unforeseen or unplanned for changes to the AI system that were Not part of the initial conformity assessment. There's also something about, and which affects the compliance of the AI system with the regime for high-risk AI systems, or the results that are coming out of the AI system. And there is something about this in that MDCG guidance, so probably it is a good idea to learn it by heart. Oh, no? And if I'm not mistaken, it does say that if you can have predetermined changes, which can then still be considered.

something that's not a substantial modification, so… from my perspective, what is… when you need to start worrying is if you have a self-learning AI system that's going to take a direction that you didn't consider in those predetermined changes. But this really does require you to go talking to your notified body about those predetermined changes.

Okay, yeah, makes sense.

Yeah, I think we're really excited about the, sort of, the concept of predetermined change control plans, and they're… they have been a reasonably big thing in the UK, and they're not such a big thing under MDR, but in the context of a system or device that continues to learn once it's on the market. having some protections in terms of the fact that as it learns and develops, there's still data that has been assessed in advance of that that can help to sort of support that on an ongoing basis. I think that's really key for manufacturers, in terms of not having to constantly come back and get changes approved.

It's, we… in the transition from MDD to MDR, there's this significant changes concept, which was a really big thing and a really big question mark for manufacturers in terms of when they were required to uplift their device from MDD to MDR after the implementation. And I think we're seeing, although it's subject to different definitions, we're seeing a similar thing here in the in the context of your device that's certified under the pre-existing regime is fine until it goes through one of these changes, and as a result, I think we're seeing similar calls for… from manufacturers, but also from the notified body sector and more broadly across the industry around what does a substantial modification actually look like.

Guidance… we're expecting guidance on that towards the end of next year, I think, is the current timeline for it. So we'll be… we'll be looking out for that to help sort of build some parameters into… to what we're looking for. And in… in the meantime, we're starting to think around how can we assist manufacturers by supporting them through approvals of predetermined changes. So I think where manufacturers are able to build, sort of, a longer-term view in terms of product updates, feature changes, and continuous learning, that can be super helpful from an assessment perspective.

Okay, we know that FDA has guidelines on predetermined control changes. Can this… are we expecting something similar to be implemented in the EU? Or maybe can manufacturers rely by that point, until then, on something similar the FDA has released? How is there… is there some kind of interconnection between the two?

I'm not aware of the… the FDA guidance specifically, but we… we… we'd be looking specifically at what's happening and what applies to the EU market, when conducting those sorts of assessments. So, yeah, it might just be a bit of patience until the equivalent guidance comes out on that side.

It would be very dangerous if companies just start relying on the FDA guidance, because no matter how much anyone can be aspired by a document from the US, we are dealing with, really, our own legislative framework here… legislative framework here, so… relying too much on purely US documentation is just going to set you up for disappointment. Although it can be good inspiration to take a first look as a company right now, and just start thinking around concepts.

Yep. Yep, yep. Yeah, I agree. Okay, then, Cecile, next question for you. For companies using foundation models or large language models. for… in their medical devices, what specific documentation should they be demanding from their AI suppliers right now, and what happens if suppliers can deliver AI Act-compliant materials?

So, maybe to start off with, there's actually an article in AI Act that says, if you start integrating large language models into models into your AI… into your AI system, and it then becomes a high-risk AI, then you're considered the provider of that AI system? So that means that you become the responsible party for whatever the general purpose AI model, as it's formally called in other AI, is going to do. So, not having documentation is going to be a problem, because then, obviously, a notified body is going to ask, questions at the minimum. So what should you… should you ask for? I think you should be focusing on the documentation that you're going to need to actually fulfill the provider requirements.

And that will depend on what you're actually having the general purpose AI system do in practice. Ideally, what you're going to look at is things about data that was used for testing, training, and validation. As well as what they can say about, about the population that was used, because you, as the provider of a high-risk AI system, are going to have to specify Whether the data that was used for testing, training, and validation is actually relevant for the intended patient population. You also really want to ask about copyright and copyright compliance policies. Ongoing debate, I know, nothing clear yet in case law thus far, but doesn't really hurt to already address.

Probably also should ask about things such as, risk assessment and mitigation measures for systemic risks. And maybe also start planning for things such as, contingencies, such as supply chain disruption of the system, and what it's going to do for your device, and what this, the provider of the General Purpose AI system has put in place in that respect.

Okay, so what should manufacturers do if the supplier simply doesn't provide the requested materials? Should they switch to some other models? Boy.

Well, that's easier said than done, I would suppose, but, I mean, it's your responsibility as the provider of the medical device high-risk AI system to make sure that you have whatever information you need to meet your obligations. So, if you're… if you really can't move forward without the information from the, from the provider of the general purpose AI system. then I think, at a certain point, you do need to make a choice whether you can move forward with the product. Yes, there is an obligation to, for the provider of the general purpose AI model to… collaborate with you, but enforcing that is going to be more difficult than just asking for the documentation. So, at that point, I think you need to talk to your legal department.

Yeah, thank you. Interesting. Yeah, Felicity, sorry, do you have anything to add?

Yeah, I think… I think that sort of raises one of the interesting distinctions under the AI Act, is it kind of introduces more… more players into the regulatory game than the MDR, and I think, that sort of… that discussion was related primarily to kind of the upstream suppliers, but also the downstream deployers of the device have their own obligations, and that, potentially one thing for manufacturers to watch out for is that they're there could be quite a lot of requests coming their way, also, from deployers of the device for information to allow them to meet their own obligations under the AI Act.

Okay, so let me push…

That's a really good point, because you also have the general obligation of human oversight if you're a deployer. So, that is going to require additional information, and ideally, they will be able to get it out of the IFU. But, there might be additional questions. In certain cases, there is also… and it won't apply for all medical devices, but there might be the obligation to conduct, a fundamental rights impact assessment as a deployer, if you're a public authority. It's probably not relevant in the Netherlands, but if you have a general health system such as the UK does in a member state, then that might be something to start looking into.

And deployers really have no idea about fundamental rights, so they might be looking at you as the provider of the AI system to answer their questions. Felicity is entirely right, there's going to be so much questions all around from everyone.

Yeah, I agree. Okay, we have to move on. The next question, with prohibited AI practices banned from February 2025, what gray areas or edge cases in medical AI applications should companies be most concerned about, and how should they conduct their risk assessment? Cecile, Felicity, or whoever wants to take this.

Do you want to take it, Felicity?

I can kick us off, but, one of the interesting, just a case study or data point that I had was an article that was published by the Dutch Data Protection Authority earlier this year that came out and said, It talks specifically about chatbots and some of the risks associated with chatbots, and, particularly the deployment of chatbots in mental health. And it said that there is risk that those chatbots could be manipulative and deceptive, and therefore, prohibited practices. So although, potentially not regulated as such as a medical device that would recover conformity assessment, under that limb, AI Act could be relevant under the prohibited practices, depending on how that sort of device or technology is deployed.

Okay. Cecile, anything to add?

Yeah, so… so also, a lot of the… several of the prohibited practices concern, monitoring. And you could have situations where you end up with patient monitoring, which also leads to employee monitoring in a healthcare setting. That's not something that you as a medical device manufacturer would run into, but it is something that employers are going to run into. So those are really… the grey areas that should take some additional, consideration to make sure that you stay out of the prohibited practices, also because the fines for undertaking prohibited practices with AI are extremely high.

Yep, yep. Look. Okay, yep, sorry, Sean, did you… Did you say something, or… No. Okay. Sorry. Okay, sorry, next question. Cecile. How should companies approach the human oversight requirements for high-risk medical AI systems? What does meaningful human control actually look like in practice, and how will this be assessed?

So, one of the things you need to consider is that this is typically something that will differ per type of medical device AI system. Because what is meaningful for one of them might not be meaningful for the other. things that you… that need to be considered from a design perspective is that the interface of the device actually needs to make it possible to have effective oversight, so it should be… there should be a real human in the loop, and that real human needs to be able to either interrupt or override the decision of the AI. That also means there needs to be an information obligation, because you can only override or interrupt the AI's decisions if you actually know what it's capable of, or what its limitations are.

And for deployers, that means that there need to be training programs and resources for the parties that are… for the employees that are performing the actual oversight. And it can come down also to very practical workplace requirements, where you make sure that there's always someone available on the clock that can actually do this, instead of that person going on vacation and nobody knowing what AI system does. Things to look at is, authority to override, capability to intervene when necessary, but also just plain understanding and plain access to the device and standing in the organization. So these are all things that companies need to be looking at, both employers and providers of AI systems.

And then the assessment, I think is going to be very case-by-case in the end.

Okay, yeah. And, Felicity, anything to add, or… Your microphone's muted.

Thank you. I don't have anything to add, but Sean looked like he might have something to… to say.

So, I wanted to take a moment to switch to audience questions. We have 23 of them. Obviously, we're not going to get to all 23 of them. I've tried to sort of, group them, logically. One is, there are a number of questions about standards. And what standards we can expect, and how to apply them, and what do you see happening with regard to standards. The questions were a little bit more specific than that, but a… Gonna just group it around standards and the interaction with them.

Yeah, I can start off on that one, if you like, Cecile. The, it was a big, sort of, mic drop moment for me when the AI office came out, I think it was late last year, to say, ISO 42001. which is, Quality Management Systems for AI Medical Devices, wasn't going to perform the same role under the AI Act. as ISO 13485 did under MDR. So that means, in layman's terms, if you achieve compliance under ISO 42001, it doesn't mean your quality management system is automatically compliant with quality management system requirements under the AI Act. Likewise for the risk management standard equivalent. So, what we have seen instead is a real EU focus in terms of technical standard development, which is standards coming out of Censenolic, and the standards that we are paying real attention to are specifically those from,

It's called JTC21, I believe, which is sort of the technical working group that are producing AI-specific standards that will be much more prescriptive and helpful in terms of determining what manufacturers need to do, and what we'll be assessing against in terms of state-of-the-art under the AI Act.

Fantastic. The next question is, MDCG Interplay MDR AI Act says IMDRF should lead in the matter of MD AI conformance evaluation, what are your expectations? This is really a question of harmonization, I guess. Or conformance evaluation? Not sure.

Yeah, I can start off on that one as well. I think ideally, I mean, I think it would be our preference in terms of aligned with our mission, in terms of accessibility to healthcare around the world, that there was more harmonization and consistency, around medical device regulation and frameworks, and I think IMDRF produces a lot of, useful, standards or guidance in that context. Ultimately, we wait to see what is actually implemented and taken account of in the EU specifically, if that's where we're assessing. So it's… it's certainly more useful and more determinative than, sort of, US regulation or, at this stage, Australian regulation, but we… we have to see how it's implemented into the EU at the same time.

It also says, I think, will likely, which leaves a lot of leeway. So it's definitely going to come down to what's actually going to happen. Don't count on anything yet.

Here's an interesting question. Can you clarify if medical scribes fall under high-risk category, as is still not considered a medical device yet in the EU?

Well, if… this is a discussion that's been going on for quite a while lately, so I think you need to look very careful on what the scribe actually does. So is it giving you summaries and thus making decisions what to take out and what not? Then it's getting really close to doing something in the definition of a medical device. If it's just… Here's records of every error. And every word that's said literally, then it's highly likely not. Because it's still… to be a medical device, high-risk AI system, it still needs to have a medical purpose. It would have to still be an AI system. So, all the general requirements for literacy and for,

With regard to the prohibited activities, which still apply, even if it's not a medical device.

Yeah, the other thing that we're seeing at the moment is a lot of ambient scribes, trying to get certified, or self-certify as Class 1 medical devices, and it's… it's quite challenging to be a Class 1 software medical device in the EU, and so I… I think, as Cecile says, take very close note in terms of what the scribe is actually trying to do. And that helps to feed into your qualification assessment.

There were a number of other questions about Class 1 medical devices, particularly the application. Let's see, wondering if you can clarify why Class 1 medical device, non-sterile, non-measuring, etc. are excluded versus included, respectively, as per in Annex 1 and Annex 3. very specific.

So, for Article 6-1, the general rule for medical device AI systems, the Class 1 devices are excluded because they don't undergo third-party conformity assessments. For NX3, you need to be… to fall under NX3, you need to fall in scope of the limited list… limited list of medical… sorry, of AI systems listed in NX3, and if you are in that scope, and also, by any chance, a medical device in Class 1, or preferably in-house. Which is more likely. Then, there is no requirement for third-party conformity assessment to fall into that scope. So then it's easier to fall into scope, but you need to be doing something of that limited list.

Hope that answers the question. I think so. So… There were a number of questions about notified body preparedness, for this, and notified, you know… You, you mentioned… And some… somebody asked, is… is there a, designated notified body already, do we have a notified body that's designated under the EUAI Act? Not that I know of, but… .

Yeah, I've, I'd be very surprised, and I'm not sure how they've achieved it, and I'd like to talk to them if they are. The, one thing that we are aware of at the moment is conversations that are happening at the European Commission level between DG Sante and DG Connect. about a technical interpretation of, Article 43-3 of the AI Act, which deals with notified bodies designation. And there's a possibility that that could be interpreted to say, if you are a notified body who's designated under the MDR, then you can also perform assessments of AI medical devices under the AI Act. It's a bit of an ambitious interpretation. It's one that we are obviously in support of, if we can get it across the line. If that interpretation gets across the line, then there will be

potentially up to 41 notified bodies who are ready to certify AI medical devices under the AI Act, because there's almost an automatic transfer across. But… yeah, that's very much up in the air as to whether that will fly at the moment.

Very interesting. Okay, well, we're really out of time. I'd like to thank all of the speakers and all of the attendees for coming today. If we didn't get to your question, we'll try to follow up by email, if it wasn't anonymous, and… Yes, and allow the speakers to say goodbye. There was one comment from somebody that said, it's very nice to see a panel that's not all men. So… Everyone have a great day, and feel free to send, any questions that you have to me, and I will forward them to the speakers. Please, please take a moment to answer the survey. Take care, everyone.

Bye.

Thanks, Ella.

Thank you. Bye! Thank you. Bye.