UTILIZING FDA WARNING LETTERS featuring GEORGE KWIECINSKI

Be right back

Oh, Sean!

How are you?

How are you?

I'm well. How are you?

I'm good

The 1st ones here. There's somebody else in the

Okay, I guess. You want us to be off camera for now, right

Oh, no, the whole thing is everybody's on. you know you can. You can turn yourself. You can turn your camera on or off

Hey! George! Hey! George!

You're still muted

I am muted. Hi, guys, how are you? I said, Hi, Dan, Hi, Sean.

Hello!

We're good. We're good.

Awesome they did

And

Super. I've got slides. We start at top of the hour, Sean

Yeah, well, so usually the 1st I don't know. 5 to 8 min is spent like welcoming everyone. And you know, just Joe likes to you know. Tell jokes, or whatever I don't know. he's fun. He should be here. But yeah, so it's it's a relaxed crew. You'll you'll get interruptions from people, and you know they'll they'll they'll probably keep you on your toes. But you know, whatever it's all, it's all in good fun. And thank you for joining

Yeah, thank you for having me. I'm excited. So super. Yeah, Dan, how have you been

So far so good.

Yeah. And I'm I guess. Have you guys chatted already? I

Yes, we did.

Okay. Super.

Yeah. And just wanted to let you know that there's starting to be some interest for the 23rd

Yeah.

Yeah, this is good. So I think that the more we promote the the session, the the more attendees we should have

Yeah.

What's the session? What are you doing

Oh, you want to tell them, or should I tell them

Then you got it.

Oh, okay. So one of the things that the industry is very interested in is it's really in dealing with warning letters. There's a lot of misconceptions as to why do people get warning letters right? And what we do is we try to educate them so in terms of the things that they have to do. So we already started with the session one. So session 2 will be around corrective actions and preventive actions. So I don't know how much you know about this terminology. Sean

So.

And

Way, too much wait

Way too much. Okay? So so we want to bring a different let's say a heightened perspective on what does it mean to really put the right level of Kappas. And what what is it? What is it? Truly a Kappa? Because people think of Kappas as something is very reactive while we're saying it's not only reactive, they also need to be proactive, and they also need to be a little bit more systemic or systematically approach. And then, when FDA is asking you to do something in the form of a Kappa, you really need to read exactly now. They're telling you exactly what that Kappa should look like. So so companies make a mistake. They don't really digest the message

so they they don't turn it into true actionable. That's why they get your responses inadequate. You didn't do this right. And what we're trying to do is help companies get to the point where they don't get that phrase. Your response is inadequate.

So tell me who you are? Because I don't actually know

Neither do I. So just kidding. I'm a former FDA investigator. I worked for the agency for 19 and a half years, and the rest of my career has been with the industry Johnson and Johnson. Sanofi Teva pharmaceuticals. And now I run my own consulting company

Oh, very nice to meet you. Where are you based?

I'm I'm based in Oklahoma City now, for personal reasons. But but I have. What I wanted to tell you is, I've also done a lot of international work Europe, Germany, Belgium, you know, China, India. So, and of course a lot of domestic work here in the United States

Okay? And what is this session that you're doing on? What did you say? The 23? rd It's the Kappa session

Is the caposition linked to warning letters

Okay. And how are you promoting this

Oh, we're doing that through Linkedin. And George has all the means to do that. So this is really more coming from Gks. Gks is the sponsor of this. I'm just the the messenger. Yeah.

So you're the speaker. Gks

Speaker right, and of course, on my own. I do a lot of promoting through articles on Linkedin and those articles trigger interest from our potential clients

Very nice. Hi! Yorg

Hey? How you doing

Very well. How are you?

Good look forward to seeing you in a month or so

Yeah, I'm I'm looking forward to being there. And it's coming together really nicely, Daniel, you know about the 10 X Conference in May 7th and 8th in

Newport Beach

Newport Beach, California. Yeah. Orange County

No idea, no idea. This is the 1st time I get to meet meet you guys. I don't even know who you are.

Well, come and join us. You'll meet everybody in intimate detail. That's what's great about the conference.

Okay.

Yeah. So I have a publication on Linkedin called Medtech, leading voice. And I started interviewing experts, particularly in the regulatory and quality area about 2 years ago, and then hosting webinars and build a list of you know, more than 10,000 and so I have a marketing agency. That's what I do. I have agreed to Co. Host the 10 x event which Joe Hage, the host of this call, has been doing for 14 years. This year, and we've reimagined it with a lot of workshops, and I'll talk to you more about it, offline, because we're getting ready to start. But I think that you would probably really benefit of you were able to go

or at least from this community, because it's right in line with what you do, and we certainly follow Kappa's and warning letters. You know, big. This group you're going to find is very actively interested in those subjects

Fantastic. Unfortunately, the 7th of May, I'm in Europe, so I won't be able to attend, but I'm sure that we can find an opportunity to meet somewhere else. At another time

Well, I mean Europe, Newport Beach, you know. It's kinda I don't know. It's a tough one.

I agree with you. You're

Who's Daniel?

So Daniel worked for the FDA for 19 and a half years, and now he's a consultant based in Oklahoma City. Is that what you said

That is correct.

And we just met him for the 1st time. But he's going to be doing a Kappa presentation, and we're going to be inviting him back to speak on this call. at some point in the next few weeks. If he's not in in Europe because he's sounds like got some really interesting stuff to share with us, and I'm promoting you, Joe.

I went. I went to Oklahoma City once for a wedding. and imagine my delight when, on my way back to the airport, I passed by the Russell Stover store

What's that?

I'm trying to know. I'm trying to figure out if I've ever been

They do candy, but specifically I know them for Easter candy

How can you guys not know Russell Stover sees candy? Come on.

Well, Russell Stover eggs are my favorite jelly beans that jelly belly. Nonsense! I don't need flavors that are like. What is this? I'm happy with just the regular fruit flavors

You don't like cardamom flavored jelly beans

I don't even know what that is

I do sean

Well, I'm not. I'm not a sugar person, so I would not necessarily be that much interested. But I now I'm curious. I'm going to find out where that store is, and

It was on a major intersection on the way to the airport

Okay, I can double check. Thank you.

Trying to get you to send some to him. Daniel

The lines.

That does not cross my mind, but

You know, since you're with the FDA, you're not used to

Awesome chat

I get the place you're inclined. Yeah, I'm I'm here for more family reasons. So it's it's good. It's it's a nice environment, and at least the Oklahoma City thunder! The Nba basketball team! They're doing very well. So that brings the name to the city. But it's it's I used to live in New Jersey. So I enjoyed really the northeast. But but he we're here for now for temporary on a temporary basis. So

Jersey during your FDA days

What's that?

Was Jersey for your FDA days

No, I was in San Juan, in the eighties in Louisiana, in New Orleans and Kansas City. and I don't know if this is a good thing to say. But in 1989 I did the 1st generic drug scandal investigation

Congratulations. That's quite a fender in your cap.

Yeah, that's right. And it was quite a quite an experience. And that was before the FDA introduced the office of criminal investigators. So I was doing criminal investigations without a gun. But that's okay. Not a problem.

Intimidating with that pen

Yeah, that's right.

That's that's not a pen

You want to worry

I didn't know anything. I think this was Dan week, but in fact, it's George's week

That's right.

Daniel will be back later

Yeah. Well, I wouldn't. Just oh, we'll be back later. Excited to meet you, sir.

Likewise

I've been looking forward to this talk with you, George.

Thank you, thank you. And I'm glad Daniel was able to join. Daniel's a great advisor to my corp, and I've been working with him and always enjoyable. I was. Gonna ask Shawn, can I share the link for Dr. Liga Zubic to join on on as well

All means.

Okay, got it.

Now. I mean, this is not a pressure situation. We're all friends here. But boy, did Sean build you up

Yeah. Well, let me know when to start. I'll give the contacts

Smart.

John and I met, or at least from my perspective, at at wraps a couple of weeks ago. But

Yeah.

Yeah. Happy to hear your story. As soon as Mark answers my question whether or not he's gonna cross the pond and come in. What's 5 weeks now? He's sinking lower and lower into the frame

I'm still not sure, so we'll see

Andre's on the phone. You might want to hit him, too, while you're at it, you know, cause he

I love the guy, but

He won't come

I think he I think he's a phantom. To be honest, I don't. I don't expect we'll ever actually meet him in person. It's some kind of hologram thing going on right

There he is!

What was that?

That was you.

That's me representing the phantom that you are, that you're not real

The phantom of the Andre

There you go!

That was good.

Very good. Jury

But I I don't know if Daniel's ever gonna come back. He's like, what's up with these people. 1st they asked me for this, and they can't hold a serious conversation to talk about sweet

Ruin it for us, Paul.

Oh, okay, I'm sorry. I'm still doing my investigation, but I have not gotten to

Good! Come back!

We're being investigated

Right, right.

George, we'd like to investigate your slides and your your show. So

Yeah.

Once you take it away

Okay, I'm gonna kick it off share my screen. Yeah. okay, can you all see my screen

And

Great. And it sounds like this is quite interactive. So please feel free to stop me and ask questions.

Don't worry. Probably go through slides pretty quick. Copy it

It'll there's a lot of slides to go through, and I stop me and we'll chat about the ones that seem most interesting. And I'm good. Start off. So just some background about myself. Why, I'm here. How I met Sean. I'm an engineer from New York City in Manhattan. Got my education here, live here. Now I've done several accelerator programs and always have had an interest in entrepreneurship. And prior to that, I did research while I was in engineering school on warning letters. And that's what we're going to talk about a lot today. That's what I spoke about at regulatory affairs, professionals, conference several weeks ago. That's where I met Sean Regulatory

Strategy Conference in Baltimore. I enjoy doing speaker proposals and attending a lot of these conferences, and shows I'm also pretty involved with International Society of Pharmaceutical Engineers. I'm at Parenteral Drug Association Week, which is next week in Palm Springs and try to interact with the American Society for Quality as well as you guys, has noted as well a couple of great advisors on board for the Company. I have started global key solutions corp, which provides regulatory data in a library fashion. 2 quality professionals. Primarily so. Dan has joined me on this mission along with Ligya, both of whom are on the call.

So thank you, Sean, for giving me the opportunity to speak to you guys today the agenda and sort of the way I'm going to go about. This is a lot like I enjoy doing a research paper. I'll give some backgrounds and just go through the inspection warning letter in 483 landscape. Then I'll show a case. Study how regulatory data is used and talk about some more recent nuanced approaches. A lot of graphs around data from documents. Stop me at any point. Some goals learning objectives, without a doubt. Let's work on understanding why warning letters are important, and if you're already there. Then what do you look for? And how do you look?

Methods on the strategy and planning side? New tools, particularly around how you can parse and contextualize documents. I'll dive deep into this, but we can take it as needed. So just landscape this part, I'll go through as essentially briefly as I can, and please stop for questions otherwise. An inspection is defined by the FDA as a situation in which the agency can enter facilities, warehouses and other areas based on the fact that they have jurisdiction over drugs, devices, biologics, tobacco, cosmetics, and veterinary products.

When the FDA investigator comes to make an inspection. They are generally walking the facility and working to conduct and understand how the operation is, and whether it falls within some metrics of compliance and guidance, but they can't cite guidance within inspections, reports. So the inspection classifications really that way, inspections can have an outcome boil down here. No action indicated voluntary action indicated and official action indicated green, yellow, red, so to speak, and quantifiably so as well.

The key understanding after this slide is that for 4, 83 s. For the actual write up. and then for warning letters. We are speaking on the Vai and Oai inspections. So those inspections that were not seen as very minor comments and essentially checked out. George, George, can I just make account? Can you go back, Sam? Go for it?

When you when you guys look at, you know, Oai's are about 10% of what the voluntary action indicated classifications are. And but if you look at the database from George, and and of course, some of us speaking from experience. those 445, that's a very conservative number, because the trend that you will see that is that a lot of these companies that are Vi are inspection after inspection, they're they're Vai, and in many cases the FDA has also used enforcement discretion not to really issue a warning letter to to a lot of these companies. So

there is a lot of risk associated with those companies that are at the Bi level, because for the most part, most companies don't really shift from Vai to Nai. They actually shift from AI to Oai because the the culture is really not there. So so I think it. The the figures are great when you look at them from a positive perspective. But the from my perspective. They're a little bit deceiving

Really excellent comments there, Dan, exactly. And actually, I believe there's a slide through here. I really enjoy this slide because it kind of shows that process of how this is what the inspector investigator really finds and issues some initial classification. And then, as Dan noted, oftentimes the agency we'll reissue that final classification more severely, very seldom less severely. in order to write the warning letter or issue that form. Yeah, overall inspection trends and going backtracking a little bit, you get the Vai or Oai inspection.

The investigator issues form 483, and you're as a firm given 15 days to respond to it. What we've seen is a decrease in inspection amounts overall yet a steady, if not increase in citations. Wow! There's limited data, but we can conclude that the amount of production of goods has not decreased. So this paper really cites that in a manufacturing environment. But that's something critical to begin to understand.

So again, less inspections with a similar amount of production, activity, or utility, import, amount per se. If you're looking at foreign and still more citations. And this is really what my research drills down to is the FDA's inspection, efficiency, and risk-based approach on selecting which firms to inspect. and really that these are the other types of inspections. Right? So there's 4 cause. There's routine inspections. There's different established programs like Bimo. And there's also a place for you to look at these inspections

Now

Yeah, I was just gonna say, the other thing you've got to really think of. And this is kind of a trend I've seen, probably in the last, maybe 5 or 10 years, is that the inspection used to be inspector came in. reviewed everything, wrote a 483, and then maybe somebody would audit that 483, and it would be done. But now what I'm seeing is a trend is your inspector comes. They take that data every day, and it goes back to the Oai, and they're re-inspecting it. And so what might have been agreed to is an observation, then get escalated to 483 s. So you're not being audited by one people. One person you're being audited

by a group of people who are experts in the area. And so, even though you might have a, you know, kind of an understanding, you can explain things. I'm seeing the less opportunity for explanation and understanding, and more. And that's where the more citations are coming from. In my opinion.

Yeah. And actually, you know, the slides don't even really need to correlate with this discussion here, but also really good to note on that topic through Covid, you know, the use of. Essentially, report requests right? I mean, just document requests and then reviews. Now, you can have the subject matter expert reviewing documents. This may have been done in less quantity before. So now, exactly, it leads to such outcomes. Okay.

Maybe maybe the other thing that can be said is that, remember, there's also been a shift of companies moving from the Us. To other countries, India, China. Singapore, Europe, even Mexico and South America. So so those companies, those sites do have a less exposure to the FDA Enforcement culture, and they go through some kind of a learning period. And if you if you notice, and I think, George, maybe this is something that we could explore later on. It's sort of compare difference in terms of inspection citations given to domestic companies versus foreign companies, because I think that there's a possibility and a high probability that we'll see that the FDA does spend more time and effort making sure that those foreign inspections are performed because there have been issues with data integrity

which is with falsification issues, with really very poor Gmb practices. So so that's also been part of the shift. If if my, if I can say that

Actually, that's right. I'll just do one more slide to cover. Why, warning letters are important, Dan. But then, essentially, let's talk about a case study in which the Congress wrote a letter to the FDA around foreign inspection investigators and inspectors. But it sounds like we've essentially covered the landscape. Warning letters are structured differently than 483 s. they are letters 483 s. Are generally can be handwritten, written right by the investigator as that chart had shown. Warning letters are really a 483 is that list of observations. Both cases you have the ability to respond as a firm. But the warning letter is really working to build that case against you. Right? Whether it's a shutdown or an injunction, seizure, recall whether it's voluntary or not. Those are the outcomes the agency is working towards with the issuance of the warning letter

Do you want to ask your question aloud?

Hi, guys, apologies. I'm a little bit sick today. Come down with a cold. But my question is, if a foreign manufacturer is no longer distributing in the Us. But they do have a facility listing, registered. They have products registered, you know, 510 k. 510 k. Exempt. Could they still be subject to an inspection, even if they're not distributing in the Us.

The the answer is, yes, so long as you have an active let's say registration, the FDA will consider that you are due to distribute product anytime you choose to do so. So the FDA will assume that the fact that you're not distributing doesn't mean that you have stopped permanently doing that. So if you, as a foreign company, do not want to be inspected by the FDA because you're no longer distributing, then you just need to Deregister, and then they're going to leave you alone.

Right

It's a good point. There's been

I've actually managed several inspections for clients that they were maintaining a active registration because their investors wanted it, and they thought it made them look better to to the investors. And then the FDA came, and they were like totally like, not ready, not prepared, couldn't explain that, and that their Qms. Wasn't being utilized because they weren't in distribution. And yeah, it can be. It can be a little bit of a mess

Yeah. I, I can just add that also, if the medical device is still active, the registration has to stay active, and when there's a expiration date on those medical devices so definitely those foreign offices are subject or companies are subject to inspection

So then I have a follow up question. So I know this is like specifically towards the FDA. But, for example, in health, Canada, anybody who's holding. Similarly, anybody who's holding an establishment license is subject every 5 to 7 years to an inspection. So as part of like the routine inspections for FDA is this kind of kind of similar expectation. It's like a set number of years that there should be a routine inspection for a facility that's registered, or that has products registered

Yeah, this is a statutory requirement that mandates the FDA to inspect facilities every 2 years. That does not happen anymore for different reasons. one of them being resources and workload. And so what the FDA has done is actually shifted a little bit to identify companies that are high risk to have than those companies being inspected more frequently than companies that are under Nai, and that have not represented risk to the agency or to the patient. So. Yes, there is a statutory requirement

Yeah. And that's what I can show some statistics on a little later. Let's essentially recap. Sounds like we all have a great understanding of the inspection. Warning. Letter 483. Landscape a direct example of use. Dan was going back to the use of inspection data based on geography. So this is an assessment from Congress to the FDA. Actually assessing inspectors that had more than 10 inspections in India and China. and what they looked at was for the inspectors that had more than 10 inspections. How frequently were they issuing the form! 483. How frequently were observations being found on a per inspection basis.

And so they actually concluded that several inspectors that had both inspection histories in the Us. And India and China had significant variation in that number close to 80% of the time in the Us. These inspectors would issue a form 483. And that's, of course, those Oai Vai inspections. Meanwhile, in India and China they would have rates between 6.7 and 11.4% for China and 9.5% for India. They use this data on a per inspector per region per issuance basis, right? So amount of 4 83 over amount of inspections

to come to this quote. This is directly from the letter that they're seriously evaluating the disturbing possibility of fraud in this case. So the reason I give this kind of staunch example is because it's a direct use of a fractional assessment with several parameters on inspectors investigators for coming to a pretty drastic conclusion and the agency. This was issued in June of 2024. There hasn't been a conclusion, as of late. so that should give a good example, I want to move on for the rest of the conversation into the more data, statistics and graph landscape. So

that's kind of where we are now. So there's a couple pretty interesting papers out there. I'm going to start with ones that I have not written. This one is from to a professor and a researcher in China at both at Pecking University. And I really enjoy this because it gives a high, level overview of warning letters. So now we're going to break into mostly warning letter assessments. This graph depicts word counts of warning letters on an annualized basis. So you can see starting at 1996 ended? 2022. How many words focused on these specific topics for the timeframe?

And really the focus after figure 2. For this paper is the understanding that instructions for further response. as well as the ideas of recommendations around both 3rd party help and just looking at guidances and readings, have increased the most particularly word count wise instructions for further response. That's again where figure 3 comes in for this paper. So again demonstrating that really, in the last 1015 years the agency has began to understand that firms have 2 problems. By the time they've gotten to a warning letter they're probably relatively incapable of solving the issues on their own, and they need guidance on how to go about doing it.

So this is the graph for seeking 3rd party help within the same timeframe. And if you talk to individuals and agency, Dan feel free to comment. But really there's been a push to essentially guide firms to how to respond, and then what to do. Right. Go seek 3rd party, help

Yeah, I was just gonna say, I mean the inadequacy of the responses. Look, companies get a warning letter 1st and foremost, not just because of the 483. It's it's 1 element. But but it's not the element that that triggers or drives the issuance of a warning letter. It's that response from the company that actually triggers the warning letter. And what I mean by that response is the inadequacy, the inability of the company to really propose solutions, corrective actions that would provide a certain level of comfort to the agency that now the agency can trust on the on, on the fact that the firm understood the issues. So the companies get the warning letter, not just because of the foreign, but because really mainly of of the poor response, and because of that poor response, the agency then says to the companies, Listen, you don't know how to respond. You don't understand your systemic issues, you need

help, and that help comes from external consultants.

Yeah. Yeah. And I think that's something to really, this research paper really makes the case of that. And going from there, your mind may be wandering in several places. One thing you may be wondering is well in what quantity, maybe over the 10 year timeframe warnings became far less popular. Right. Maybe the agency has employed new practices or firms simply just became more compliant. That is certainly not the case. This is from the paper I have published back in November of this year. And what it goes to show in this graph, while it's not a line by line, these figures should help provide better context.

Warning letters issued as a fraction of inspection volume have pretty drastically increased from the last 5 years through Covid, almost doubling for overall, but definitely doubling for devices and drugs. And before I even dive further on this particular topic, why this matters moving forward? Of course we can recognize 3rd party oversight and recommendation from the agency has arrived next steps are directly in the document. and instructions have added clarity. Volume has not slowed down, and moving forward.

This serves as a baseline for starting to understand. Okay, well, is there a way to establish a baseline of compliance for firms? Obviously, if we can have all firms become more compliant, the agency can continue to become more efficient, and industry can advance at a faster pace. That's really the goal. Behind all of these graphs, papers, and statistics is those 2 things more efficiency. faster advancement. So there's another paper that entirely covers mathematically establishing a baseline for compliance. I like this graph in particular, because you end up seeing that if an action is issued at time, t. 1 to firm one, this is generally at least mathematically, how a cohort of companies would react.

the firm would be essentially forced to become more compliant, or at least remediate the actions if they want to stay operating. At least, that's been the status quo. But we should certainly have a discussion around drug shortages and firms that have simply become so big. They've monopolized a single drug pathway. So now they don't need to stay compliant even upon warning letter issuance. But this is without that assumption, right? This is assuming. The firm does listen to the agency. and then, assuming that friends, friends, which is about 50 out of the 100 extrapolated lines in this graph. would also listen and understand. Thanks to the availability of the data.

what to do. Right? So there's a group of companies that would at least be aware enough to say, okay, this is what my friend or my competitor. However, you want to call this variable friends. This is what they did to become more compliant. We will do it for safety right? And then the ideal of this and another graph, I think figure 5 in this specific paper shows this over time, you know, t, 1 t, one t, 1, basically over a series of years. And why? You can't just have routine inspections. But they must be random, at least partially in nature. So this paper provides excellent support for those types of activities. By the FDA, right?

This is the paper I I jumped to. And now we're jumping back a little bit. But essentially the further understanding that okay, you have an understanding that compliance comes from a warning letter. You see what the agency is doing around warning letters. What about pure warning letter volume? Well, we've seen a couple of things. The agency has become more efficient on a volume metric basis. Right? So if you just assess the amount of inspections that were conducted from 2019 to 2023 as a ratio of the amount of warning letters. The increases are staunch, particularly over Covid, right? This is when investigators didn't go out, and

in inspections were really only made under extremely risk based circumstances. And then, even on top of that, a lot of document requesting and other activities. Other technologies were utilized in order to make these assessments and write warning letters that has in some way stopped. But even the Government Accountability Office has published through 2022, and 2023, that, leaning on foreign inspection reports doing more document requests. activities that continue to enforce compliance but aren't. Direct inspections are efficient. So they should continue. So that's been a a bit of a deviation there.

So again, just to recap these types of analysis by volume essentially allows you to understand that these documents are credible and useful. So now we want to look at the content within the documents. This brings you to part 3 and part 4. So now we're really thinking about. Okay. Great. I've walked you through the warning letter landscape. We understand they they exist. And why? They're important. Now, what can we get out of them quite literally, what can we get out of them in the classic realm? And then part 4 is more of utilizing nlp and more vectorized based ways of extraction, but classically, warning letters do a couple of things. They cite Cfr codes so you can look at the codes of Federal regulation that are within warning letters, and what the agency is essentially writing up against. Right? So

this is a bit of a recap on this multimodal approach. So yeah. spoke a bit about that again, also touching on the fact that the Government Accountability Office wants to see foreign inspection reports and other things utilized for action issuance. These types of things are not noted in the inspections dashboard, but warning letters can come from them. Other things like that. Remote assessments are key indication of why the rates were so high for warning letter to inspection. Issuance over the covid pandemic, and also another reason. Rates were so high over the Covid pandemic was. Of course you couldn't go inspect firms enough about Covid.

Okay, moving through this into this essential extraction base. This is still quantity focus. And I'm essentially gonna move through into this is again touching on the rise of issuance. I kind of spoke through a lot of these things. So assessing

Sorry. I have a quick question just in terms of the what you just said about the remote audits. And what is the correlation between higher citations. What was the cause of that? Yeah. So what I was trying

Dictate.

Please let me.

Was that remote inspections document requests. These are reasons why that graph I had shown why warning letter issuance versus inspection volume was so high because a remote document request, and then the issuance of a warning letter doesn't. That's 1 warning letter and 0 inspections. So the fraction ends up being higher as a point up quantity over inspections

Oh, thank you!

Yeah, yeah. And then, like, I had said, moving on to assessing data from warning letters, how do we move from there? So warning letters are generally written, and they cite codes of Federal regulations and predicate rules. and also the agency publishes data on codes of Federal regulations against 483 s. And some inspection data. So what you can end up doing and what I'll sort of gently walk through now is you can take Cfr codes for inspections for 483 s. And for warning letters. and like I had been doing, you can normalize the amount

over the total amount of inspections, the total amount of 483 s. And the total amount of warning letters for any given filter right? So we want to look at just drugs. Perhaps we want to look at just a certain period of time, maybe a certain program, etcetera. So this is one walkthrough of that situation. Right? So here's 21 Cfr, 2, 11, 1, 9, 2, production record review. And this is the quantities of that code within 483 s. So 21 Cfr. 2, 11, 1, 9, 2 was cited a hundred 74 times in 4, 83 s. And then, for our date range of 2019 to 2023, there were 328 total related inspections in the drug landscape

with Oai and Vai. And so you can take that fraction. You can do the same thing for warning letters. So how many warning letters had this specific Cfr code for all of our filters over the filtered amount of total warning letters. You end up with these 2 rates. You also end up with these 2 fractions. and then you can essentially make an assessment how frequently, how much is the agency either making observations on this or industry reaction on this? You maybe can't always drill down to a direct root cause, but you can certainly start to understand the trend or a historic trend.

So one other way to visualize how this looks is a Sankey diagram example. So this provides a really nice metric. For how many? 483 s. Had 21 Cfr. 211.1 9 2. That's 174. And then from those 174 12 companies went on to have a warning letter with the same Cfr. Code. and then, from the total amount of 483 s. There were also 12 warning letters with this Cfr code. but the 483 s. Did not cite the Cfr code. So this is a particular example where resolution rate from 4 83 S. With the Cfr. Code to warning letters is particularly high, and the amount of use of this Cfr code within warning letters

is also not particularly high, so the likelihood firms are able to resolve. Such an issue is pretty high. That's what I touched on. Okay. so that's a direct extraction and analysis of Cfr codes. Maybe this left you thinking, okay, that's great, George. But what else? Why is this important? Why would this relationship be meaningful to me if I already read the warning letters, or I know this. So that kind of brings us on to documents and data sets where we're moving into a world where

you know, as soon as someone talks about AI, particularly in our quality environment or Gmp manufacturing environment, right? Then, we need to do a risk assessment. What are the outcomes of using AI to make any particular decision. Human in the loop like these are probably the things that might start running through the quality team's mind, right? So the reason where we can use this data is it doesn't use AI. And so if you're going to make particular risk assessment decisions or want to integrate certain document writing processes or use this type of data, you can essentially have real facts. help drive your conclusions. So what that really means moving forward is, you can take the methods used for our Cfr code and start to think about if you're going to use them for a keyword, for example. So

you know, I put a tool here. Lang chain is essentially one non AI extraction based tool. You can use to make queries against warning letter data. But there's all kinds of ways. You can essentially build a technical stack to assess this data, what becomes important out of it. Beyond that is what you do with it. So I'm going to kind of show you, we can extract data. We can provide a prompt. You could use AI, and you can go through the keyword methodology. So you're essentially using a certain list of tools to extract subject line keywords from warning letters

kind of the visual version, right? We take the data, we extract and store the keywords. And then we relation all of this data what you end up with. If you essentially repeat this process, time over time is a bunch of subject top terms. So this is what you end up getting. If you took all of the drug, related warning letters and wanted to go through top keywords top subjects for these documents. Now this is where we can relate it back to Cfr codes and predicate rules. because you can conduct an assessment of the warning letters

that had both the keyword and the Cfr. Code, obviously 210 and 211. 0, my God! George Duh! Yes, duh, because they should have the most amount of keywords, because they're always frequently cited. Everything is either 210, or 2, 11 in this environment for these keywords. So you end up with essentially 100% co-occurrence rate for the document set. And then you can start to essentially understand whether or not there's certain keywords like prescription drugs and others. I think these are a little off But for prescription drugs and the exact Cfr code that aligns with it. Right? So 201.5 is adequate directions for use. And then what relates to adequate directions for use on a Cfr code basis.

So what we're doing is aligning the predicate rule the Cfr code with certain subjects and topics. Why am I presenting this to you? And this is something I'd love for anyone to chime in on. But as we know, the FDA doesn't really push out new laws. There's not a new Cfr code every day. So what ends up happening is a guidance is issued. Industry reads the guidance industry comments on the guidance. There's lots of buzz about the guidance, and then nothing right. You don't hear about the guidance. The guidance doesn't get cited in the 483. The guidance doesn't get cited in the warning letter. What ends up happening is the investigator needs to find a way to take the guidance, and essentially cite on past regulations some form of the codes of Federal regulations. So you end up with a sort of vague Cfr code citation. Maybe some comments and text around it. But you don't actually end up knowing. Oh, look! They didn't follow this part of the guidance

right? So that's really why I'm presenting this type of square to you today is because we want to slowly start taking guidances and other text based documents and relating them to the exact Cfr codes and potential citations. So we can understand

One while you were talking I I visited your site. And the 1st thing that struck me is you you start off by talking about pharmaceuticals. In my quick glance. I didn't see you talking about medical devices really much at all. Of course, there's combination products and the like where it's relevant. I'm curious about your customer base, or if you're still in Beta, is your customer. the manufacturer, or the consultants that serve them. And this is why I ask, because I, in my time in industry, had folks with. you know, true value add concepts like yours.

But I'm thinking I don't have a warning letter right now. So do I really need this? I mean, isn't it relevant? If I have a 43 year old warning letter so that I can parse the data and understand what to do. So it it seems very like now I need it now. I don't. Now I need it now I don't. What's your business model? How does that work

Yeah, Joe, this is a bit of a shift from the research. And what I was speaking on

Of course I know

Is there any particular question on what I was speaking on prior to this? Otherwise I'm I'm happy to spend the remainder of the chat on this topic. Yeah.

What funny you kind of put me on the spot like, were you listening to what I said? Yes, very much so, but because we're all different functions. We approach things with different eyes. I'm a marketing strategist. And I, you know, product, price place promotion. You've got an awesome product

Thank you.

I don't understand how. if I were on your team, how I would market it to who you know, who is your target? What is the primary benefit they're going to derive. Clearly. But yeah, there's a lot of this data. But who? And I would think that a consultant who has many companies coming at them with warning letters and other regulatory concerns come to them for advice, and that's across industry across many companies. But would I, as a manufacturer need my own. I imagine you're selling seats or licenses. Would I need one

Yeah, Joe, we work with both consultants and manufacturing firms. The business model is geared to accommodate both types of stakeholders, however, generally the status quo and industry has been that manufacturers and larger companies purchase the product, and then consultants, as well as these 3rd party reviewers generally are able to have logons and seats under the companies. They do the work for

So if you had a an outbound sales team, do they focus like, Hey, we're gonna go get some manufacturing accounts, or we're gonna go look for the large consultants to, you know you, you can't focus on everything. So where? Where's the board? And you and your team saying, this is where we need to focus. This is where the money is

Though, if I can add something coming from a large global industry, global companies, they usually have a groups working as a global department. So, for example, one of our primary interest group is auditors, auditors who are dispatch around the international markets to audit. They manufacture Cdmo Cmos. They would benefit from reviewing various type of warning. Letters 483 s. Because they work on like you said, for consultants. Very important, but the same with the global companies. They train the auditors for a specific audit specific subject of inspection or audit. However, they also cross train. They wanna

have a knowledge to be able to answer FDA questions on inspection. So I'm bridging here between auditors for a global large pharma going to audit entire market of the suppliers when they need a knowledge quite diverse. So they can well prepare companies, those global companies to the FDA or another regulatory body inspections. So similar to consultants. The data intelligence data knowledge for a big pharma is extremely important. They wanna promote the company that they understand. They know they can predict their own trends, so they use various type of resources to be prepared for both audits and inspections. If I

answer your questions

No, no, you did. And and I think you're saying among manufacturers, it makes sense for larger ones with lots of products. This wouldn't be something for startups to

Yeah, that's their 1st

Couple dollars in

Right, and I was

Yes, and it all depends again, because you also have to look at supplier management activities. So when you have all of this information in one location. and you can rely on that information and the ability that you can do to to search and research on the based on the information you have you? If you're part of a supplier management group, and you're trying to figure out, you know, who to get Apis from, or who to get certain products from you're you're going to use this product as part of your research. And this is one part that is going to help you. Number one number 2. The industry has consistently gone through ups and downs because

there are many factors that actually make the industry not be as consistent as it should be. If we were aerospace we we would be in in full decline, because we would have like a crash every every 5 min, right? But as an industry. Yeah, we were we. We are for rejections, and we are for you know, to perform badly. And and the challenge of the industry has moving forward, especially with biological product, is really to be as as the best as they can be. You know, George talked about 1, 92. Do you know how many years we've been talking about 1 92? And what and how is it the company should be able to do good production records review could do. How could they go do good investigations.

this process that George is putting together? It's going to allow companies to really look at warning letters, not only from a reactive perspective, but also from a proactive perspective. And this is part of the the selling that we have to do, and that we're going to do. Because what we really want is for companies to engage more into self regulation. less reactive and to be more product. That's really where they're gonna make their money

Thank you for that. And, Rama, I see you have your handwritings

Hi, Joe, thanks. So I've spoken with the manufacturer, and they've been in operation for like 15 years. And they haven't had an FDA inspection since 2017, you know. It's low, low risk. Medical devices. no complaints, really, no adverse events, never had any kind of concerns, but they were mentioning that they haven't had an FDA inspection because they were Mdsap. Certified. And I was just wondering I've never, really, you know, heard that as a factor in terms of the frequency of FDA inspections is Mdsap certification something that would impact the frequency

I'm not aware of that. So I I am hesitant to comment. But it. What I'm thinking is that it depends on the the type of product that they sell the type of medical device because it's a if it's a. you know, 5, 10 K innocuous product. I don't see the FDA sort of again because of the challenges they have with resources. I don't see the FDA sort of prioritizing, but are they due for an inspection, I can tell you right now. I have pharmaceutical clients that that half issued Pll reports on specific activities, and I, under normal circumstances, I would have expected the FDA to do a follow up

inspection, and that inspection has not occurred. So I think again, the agency is probably trying to maneuver as much as it can within a risk-based approach. So I don't know the answer to your question, but I'm speculating that from all normal circumstances they should have been inspected already.

Yeah, I thought so as well. It's just since 2,017. That's quite an that's quite a period of time.

Agree with you. I totally agree with you, and I tell you I've seen clients that have submitted new product. the products to the market, and they have not received a pre-approval inspection, which to me is okay. It is what it is

Say, welcome. You have a comment

Yeah, just to give some clarity on the prior question, Mdsap is actually a program within the medical device industry that the FDA is one of 5 countries that participate in, and one of the benefits of participating in Mdsap is, you do get removed from the FDA's routine surveillance schedule. However, they still have the opportunity to perform an inspection. If they have a fork, cause

Thank you for that

That makes a lot of sense. Yes.

Somebody sent you a heart

Yeah, I was just planning to say this. So yeah, very st, that I definitely agree

Yeah. I guess we're just about at time. I don't have much else to conclude. And again, thank you, Sean, for the opportunity to present today. Thank you all for the fruitful discussion. I would also conclude. you know, I was talking about AI vector based searches. And this is really the goal right is to take guidances and start to see past documents and past records and examples like the abiomed case where AI and Ml. Was a guidance the FDA issued. And here's a direct example with what Abiomed did within their pacemaker app. So things like this are what we want to monitor for. and the real reason here is to help back your arguments, provide insight and increase the baseline for compliance.

Again. Thank you. Thank you, guys for the fruitful discussion. I appreciate it

You, George, and and thanks for bringing along your friends, Daniel and

Yeah.

Is it pronounced? Lesia

Yeah, it's perfectly is good. It's a Polish name.

Okay, well, thank you both

Thank you. Thomas.

And for those of you who send me a Linkedin invite, I'll respond to them later on. I after this call I need to do something, but I'll we'll be in touch

John, bring us home.

Hey? I was so glad that you showed up. I met a lot of interesting people at the Wraps convention, and hopefully, we'll have some more of them on of everyone, and thank you for everyone who showed up today. And I encourage you to think about joining us in California. If you're able to, George, that includes you. And I'll talk to you soon. Thanks

Thanks. Everyone have a great weekend

Thank you. Guys.

So much.

Great Webinar. Thank you for hosting

Have a great day. Guys.

Bye-bye.