Unpacking Averages: Exploring New Data on FDA Responses to FOIA Requests

Federal agencies in health care publish large amounts of data, and my posts typically analyze that data.  To provide more value to readers, I’ve started submitting FOIA requests for unpublished data to produce additional insights into how FDA works.  And what better first topic than data on FDA responses to FOIA requests.

Information is important, and thus so is access to it.  Our democracy needs to know what’s going on in our government, and businesses trying to navigate the FDA regulatory process likewise need to understand the regulatory process.  For both purposes, the FOIA process should be fair and efficient.

FDA has been releasing data on its FOIA process, specifically its FOIA logs, for a few years.  For data analysis purposes, those data are missing some important fields such as the date of the final decision.  Further, when it comes to looking at the data on the closed cases, the data only go back four years.  In my experience, the pandemic years were anomalous in so many ways that we can’t treat any data from the last three years as typical.  As a result, I wanted to go back 10 years.

I submitted my request on December 21, 2022.  When I told one of my friends that I had made the request, he joked that he hoped I got the results before either of us retired.  I thought the same thing.  We were wrong.  I received the lion’s share of the data on January 11, 2023, less than a month later.

One of the reasons FDA was able to respond so quickly is that all the data I needed were in spreadsheets in the FDA’s Division of Freedom of Information. The staff did not need to involve anyone else at the agency, there was no confidential information to be protected, and there was no real manual labor involved.

But I also want to say that the office was easy to work with and very helpful.  We went back and forth to clarify exactly what I was looking for, and I received a response from the agency typically right away, the same day.  We on the outside are often quick to complain when we don’t think FDA is doing what they should be doing.  So, I want to make sure to take the time to say that the Division was incredibly responsive.

Since there’s a lot of data here (over 118,000 requests during the ten-year period) and to my knowledge the data haven’t been analyzed before, I’m going to do three monthly blog posts organized on the following topics:

1. High level analysis of the response times, broken out by most frequent requesters and by types of responses (this March post);
2. Using topic modeling, an analysis of response times by topic (April post); and
3. Continuing the use of topic modeling, analysis of outcomes by topic, including which topics succeed and which don’t (May post).

Exploring the Data at a High Level

FDA tracks the number of requests year to year and presents them here and here, so I won’t duplicate that analysis in this post.  If you want to see how the requests have ebbed and flowed over the years, you can find that there.  Spoiler alert, the volume doesn’t really seem to change a lot.

The first thing I wanted to see was review times. Using the raw data, I simply compared the date the request was resolved with the date the request was made, and then plotted those review times here.

These data are from the time between January 1, 2013 and January 1, 2023. 

I personally think the chart is a bit hard to read because of the two extremes.  The first extreme is the height of the first column so dwarfing the other columns.  The second extreme is the length of the tail on the x-axis that goes all the way out to 3600 days (which is close to 10 years).  But the chart does at least make two points:

1. About 80% of requests are completed within 200 days. 
2. But equally interesting, there are a material number of requests that seem to go on forever.

Zoomed In Picture

I was interested in eliminating the two extremes to focus more granularly on what’s going on in the lower left-hand corner.  To do that, I first got rid of all those requests that were “closed” within the first 30 days.  As you will see in a bit, the word “closed” has a specific meaning that records were found and produced, as opposed to other ways in which a request might end, for example in a declaration that there were no responsive records.  I picked 30 days because that’s roughly equivalent to the time FDA has under the regulations (21 CFR 20.41) to provide at least an initial response.  I say “roughly” because the actual regulation says 20 working days plus included weekends and holidays.  It turns out, setting these parameters removes over 46,000 records, so it trims that first column nicely.

Then, to prune the extremely long tail, I picked 1200 days as the maximum I would display, which is just over three years.  No particular magic to that selection because the tail trails off pretty gradually.  But that did remove about 1800 requests.  I was trying to keep that number low because I’m just trying to get rid of outliers here.

Having zoomed into that lower left-hand corner, I thought I would then layer on how the cases are disposed because that’s ultimately the kind of detail in which I’m interested.  FDA has 24 different categories of resolution that, quite honestly in some cases, look pretty similar to me.  I wonder if they’ve evolved over the 10 years.  That said, quite a few of them were rarely used, so I just created a miscellaneous category, and kept the top 10 resolutions.

With those changes, here’s what I came up with.

I should explain why a large chunk is showing as being “open,” but also strangely showing as being resolved in the first 100 days.  Open is a special category, and includes those which remained open as of January 1, 2023 (the end of my time period).  This chart says that many of those that are open as of that date, were less than 100 days old.

Closed, meaning fulfilled, was the most popular category.  Withdrawn without charge, another big category at almost 10,000 records in the first 100 days, probably includes those who made a request, were informed that it would be expensive to pursue, and decided against spending the money.  There’s also a very large bar, about 5000 in that first hundred days, where it appears there were no responsive records.

Many of these resolutions that involve something other than “closed” come in the first 100 days.  But clearly there are also many people who do not get the records even though it takes more than 100 days to reach a resolution.  In other words, having to wait a long time doesn’t guarantee an affirmative response.  My assumption going in was that if FDA was going to say no, they would say it relatively promptly.  But that isn’t always true apparently.

Requester Analysis

It is not my goal to shine a spotlight or embarrass anyone by talking about organizations that are making these requests, but it is a matter of public record and I think readers are interested in who is doing so.  While many organizations make these requests, I thought I might simply show the organizations that are making the most.

It probably comes as no surprise that those in the information merchant business are the most frequent requesters.  Here, I’m using the names exactly as the FDA office enters them in the log.  But understand that beyond this short list, there’s a very long tail. Almost 27,000 firms submitted fewer than 300 requests over the ten-year period, which was the cut off to make this top 20 list.

As a result, my next question was, how do these frequent requesters compare to one another when it comes to their review times?  I figured they were all professional and knew exactly what they were doing and would write a request in a way that was probably about as efficient as any out there.  Plus, given the volume of requests they made and the law of averages, my supposition was that they’d be fairly similar.  But again, I was wrong.

To me that looks like significant variability, although there’s a group toward the middle that are all pretty consistent in getting about a 40 to 50 day response time.  But at the top and at the bottom there are firms doing much better and firms doing much worse.

I was curious about the very top.  It looks to me like they get their responses almost instantaneously.  How could that be true for these organizations that submit so many FOIA requests?  I picked the very top one – the “FDA review/FDA webview” requester – and started to study the nature of their particular requests.  Turns out there’s a logical reason for their success at getting quick reviews.  Before FDA was routinely posting the FOIA logs on the agency’s website, this requester was in the business of, on a weekly or monthly basis, requesting the FOIA logs and presumably publishing them on the company’s website.  As a result, if FDA knew on a weekly basis that it was going to get such a request, I can imagine it taking five minutes for the staff to cut and paste the data out of their tracking system and hit send.  It therefore appears that over several years, this firm got those FOIA logs essentially the day they were requested.

To look toward the bottom of the list, I’m familiar with FOI Services, and it’s my understanding that they request a lot of copies of 510(k)s that have been cleared and other materials that require review and redaction before they are publicly released.  Their average review time over 10 years is 180 days, which makes sense for that sort of request.

Harrington Films appears to be Kohl Harrington, a documentary filmmaker, that apparently submits lots of FOIA requests on sensitive topics and then in some cases must battle FDA in court to get the information released.  Thus, it’s not surprising their review times are so much higher.

That all makes sense.  They each have significantly different information needs and the requests vary and thus the review times vary.

Open FOIAs

I want to return to the topic of those FOIA requests that, as of January 1, 2023, were still open.  I wondered, given that some in the ten-year data set had been under FDA review for many years, would the open ones include mostly the difficult ones that were also associated with longer reviews to that point? Turns out there’s a bit of truth to that.

There are, of course, a bunch that are in their first 200 days.  But notice that it doesn’t drop off abruptly after that first 200 days like the larger chart above does.  This chart shows that there are quite a few that have already been under review for quite some time and are still sitting there, open and unresolved.  Indeed, there are a few that have been open for over eight years.

I myself have been in the position where I submitted a FOIA request because a client needed the information, we didn’t get it, we didn’t get it, and eventually we made do with whatever we had.  And I would then frankly forget about the request.  Then, years and years later I would get contacted by the FOIA office to ask me if I still wanted the information.

I know one of the big obstacles here is the low staffing in the FOIA office.  I think they have something like 10 people working there.  If the requester is a commercial organization, FDA can charge for their “costs of search, review, and duplication.” 21 CFR 20.45.  I don’t claim to be an expert in the relevant administrative procedures, but it’s unclear to me why FDA couldn’t staff up for this, although I suspect that much of the legwork must happen in the offices where the records are held as opposed to among the FOIA staff themselves.  However, I don’t see anything to obviously explain why the fees that FDA is allowed to charge can’t be used to pay for time by people outside of the FOIA division.  So why doesn’t FDA have the resources necessary to get the old ones done?

Conclusion

I would say that FDA does a good job about 80% of the time.  Being a data scientist, I’m not just making that number up.  The agency seems to have addressed about 97,000/119,000 requests in the first 200 days over the course of 10 years.  That includes many that are resolved immediately, some that take maybe 40 days, while others take longer.  But getting the various types of requests done in the first 200 days I would say is good.

I would also say in about 8% of requests FDA does acceptably in that the request is resolved in the next roughly 200 days, meaning fewer than 400 days total.  That’s really pushing it, but I understand that sometimes everyone just gets busy and that FDA has competing needs for its time and attention.

But that leaves about 12% of the time, where resolution requires more than 400 days, over 13 months.  At that point, I think you have to say the system is broken because for nearly anyone, whether media, commercial, or academic, waiting over a year renders the information often useless.  There must be some way that we can keep requests from going past a year.

I suspect the people in the FOIA office can’t solve this by themselves because it requires the active engagement by the other offices throughout FDA.  This requires FDA senior leadership to say this is a problem we need to fix.  FDA just doesn’t seem to be making any progress.

Stay tuned for next month, when I start to peel the onion on the topics that are the subject of these FOIA requests through natural language processing.  That’s where it starts to get real fun.