Health Law Advisor

As we noted in our previous blog post, HealthBench, an open-source benchmark developed by OpenAI, measures model performance across realistic health care conversations, providing a comprehensive assessment of both capabilities and safety guardrails that better align with the way physicians actually practice medicine. In this post, we discuss the legal and regulatory questions HealthBench addresses, the tool’s practical applications within the health care industry, and its significance in shaping the future of artificial intelligence (AI) in medicine.

The Evolution of Health Care AI Benchmarking

Artificial Intelligence (AI) foundation models have demonstrated impressive performance on medical knowledge tests in recent years, with developers proudly announcing their systems had “passed” or even “outperformed” physicians on standardized medical licensing exams. Headlines touted AI systems achieving scores of 90% or higher on the United States Medical Licensing Examination (USMLE) and similar assessments. However, these multiple-choice evaluations presented a fundamentally misleading picture of AI readiness for health care applications. As we previously noted in our analysis of AI/ML growth in medicine, a significant gap remains between theoretical capabilities demonstrated in controlled environments and practical deployment in clinical settings.

These early benchmarks—predominantly structured as multiple-choice exams or narrow clinical questions—failed to capture how physicians actually practice medicine. Real-world medical practice involves nuanced conversations, contextual decision-making, appropriate hedging in the face of uncertainty, and patient-specific considerations that extend far beyond selecting the correct answer from a predefined list. The gap between benchmark performance and clinical reality remains largely unexamined.

Those in the tech world and in medicine alike see potential in the use of AI chatbots to support mental health—especially when human support is unavailable, or therapy is unwanted. Others, however, see the risks—especially when chatbots designed for entertainment purposes can disguise themselves as therapists.

So far, some lawmakers agree with the latter. In April, U.S. Senators Peter Welch (D-Vt.) and Alex Padilla (D-Calif.) sent letters to the CEOs of three leading artificial intelligence (AI) chatbot companies asking them to outline, in writing, the steps they are taking to ensure that the human interactions with these AI tools “are not compromising the mental health and safety of minors and their loved ones.”

The concern was real: in October 2024, a Florida parent filed a wrongful death lawsuit in federal district court, alleging that her son committed suicide with a family member’s gun after interacting with an AI chatbot that enabled users to interact with “conversational AI agents, or ‘characters.’” The boy’s mental health allegedly declined to the point where his primary relationships “were with the AI bots which Defendants worked hard to convince him were real people.”

On May 17, 2024, Colorado Governor Jared Polis signed Colorado’s historic artificial intelligence (AI) consumer protection bill, SB 24-205, colloquially known as “Colorado’s AI Act” (“CAIA”), into law. As we noted at the time, CAIA aims to prevent algorithmic discrimination in AI decision-making that affects “consequential decisions”—including those with a material, legal, or similarly significant effect with respect to health care services and employment decision-making. The bill is scheduled to take effect February 1, 2026.

The same day he signed CAIA, however, Governor Polis addressed a “signing statement” letter to Colorado’s General Assembly articulating his reservations. He urged sponsors, stakeholders, industry leaders, and more to “fine tune” the measure over the next two years to sufficiently protect technology, competition, and innovation in the state.

As the local and national political climate steers toward a less restrictive AI policy, Governor Polis drafted another letter to the Colorado legislature. On May 5, 2025, Polis—along with Attorney General Phil Weiser, Denver Mayor Mike Johnston, and others—requested that CAIA’s effective date be delayed until January 2027.

New York State appears poised to become the fourth state to explicitly regulate consumer health data not covered by the federal Health Insurance Portability and Accountability Act (HIPAA). In May of 2023, Washington State enacted the My Health My Data Act; in June of 2023, Connecticut amended its Data Privacy Act; and in March of 2024, Nevada passed Senate Bill 370. In many respects, NY HIPA is broader in scope and effect than its three predecessors.

New York’s S929 (Health Information Privacy Act or NY HIPA), sponsored by state Senator Liz Krueger (D), establishes requirements for communications to individuals regarding the disposition of their health information; and requires written consent or a designated necessary purpose for the processing of such health information. NY HIPA addresses vulnerabilities unaddressed by HIPAA because it applies to a broader range of private companies and protects health information at risk of disclosure through the commercialization of health data.

For decades, FDA’s Center for Devices and Radiological Health (CDRH) has been recognizing standards that can be referenced in premarket medical device submissions.  Congress broadly directed federal agencies to begin relying on standards in 1996, through the National Technology Transfer and Advancement Act, but the informal practice dates back to the 1970s.  Congress specifically directed FDA to begin using standards for medical device submissions through the Food and Drug Administration Modernization Act of 1997 (FDAMA). 

Being a curious person, I wanted to see what FDA has done with that authority by looking at the CDRH database for Recognized Consensus Standards: Medical Devices.  My main takeaway is that CDRH is not yet investing enough time and energy in recognizing standards that support digital health and AI.

Findings

I downloaded the data set on September 20, 2024, and looked when standards were recognized by FDA and to which therapeutic or functional areas they related.

New from the Diagnosing Health Care Podcast: One year ago, on October 30, 2023, President Joe Biden signed an executive order laying the groundwork both for how federal agencies should responsibly incorporate artificial intelligence (AI) within their workflows and how each agency should regulate the use of AI in the industries it oversees.

What has happened in the past year, and how might things change in the next?

On this episode, Epstein Becker Green attorneys Lynn Shapiro Snyder, Eleanor Chung, and Rachel Snyder Good reflect on what is new in health care AI as a result of the 2023 executive order and discuss what industry stakeholders should be doing to comply and prepare for future federal regulation of AI in health care.

I may be jumping the gun here, but I’m anxious to understand how the new flurry of AI medical devices is performing in the marketplace, or more specifically, whether the devices are failing to perform in a way that jeopardizes health.

FDA keeps a list these days of medical devices that involve AI, and here’s the recent growth in clearances or other approvals. 

Click to enlarge the image. 

Note for calendar year 2024, we only have first-quarter data.

The growth is notable. As these devices enter the market, they are subject to all the typical medical device postmarket regulatory ...

On May 17, 2024, Colorado Governor Jared Polis signed into law SB 24-205—concerning consumer protections in interactions with artificial intelligence systems—after the Senate passed the bill on May 3. The law adds a new part 17, “Artificial Intelligence,” to Article I, Title 6 of the Colorado Consumer Protection Act, to take effect on February 1, 2026. This makes Colorado “among the first in the country to attempt to regulate the burgeoning artificial intelligence industry on such a scale,” Polis said in a letter to the Colorado General Assembly.

The new law will ...

Turns out, ignorance really is bliss, at least according to the Office of Civil Rights (“OCR”) within the Department of Health and Human Services (“HHS”), in publishing its final rule on algorithmic discrimination by payers and providers. Our concern is that the final rule,  based on section 1557 of the Affordable Care Act, creates a double standard where more sophisticated organizations are held to a higher level of compliance.  Set to become effective 300 days after publication, health care providers and payers will have a lot of work to do in that time.

In this post, we will lay ...

On October 30, 2023, President Joe Biden signed the first ever Executive Order (EO) that specifically directs federal agencies on the use and regulation of Artificial Intelligence (AI). A Fact Sheet for this EO is also available.

This EO is a significant milestone as companies and other organizations globally grapple with the trustworthy use and creation of AI.  Previous Biden-Harris Administration action on AI have been guidance of principles (e.g., the AI Bill of Rights) or have been targeted guidance on a particular aspect of AI such as the Executive Order Addressing Racial ...

Introduction

Hardly a day goes by when we don’t see some media report of health care providers experimenting with machine learning, and more recently with generative AI, in the context of patient care. The allure is obvious. But the question is, to what extent do health care providers need to worry about FDA requirements as they use AI?

In the absence of a federal law directly aimed at regulating artificial intelligence (AI), the Federal Trade Commission (FTC) is seeking to position itself as one of the primary regulators of this emergent technology through existing laws under the FTC’s ambit. As we recently wrote, the FTC announced the establishment of an Office of Technology, designed to provide technology expertise and support the FTC in enforcement actions. In a May 3, 2023 opinion piece published in the New York Times entitled “We Must Regulate A.I. Here’s How,” Lina Khan, the Chairperson of the FTC, outlined at least three potential avenues for FTC enforcement and oversight of artificial intelligence technology.

On February 17, 2023, the Federal Trade Commission (“FTC”) announced the creation of the Office of Technology (the “OT”), which will be headed by Stephanie T. Nguyen as Chief Technology Officer. This development comes on the heels of increasing FTC scrutiny of technology companies. The OT will provide technical expertise and strengthen the FTC’s ability to enforce competition and consumer protection laws across a wide variety of technology-related topics, such as artificial intelligence (“AI”), automated decision systems, digital advertising, and the collection and sale of data. In addition to assisting with enforcement matters, the OT will be responsible for, among other things, policy and research initiatives, and advising the FTC’s Office of Congressional Relations and its Office of International Affairs. 

The success of an artificial intelligence (AI) algorithm depends in large part upon trust, yet many AI technologies function as opaque ‘black boxes.’ Indeed, some are intentionally designed that way. This charts a mistaken course.

The application of artificial intelligence technologies to health care delivery, coding and population management may profoundly alter the manner in which clinicians and others interact with patients, and seek reimbursement. While on one hand, AI may promote better treatment decisions and streamline onerous coding and claims submission, there are risks associated with unintended bias that may be lurking in the algorithms. AI is trained on data. To the extent that data encodes historical bias, that bias may cause unintended errors when applied to new patients. This can result in ...

After a Congressional override of a Presidential veto, the National Defense Authorization Act became law on January 1, 2021 (NDAA). Notably, the NDAA not only provides appropriations for military and defense purposes but, under Division E, it also includes the most significant U.S. legislation concerning artificial intelligence (AI) to date: The National Artificial Intelligence Initiative Act of 2020 (NAIIA).

The NAIIA sets forth a multi-pronged national strategy and funding approach to spur AI research, development and innovation within the U.S., train and prepare an ...

Based on their extensive experience advising health care industry clients, Epstein Becker Green attorneys and strategic advisors from EBG Advisors are predicting the “hot” health care sectors for investment, growth, and consolidation in 2020.  These predictions for 2020 are largely based on the increasing confluence of the following three key “drivers” of health industry transformation that is substantially underway:

1. The ongoing national imperative of reducing the cost of health care, via disease prevention and detection, and cost-effective, quality treatment, including more efficient care in ambulatory and retail settings;
2. Extraordinary advances in technologies which enhance disease prevention, detection and cost-effective treatment (e.g., artificial intelligence (AI)-driven diagnosis and treatment, virtual care, electronic medical record (EMR) systems, medical devices, gene therapy, and precision medicine); and
3. The aging baby-boomer population, with tens of millions of Americans entering into their 70s, 80s, and above.

As stakeholders, legislators and policymakers wrestle with the myriad of issues related to the provision of remote health care, clinical and technological advancements continue apace. What was once an industry focused primarily on the provision of primary care through existing remote platforms is morphing into a highly sophisticated brew of clinical and technological innovation.  In that regard, several trends have caught my attention. While these trends may not squarely fall within the accepted definitions of “telehealth”, they are worth noting because they raise many ...