Health Law Advisor

As we noted in our previous blog post, HealthBench, an open-source benchmark developed by OpenAI, measures model performance across realistic health care conversations, providing a comprehensive assessment of both capabilities and safety guardrails that better align with the way physicians actually practice medicine. In this post, we discuss the legal and regulatory questions HealthBench addresses, the tool’s practical applications within the health care industry, and its significance in shaping the future of artificial intelligence (AI) in medicine.

The Evolution of Health Care AI Benchmarking

Artificial Intelligence (AI) foundation models have demonstrated impressive performance on medical knowledge tests in recent years, with developers proudly announcing their systems had “passed” or even “outperformed” physicians on standardized medical licensing exams. Headlines touted AI systems achieving scores of 90% or higher on the United States Medical Licensing Examination (USMLE) and similar assessments. However, these multiple-choice evaluations presented a fundamentally misleading picture of AI readiness for health care applications. As we previously noted in our analysis of AI/ML growth in medicine, a significant gap remains between theoretical capabilities demonstrated in controlled environments and practical deployment in clinical settings.

These early benchmarks—predominantly structured as multiple-choice exams or narrow clinical questions—failed to capture how physicians actually practice medicine. Real-world medical practice involves nuanced conversations, contextual decision-making, appropriate hedging in the face of uncertainty, and patient-specific considerations that extend far beyond selecting the correct answer from a predefined list. The gap between benchmark performance and clinical reality remains largely unexamined.

“ERISA, you’ll need a lawyer for that.” Our practice group’s tagline is meant to be a shorthand for the alphabet soup of laws that apply to employee benefits, including the Employee Retirement Income Security Act (ERISA). Employee benefits compliance has many traps for the unwary and is ever evolving. Below, we have provided a primer on current issues of importance in the employee benefits area to help in-house attorneys identify potential risks, mitigate them, and know when to call an outside ERISA lawyer.

1. What Is Old Is New: Get Your Health Plan Governance in Order

Employers that sponsor self-funded health plans have a host of complicated obligations. There are greater potential legal, regulatory, and fiduciary risks than in years past with managing health plans because of increased congressional legislation, increased Department of Labor (DOL) focus on group health plan compliance, and increased group health plan litigation, often by the same plaintiffs’ firms that have been suing 401(k) plans in fee litigation the past 20 years or more.