AI Decision-Making in Health Fields: Regulation Coming to States this Year

It is no secret that healthcare delivery and payment in the US are widely derided by many Americans.   Even before the killing of a health insurance CEO at the end of 2024, state efforts were underway already to set guardrails in place for use of AI in the healthcare and health insurance settings.  California, leading as it often does, enacted a law that took effect 1/1/2025 and prohibits use of AI in an insurer's decision making about “medical necessity" of a recommended treatment.  Other states tried to pass AI-in-insurance bills in 2024, and may see them come back around in 2025.  Also, at least one state (Colorado) enacted a comprehensive AI bill in 2024 that will take effect in 2026 and will have a direct impact on the use of AI in healthcare.  That new law will require both developers and “deployers” of AI to implement risk-based measures aimed at preventing algorithmic discrimination.  The higher the risk in a system, the more safety measures and disclosures are required.  Healthcare is defined as a high-risk system.  

The cumulative effect of these laws is not yet known, of course.  They are too new to measure.  However, we know that Colorado's law is being widely studied as a model, and its approach (risk-based model with higher compliance requirements for higher-risk AI usage), is similar to the legislation governing AI use across the whole bloc of the EU.

WHY IT MATTERS

Companies that use AI, even outside health-related fields, should keep an eye on these laws.  They are defining what will come to be seen as best practices, including conducting and documenting risk assessments and impact assessments, implementing and documenting risk management systems, drafting consumer disclosures about AI decision-making and its effects (and potential for risk/wrong outcomes), categorization of systems based on the kinds of harms they can pose to humans, and self-disclosing certain risk issues to regulators after discovering them.  It is notable, as well, that these laws may apply to both developers and the customers who buy and use AIs.  

Healthcare regulation is in for quite a few changes in 2025 for many different reasons.  AI is only one piece of the pie.  However, it is an issue that has quite a lot of interest at the state level, which is likely to increase as AI decisions from providers and insurers become more common.