Vendor Risk Index · Compliance Audits

Evaluating AI Claims Vendors: The Regulatory Compliance Index

An objective statutory analysis auditing popular commercial claims and legal-tech AI tools—including EvenUp, Stream Claims, Legora, and Shift Technology—against state adjuster licensing laws, the NAIC AI Model Bulletin, and bad-faith standards.

The Non-Delegation Doctrine

Under Section 1 of the NAIC Model Bulletin on Artificial Intelligence Systems, an insurance carrier remains directly and non-delegably liable for any statutory violations, unfair settlement practices, or unlicensed adjusting conduct caused by third-party vendor software.

Market Overview: The AI Claims Landscape Under Scrutiny

As insurance carriers and TPAs adopt artificial intelligence for intake, document classification, medical bill review, and demand package evaluation, procurement teams face a critical compliance gap: general software vendor questionnaires do not evaluate insurance-specific statutory liabilities.

Below is an objective regulatory analysis examining the operational architectures, public representations, and compliance boundaries of prominent market participants.

1. EvenUp (Plaintiff Demand Generation)

Primary Function: Generates personal injury demand packages, medical chronologies, and claimed damages for plaintiff law firms.

Regulatory Dimension Operational Reality & Carrier Scrutiny
Evidentiary Accuracy & Hallucinations Public reporting and whistleblower disclosures in late 2024 revealed that automated demand generators can introduce phantom injuries, misread medical ICD codes, or duplicate chiropractic charges. Insurance examiners receiving AI-generated demands must maintain rigorous verification controls against original provider invoices.
Algorithmic Negotiation Anchor Plaintiff AI platforms use proprietary settlement databases to generate elevated demand anchors. Insurers facing EvenUp demands risk bad faith if they rely on black-box counter-algorithms instead of verifiable, page-by-page document reconciliations.
Legal Responsibility EvenUp disclaims legal liability in its terms of service, placing 100% of ethical and legal responsibility on submitting attorneys. Insurers must similarly ensure defense adjusters do not accept asserted summaries without primary-source substantiation.

2. Stream Claims (stream.claims)

Primary Function: Workers' compensation and casualty document summarization, medical chronology generation, and workflow triage.

Regulatory Dimension Operational Reality & Carrier Scrutiny
Licensing & Discretionary Adjusting To remain compliant with state adjuster licensing statutes (e.g., Cal. Ins. Code § 14021), platforms summarizing workers' comp files must operate strictly as clerical assistance. Any automated recommendation regarding disability ratings, causality, or settlement reserves triggers statutory adjusting oversight.
Data Privacy & Cloud Transmission Workers' comp records contain detailed occupational health records, psychiatric evaluations, and Social Security numbers. Enterprise diligence requires verifying whether claimant data is isolated or transmitted to multi-tenant third-party LLMs.
Explainability Standards When adjusters use AI chronologies to justify indemnity reserve changes, the chronology must provide direct visual links to the underlying doctor's narrative to survive state workers' compensation board audits.

3. Legora (Legal AI Drafting & Prediction)

Primary Function: AI assistant for law firms and legal teams, providing drafting, demand analysis, and case valuation prediction.

Regulatory Dimension Operational Reality & Carrier Scrutiny
Unauthorized Practice of Law (UPL) Boundaries Like many legal-tech providers, Legora explicitly disclaims that it is a law firm or provides legal advice. Both plaintiff and defense firms utilizing predictive valuation engines must ensure that licensed attorneys independently formulate legal theories and valuations.
Black-Box Settlement Predictions Using predictive AI to recommend settlement values can violate state UCSPA statutes if used by insurers to justify lowball offers without factual, case-specific evidence.

4. Shift Technology (Fraud Detection & Claims Automation)

Primary Function: Enterprise AI for insurance fraud detection, subrogation, and automated claims handling.

Regulatory Dimension Operational Reality & Carrier Scrutiny
Algorithmic Bias & Colorado SB 21-169 Fraud scoring algorithms that disproportionately flag claims from specific demographic or socioeconomic groups face severe scrutiny under state anti-bias laws. Carriers must require vendors to provide empirical bias audits and validation studies.
The Right to Explanation If a claim is investigated or denied based on a fraud score, state insurance departments require the insurer to articulate the factual basis. Carriers cannot rely on an unexplainable “fraud likelihood score” without documentary proof.

Comparative Compliance Scorecard for Insurers

The following scorecard outlines the key statutory questions carrier compliance committees must require every claims AI vendor to answer:

Vendor Diligence Requirement Statutory Basis Compliant Architectural Standard
Adjuster Discretion State Licensing Codes (Cal. § 14021, Tex. § 4101) Zero automated valuations or denial mandates. Software assists with evidence extraction; licensed adjusters retain 100% authority.
Explanatory Provenance NAIC Model #900 & State UCSPA Every single date, dollar figure, and diagnostic code links directly to an underlying Bates-numbered document page.
Algorithmic Fairness Colorado SB 21-169 & NY DFS Circular 7 No black-box regional multipliers or demographic discounting models. Transparent, reproducible factual reconciliation.
Data Custody & Privacy NAIC Model #668 & HIPAA Privacy Zero data egress to multi-tenant public APIs. Processing remains inside single-tenant, isolated infrastructure with zero model retention.

Conclusion: The Defensible Standard

As state insurance commissioners expand market conduct examinations into claims artificial intelligence, the era of unscrutinized black-box software is ending. Carriers that align their technology procurement with transparent, source-linked, examiner-controlled standards protect themselves against catastrophic regulatory penalties, voided releases, and bad-faith liability.