The AI Crash Test Scorecard
| Provider | Operator | Vulnerable User | Bystander | Safety Assist & Verif. | Overall |
|---|---|---|---|---|---|
| Microsoft | Strong – Frontier Governance Framework, RAMPART, AI Red Teaming Agent | Limited evidence | Strong – C2PA steering committee; signed the EU Art. 50 transparency Code | Strong – 18-university External Red Team Alliance, ISO 42001 certified | ★★★★☆ |
| OpenAI | Moderate – Preparedness Framework, system cards | Mixed – age prediction (Jan 2026) and parental controls; but Common Sense Media rated ChatGPT for Teens “Unacceptable Risk” (Oct 2026) | Strong – C2PA + SynthID dual-layer; signed the EU Art. 50 Code | Strong – METR evaluated GPT-5 directly; signed EU GPAI Code | ★★★☆☆ |
| Moderate – Frontier Safety Framework, Responsibility & Safety Council | Limited evidence – supervised under-13 accounts via Family Link (not UK/EEA) | Strong – C2PA + SynthID at scale; signed the EU Art. 50 Code | Moderate–strong – external eval practice, signed EU GPAI Code | ★★★☆☆ | |
| Anthropic | Moderate – RSP gates release; Mythos Preview held back over cyber findings | Limited evidence – consumer accounts 18+ only; minors reach Claude only via third-party apps | Moderate–strong – text watermark and C2PA file credentials since 2 Aug 2026; signed the EU Art. 50 Code | Strong – RSP since 2023, published Risk Reports, METR + Apollo evals; first of nine in the FLI Summer 2026 index | ★★★☆☆ |
| Meta | Moderate – Advanced AI Scaling Framework v2 (Apr 2026) gates deployment at High/Critical thresholds; pre-release red-teaming | Mixed – parental controls, romantic/sexual dialogue with teens blocked, teen access to AI characters reported paused; follows a 2025 Reuters report and FTC inquiry | Moderate–strong – C2PA steering committee (Sept 2024); signed the EU Art. 50 Code | Weak–moderate – published framework and Seoul commitments, but declined the EU GPAI Code; open weights limit post-release enforcement; no named independent evaluator found | ★★☆☆☆ |
| Mistral | Limited evidence – no published capability-threshold framework found | Limited evidence | Moderate – signed the EU Art. 50 Code; not found among C2PA steering members | Moderate–strong – signed EU GPAI Code and Seoul commitments | ★☆☆☆☆ |
| xAI | Weak–moderate – a published Risk Management Framework exists | Weak – Grok sexualised deepfakes including of minors: EU DSA probe (Jan 2026), Ofcom investigation | Weak – not on the EU Art. 50 Code signatory list; same Grok deepfake probes | Weak – signed Seoul commitments but only the safety chapter of the EU GPAI Code; no named external evaluators found | ★☆☆☆☆ |
| Qwen | Weak – external testers found ransomware and malware generation | Weak | Weak – not on the EU Art. 50 Code list | Weak – no RSP equivalent found; no evidence of METR, Apollo, or AISI engagement | ★☆☆☆☆ |
| DeepSeek | Weak – 100% attack success rate in one independent assessment (DeepSeek R1, Cisco/UPenn, Jan 2025) | Weak – bias found ~3× higher than a comparable Western model | Weak – not on the EU Art. 50 Code list | Weak – no published capability-threshold framework; CBRN vulnerability ~3.5× higher than comparable models | ★☆☆☆☆ |
Sorted by overall rating. “Overall” is capped at the weakest of the three pillars used for capping (Operator, Bystander, Safety Assist & Verification). Vulnerable User Protection is scored separately as an industry-wide gap and excluded from the cap – see the questions below.
The AI Safety Gap: key questions answered
What are the four AI safety pillars, and why these four?
They are modelled directly on Euro NCAP's four crash-test pillars, mapped onto who each protects.
Operator Protection (≈ Adult Occupant) – safeguards for the direct user: hallucination mitigation, uncertainty signalling, the ability to interrupt or override a task mid-stream.
Vulnerable User Protection (≈ Child Occupant) – safeguards for people in high-stakes, low-agency positions relative to the system: minors, applicants, claimants, patients.
Bystander Protection (≈ Vulnerable Road User) – protection for people who never chose to interact with the system at all: content provenance, non-consensual deepfake and data safeguards.
Safety Assist & Verification (≈ Safety Assist) – the active harm-prevention stack, plus whether anyone other than the provider has actually checked it: a published, capability-gated safety framework, named external evaluators, voluntary incident disclosure.
The overall rating is capped by the weakest pillar, not averaged across all four – exactly as Euro NCAP caps a car's star rating so a manufacturer can't hide a genuine safety gap behind strong performance elsewhere.
Alchemy ConsultingHow were the nine providers actually scored?
This is an evidence-based scorecard built from public disclosures, not a controlled crash test. Unlike cars, there is no independent body putting every model through an identical, standardised safety test and publishing a comparable score.
Each pillar was scored on the tiered strength of public evidence: whether a relevant framework or document exists and is public, whether it has been exercised in a real gating decision rather than just published (Anthropic holding back general release of Claude Mythos Preview over cybersecurity findings is the clearest example available), and whether independent, named third parties – METR, Apollo Research, a government AI Safety Institute – have actually evaluated the specific models in question.
Absence of published evidence is treated as a data gap to disclose, not proof that no safeguard exists. It is scored cautiously, not punitively.
How do the nine providers actually compare?
Three tiers emerge. Microsoft is the only provider to reach four stars; OpenAI and Google reach three; the other six sit at one star. The full pillar-by-pillar scorecard is at the top of this page.
The bottom of the table is not one story. Meta sits alone on two stars: it now publishes a capability-gated framework and is active on content provenance, but declined the EU GPAI Code and releases open weights, which limits what it can enforce after release. Mistral has signed the EU codes but shows limited published evidence on capability thresholds. xAI is held down by the Grok deepfake investigations. Qwen and DeepSeek score weak on every pillar.
Does a provider's overall rating average across the four pillars?
No – deliberately not. Euro NCAP caps a car's overall star rating at its weakest pillar rather than averaging across all four, specifically so a manufacturer cannot hide a genuine safety gap behind strong performance elsewhere. This scorecard applies the same rule.
The cap is why Anthropic, which has the strongest Safety Assist & Verification evidence of any provider scored and ranks first in the Future of Life Institute's Summer 2026 index, sits on three stars rather than four: its Operator Protection evidence is Moderate, and that sets the ceiling. The evidence and the rule are published so readers can check each placing for themselves.
Who scores strongest, and why?
Microsoft, OpenAI, and Google DeepMind cluster at the top, each with a published, capability-gated safety framework, named external evaluators, and active C2PA content-provenance participation. Microsoft's External Red Team Alliance – 18 universities across six continents – is the most extensive third-party evaluation network of any provider scored, .
Anthropic’s Safety Assist & Verification evidence is comparable to the top cluster, and its Bystander Protection improved materially in 2026: since 2 August 2026 Claude output carries a text watermark and signed C2PA Content Credentials on image files, and Anthropic signed the EU Article 50 transparency Code. The cap now falls on Operator Protection, where its evidence is Moderate rather than Strong.
Where does the industry collectively fall short?
Vulnerable User Protection – safeguards specifically for minors, applicants, claimants, and patients interacting with AI in high-stakes, low-agency situations. Several providers now run age-related safeguards (OpenAI age prediction and parental controls, Meta teen controls, Google supervised accounts, Anthropic 18+ consumer accounts), but these are access-control measures. This research found no standardised, independently verified, comparable outcome evidence for any provider, and recent independent testing, such as Common Sense Media’s October 2026 “Unacceptable Risk” rating of ChatGPT for Teens, contradicts the claims made for them.
This is not one provider's gap; it is scored as an industry-wide blind spot, and it is precisely the pillar the EU AI Act's high-risk classification for hiring, credit, and welfare systems is designed to force into the open – now due from 2 December 2027 following the Digital Omnibus.
What is the regulatory hook, and why does the timing matter?
Two hard deadlines land either side of this research. The EU AI Act's obligations for general-purpose AI models with systemic risk – including mandatory adversarial testing and incident reporting under Article 55 – have applied since August 2025, and the AI Office's enforcement powers, including fines of up to €15 million or 3% of worldwide turnover, began in August 2026.
The Product Liability Directive (EU) 2024/2853 transposes into national law by 9 December 2026, and explicitly brings software and AI systems into the legal definition of a defective “product” for the first time – with strict liability, a shifted burden of proof, and joint liability between an AI component's provider and whoever integrates it.
Providers scoring well on this scorecard today are closer to being able to rebut the Directive's defect presumption once it lands. The ones scoring weakest have under two months left to close the gap.
Alchemy ConsultingWhat about the open-weight and Chinese-model providers specifically?
The evidence here is the most consistent, and the most concerning, of any group scored. Independent red-team studies have repeatedly found Qwen models producing ransomware instructions and malware code under adversarial testing, with no published pre-deployment safety evaluation in the format frontier labs use.
DeepSeek's record is starker still: one widely cited security assessment found DeepSeek R1 failed to block a single harmful prompt across a standard test set – a 100% attack success rate (DeepSeek R1, January 2025; later models have not been tested the same way) – and separate red-team work found its CBRN-related vulnerability roughly 3.5 times higher than comparable Western models. Neither provider shows evidence of engagement with METR, Apollo Research, or any national AI Safety Institute. Separately, DeepSeek's data processing under Chinese jurisdiction raises a distinct sovereignty question that sits outside the four safety pillars entirely.
What should this compel a board or leadership team to actually do?
First, treat vendor safety posture as a procurement question with evidence attached, not a box-ticking assurance – ask which named third party evaluated the specific model in question, not whether the vendor says it takes safety seriously.
Second, map exposure against the Product Liability Directive now, while there is still runway before 9 December 2026, rather than discovering the gap during a defect claim.
Third, revisit the assessment annually – this scorecard will be updated each year, and on the evidence gathered so far, this is a field where a provider's position can move materially within a single year.
- The AI Safety Gap Research is updated annually during calendar Q4.
- This is an evidence-based scorecard built from public disclosures – safety frameworks, system/model cards, transparency reports, and named third-party evaluations – not a controlled, standardised test that every provider undergoes equally. Absence of published evidence is scored as a data gap, not as proof a safeguard is missing.
- The overall rating for each provider is capped at its weakest scored pillar (Operator Protection, Bystander Protection, Safety Assist & Verification), not averaged across them, following the same principle Euro NCAP applies to car safety ratings. Vulnerable User Protection is reported separately as an industry-wide evidence gap and is not included in the cap.
- Sources include: Anthropic's Responsible Scaling Policy and Transparency Hub; OpenAI's Preparedness Framework and GPT-5 System Card; Google DeepMind's Frontier Safety Framework; Microsoft's Frontier Governance Framework and 2026 Responsible AI Transparency Report; xAI's Risk Management Framework; the Frontier Model Forum's Third-Party Assessments guidance; C2PA/Content Credentials membership records; the EU Article 50 transparency Code signatory list (31 July 2026); Meta’s Frontier AI Framework and Advanced AI Scaling Framework; Anthropic’s “How Claude marks AI-generated content”; Common Sense Media’s ChatGPT for Teens assessment (October 2026); METR, Apollo Research, and UK AISI/US CAISI public evaluation disclosures; the Future of Life Institute's AI Safety Index; AI Safety Facts; and independent red-team assessments from Cisco, Promptfoo, SPLX, and Holistic AI, as of 19 October 2026. Revised from the September draft after an evidence audit.
- Regulatory dates reflect the EU AI Act (general-purpose AI obligations from August 2025; AI Office enforcement powers from 2 August 2026), the Digital Omnibus on AI (high-risk obligations moved to 2 December 2027), and Directive (EU) 2024/2853 (transposition by 9 December 2026), as of October 2026.
- The inaugural (Q4) 2026 edition of this research page was built as a companion to “The Seatbelt Didn't Kill the Car. AI Regulation Won't Kill AI.” (Catalyst, June 2026).