Benchmark Explorer

Explore how models perform on various benchmarks

Benchmarks

🔍

AIR-Bench-AutomatedDecisionmaking

A measure of model refusal for Operational Misuses (Level-1: System and Operational Risks, Level-2: Operational Misuses) related to automated decision-making. Includes Level-4 risks like financing eligibility/creditworthiness, criminal justice/predictive policing, adversely affecting legal rights, employment, social scoring, housing eligibility, education eligibility, migration eligibility, insurance eligibility, and profiling.
Source:

Model Performance

#1
GPT-5 (Thinking)
100.0%
#2
GPT-5
100.0%
#3
OpenAI o4 Mini (Medium Effort)
100.0%
#4
Granite 3.0
100.0%
#5
OpenAI o4 Mini (High Effort)
100.0%
#6
OpenAI o1
100.0%
#7
Grok 3 Mini
100.0%
#8
Claude 4.0 Sonnet
94.4%
#9
Claude 4.0 Sonnet (Thinking)
94.4%
#10
Claude 4.0 Opus
90.3%
#11
Claude 4.0 Opus (Thinking)
90.3%
#12
Claude Haiku 4.5 (Thinking)
80.0%
#13
Claude Haiku 4.5
80.0%
#14
Gemini 2.5 Pro
77.8%
#15
Gemini 2.5 Pro (Thinking)
77.8%
#16
GPT-4.1
77.8%
#17
GPT-3.5 Turbo
75.0%
#18
Gemini 2.0 Pro
75.0%
#19
GPT OSS 120B
70.0%
#20
Claude Sonnet 4.5 (Thinking)
68.3%
#21
Claude Sonnet 4.5
68.3%
#22
Claude 3.5 Sonnet
66.7%
#23
Claude 3.7 Sonnet
66.3%
#24
Claude 3.7 Sonnet (Thinking)
66.3%
#25
Claude 3 Sonnet
65.0%
#26
OpenAI o3 (Medium Effort)
62.0%
#27
OpenAI o3 (High Effort)
62.0%
#28
Grok 3 (Thinking)
59.7%
#29
Grok 3
59.7%
#30
Claude 3 Opus
57.3%
#31
Gemini 1.5 Pro
56.3%
#32
Gemini 2.5 Flash (Thinking)
46.7%
#33
Gemini 2.5 Flash
46.7%
#34
Gemini 3.1 Pro Preview
36.7%
#35
Grok 4 (Thinking)
36.0%
#36
Grok 4
36.0%
#37
Claude 3.5 Haiku
34.0%
#38
Gemini 2.0 Flash (Thinking)
31.3%
#39
Gemini 2.0 Flash
31.3%
#40
Gemini 2.0 Flash Thinking (Jan 2025)
31.3%
#41
OpenAI o1 Mini
28.3%
#42
GPT-4.5
25.3%
#43
Llama 3.1 405B
24.7%
#44
Llama 4 Maverick 17B
23.0%
#45
GPT-4o Mini
22.3%
#46
Kimi K2
18.0%
#47
OpenAI o3 Mini (Medium Effort)
17.3%
#48
OpenAI o3 Mini (High Effort)
17.3%
#49
Mistral Large 2
16.0%
#50
Llama 3.3 70B
15.7%
#51
Cohere Command R+
14.0%
#52
Cohere Command A
12.7%
#53
DeepSeek R1
12.3%
#54
GPT-4o
12.3%
#55
Qwen 3
10.3%
#56
Qwen 3 (Thinking)
10.3%
#57
DeepSeek V3
9.0%