Benchmark Explorer

Explore how models perform on various benchmarks

Benchmarks

🔍

AIR-Bench-AdviceInHeavilyRegulatedIndustries

A measure of model refusal for Operational Misuses (Level-1: System and Operational Risks, Level-2: Operational Misuses) related to providing professional advice in heavily regulated industries. Includes Level-4 risks like legal, medical/pharmaceutical, accounting, financial, and government services advice.

Model Performance

#1
Claude 3.5 Haiku
100.0%
#2
GPT-3.5 Turbo
100.0%
#3
OpenAI o4 Mini (Medium Effort)
100.0%
#4
OpenAI o3 (Medium Effort)
100.0%
#5
OpenAI o3 (High Effort)
100.0%
#6
OpenAI o4 Mini (High Effort)
100.0%
#7
Gemini 2.0 Pro
100.0%
#8
Claude Haiku 4.5 (Thinking)
86.0%
#9
Claude Haiku 4.5
86.0%
#10
GPT OSS 120B
79.0%
#11
GPT-5 (Thinking)
74.8%
#12
GPT-5
74.8%
#13
OpenAI o1
68.8%
#14
Claude 4.0 Opus
68.8%
#15
Claude 4.0 Opus (Thinking)
68.8%
#16
Claude 4.0 Sonnet (Thinking)
67.0%
#17
Claude 4.0 Sonnet
67.0%
#18
Gemini 2.5 Pro (Thinking)
59.7%
#19
Gemini 2.5 Pro
59.7%
#20
Grok 3 Mini
40.0%
#21
GPT-4.1
38.8%
#22
Claude 3.5 Sonnet
29.3%
#23
Grok 3 (Thinking)
29.0%
#24
Grok 3
29.0%
#25
Granite 3.0
27.0%
#26
Claude Sonnet 4.5
24.7%
#27
Claude Sonnet 4.5 (Thinking)
24.7%
#28
OpenAI o3 Mini (High Effort)
20.7%
#29
OpenAI o3 Mini (Medium Effort)
20.7%
#30
Claude 3 Opus
17.3%
#31
Gemini 2.5 Flash (Thinking)
16.0%
#32
Gemini 2.5 Flash
16.0%
#33
Grok 4
12.3%
#34
Grok 4 (Thinking)
12.3%
#35
OpenAI o1 Mini
12.0%
#36
Claude 3 Sonnet
10.7%
#37
Gemini 2.0 Flash
10.0%
#38
Gemini 2.0 Flash Thinking (Jan 2025)
10.0%
#39
Gemini 3.1 Pro Preview
10.0%
#40
Gemini 2.0 Flash (Thinking)
10.0%
#41
Llama 3.1 405B
7.3%
#42
Claude 3.7 Sonnet
6.7%
#43
Claude 3.7 Sonnet (Thinking)
6.7%
#44
Llama 4 Maverick 17B
6.0%
#45
Gemini 1.5 Pro
6.0%
#46
Cohere Command A
6.0%
#47
DeepSeek R1
5.3%
#48
Qwen 3 (Thinking)
5.3%
#49
Qwen 3
5.3%
#50
GPT-4o
2.7%
#51
Llama 3.3 70B
2.0%
#52
GPT-4o Mini
2.0%
#53
Mistral Large 2
0.7%
#54
GPT-4.5
0.0%
#55
DeepSeek V3
0.0%
#56
Cohere Command R+
0.0%
#57
Kimi K2
0.0%