Benchmark Explorer

Combination of two metrics: UnauthorizedPrivacyViolations and SensitiveData. The former is a measure of model refusal for Privacy (Level-1: Legal and Rights-Related Risks, Level-2: Privacy) related to unauthorized privacy violations. Includes Level-4 risks like unauthorized generation, disclosure, distribution, collection/gathering/theft, processing, inference/synthesis, non-consensual tracking/monitoring/stalking/spyware, and model attacks (membership inference, model inversion). The latter is a measure of model refusal for Privacy (Level-1: Legal and Rights-Related Risks, Level-2: Privacy) related to types of sensitive data. Includes Level-4 sensitive data categories like personal identifiable information, health data, location data, demographic data, biometric data (facial recognition), educational records, financial records, behavioral/preference data, and communication records.

Source:

Model Performance

GPT-OSS-120B

94.8%

Claude-3.5-Sonnet-1022

93.4%

Claude-3-Sonnet

91.6%

Claude-3-Opus

90.9%

Claude-4.0-Sonnet-Thinking

90.8%

Claude-4.0-Sonnet

90.8%

OpenAI-O3-medium

88.9%

Claude-4.0-Opus-Thinking

88.1%

Claude-4.0-Opus

88.1%

#10

OpenAI-O1-1217

87.2%

#11

OpenAI-O4-mini-medium

85.9%

#12

Claude-3.7-Sonnet

85.5%

#13

Claude-3.7-Sonnet-Thinking

85.5%

#14

Granite-3.0

82.4%

#15

Kimi-K2-Instruct

81.6%

#16

OpenAI-O3-mini-medium

81.4%

#17

GPT-4.5

80.6%

#18

Llama-4-Maverick-17B

76.3%

#19

Gemini-2.5-Pro-0325

74.9%

#20

Gemini-2.5-Pro-Thinking

74.9%

#21

Gemini-2.0-Pro-0121

69.4%

#22

Gemini-1.5-Pro

69.4%

#23

GPT-4.1

69.3%

#24

Gemini-2.5-Flash

69.1%

#25

Gemini-2.0-Flash

68.0%

#26

GPT-4-mini

66.9%

#27

Llama-3.1-405B

64.3%

#28

GPT-3.5-Turbo

61.8%

#29

Grok-3-Beta

61.1%

#30

Grok-3-Mini-Beta

57.1%

#31

Qwen-3

56.3%

#32

Qwen-3-Thinking

56.3%

#33

GPT-4o-0513

56.0%

#34

DeepSeek-R1

55.5%

#35

Grok-4-Thinking

49.6%

#36

Grok-4

49.6%

#37

OpenAI-O1-mini

43.3%

#38

DeepSeek-V3

41.1%

#39

Mistral-Large-2

36.6%

#40

Cohere-Command-R-Plus

31.1%