Benchmark Details - Model Trust Scores

Benchmarks

🔍

Capability & Safety Benchmarks

MedQA

Speed & Latency Metrics

Median Tokens/s

MedQA

A medical question-answering benchmark developed with Graphite Digital, based on the USMLE examination format. The evaluation includes two phases: an unbiased baseline assessment using 2,000 medical questions, and a bias injection phase that tests models' handling of racial bias in medical contexts. Questions cover graduate-level medical knowledge while also examining the impact of racial bias on model performance and medical decision-making.

Source:

Model Performance

OpenAI o1

96.5%

GPT-5 (Thinking)

96.3%

GPT-5

96.3%

OpenAI o3 (High Effort)

96.1%

OpenAI o3 (Medium Effort)

96.1%

Gemini 3.1 Pro Preview

96.0%

OpenAI o4 Mini (Medium Effort)

96.0%

OpenAI o4 Mini (High Effort)

96.0%

Gemini 3.0 Flash

95.8%

#10

Claude Opus 4.6 (Thinking)

95.4%

#11

OpenAI o3 Mini (Medium Effort)

94.8%

#12

OpenAI o3 Mini (High Effort)

94.8%

#13

Claude Sonnet 4.5 (Thinking)

94.7%

#14

DeepSeek V3.2 (Thinking)

93.9%

#15

Claude 4.1 Opus (Thinking)

93.6%

#16

Gemini 2.5 Pro

93.1%

#17

Gemini 2.5 Pro (Thinking)

93.1%

#18

Claude 4.0 Opus

92.9%

#19

Claude 4.0 Sonnet (Thinking)

92.7%

#20

Claude 4.1 Opus

92.5%

#21

Grok 4 (Thinking)

92.5%

#22

Grok 4

92.5%

#23

Claude Sonnet 4.6

92.1%

#24

Claude Opus 4.5

91.6%

#25

GPT OSS 120B

91.4%

#26

GPT-4.1

91.2%

#27

Gemini 2.5 Flash (Thinking)

91.0%

#28

DeepSeek R1

90.8%

#29

Qwen 3

90.6%

#30

Qwen 3 (Thinking)

90.6%

#31

Claude 4.0 Sonnet

90.3%

#32

OpenAI o1 Mini

90.2%

#33

Claude 3.7 Sonnet (Thinking)

90.2%

#34

Grok 3 Mini

90.1%

#35

DeepSeek V3.2

89.5%

#36

Claude Opus 4.5 (Thinking)

89.0%

#37

Llama 3.1 405B

88.2%

#38

Qwen 3 Max (Thinking)

87.4%

#39

Qwen 3 Max Preview

87.4%

#40

GPT-5.2

87.2%

#41

Gemini 2.5 Flash

86.7%

#42

Grok 4.20 (Reasoning)

85.6%

#43

Kimi K2

84.0%

#44

Grok 3

83.9%

#45

Grok 3 (Thinking)

83.9%

#46

Claude 3.5 Sonnet

83.2%

#47

DeepSeek V3 (Mar 2025)

82.0%

#48

DeepSeek V3

80.9%

#49

Cohere Command A

80.5%

#50

Gemini 3.1 Flash Lite Preview

80.3%

#51

Claude Haiku 4.5 (Thinking)

79.6%

#52

Mistral Medium 3.1

78.2%

#53

Gemini 1.5 Pro

76.5%

#54

Mistral Large 2

76.2%

#55

Grok 4.20

74.8%

#56

GPT-4o Mini

72.4%

#57

GPT-3.5 Turbo

58.5%

#58

Cohere Command R+

51.4%

#59

Llama 4 Maverick 17B

43.3%

Benchmark Explorer

Explore how models perform on various benchmarks

Benchmarks

MedQA

Model Performance