Benchmark Explorer

A comprehensive contract analysis benchmark developed with SpeedLegal that evaluates three key aspects of contract processing: extraction of specific legal terms, matching contract language against standards, and correction of non-standard contract language. The evaluation covers five contract types (NDAs, DPAs, MSAs, Sales Agreements, and Employment Agreements) and tests models' ability to understand, analyze, and modify legal documents according to industry standards.

Source:

Model Performance

Llama-3.1-405B

75.2%

Claude-3-Opus

74.0%

Claude-3.7-Sonnet-Thinking

73.0%

OpenAI-O1-mini

72.8%

OpenAI-O1-1217

72.7%

GPT-4-mini

72.4%

Claude-4.0-Sonnet

72.4%

OpenAI-O3-mini-high

69.3%

OpenAI-O3-mini-medium

69.3%

#10

OpenAI-O4-mini-high

68.9%

#11

OpenAI-O4-mini-medium

68.9%

#12

Claude-4.0-Opus

68.8%

#13

Claude-3.5-Haiku

68.7%

#14

Claude-3.5-Sonnet-1022

68.7%

#15

DeepSeek-V3

68.4%

#16

GPT-4.1

68.4%

#17

Cohere-Command-R-Plus

68.2%

#18

Claude-3.7-Sonnet

68.1%

#19

Gemini-1.5-Pro

68.0%

#20

Cohere-Command-A

67.7%

#21

Claude-3-Sonnet

67.6%

#22

Mistral-Large-2

67.0%

#23

Gemini-2.0-Flash

66.8%

#24

Claude-4.1-Opus

66.5%

#25

Kimi-K2-Instruct

66.4%

#26

Claude-4.1-Opus-Thinking

66.2%

#27

Mistral-Medium-3.1

66.2%

#28

OpenAI-O3-medium

66.0%

#29

Claude-4.0-Sonnet-Thinking

66.0%

#30

Llama-3.3-70B

66.0%

#31

OpenAI-O3-high

66.0%

#32

Grok-4-Thinking

66.0%

#33

Grok-4

66.0%

#34

Gemini-2.5-Pro-0325

64.7%

#35

Gemini-2.5-Pro-Thinking

64.7%

#36

DeepSeek-V3-0324

64.7%

#37

Qwen-3

63.6%

#38

Qwen-3-Thinking

63.6%

#39

Grok-3-Beta

63.5%

#40

Grok-3-Think

63.5%

#41

Gemini-2.5-Flash

63.4%

#42

Grok-3-Mini-Beta

63.4%

#43

Llama-4-Maverick-17B

63.3%

#44

DeepSeek-R1

62.0%

#45

GPT-3.5-Turbo

61.6%

#46

Gemini-2.5-Flash-Thinking

60.6%

#47

Llama-2-70B

59.7%

#48

Llama-2-13B

55.9%

#49

Llama-2-7B

42.5%