updated Fri Sept 4

LLM Leaderboard

This LLM leaderboard displays the latest public benchmark performance for SOTA model versions released after April 2024. The data comes from model providers as well as independently run evaluations by Vellum or the open-source community. We feature results from non-saturated benchmarks, excluding outdated benchmarks (e.g. MMLU).

Best Overall (Humanity's Last Exam)

70%53%35%18%0%

Claude Fable 5.1

Claude Mythos 5.1

64.7

Claude Opus 5

64.5

Claude Mythos 5

57.9

Claude Opus 4.8

57.4

Claude Sonnet 5

57.2

GPT-6 Astra

Kimi K3

55.3

GLM-5.3-Flash

54.7

GLM 5.2

Kimi K2.6

51.6

DeepSeek V4 Flash

48.2

DeepSeek V4 Pro

47.2

GPT-5.6 Sol

45.8

Gemini 3 Pro

44.9

Kimi K2 Thinking

44.4

Gemini 3.1 Pro

43.1

GPT-5.5 Pro

41.4

GPT-5.5

40.2

Gemini 3.5 Flash

Best Overall (Humanity's Last Exam)
Model	Score
Claude Fable 5.1	65%
Claude Mythos 5.1	65%
Claude Opus 5	64.7%
Claude Mythos 5	64.5%
Claude Opus 4.8	57.9%
Claude Sonnet 5	57.4%
GPT-6 Astra	57.2%
Kimi K3	56%
GLM-5.3-Flash	55.3%
GLM 5.2	54.7%
Kimi K2.6	54%
DeepSeek V4 Flash	51.6%
DeepSeek V4 Pro	48.2%
GPT-5.6 Sol	47.2%
Gemini 3 Pro	45.8%
Kimi K2 Thinking	44.9%
Gemini 3.1 Pro	44.4%
GPT-5.5 Pro	43.1%
GPT-5.5	41.4%
Gemini 3.5 Flash	40.2%

Top models per tasks

Best in Reasoning (GPQA Diamond)

100%95%91%86%81%

96.2

Claude Sonnet 5

GPT-6 Astra

95.4

Claude 3 Opus

94.6

GPT-5.6 Sol

94.3

Gemini 3.1 Pro

Best in Reasoning (GPQA Diamond)
Model	Score
Claude Sonnet 5	96.2%
GPT-6 Astra	96%
Claude 3 Opus	95.4%
GPT-5.6 Sol	94.6%
Gemini 3.1 Pro	94.3%

Best in Agentic Coding (SWE Bench)

100%94%88%82%77%

96.2

GPT-5.6 Sol

95.5

Claude Mythos 5

Claude Fable 5

GPT-5.6 Luna

88.6

Claude Opus 4.8

Best in Agentic Coding (SWE Bench)
Model	Score
GPT-5.6 Sol	96.2%
Claude Mythos 5	95.5%
Claude Fable 5	95%
GPT-5.6 Luna	93%
Claude Opus 4.8	88.6%

New

Best for Work Automations (AutoBench)

50%38%25%13%0%

48.8

GLM-5.3-Flash

41.4

GPT-6 Astra

31.4

Claude Fable 5.1

31.4

Claude Mythos 5.1

30.4

Gemini 3.7 Flash

Best for Work Automations (AutoBench)
Model	Score
GLM-5.3-Flash	48.8%
GPT-6 Astra	41.4%
Claude Fable 5.1	31.4%
Claude Mythos 5.1	31.4%
Gemini 3.7 Flash	30.4%

New

Best in Computer Use (OSWorld)

85%81%76%72%68%

Claude Fable 5

83.4

Claude Opus 4.8

81.2

Claude Sonnet 5

78.7

GPT-5.5

78.5

Claude Sonnet 4.6

Best in Computer Use (OSWorld)
Model	Score
Claude Fable 5	85%
Claude Opus 4.8	83.4%
Claude Sonnet 5	81.2%
GPT-5.5	78.7%
Claude Sonnet 4.6	78.5%

New

Best in Browsing (BrowseComp)

95%90%86%81%77%

92.2

GPT-5.6 Sol

91.5

GPT-6 Astra

91.2

Kimi K3

90.8

Claude Opus 5

Claude Fable 5

Best in Browsing (BrowseComp)
Model	Score
GPT-5.6 Sol	92.2%
GPT-6 Astra	91.5%
Kimi K3	91.2%
Claude Opus 5	90.8%
Claude Fable 5	88%

New

Best in Terminal Use (Terminal-Bench 2.1)

90%86%81%77%72%

88.8

GPT-5.6 Sol

88.3

Kimi K3

Claude Mythos 5

85.8

Gemini 3.7 Flash

84.3

Claude Fable 5

Best in Terminal Use (Terminal-Bench 2.1)
Model	Score
GPT-5.6 Sol	88.8%
Kimi K3	88.3%
Claude Mythos 5	88%
Gemini 3.7 Flash	85.8%
Claude Fable 5	84.3%

Fastest and most affordable models

Fastest Models (Tokens/sec)

Llama 4 Scout2600 t/s

Llama 3.1 405b969 t/s

GLM 5.2347 t/s

Kimi K2.6342.6 t/s

Kimi K2.5337.7 t/s

Lowest Latency (TTFT)

GPT-5.3 Codex0.003s

Nova Micro0.3s

Llama 4 Scout0.33s

Gemini 2.0 Flash0.34s

GPT-4o mini0.35s

Cheapest Models (per 1M tokens)

Nova Micro$0.04 / $0.14

Gemini 1.5 Flash$0.075 / $0.3

Gemini 2.0 Flash$0.1 / $0.4

GPT-4.1 nano$0.1 / $0.4

Llama 4 Scout$0.11 / $0.34

Compare models

Side-by-side comparison of the latest models released in the last 9 months.

	Claude Mythos 5.1	Claude Fable 5.1
Context size	1000000	1000000
Cutoff date	2026-06	2026-06
I/O cost	$10 / $50	$10 / $50
Max output	1000000	1000000
Latency	-	-
Speed	-	-

Best Overall (HLE)

Claude Mythos 5.1

Claude Fable 5.1

Best in Terminal Use (Terminal-Bench 2.1)

Claude Mythos 5.1

60.9

Claude Fable 5.1

55.8

Best in Agentic Coding (SWE-Bench)

Claude Mythos 5.1-

Claude Fable 5.1-

Best in Reasoning (GPQA Diamond)

Claude Mythos 5.1-

Claude Fable 5.1-

Compare Personal AI harnesses

Compare with

	Vellum	Hermes	OpenClaw	Claude Cowork	Hermes
Open source	MIT	MIT	Apache 2.0	Proprietary	MIT
Time to set up	Easy	Moderate	Difficult	Easy	Moderate
Native channels	iOS, MacOS, Web, Voice, Email, Telegram, Slack, CLI	CLI / TUI	CLI, MacOS, Web	CLI, MacOS, Windows, Web	CLI / TUI
Memory	Managed memory	SQLite + markdown — you build the memory stack	Basic memory, context loss	Limited	SQLite + markdown — you build the memory stack
Security	Built-in security	DIY	DIY	No sandboxing	DIY
Hosting	Cloud or self-hosted	Self-hosted only	Self-hosted only	Anthropic cloud	Self-hosted only
Native integrations	Managed OAuth connections	No managed connectors	No managed connectors	MCP only	No managed connectors
Schedules	Cron + Heartbeat	Cron + Heartbeat	Cron + Heartbeat	Cron only	Cron + Heartbeat
Pricing	Free + API costs, Paid plans available	Free + DIY Hosting Costs + API costs	Free + DIY Hosting Costs + API costs	Paid plans available + API costs	Free + DIY Hosting Costs + API costs

Model Comparison

Model	Context size	Cutoff date	I/O cost	Max output	Latency	Speed
Claude Fable 5.1	1000000	2026-06	$10 / $50	1000000	-	-
Claude Mythos 5.1	1000000	2026-06	$10 / $50	1000000	-	-
Claude Opus 5	1,000,000	May 2026	$5 / $25	128,000	-	-
GPT-6 Astra	-	-	$10 / $50	-	-	-
Kimi K3	1,048,576	-	$3 / $15	-	4.46s	35.2 t/s
GLM-5.3-Flash	1,000,000	-	$0.15 / $0.5	-	-	-
GLM 5.2	1,000,000	Mar 2026	$0.95 / $3	128,000	1.14s	347 t/s
DeepSeek V4 Flash	1000000	Jan 2026	$0.14 / $0.28	384000	1.42s	107.9 t/s
DeepSeek V4 Pro	1,000,000	Jan 2026	$0.435 / $0.87	384,000	1.2s	174.9 t/s
GPT-5.6 Sol	1,050,000	Feb 2026	$5 / $30	128,000	-	-
Gemini 3.1 Pro	1,000,000	Jan 2026	$2 / $12	65,536	20.34s	136.2 t/s
Gemini 3.5 Flash	1,000,000	Jan 2026	$1.5 / $9	65,536	23.16s	175.4 t/s
Gemini 3.7 Flash	1,048,576	Mar 2026	$0.75 / $3.75	65,536	-	-
GPT-5.6 Luna	1,050,000	Feb 2026	$0.2 / $1.2	128,000	-	-
GPT-5.6 Terra	1,050,000	Feb 2026	$2 / $12	128,000	-	-

Context window, cost and speed comparison

Models	Context Window	Input Cost / 1M tokens	Output Cost / 1M tokens	Speed (tokens/second)	Latency
Claude Fable 5.1	1000000	$10	$50	n/a	n/a
Claude Mythos 5.1	1000000	$10	$50	n/a	n/a
Claude Opus 5	1,000,000	$5	$25	n/a	n/a
GPT-6 Astra	n/a	$10	$50	n/a	n/a
Kimi K3	1,048,576	$3	$15	35.2 t/s	4.46 seconds
GLM-5.3-Flash	1,000,000	$0.15	$0.5	n/a	n/a
GLM 5.2	1,000,000	$0.95	$3	347 t/s	1.14 seconds
DeepSeek V4 Flash	1000000	$0.14	$0.28	107.9 t/s	1.42 seconds
DeepSeek V4 Pro	1,000,000	$0.435	$0.87	174.9 t/s	1.2 seconds
GPT-5.6 Sol	1,050,000	$5	$30	n/a	n/a
Gemini 3.1 Pro	1,000,000	$2	$12	136.2 t/s	20.34 seconds
Gemini 3.5 Flash	1,000,000	$1.5	$9	175.4 t/s	23.16 seconds
Gemini 3.7 Flash	1,048,576	$0.75	$3.75	n/a	n/a
GPT-5.6 Luna	1,050,000	$0.2	$1.2	n/a	n/a
GPT-5.6 Terra	1,050,000	$2	$12	n/a	n/a

Benchmark glossary

Humanity's Last Exam: A crowd-sourced exam of extremely hard questions spanning every academic discipline. Designed to be the final exam before superhuman AI.
GPQA Diamond: Graduate-level science questions curated by domain experts. Tests advanced reasoning across physics, chemistry, and biology.
SWE-Bench Verified: Real GitHub issues from popular Python repos that the model must resolve end-to-end. Measures agentic software engineering ability.
AutoBench: Automation benchmark evaluating a model's ability to complete real-world work automation tasks using tools and multi-step workflows.
OSWorld-Verified: Real-world computer use tasks requiring GUI interaction in desktop environments. Measures end-to-end task completion on a real OS.
BrowseComp: Agentic web search benchmark testing a model's ability to browse and extract information from the web to answer complex questions.
Terminal-Bench 2.1: Terminal and tool use benchmark evaluating a model's ability to execute multi-step tasks in a terminal environment.

LLM Leaderboard

Best Overall (Humanity's Last Exam)

Top models per tasks

Best in Reasoning (GPQA Diamond)

Best in Agentic Coding (SWE Bench)

Best for Work Automations (AutoBench)

Best in Computer Use (OSWorld)

Best in Browsing (BrowseComp)

Best in Terminal Use (Terminal-Bench 2.1)

Fastest and most affordable models

Fastest Models (Tokens/sec)

Lowest Latency (TTFT)

Cheapest Models (per 1M tokens)

Compare models

Compare Personal AI harnesses

Model Comparison

Context window, cost and speed comparison

Benchmark glossary

The Personal AI you were promised