updated Mon Sept 28

LLM Leaderboard

This LLM leaderboard displays the latest public benchmark performance for SOTA model versions released after April 2024. The data comes from model providers as well as independently run evaluations by Vellum or the open-source community. We feature results from non-saturated benchmarks, excluding outdated benchmarks (e.g. MMLU).

Best Overall (Humanity's Last Exam)

70%53%35%18%0%

67.7

Claude Opus 5.5

Claude Fable 5.1

Claude Mythos 5.1

64.7

Claude Opus 5

64.5

Claude Mythos 5

64.5

Claude Sonnet 5.5

57.9

Claude Opus 4.8

57.4

Claude Sonnet 5

57.2

GPT-6 Astra

Kimi K3

55.3

GLM-5.3-Flash

54.9

Gemini 3.8 Flash

54.7

GLM 5.2

Kimi K2.6

52.1

GPT-5.4

51.6

DeepSeek V4 Flash

48.2

DeepSeek V4 Pro

47.2

GPT-5.6 Sol

45.8

Gemini 3 Pro

44.9

Kimi K2 Thinking

Best Overall (Humanity's Last Exam)
Model	Score
Claude Opus 5.5	67.7%
Claude Fable 5.1	65%
Claude Mythos 5.1	65%
Claude Opus 5	64.7%
Claude Mythos 5	64.5%
Claude Sonnet 5.5	64.5%
Claude Opus 4.8	57.9%
Claude Sonnet 5	57.4%
GPT-6 Astra	57.2%
Kimi K3	56%
GLM-5.3-Flash	55.3%
Gemini 3.8 Flash	54.9%
GLM 5.2	54.7%
Kimi K2.6	54%
GPT-5.4	52.1%
DeepSeek V4 Flash	51.6%
DeepSeek V4 Pro	48.2%
GPT-5.6 Sol	47.2%
Gemini 3 Pro	45.8%
Kimi K2 Thinking	44.9%

Top models per tasks

Best in Reasoning (GPQA Diamond)

100%95%91%86%81%

96.2

Claude Sonnet 5

GPT-6 Astra

95.4

Claude 3 Opus

94.6

GPT-5.6 Sol

94.3

Gemini 3.1 Pro

Best in Reasoning (GPQA Diamond)
Model	Score
Claude Sonnet 5	96.2%
GPT-6 Astra	96%
Claude 3 Opus	95.4%
GPT-5.6 Sol	94.6%
Gemini 3.1 Pro	94.3%

Best in Agentic Coding (SWE Bench)

100%94%88%82%77%

96.2

GPT-5.6 Sol

95.5

Claude Mythos 5

Claude Fable 5

GPT-5.6 Luna

88.6

Claude Opus 4.8

Best in Agentic Coding (SWE Bench)
Model	Score
GPT-5.6 Sol	96.2%
Claude Mythos 5	95.5%
Claude Fable 5	95%
GPT-5.6 Luna	93%
Claude Opus 4.8	88.6%

New

Best for Work Automations (AutoBench)

50%38%25%13%0%

48.8

GLM-5.3-Flash

41.4

GPT-6 Astra

Claude Opus 5.5

33.2

GPT-6 Sol

31.4

Claude Fable 5.1

Best for Work Automations (AutoBench)
Model	Score
GLM-5.3-Flash	48.8%
GPT-6 Astra	41.4%
Claude Opus 5.5	40%
GPT-6 Sol	33.2%
Claude Fable 5.1	31.4%

New

Best in Computer Use (OSWorld)

85%81%76%72%68%

Claude Fable 5

83.4

Claude Opus 4.8

81.2

Claude Sonnet 5

78.7

GPT-5.5

78.5

Claude Sonnet 4.6

Best in Computer Use (OSWorld)
Model	Score
Claude Fable 5	85%
Claude Opus 4.8	83.4%
Claude Sonnet 5	81.2%
GPT-5.5	78.7%
Claude Sonnet 4.6	78.5%

New

Best in Browsing (BrowseComp)

95%90%86%81%77%

92.2

GPT-5.6 Sol

91.5

GPT-6 Astra

91.2

Kimi K3

90.8

Claude Opus 5

Claude Fable 5

Best in Browsing (BrowseComp)
Model	Score
GPT-5.6 Sol	92.2%
GPT-6 Astra	91.5%
Kimi K3	91.2%
Claude Opus 5	90.8%
Claude Fable 5	88%

New

Best in Terminal Use (Terminal-Bench 2.1)

90%87%83%80%77%

89.4

Gemini 3.8 Flash

88.8

GPT-5.6 Sol

88.3

Kimi K3

Claude Mythos 5

85.8

Gemini 3.7 Flash

Best in Terminal Use (Terminal-Bench 2.1)
Model	Score
Gemini 3.8 Flash	89.4%
GPT-5.6 Sol	88.8%
Kimi K3	88.3%
Claude Mythos 5	88%
Gemini 3.7 Flash	85.8%

Fastest and most affordable models

Fastest Models (Tokens/sec)

Llama 4 Scout2600 t/s

Llama 3.1 405b969 t/s

GLM 5.2347 t/s

Kimi K2.6342.6 t/s

Kimi K2.5337.7 t/s

Lowest Latency (TTFT)

GPT-5.3 Codex0.003s

Nova Micro0.3s

Llama 4 Scout0.33s

Gemini 2.0 Flash0.34s

GPT-4o mini0.35s

Cheapest Models (per 1M tokens)

Nova Micro$0.04 / $0.14

Gemini 1.5 Flash$0.075 / $0.3

Gemini 2.0 Flash$0.1 / $0.4

GPT-4.1 nano$0.1 / $0.4

GPT-6 Luna$0.1 / $0.5

Compare models

Side-by-side comparison of the latest models released in the last 9 months.

	Claude Opus 5.5	Claude Mythos 5.1
Context size	1000000	1000000
Cutoff date	Jun 2026	2026-06
I/O cost	$4 / $20	$10 / $50
Max output	-	1000000
Latency	-	-
Speed	-	-

Best Overall (HLE)

Claude Opus 5.5

67.7

Claude Mythos 5.1

Best in Terminal Use (Terminal-Bench 2.1)

Claude Opus 5.5

66.4

Claude Mythos 5.1

60.9

Best in Agentic Coding (SWE-Bench)

Claude Opus 5.5-

Claude Mythos 5.1-

Best in Reasoning (GPQA Diamond)

Claude Opus 5.5-

Claude Mythos 5.1-

Compare Personal AI harnesses

Compare with

	Vellum	Hermes	OpenClaw	Claude Cowork	Hermes
Open source	MIT	MIT	Apache 2.0	Proprietary	MIT
Time to set up	Easy	Moderate	Difficult	Easy	Moderate
Native channels	iOS, MacOS, Web, Voice, Email, Telegram, Slack, CLI	CLI / TUI	CLI, MacOS, Web	CLI, MacOS, Windows, Web	CLI / TUI
Memory	Managed memory	SQLite + markdown — you build the memory stack	Basic memory, context loss	Limited	SQLite + markdown — you build the memory stack
Security	Built-in security	DIY	DIY	No sandboxing	DIY
Hosting	Cloud or self-hosted	Self-hosted only	Self-hosted only	Anthropic cloud	Self-hosted only
Native integrations	Managed OAuth connections	No managed connectors	No managed connectors	MCP only	No managed connectors
Schedules	Cron + Heartbeat	Cron + Heartbeat	Cron + Heartbeat	Cron only	Cron + Heartbeat
Pricing	Free + API costs, Paid plans available	Free + DIY Hosting Costs + API costs	Free + DIY Hosting Costs + API costs	Paid plans available + API costs	Free + DIY Hosting Costs + API costs

Model Comparison

Model	Context size	Cutoff date	I/O cost	Max output	Latency	Speed
Claude Opus 5.5	1000000	Jun 2026	$4 / $20	-	-	-
Claude Fable 5.1	1000000	2026-06	$10 / $50	1000000	-	-
Claude Mythos 5.1	1000000	2026-06	$10 / $50	1000000	-	-
Claude Sonnet 5.5	1000000	Jun 2026	$2 / $10	-	-	-
GPT-6 Astra	-	-	$10 / $50	-	-	-
GLM-5.3-Flash	1,000,000	-	$0.15 / $0.5	-	-	-
Gemini 3.8 Flash	1048576	-	$0.75 / $3.75	-	-	-
GPT-5.4	1050000	-	$2.5 / $15	-	-	-
GPT-5.4 mini	400000	-	$0.75 / $4.5	-	-	-
GPT-5.4 nano	400000	-	$0.2 / $1.25	-	-	-
Gemini 3.8 Flash Cyber	-	-	-	-	-	-
GPT-5.4 Pro	-	-	$30 / $180	-	-	-
GPT-5.6 Cyber	-	-	-	-	-	-
GPT-6 Luna	1050000	May 2026	$0.1 / $0.5	-	-	-
GPT-6 Sol	1050000	Apr 2026	$2 / $10	-	-	-

Context window, cost and speed comparison

Models	Context Window	Input Cost / 1M tokens	Output Cost / 1M tokens	Speed (tokens/second)	Latency
Claude Opus 5.5	1000000	$4	$20	n/a	n/a
Claude Fable 5.1	1000000	$10	$50	n/a	n/a
Claude Mythos 5.1	1000000	$10	$50	n/a	n/a
Claude Sonnet 5.5	1000000	$2	$10	n/a	n/a
GPT-6 Astra	n/a	$10	$50	n/a	n/a
GLM-5.3-Flash	1,000,000	$0.15	$0.5	n/a	n/a
Gemini 3.8 Flash	1048576	$0.75	$3.75	n/a	n/a
GPT-5.4	1050000	$2.5	$15	n/a	n/a
GPT-5.4 mini	400000	$0.75	$4.5	n/a	n/a
GPT-5.4 nano	400000	$0.2	$1.25	n/a	n/a
Gemini 3.8 Flash Cyber	n/a	n/a	n/a	n/a	n/a
GPT-5.4 Pro	n/a	$30	$180	n/a	n/a
GPT-5.6 Cyber	n/a	n/a	n/a	n/a	n/a
GPT-6 Luna	1050000	$0.1	$0.5	n/a	n/a
GPT-6 Sol	1050000	$2	$10	n/a	n/a

Benchmark glossary

Humanity's Last Exam: A crowd-sourced exam of extremely hard questions spanning every academic discipline. Designed to be the final exam before superhuman AI.
GPQA Diamond: Graduate-level science questions curated by domain experts. Tests advanced reasoning across physics, chemistry, and biology.
SWE-Bench Verified: Real GitHub issues from popular Python repos that the model must resolve end-to-end. Measures agentic software engineering ability.
AutoBench: Automation benchmark evaluating a model's ability to complete real-world work automation tasks using tools and multi-step workflows.
OSWorld-Verified: Real-world computer use tasks requiring GUI interaction in desktop environments. Measures end-to-end task completion on a real OS.
BrowseComp: Agentic web search benchmark testing a model's ability to browse and extract information from the web to answer complex questions.
Terminal-Bench 2.1: Terminal and tool use benchmark evaluating a model's ability to execute multi-step tasks in a terminal environment.

LLM Leaderboard

Best Overall (Humanity's Last Exam)

Top models per tasks

Best in Reasoning (GPQA Diamond)

Best in Agentic Coding (SWE Bench)

Best for Work Automations (AutoBench)

Best in Computer Use (OSWorld)

Best in Browsing (BrowseComp)

Best in Terminal Use (Terminal-Bench 2.1)

Fastest and most affordable models

Fastest Models (Tokens/sec)

Lowest Latency (TTFT)

Cheapest Models (per 1M tokens)

Compare models

Compare Personal AI harnesses

Model Comparison

Context window, cost and speed comparison

Benchmark glossary

The Personal AI you were promised