last update: 18 December 2024
LLM Leaderboard
Top Models per Task
Best in Multitask Reasoning (MMLU)
Best in Coding (Human Eval)
Best in Math (MATH)
Fastest and Most Affordable Models
Lowest Latency (TTFT)
Cheapest Models
Compare Models
Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form.
vs
Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form.
Model
Context size
Cutoff date
Input/Output cost
Max output tokens
Latency (TTFT)
Throughput
OpenAI o1
200,000
October, 2023
$15 input & $7.50 cashed
/
$60
100,000
/
/
Gemini 2.0 Flash
1,000,000
December 2024
/
8,192
0.48s
169 t/s
Llama 3.3 70b
128,000
July 2024
$0.59
/
$0.79
32,768
n/a
250 t/s via Groq
AWS Nova Micro
128,000
July 2024
$0.000035
/
0.00014
4096
0.3s
n/a
AWS Nova Lite
300,000
July 2024
$0.00006
/
$0.00024
4096
0.4s
150 t/s
AWS Nova Pro
300,000
July 2024
$0.0008
/
$0.0032
4096
0.4s
97 t/s
Claude 3.5 Haiku
200,000
July 2024
$0.80
/
$4
4096
0.77s
60 t/s
Llama 3.1 405b
128,000
Dec 2023
$5
/
$10
4096
0.59s
163 t/s via SambaNova
Llama 3.1 70b
128,000
Dec 2023
$0.6
/
4096
0.38s
2,100 t/s (Cerebras)
Llama 3.1 8b
128,000
Dec 2023
$0.05
/
$0.08
4096
0.32s
~1800 t/s (Cerebras)
Gemini 1.5 Flash
1,000,000
May 2024
$0.075
/
$0.3
4096
1.06S
166 t/s
Gemini 1.5 Pro
2,000,000
May 2024
$3.5
/
$10.5
4096
1.12s
61 t/s
GPT-3.5 Turbo
16,400
Sept 2023
$0.5
/
$1.5
4096
0.37s
84 t/s
GPT-4o mini
128,000
Oct 2023
$0.15
/
$0.6
4096
0.56s
97 t/s
GPT-Turbo
128,000
Dec 2023
$10
/
$30
4096
0.6s
28 t/s
GPT-4o
128,000
Oct 2023
$2.5
/
$10
4096
0.48s
79 t/s
Claude 3 Haiku
200,000
Apr 2024
$0.25
/
$1.25
4096
0.55s
133 t/s
Claude 3.5 Sonnet
200,000
Apr 2024
$3
/
$15
4096
1.22s
78 t/s
Claude 3 Opus
200,000
Aug 2023
$15
/
$75
4096
1.99s
25 t/s
GPT-4
8192
Dec 2023
$30
/
$60
4096
0.59s
125 t/s
OpenAI o1
200,000
October, 2023
$15 input & $7.50 cashed
/
$60
100,000
/
/
Gemini 2.0 Flash
1,000,000
December 2024
/
8,192
0.48s
169 t/s
Llama 3.3 70b
128,000
July 2024
$0.59
/
$0.79
32,768
n/a
250 t/s via Groq
AWS Nova Micro
128,000
July 2024
$0.000035
/
0.00014
4096
0.3s
n/a
AWS Nova Lite
300,000
July 2024
$0.00006
/
$0.00024
4096
0.4s
150 t/s
AWS Nova Pro
300,000
July 2024
$0.0008
/
$0.0032
4096
0.4s
97 t/s
Claude 3.5 Haiku
200,000
July 2024
$0.80
/
$4
4096
0.77s
60 t/s
Llama 3.1 405b
128,000
Dec 2023
$5
/
$10
4096
0.59s
163 t/s via SambaNova
Llama 3.1 70b
128,000
Dec 2023
$0.6
/
4096
0.38s
2,100 t/s (Cerebras)
Llama 3.1 8b
128,000
Dec 2023
$0.05
/
$0.08
4096
0.32s
~1800 t/s (Cerebras)
Gemini 1.5 Flash
1,000,000
May 2024
$0.075
/
$0.3
4096
1.06S
166 t/s
Gemini 1.5 Pro
2,000,000
May 2024
$3.5
/
$10.5
4096
1.12s
61 t/s
GPT-3.5 Turbo
16,400
Sept 2023
$0.5
/
$1.5
4096
0.37s
84 t/s
GPT-4o mini
128,000
Oct 2023
$0.15
/
$0.6
4096
0.56s
97 t/s
GPT-Turbo
128,000
Dec 2023
$10
/
$30
4096
0.6s
28 t/s
GPT-4o
128,000
Oct 2023
$2.5
/
$10
4096
0.48s
79 t/s
Claude 3 Haiku
200,000
Apr 2024
$0.25
/
$1.25
4096
0.55s
133 t/s
Claude 3.5 Sonnet
200,000
Apr 2024
$3
/
$15
4096
1.22s
78 t/s
Claude 3 Opus
200,000
Aug 2023
$15
/
$75
4096
1.99s
25 t/s
GPT-4
8192
Dec 2023
$30
/
$60
4096
0.59s
125 t/s
Standard Benchmarks
Model Comparison
* For some models, we don't show an average value because there is missing data. Also, every model on this list is the latest version.
x
Thank you!
Your submission has been received!
Your submission has been received!
Oops! Something went wrong while submitting the form.
Cost and Context Window Comparison
Comparison of context window and cost per 1M tokens.
HumanEval: Coding Leaderboard
Comparison of pre-trained proprietary and open-source models for code generation.
Sources
This leaderboard shows a comparison of capabilities, price and context window for leading commercial and open-source LLMs, based on the benchmark data provided in the models' technical reports. Updated March 2024.
Technical reports
Pricing info