Yesterday, OpenAI launched its most anticipated model: GPT-4.5—and it didn’t feel all that exciting. It came across more like a mid-cycle refresh than a flagship release. That might mean bigger things are on the horizon—and given OpenAI’s usual pace, we probably won’t have to wait long to find out.

The main improvements? Fewer hallucinations and a more natural conversational feel. This human-like upgrade seemed like a small change today, but many are saying that it actually feels more human, and that is a big thing.

But then we looked at the price. GPT-4.5 is 75x more expensive than GPT-4o. After the announcement, we were all left wondering: Did we miss something that justifies the cost? Still unclear.

Last week, Claude 3.7 Sonnet dropped—probably the best model for coding right now—and the hype was real. Many were expecting OpenAI to respond with a strong competitor. Maybe that’s still coming. We’ll see.

For now, here’s our breakdown of pricing, latency, standard benchmarks—and our own independent evals on adaptive reasoning and math equations.

Results

In this analysis we compared Claude 3.7 Sonnet with GPT-4.5 on price, latency, benchmarks and small evaluations on reasoning and math tasks. For good measure we included some other models in the analysis. Here are our findings:

• Pricing: Claude 3.7 Sonnet is significantly more affordable than GPT-4.5. GPT-4.5 is 25 times more expensive for input tokens and 10 times more expensive for output tokens compared to Claude 3.7 Sonnet. Claude 3.7 Sonnet is both a general purpose and reasoning model, so it feels like there is a clear choice here when it comes to pricing.
• Latency & Speed: Claude 3.7 Sonnet has twice the throughput of GPT-4.5 while maintaining similar latency, making it a more efficient alternative.
• Benchmarks: Looking at the benchmarks, it's clear that Claude 3.7 Sonnet is significantly better at coding than GPT-4.5. While math isn’t Claude's strongest area, it still outperforms GPT-4.5 on AIME’24 problems. For reasoning and multimodal tasks, the two models are closely matched, with only slight differences in performance.
• Hardest SAT math equations: GPT-4.5 is on par with reasoning models like DeepSeek when it comes to solving math equations. This is great because we can see that a general purpose model can do as well as a reasoner model on this task.
• Adaptive puzzle reasoning: For this evaluation, we took very famous puzzles and changed one parameter that made them trivial. If a model really reasons, solving this puzzles should be very easy. Yet, most struggled. However, Claude 3.7 Sonnet is the model that handled this new context most effectively. This suggests it either follows instructions better or depends less on training data. This could be an isolated scenario with reasoning tasks, because when it comes to coding, just ask any developer—they’ll all say Claude 3.7 Sonnet struggles to follow instructions. Surprisingly, GPT-4.5 outperformed o1 and o3-mini.

In the next sections, we break down our methodology, share detailed evaluation results, and highlight key observations!

Methodology

In the next two sections we will cover three analysis:

• Latency, Speed & Cost comparison
• Standard benchmark comparison (example: what is the reported performance for math tasks between Claude 3.7 Sonnet vs GPT-4.5?)
• Independent evaluation experiments:
  • Hardest SAT math equations
  • Adaptive puzzle reasoning

Evaluations with Vellum

To conduct these evaluations, we used Vellum’s AI development platform, where we:

• Configured all 0-shot prompt variations for both models using the LLM Playground.
• Built the evaluation dataset & configured our evaluation experiment using the Evaluation Suite in Vellum. We used an LLM-as-a-judge to analyze generated answers to correct responses from our benchmark dataset for the math/reasoning problems.

We then compiled and presented the findings using the Evaluation Reports generated at the end of each evaluation run.

You can skip to the section that interests you most using the "Table of Contents" panel on the left or scroll down to explore the full comparison between GPT-4.5 and Claude 3.7 Sonnet (and some other models for good measure!).

Pricing

Pricing-wise, it’s hard to justify GPT-4.5’s cost. It’s nearly 75x more expensive than GPT-4o, yet the performance gains feel minor in comparison. What’s even stranger is that models built for reasoning—like Claude 3.7 Sonnet—offer a standard mode with latency similar to GPT-4.5.

Latency

Claude 3.7 Standard and GPT-4.5 have similar latency. All of these general purpose models are fast enough for real-time features. Even o3-mini can be justified for some real-time tasks.

Throughput

Claude 3.7 Sonnet has the highest throughput and is 2x faster than GPT-4.5, making it a strong choice for AI applications that require quick responses, high scalability, and real-time interactions. This speed advantage is especially useful for chatbots, virtual assistants, and other AI-driven tools where low latency enhances user experience.

Benchmarks

Looking at the benchmarks, it's clear that Claude 3.7 Sonnet is significantly better at coding than GPT-4.5. While math isn’t its strongest area, it still outperforms GPT-4.5 on AIME’24 problems. For reasoning and multimodal tasks, the two models are closely matched, with only slight differences in performance.

Independent Evaluations

Task 1: Math

For this task, we’ll compare the models on how well they solve some of the hardest SAT math questions. This is the 0-shot prompt that we used for both models:

You are a helpful assistant who is the best at solving math equations. You must output only the answer, without explanations. Here’s the <question>

We then ran all 50 math questions and here’s what we got:

‍

‍

From the above we can clearly see that:

• GPT-4.5 is on pair to DeepSeek R1 on this math dataset, which very interesting given that it’s not a thinking model
• Claude 3.7 Sonnet has the lowest performance on math tasks, which further confirms Anthropic’s statement that they did not train this model to be good at math tasks.

Task 2: Reasoning with new context

We tested OpenAI-o1, DeepSeek-R1, Claude 3.7 Sonnet, and OpenAI o3-mini on 28 well-known puzzles. For this evaluation, we changed some portion of the puzzles, and made them trivial. We wanted to see if the models still overfit on training data or will adapt to new contexts.

For example, we modified the Monty Hall problem:

Suppose you're on a game show, and you're given the choice of three doors: Behind one door is a gold bar; behind the others, rotten vegetables. You pick a door, say No. 1, and the host asks you, 'Do you want to pick door No. 2 instead?' What choice of door now gives you the biggest advantage?

In the original Monty Hall problem, the host reveals an extra door. In this case, it does not, and since there is no additional information provided, your odds remain the same.

The correct answer here is: “It is not an advantage to switch. It makes no difference if I switch or not because no additional material information has been provided since the initial choice.”

Most models had trouble working with the new context, but Claude 3.7 Sonnet performed noticeably better:

‍

‍

From the above we can clearly see that:

• Claude 3.7 Sonnet (Without extended thinking) does so much better than every other model. This is very interesting, because it’s the first time we’re seeing a model perform this well on this evaluation dataset. Here we are testing whether these reasoning models can work with new context, and performing well on it can mean that they can really follow instructions and/or they are less reliant on training data.
• This could be isolated to reasoning tasks. For coding, ask any developer and they’ll all say that Claude 3.7 Sonnet doesn’t follow instructions well.
• Surprisingly GPT-4.5 does better than o1, and o3-mini which could be a testament to the fact that reasoning models can overthink. A lot.

Evaluate with Vellum

The benchmarks in this article tell a clear story—Claude 3.7 Sonnet is great for coding, GPT-4.5 holds its own in math, and reasoning models can sometimes "overthink" problems. But benchmarks alone don’t tell you how these models will actually perform in real-world workflows. Testing prompts in isolation is fine, but AI applications aren’t just about a single input-output pair. They involve sequences of decisions, multiple model calls, and real user interactions—and testing that end-to-end is where things get challenging.

Take reasoning tasks, for example. Our adaptive puzzle evaluations showed that Claude 3.7 Sonnet could handle modified problems better than other models, meaning it likely relies less on memorized training data. But how does that translate to a real product? If you're using an LLM for decision-making, retrieval-augmented generation (RAG), or AI agents, you need to test not just whether a single prompt works, but whether the entire workflow holds up under different conditions.

With Vellum, you can move beyond simple prompt testing and evaluate full AI workflows at scale. Using Vellum’s Evaluation Suite, we were able to:

• Run structured experiments with different reasoning tasks, coding challenges, and math problems, all within the same environment.
• Compare models side by side with LLM-as-a-judge scoring, ensuring objective evaluations beyond just human intuition.
• Identify failure patterns across workflows—not just individual prompts—helping refine deployment strategies before models hit production.

This kind of end-to-end evaluation is key. A model that does well on a reasoning benchmark might still fail when integrated into an AI agent or customer-facing tool. If you're relying on LLMs for real-world applications, you need a system-level view—not just a single prompt test. Vellum makes that possible.

Conclusion

GPT-4.5’s launch felt more like an incremental update than a major breakthrough. While it improves on hallucinations and conversational flow, the pricing raises big questions—it's 75x more expensive than GPT-4o, but the performance improvements don’t seem to match the cost. Meanwhile, Claude 3.7 Sonnet has emerged as the go-to model for coding, with better pricing, higher throughput, and strong reasoning capabilities.

Our evaluations confirmed these trends: Claude 3.7 Sonnet leads in coding and reasoning with new context, while GPT-4.5 holds its own in math but struggles to justify its price tag. The real takeaway? The landscape is shifting fast. If this was just a mid-cycle refresh, we’re likely on the verge of something much bigger from OpenAI.