#INSIGHTS
From idea to implementation:what does 2025 hold for AI?
Like the year before it, 2024 was defined by AI.
Notably, AI moved decisively from theory to practice: hundreds of AI-focused companies emerged and countless established companies launched AI-powered features. AI became the commanding element of product roadmaps, venture trends, and value propositions.
We believe AI’s role will become even more pivotal in 2025.
As models improve and tooling matures, it’s becoming easier for builders to use this new technology. But to truly understand what’s ahead, let’s take a look at the immediate past.
What did builders achieve (and not achieve) with AI in 2024, and what do these learnings signify for the future?
To answer these questions, we surveyed over 1,250+ AI developers & builders.
Notably, AI moved decisively from theory to practice: hundreds of AI-focused companies emerged and countless established companies launched AI-powered features. AI became the commanding element of product roadmaps, venture trends, and value propositions.
We believe AI’s role will become even more pivotal in 2025.
As models improve and tooling matures, it’s becoming easier for builders to use this new technology. But to truly understand what’s ahead, let’s take a look at the immediate past.
What did builders achieve (and not achieve) with AI in 2024, and what do these learnings signify for the future?
To answer these questions, we surveyed over 1,250+ AI developers & builders.
#STAGES
Let’s take a heartbeat:Where Are Companies in the AI Development Journey?
There is no doubt that AI is flooding recent product launches. However, where is the average company with their AI development lifecycles? In production? Beta-testing? Still In ideation? Turns out, the results are quite mixed:
Where are you in the AI development process?
25.1%
Deployed in production
25%
Still building our strategy (pre-building)
21%
Building our Proof of Concept (PoC)
14.1%
Beta testing with users
7.9%
Talking to users and gathering requirements
7%
Evaluating our PoC
25.1% of businesses have deployed an AI applications (both internal and external) into production. This includes both small companies (23% in production) and big companies (29% in production). Meanwhile, a similar number are building a proof of concept, and a few more are beta-testing.
We have a few theories why the results are so split. For a few businesses, there may be no good AI use case available, keeping them stuck in a pre-building stage. For others, implementing AI was straight-forward, moving features to production within the year.
But for most, we think what’s holding them back isn’t AI, talent, or vision—but tooling.
As future data will indicate, 2025 might be the year of AI tooling.
We have a few theories why the results are so split. For a few businesses, there may be no good AI use case available, keeping them stuck in a pre-building stage. For others, implementing AI was straight-forward, moving features to production within the year.
But for most, we think what’s holding them back isn’t AI, talent, or vision—but tooling.
As future data will indicate, 2025 might be the year of AI tooling.
#APPS
What are companies building with AI?
Understanding what companies are building with AI is fascinating for a few reasons.
First, 2024 was highlighted by other AI modalities (think images, audio, and video) hitting the market. Additionally, developers had more access to community knowledge and documentation on how to successfully deploy AI. Competing model providers shipped models that are better at reasoning tasks, making them useful for more complex use-cases.
These improved assets beg the question: what was actually built?
Let’s dive into what AI is actually being used for:
First, 2024 was highlighted by other AI modalities (think images, audio, and video) hitting the market. Additionally, developers had more access to community knowledge and documentation on how to successfully deploy AI. Competing model providers shipped models that are better at reasoning tasks, making them useful for more complex use-cases.
These improved assets beg the question: what was actually built?
Let’s dive into what AI is actually being used for:
What types of AI applications are you building today?
59.7%
Document parsing and analysis
51.4%
Customer service/chatbots
43.8%
Analytics with natural language
41.9%
Content generation
25.9%
Recommendation systems
25.3%
Code generation and automation
23.7%
Research automation
15%
Compliance automation
The leader of the pack—document parsing and analysis—is hardly surprising.
Given that early models available in 2024 were adept for the task, companies with parsing requirements have had a head-start on shipping AI features. Model providers introduced new developer features, such as structured outputs and prompt caching, making it easier for teams to build more reliable customer service chatbots. As a result, this use case remains one of the top three apps being built in 2024.
Finally, with context windows of 128k tokens becoming the new standard and smaller models nearing the performance of top models, 'analytics with natural language' has become easier and more cost-effective, securing its place as the third most-built use case for 2024.
What’s even more intriguing is the impact on analytics and querying, where procedural tasks have likely benefited from recent models like O1. The same effect may extend to code generation, research automation, and compliance automation—fields that have a low tolerance for error.
Given that early models available in 2024 were adept for the task, companies with parsing requirements have had a head-start on shipping AI features. Model providers introduced new developer features, such as structured outputs and prompt caching, making it easier for teams to build more reliable customer service chatbots. As a result, this use case remains one of the top three apps being built in 2024.
Finally, with context windows of 128k tokens becoming the new standard and smaller models nearing the performance of top models, 'analytics with natural language' has become easier and more cost-effective, securing its place as the third most-built use case for 2024.
What’s even more intriguing is the impact on analytics and querying, where procedural tasks have likely benefited from recent models like O1. The same effect may extend to code generation, research automation, and compliance automation—fields that have a low tolerance for error.
#MODALITIES
Let’s talk about modalities
A big question for 2024 was if GenAI has moved beyond text to embrace multimodal use. The numbers present a resounding yes:
What modalities is your application using?
I don’t know
1.6%
Video
16.3%
Audio
27.7%
Files (e.g., PDFs, Word Docs)
62.1%
Images
49.8%
Text
93.8%
While text and text-like files remain the frontrunners, images have nearly 50% adoption, followed by audio and video at 27.6% and 16.3%, respectively. With LLMs now handling voice and image inputs, along with generative models for creating both we're not surprised to see a surge in multi-modal adoption.
#MODELS
What hosting providers & models are companies using
Last year, OpenAI was the undisputed leader in the AI race, and according to the data - it continues to lead the pack. With good reason, their models continue to top many benchmarks—just take a look at our LLM leaderboard here.
In 2024, companies had more optionality how they wanted to access a model—either directly from the model publisher (e.g. GPT 4o via OpenAI) or through another platform (e.g. Azure for OpenAI). The results from the survey tell an interesting story:
In 2024, companies had more optionality how they wanted to access a model—either directly from the model publisher (e.g. GPT 4o via OpenAI) or through another platform (e.g. Azure for OpenAI). The results from the survey tell an interesting story:
Which API providers are you using?
OpenAI
63.3%
Microsoft / Azure
33.8%
Anthropic
32.3%
AWS / Bedrock
25.6%
GCP / Vertex
15.2%
Groq
10.7%
I don’t know
6.2%
Together AI
4.2%
None
3.8%
The usual suspects—OpenAI, Azure, Anthropic, Bedrock etc.—are leading. Notable mentions also went to model providers hosting open-source models (e.g. Llama 3.2 70b) such as Groq, Fireworks AI, and Together AI.
Generally speaking, larger companies tend to use AI via cloud solutions; hardly surprising given that the rest of their stack is located there.
Generally speaking, larger companies tend to use AI via cloud solutions; hardly surprising given that the rest of their stack is located there.
Azure adoption by company size
25%
1-10 employees
23%
11-50 employees
28%
51-200 employees
41%
201-500 employees
43%
501-1,000 employees
44%
1,001-5,000 employees
48%
5,000+ employees
#TOOLING
How are companies enabling AI development
Beyond AI providers and models, we were curious to learn how companies are empowering their developers to build with AI. Some may invest in building internal tools from scratch, while others leverage pre-existing frameworks or platforms.
Let’s take a closer look at how companies are actually developing with AI:
Let’s take a closer look at how companies are actually developing with AI:
How do you develop your AI products?
Internal tooling
52.2%
Third-party AI dev platform or framework
29.9%
Defining prompts and orchestration logic without any tooling
17.9%
Notably, most teams are leveraging internal tooling over third-party tooling. And some teams (just under 20%!) are using no tooling at all—which is both impressive as a fact and as an implication: that most teams have actually embraced tooling in 2024.
Why might tooling be necessary? Because building with AI comes with challenges, a worthy topic in its own right, which leads us to our next section.
Why might tooling be necessary? Because building with AI comes with challenges, a worthy topic in its own right, which leads us to our next section.
#CHALLENGES
What were the top challenges of working with AI?
Ah, challenges—let’s dive into what caused us the most headaches in 2024. The results were illuminating.
What challenges have you faced when building AI products?
57.4%
Managing Al "hallucinations" and prompts
42.5%
Prioritizing use cases with the most impact
38%
Lack of technical expertise
33.4%
Model speed and performance
32.5%
Data access/security
21.2%
Securing buy-in from key stakeholders
Hallucinations—the boogeyman of large learning models from the start—remains a chief concern for most companies, with over half citing it as an issue. It’s followed by more organizational issues, such as prioritizing user cases and needing technical expertise.
As model accuracy improves, hallucinations might be replaced by more procedural issues of how AI is leveraged. Accordingly, that could inspire more AI framework solutions that expedite the discovery process and make it easier to launch applications into production.
To that end, we were curious to see which frameworks and platforms companies are using to build with AI. These frameworks and platforms— Vellum, Langchain, Llama Index, N8N, Langfuse, Flowise, CrewAI, and Voiceflow—were among the most mentioned in the survey.
As model accuracy improves, hallucinations might be replaced by more procedural issues of how AI is leveraged. Accordingly, that could inspire more AI framework solutions that expedite the discovery process and make it easier to launch applications into production.
To that end, we were curious to see which frameworks and platforms companies are using to build with AI. These frameworks and platforms— Vellum, Langchain, Llama Index, N8N, Langfuse, Flowise, CrewAI, and Voiceflow—were among the most mentioned in the survey.
AI Development Tooling
.png)
#EVALS
Do developers actually perform evaluations?
One way to tackle mistakes—including hallucinations in certain situations—is to perform evaluations, where specific metrics are used to test the correctness of a generated response.
But, did developers leverage evaluations in 2024?
But, did developers leverage evaluations in 2024?
Do you perform evaluations on your AI or applications?
Yes
57.4%
Planning to
30.9%
No
11.7%
Generally speaking, yes! Developers are using evaluations—at least, half of them. Moreover, less than 12% of developers have no interest in evaluations; the others plan to eventually implement them.
This brings up the next question: how do developers conduct evaluations?
This brings up the next question: how do developers conduct evaluations?
Which methods do you use for evaluation?
75.6%
Manual testing and reviews
47.9%
User feedbck sessions
38%
Automated evalution tools
27%
A/B testing
21.8%
Open-source eval framework
10.5%
Third-party evaluation platforms
1.7%
Other
This is one of the bigger surprises: despite the higher investment in toolings in 2024, most users are still using manual checks for performing evaluations. Automated evaluation tools, meanwhile, only had 38% market penetration. Both open-source evaluation frameworks and third-party evaluation platforms have yet to see high adoption.
We expect this trend to shift over time. While 2024 may have been the year companies took AI more seriously, 2025 is likely to be the year of AI tooling—particularly tools that enable developers to evaluate complex AI systems.
We expect this trend to shift over time. While 2024 may have been the year companies took AI more seriously, 2025 is likely to be the year of AI tooling—particularly tools that enable developers to evaluate complex AI systems.
#MONITORING
AI models can go rogue.how are developers monitoring outputs?
Evaluating your AI product during development is crucial, but the process doesn't stop once it's deployed to production. After launch, your users will likely encounter unexpected outputs from your GenAI—some that weren’t accounted for in the training eval set.
In 2024, many companies became aware about the need to monitor their AI outputs in prod and capture user-feedback to improve the performance, as we can see in the chart below:
In 2024, many companies became aware about the need to monitor their AI outputs in prod and capture user-feedback to improve the performance, as we can see in the chart below:
Do you monitor your AI models in production?
Yes
52.7%
Not applicable (not in production yet)
30.9%
No
15.2%
For those monitoring their AI models in production, we wanted to explore the different approaches used to set up their monitoring layer. The majority (55.3%) rely on in-house monitoring solutions, suggesting a preference for custom-built, tailored monitoring setups.
Third-party monitoring tools follow at 19.4%, highlighting that many companies are opting for external solutions, while cloud provider services and open-source tools are used by 13.6% and 9%, respectively.
Third-party monitoring tools follow at 19.4%, highlighting that many companies are opting for external solutions, while cloud provider services and open-source tools are used by 13.6% and 9%, respectively.
How do you monitor/observe your AI models?
55.3%
In-house monitoring solutions
19.4%
Third-party monitoring tools
13.6%
Cloud provider services
9%
Open-source monitoring tools
2.6%
Other
#ROLES
Who is involved in the AI development process?
AI development represents a new paradigm, involving multiple parts of the organization in the process. Unlike traditional software development, there's a greater need for cross-functional collaboration due to the unpredictable nature of GenAI models.
To ensure your AI performs well in production, you'll need to collaborate with non-technical teams—but which ones?
To ensure your AI performs well in production, you'll need to collaborate with non-technical teams—but which ones?
Who participates in the AI development process on your team?
Other
3%
Design
38.2%
Product
55.4%
Leadership / execs
60.8%
Subject Matter Experts (SMEs)
57.5%
Engineering
82.3%
Our data shows that product development teams (engineering, product, and design), leadership, and subject matter experts (SMEs) are all key players in AI product development.
This is largely driven by the use of natural language for writing prompts instead of code, as well as the critical role of SMEs in ensuring the AI meets specific requirements.
Building with GenAI is undeniably a collaborative effort. If you want your AI to perform reliably and truly deliver value to customers, there’s no way around it—you must work closely with product teams and subject matter experts (SMEs).
This is largely driven by the use of natural language for writing prompts instead of code, as well as the critical role of SMEs in ensuring the AI meets specific requirements.
Building with GenAI is undeniably a collaborative effort. If you want your AI to perform reliably and truly deliver value to customers, there’s no way around it—you must work closely with product teams and subject matter experts (SMEs).
#ARCHITECTURE
Prompting & RAG Dominate,Fine-Tuned Models Lag
Improvements from foundation model providers and the open source movement is resulting in AI costs reducing over time (for comparable performance).
This is resulting in lower uptake of fine-tuning than originally expected, development teams still predominantly use prompts to build their AI applications.
This is resulting in lower uptake of fine-tuning than originally expected, development teams still predominantly use prompts to build their AI applications.
Is your company using fine-tuned models?
No
53.5%
I don't know
14%
Yes
32.5%
When it comes to knowledge retrieval, RAG remains the go-to solution for helping companies—especially enterprises—ground their AI and deliver more factually accurate answers.
Are you using vector databases?
59.7%
Using vector databases
21.83%
I don't know
19.46%
Not using vector databases
Vector Databases
#IMPACT
Measuring AI impact may take time
With GenAI, product development teams can tackle old problems with new solutions. What was once impossible before the AI boom is now accessible to everyone—today anyone can develop with AI.
But what’s the biggest impact of all these AI initiatives?
But what’s the biggest impact of all these AI initiatives?
What is the biggest impact from your AI product(s)?
31.6%
Competitor advantage
27.1
Big cost and time savings
24.2%
No measurable impact yet
12.6%
Higher user adoption rates
4.5%
Other
Nothing surprising here—competitive advantage and significant cost and time savings lead the pack at 31.6% and 27.1%, respectively. What’s interesting, though, is that nearly a quarter of respondents said there’s been no measurable impact yet. Are they just starting to innovate with AI, or are these investments difficult to measure? We’ll have to wait and see.
One thing we do know for sure: the AI development train isn’t slowing down, and companies are planning to dive even deeper in 2025. Here’s what they have in store!
One thing we do know for sure: the AI development train isn’t slowing down, and companies are planning to dive even deeper in 2025. Here’s what they have in store!
#PLANS
What’s in store for 2025?
We asked our respondents what they were planning to build with AI in 2025.
This question gives us a peak into where companies see their AI investments going; in a year, we’ll be able to learn if these predictions actually amounted to reality.
This question gives us a peak into where companies see their AI investments going; in a year, we’ll be able to learn if these predictions actually amounted to reality.
What are your plans for 2025?
58.8%
Build more customer-facing use cases
55.2%
Build more complex workflows (agentic)
41.9%
Upskill your team
37.9%
Build your org's own AI for internal use cases
33%
Use third-party AI tools to improve internal operations
17.1%
Hire more AI developers
The trend? Companies are preparing to tackle more customer-facing challenges with GenAI, driving even greater cost and time savings in the year ahead, just as they did before.
But what's on everyone’s mind?
Agents.
RAG has already paved the way for how models can leverage external tools to ground responses and provide factually accurate answers. Now, with agentic workflows, we’re expanding the AI’s toolset to help it make smarter decisions and generate better outputs.
Building agentic workflows will unlock new opportunities, especially for companies dealing with data-heavy operations.
Additionally, upskilling teams and developing internal AI processes are also high priorities, reflecting a strong need for companies to invest in their talent and prepare them for AI-driven roles in the future.
But what's on everyone’s mind?
Agents.
RAG has already paved the way for how models can leverage external tools to ground responses and provide factually accurate answers. Now, with agentic workflows, we’re expanding the AI’s toolset to help it make smarter decisions and generate better outputs.
Building agentic workflows will unlock new opportunities, especially for companies dealing with data-heavy operations.
Additionally, upskilling teams and developing internal AI processes are also high priorities, reflecting a strong need for companies to invest in their talent and prepare them for AI-driven roles in the future.
#INSIGHTS
Conclusion
As we wrap up our review of AI development in 2024, it’s clear that AI has firmly established itself in production applications, transforming industries and workflows.
The use cases are diverse, ranging from document parsing to customer service chatbots and analysis using natural language.
This year’s advancements highlight the growing need for a multi-disciplinary approach, bringing together developers, product teams, leadership, and SMEs. As AI becomes integral to development processes, effective collaboration will be key to ensuring it stays powerful and aligned with organizational goals.
The use cases are diverse, ranging from document parsing to customer service chatbots and analysis using natural language.
This year’s advancements highlight the growing need for a multi-disciplinary approach, bringing together developers, product teams, leadership, and SMEs. As AI becomes integral to development processes, effective collaboration will be key to ensuring it stays powerful and aligned with organizational goals.
Looking ahead to 2025, the focus will shift to creating more customer-facing products and developing complex agentic workflows. As these systems grow more intricate, the challenges will become even more pronounced, driving a greater reliance on AI tooling to tackle issues like hallucinations, evaluations, and observability.
Enhanced tooling will drive broader adoption, unlocking new possibilities for AI use cases.
2024 has been a transformative year, laying the foundation for future advancements that will continue to shape the AI landscape in the years to come.
Enhanced tooling will drive broader adoption, unlocking new possibilities for AI use cases.
2024 has been a transformative year, laying the foundation for future advancements that will continue to shape the AI landscape in the years to come.
Methodology
The insights in this report were gathered from a public survey of 1,285 people in December 2024.
Top 5 industries
- Technology: 46%
- Healthcare: 10%
- Finance: 10%
- Retail: 4%
- Legal: 2%
Top 5 teams
- Engineering: 32%
- Management: 21%
- Data Science: 16%
- Product Management: 10%
- Subject Matter Expert: 10%
Top 5 geographies
- North America: 55%
- Europe: 29%
- Asia: 8%
- South America: 5%
- Australia: 3%
Breakdown by company size
- 1-50 Employees: 48%
- 51-500 employees: 20%
- 500+ employees: 32%
Acknowledgements
This report wouldn’t have been possible without the support of our partners Llama Index, Weaviate, and Fireworks AI, who generously amplified the survey within their developer communities. A special thank you also goes to Neuron AI and TLDR AI, our media partners, for helping us spread the word and reach a wider audience.
Lastly, a huge thank you to everyone who completed the survey and contributed to this report.
Lastly, a huge thank you to everyone who completed the survey and contributed to this report.
