AI engineers use language models across internal and customer-facing products, with most applying them to several use cases.
2
Retrieval-augmented generation is the most common customization method, while fine-tuning is more widespread than Barr expected.
3
Evaluation is the most painful part of AI engineering, while agents are still early in production despite strong interest in adopting them.
Summary
Barr Yaron presents early findings from Amplify Partners' 2025 State of AI Engineering survey, based on 500 respondents. The respondents have varied job titles, and many experienced software engineers have only recently started working with AI. Most use language models for both internal and external applications, often across several use cases. Retrieval-augmented generation is the most common customization method, while fine-tuning is also widespread among researchers and research engineers. Teams update models and prompts frequently, but 31% have no prompt-management system. Text remains far ahead of image, audio, and video in workplace adoption. Agents have attracted strong interest, although fewer than 20% say they work well at work. Monitoring, human review, offline evaluations, and internal metrics are common, and evaluation is the most frequently cited source of pain. The survey also covers vector databases, disclosure by AI agents, model architecture, open and closed models, and AI-generated relationships.
AI engineering has a broad community and many newcomers
The survey included 500 respondents, with engineers forming the largest group, but the audience had many different job titles. Barr says titles are currently unreliable because people with different labels often do the same work. Interest in the term AI engineering rose after ChatGPT launched in late 2022. Many experienced software engineers are still new to AI: among those with more than 10 years of software experience, nearly half have worked with AI for three years or less, and one in 10 started within the past year.
Teams use language models across several kinds of work
More than half of respondents use LLMs for both internal and external use cases. Code generation and code intelligence, along with writing assistance and content generation, are the leading use cases. Barr describes the main pattern as heterogeneity. Of the people using LLMs, 94% use them for at least two use cases and 82% use them for at least three. For customer-facing products, three of the five most-used models and half of the top 10 came from OpenAI.
RAG is common, and fine-tuning is more widespread than expected
Besides few-shot learning, retrieval-augmented generation is the most popular way respondents customize their systems, with 70% reporting its use. Barr was surprised by how much fine-tuning is happening across the sample. Researchers and research engineers fine-tune by far the most. Among people who fine-tune, 40% mention LoRA or QLoRA, while other approaches include DPO, reinforcement fine-tuning, and supervised fine-tuning. Respondents also report hybrid methods.
Teams change models and prompts often, but prompt management is inconsistent
More than half of respondents update their models at least monthly, including 17% who update them weekly. Prompt changes happen even more often: 70% update prompts at least monthly, and one in 10 does so daily. Despite that activity, 31% have no way to manage their prompts. Barr connects the pressure to frequent model releases and the possibility that a new result or blog post makes an existing prompt feel inadequate.
Text adoption is far ahead of image, audio, and video
Image, video, and audio models lag text models by significant margins in workplace use and production traction. Barr calls this the multimodal production gap. Among respondents not currently using a modality, audio has the strongest future intent: 37% of non-users plan to adopt it eventually. Barr expects adoption to rise as models become better and easier to access.
Agents attract interest even though few teams say they work well
Barr defines an AI agent as a system where an LLM controls the core decision-making or workflow. Although 80% of respondents say LLMs work well at work, fewer than 20% say the same about agents. Fewer than one in 10 say they will never use agents, so most respondents at least plan to adopt them. Agents already in production usually have write access with a human in the loop, while some can take actions independently.
Most respondents use several ways to monitor AI systems. Sixty percent use standard observability, more than half rely on offline evaluations, and teams also collect user data and use benchmarks. Human review remains the most popular way to evaluate model and system quality. For model-usage monitoring, internal metrics are the usual choice. When asked about the most painful part of AI engineering, respondents put evaluation at the top.
Specialized vector databases are common in production systems
Sixty-five percent of respondents use a dedicated vector database to store and retrieve context. Barr says this suggests that specialized vector databases provide enough value over general-purpose databases with vector extensions for many applications. Among dedicated-vector-database users, 35% primarily self-host and 30% primarily use a third-party provider.
"And finally, we asked folks, what is the number one most painful thing about AI engineering today? And evaluation topped that list."11:14
Who should watch
You are deciding where your team should use LLMs, RAG, fine-tuning, or vector databases and want comparison data from other practitioners.
Your team is building agents and needs a view of current production adoption, human approval patterns, monitoring, and tool permissions.
You are struggling with evaluations or prompt changes and want to see how widespread those problems are among AI engineers.
