Adding generative AI introduces model, data, hosting, evaluation, and latency decisions that ordinary applications do not have.
2
A chatbot is easy to build, but checking that its answers are correct requires techniques such as prompt engineering, guardrails, retrieval, fine-tuning, and repeated evaluation.
3
Agent costs can grow quickly, so teams should compare models, control parallel work, externalize prompts, and trace every model call.
Summary
Juan Peredo describes the practical problems he encountered while building generative AI applications over roughly a year and a half. AI coding tools can speed up development, but putting a model into a product adds decisions about model choice, retrieval, fine-tuning, hallucinations, evaluation, GPU hosting, and cost. He explains why correct chatbot answers are difficult to guarantee and compares prompt engineering, guardrails, retrieval augmented generation, and fine-tuning. He recommends evaluating models throughout development, keeping prompts outside application code, and using shared prompt repositories. Agents can call tools and complete multi-step tasks, but sequential model calls create latency and cost. Peredo shows how model selection can change a call center example from almost $300,000 per month to almost $50,000 per month. He ends with observability, using traces to find failures that ordinary debugging would miss.
AI coding tools speed up starts, but they do not build complete applications
Peredo rejects the claim that a single prompt will build an entire website in about a minute. Large language models provide code that gets developers started, while tools such as Codium and Cursor can work inside an IDE and help build applications faster. The right model depends on the task. A basic boilerplate website may work with almost any coding model, while a complex or newly released technology needs a model with stronger current knowledge. These tools help with development, but they also introduce more choices once the model becomes part of the application itself.
Adding a model creates a second layer of application decisions
A conventional application already has a frontend, backend, infrastructure, storage, CPU, and memory. An AI application adds decisions about which model to use, whether to fine-tune it, how to write prompts, whether retrieval augmented generation is needed, and how to detect hallucinations. Models change frequently, and performance on a benchmark does not prove that a model fits a particular application. Teams have to evaluate the model already in production alongside new models. Hosting also changes because GPU capacity, cloud versus on-premises deployment, and the cost of running the hardware become part of the design.
Model hosting choices affect experimentation and operating cost
For local exploration, Peredo recommends tools that let developers download and run models on their own machines, provided the hardware is powerful enough. Cloud deployment adds questions about the provider and GPU availability. He mentions Modal as a way to package Python code with decorators and deploy it through a CLI, with the platform handling containers and cloud deployment. SkyPilot can spread clusters and containers across clouds to reduce cost. Developers can also run models locally with Hugging Face Transformers. Hosting is therefore a product decision, since the chosen model and deployment method affect both the bill and the application.
A working chatbot still needs extensive answer validation
Peredo says building a chatbot can take an hour or two, while making sure it gives correct answers is much harder. He gives a failure from his own work: when asked what three times three is, one language model answered 33. Prompt engineering adds instructions to the user's question, but a model may still ignore them. A guardrail can use a second model as a classifier for unsafe questions or bad answers, though that adds another call, latency, and cost. Retrieval augmented generation supplies chunks from a vector database, but outdated source data produces outdated answers. Fine-tuning costs more, takes longer, and can reduce quality if the added data confuses the model.
Evaluation belongs at every stage of application development
Peredo recommends evaluating both models and the application throughout development. Ollama can help developers explore many models locally. Hugging Face provides open-source models and Spaces that developers can inspect and reuse. OpenRouter can run a prompt against a selected model and choose among providers offering that model, including a cheaper provider for testing. Peredo says he uses OpenRouter every day and had spent about $10 of an initial $50 balance after six months. For ongoing comparison, LangSmith records model runs and lets teams compare performance over time. He treats evaluation as a continuing activity rather than a final check before release.
Hard-coding prompts makes them harder to review and change. Peredo recommends externalizing them in a repository such as LangChain Hub. An education specialist who does not code can then edit prompts, run the model, inspect the output, and commit an approved version that flows into the application. External prompts also make model changes easier to handle. A new model can produce different results even when it belongs to the same model family, so developers need to test and adjust prompts without changing the surrounding application code. Peredo's conclusion is direct: prompts should not be encoded in application code.
Agents trade fixed application paths for tool-using model decisions
An agent lets a language model interact with the real world through tools, which are functions the model can call. Peredo gives the example of receiving a book, translating it into several languages, and creating audio. The assistant decides which tools to call and when to finish, rather than requiring every path to be hard-coded. This flexibility comes with latency. If an agent makes four or five model calls and each takes several seconds, the total process can take 12 to 15 seconds. Frameworks such as LangChain, LlamaIndex, and Lflow can support concurrent calls and branching. Running independent tasks in parallel can shorten the wait.
Model choice can change an agent's monthly bill by hundreds of thousands of dollars
Peredo calculates a call-center example with 3,000 calls each day and 15 tool calls per interaction. Using OpenAI o1 with 1,500 input tokens and 3,000 output tokens per model call produces an estimated $24 per call, about $9,270 per day, or almost $300,000 per month. Replacing it with Llama 3.3 70B changes the estimate to about 52 cents per call and almost $50,000 per month. He also compares GPU and CPU hosting costs, noting that GPUs can cost dollars per hour while ordinary cloud CPU instances can cost cents. Smaller models can reduce cost and latency when they can still complete the task.
Agent traces expose failures that ordinary debugging misses
Peredo describes agents as probabilistic systems with many unknowns, unlike a deterministic application that follows the same sequence each time. In one example, an agent that retrieved a user's information began failing because the input used lowercase 'john' instead of capitalized 'John'. The fix was a prompt change adding 'ignore case'. Externalized prompts made that change easy. Without traces, following a failure across several model calls would have been difficult even with print statements and a debugger. Tools such as LangSmith can record the order of model calls, metadata, inputs, outputs, and errors. Peredo says teams should build this observability because they will need it.
"There's no technique that will guarantee that the answer of your chatbot is going to be good."12:31
Who should watch
You are adding a language model to an existing product and need to account for hosting, hallucinations, evaluation, and changing model behavior.
You are building a chatbot or agent and want concrete ways to test answers, reduce latency, and trace failed calls.
You are estimating the cost of a model-powered service and need to compare provider pricing, GPU hosting, and smaller models.
