# How to Become an AI Engineer from a Fullstack Background Reid Mayo, Rema AI | AI Engineer Summit 2023 | 10:19 Source: https://www.youtube.com/watch?v=a0NwFm5NxGo Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks. Page: https://aietalks.com/talks/how-to-become-an-ai-engineer-from-a-fullstack-background Published: 2024-02-02 Tags: agents, coding-agents, evals, fine-tuning, prompt-engineering ## TL;DR - Full-stack engineers can build useful AI systems with foundation models without first becoming traditional machine learning specialists. - A practical learning path should begin with fundamentals, prompt engineering, OpenAI, LangChain, evaluations, and fine-tuning. - Engineers should prototype with hosted models, collect usage data, then consider smaller open-source models when cost, latency, privacy, or control become problems. ## Summary Reid Mayo presents a syllabus for full-stack engineers who want to move into AI engineering without prior machine learning experience. The course starts with an overview of large language models, then covers prompt engineering, OpenAI, LangChain, evaluations, and fine-tuning. Mayo treats fundamentals as the base for composing more sophisticated systems, and recommends using ChatGPT as a Socratic tutor while studying. He compares evaluations to software tests and argues that teams need them before changing black-box models. The practical path is to prototype quickly with OpenAI models, gather usage and training data, and later fine-tune a smaller open-source model if production creates cost, latency, rate-limit, privacy, or control problems. After the boot camp material, he points learners toward fast.ai and Hugging Face for deeper theory and training models from scratch. ## Key ideas ### Foundation models let full-stack engineers build useful AI systems [00:48](https://www.youtube.com/watch?v=a0NwFm5NxGo&t=48s) Mayo assumes the learner already knows how to build modern applications across infrastructure, databases, back ends, and clients, but has no AI or machine learning background. He points to Swyx's essay "The Rise of the AI Engineer" as the motivation for the course. The change he describes is that full-stack engineers can now deploy useful AI solutions by using foundation models. In the past, similar systems would have required substantial traditional machine learning experience and expensive upfront data collection. The syllabus therefore starts from zero on AI while building on an engineer's existing software background. ### Learning should focus on fundamentals and use ChatGPT as a tutor [01:25](https://www.youtube.com/watch?v=a0NwFm5NxGo&t=85s) The course applies learning advice from The Art of Learning. Mayo recommends limiting distractions and avoiding low-information material because the syllabus has limited time. He puts heavy emphasis on fundamentals, since understanding the building blocks makes it possible to compose more sophisticated AI products. He also recommends using ChatGPT as a private tutor. When a new concept appears, the learner should use a Socratic method and keep asking questions until the concept is understood thoroughly. Mayo notes that many concepts predate ChatGPT's January 2022 knowledge cutoff. ### Prompt engineering improves model output and prepares engineers for fine-tuning [02:57](https://www.youtube.com/watch?v=a0NwFm5NxGo&t=177s) Mayo says prompt engineering can feel strange to programmers who are used to symbolic systems and code logic. Language models are neural architectures, so different techniques are needed to get better results. He argues that prompt engineering objectively improves the output of language models. He also warns against skipping directly to fine-tuning. Fine-tuning quality often improves when training data uses the best-performing prompts. The recommended study path includes an overview from Prompt Engineering Guide founder Elvis Cavalea, the guide itself, its graduate job classification case study, and OpenAI's learning materials. ### OpenAI is the fastest way to learn what current AI systems can do [04:30](https://www.youtube.com/watch?v=a0NwFm5NxGo&t=270s) Mayo recommends reading the OpenAI documentation and API reference cover to cover. He describes OpenAI as strong in two areas: providing state-of-the-art models and making them accessible. Learning the APIs lets engineers understand what is possible and start building quickly. He still says the platform has practical limitations, which the syllabus addresses later. The OpenAI cookbook is useful for hands-on examples, but Mayo advises moving through it quickly and returning to it when a project needs a particular example. ### LangChain teaches how to assemble AI applications from modular parts [05:28](https://www.youtube.com/watch?v=a0NwFm5NxGo&t=328s) Mayo describes LangChain as an application framework for putting AI components together in an organized way. It connects models, prompts, short- and long-term memory, retrieval-augmented generation, conversations, and other parts of an AI system. The framework can also integrate components that it does not support directly, including proprietary components. Mayo calls LangChain the glue layer for much of the AI ecosystem, so studying its documentation and codebase teaches practical AI engineering patterns. He recommends reading both the Python and JavaScript documentation, inspecting the GitHub code, and following tutorials from Mayo Oshin. ### AI evaluations are the tests that make model changes measurable [07:21](https://www.youtube.com/watch?v=a0NwFm5NxGo&t=441s) For someone coming from full-stack engineering, Mayo compares evaluations to software tests. Before fine-tuning a black-box model, engineers need a repeatable way to assess changes. Otherwise, they cannot tell whether a modification improves the system or introduces a regression. Mayo recommends OpenAI's cookbook examples for writing evaluations and its evaluation framework for creating custom evaluation suites. He also warns that AI outputs often require creative evaluation methods, because ordinary exact-match tests may not capture whether a response is useful. ### Hosted models are a starting point, while smaller open models can reduce production costs [08:18](https://www.youtube.com/watch?v=a0NwFm5NxGo&t=498s) Mayo recommends completing OpenAI's fine-tuning cookbook before studying fine-tuning for open-source models. Hosted models can become expensive and can create latency or rate-limit problems in production. Privacy and control can also make dependence on one provider unsuitable. His suggested pattern is to prototype and ship with OpenAI, gather usage and training data, then test whether a smaller, cheaper open-source model can match or beat the hosted model on the target task. He cites an OpenPipe case study where a fine-tuned Llama 2 model cost $19 for a task compared with about $244,000 for OpenAI's state-of-the-art model. ### Advanced study comes after deploying the basic skills [09:39](https://www.youtube.com/watch?v=a0NwFm5NxGo&t=579s) Mayo recommends using the boot camp material first and deploying those skills in real projects before moving to advanced study. For deeper machine learning knowledge, he points to fast.ai's practical deep learning course and Hugging Face's natural language processing course and documentation. These materials cover deep learning theory and further fine-tuning. They also teach learners how to train models from scratch. The intended sequence is practical application first, followed by deeper theory when the learner is ready to extend beyond the basics. ## Notable quotes - "Full-stack Engineers can now deploy a wide variety of legitimately useful AI solutions by leveraging new foundational models." (01:06) - "The bottom line is that prompt engineering objectively increases the quality of neural architectures output such as language models." (03:16) - "Coming from a full stack background evals are basically your software tests." (07:21) - "An efficient pattern is to prototype and ship a solution quickly using OpenAI's models, start gathering usage and training data, then if the solution needs to start scaling see if you can fine-tune a smaller and cheaper open source model." (08:35) ## Tools & references mentioned - Swyx - The Rise of the AI Engineer - The Art of Learning - ChatGPT - Cohere - Transformer architecture - Prompt Engineering Guide - Elvis Cavalea - OpenAI - OpenAI Cookbook - LangChain - Mayo Oshin - OpenPipe - Meta Llama 2 - fast.ai - Hugging Face ## Who should watch - You are a full-stack engineer who can build applications but has little or no background in AI or machine learning. - You want a structured syllabus instead of collecting disconnected tutorials about language models. - You are choosing between hosted models and open-source models, and need a practical path through evaluation, fine-tuning, cost, latency, privacy, and control concerns. ## Related talks - [From Software Developer to AI Engineer](https://aietalks.com/talks/from-software-developer-to-ai-engineer) (Antje Barth, AWS, 19:48) - [The 1,000x AI Engineer](https://aietalks.com/talks/the-1-000x-ai-engineer) (Swyx, AI Engineer Summit, latent.space, smol.ai, 09:27) - [Building Applications with AI Agents](https://aietalks.com/talks/building-applications-with-ai-agents) (Michael Albada, Microsoft, 15:50) - [AI Engineering 201: The Rest of the Owl](https://aietalks.com/talks/ai-engineering-201-the-rest-of-the-owl) (Charles Frye, Full Stack LLM Bootcamp, 56:57) - [How to Build AI Agents that Actually Work](https://aietalks.com/talks/how-to-build-ai-agents-that-actually-work) (Patrick Dougherty, Rosco, 17:44)