# AI Engineering 101

Noah Hein, Latent Space University | AI Engineer Summit 2023 | 3:02:24

Source: https://www.youtube.com/watch?v=C0ZUdFg-iTo
Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks.
Page: https://aietalks.com/talks/ai-engineering-101
Published: 2023-11-06
Tags: code-generation, embeddings, multimodal, rag

## TL;DR
- AI engineering basics start with calling language-model APIs and connecting their responses to a usable product.
- Embeddings, chunking, and retrieval let an application provide a model with relevant information without sending an entire data set in every prompt.
- Code generation, image generation, and speech-to-text extend a text chatbot into a broader AI application.

## Summary
Noah Hein teaches a practical introduction to AI engineering through a Telegram bot. Participants first configure Python, Telegram, BotFather, and API keys, then connect a bot to GPT-3.5 Turbo. The workshop explains how message arrays provide short-term conversational memory and why that memory eventually needs retrieval and context management. Hein then builds a retrieval-augmented system over scraped MDN documentation, covering tokens, embeddings, cosine distance, chunking, and prompt construction. The later projects use GPT-4 for few-shot code generation, DALL-E for images, and Whisper for speech-to-text. The examples are intentionally small, but they expose the implementation details behind common AI products. Hein is also direct about the limits: models can hallucinate, image generators struggle with hands, counting, and text, and generated code or images still need checking.

## Key ideas
### AI engineering begins with practical building blocks that teams already expect
[01:45](https://www.youtube.com/watch?v=C0ZUdFg-iTo&t=105s)
Hein defines the course as a starting point for people who are new to AI engineering. The selection criterion is knowing how to build things that are already known to work, or knowing where to find the information when a task is unfamiliar. The workshop is a sampler rather than a path to expertise. It covers five areas through APIs, then leaves participants with projects they can extend at home. The opening bot gives the work a concrete target: combine several AI API calls into one Telegram application that people can use and share.

### A Telegram bot makes the API workflow concrete
[10:48](https://www.youtube.com/watch?v=C0ZUdFg-iTo&t=648s)
The first project connects Telegram to OpenAI. Participants create a bot with BotFather, receive a Telegram HTTP API key, clone the GitHub repository, and put the bot token and OpenAI key into environment variables. Hein explains that Telegram handlers map incoming events to Python functions. A command handler catches slash commands such as /start, while the application polls Telegram for new messages. The initial bot only answers /start, which gives the group a working baseline before adding chat completion.

### A message array gives the chatbot its conversational memory
[32:17](https://www.youtube.com/watch?v=C0ZUdFg-iTo&t=1937s)
The chat implementation stores each user message and assistant response in a messages array. The system message says the model is a helpful assistant, then user and assistant messages alternate. The array is sent to GPT-3.5 Turbo with each request, so a follow-up such as asking for Simon Cowell's net worth can refer to the earlier answer. Hein stresses that the model does not remember anything outside this array. Restarting the server clears it. He also explains that this approach eventually hits the model's context limit, where the application needs a different memory strategy.

### Tokens and embeddings turn text into data the model can search
[46:48](https://www.youtube.com/watch?v=C0ZUdFg-iTo&t=2808s)
Hein describes tokens as the atomic units used by language models and says API charges depend on how many tokens an application consumes. A token is roughly four characters of English text, although tokenizers behave differently across languages and models. Embeddings are lists of floating-point numbers that represent semantic meaning. Words such as dogs and cats have nearby vectors, while unrelated concepts are farther apart. The workshop uses OpenAI's tokenizer and embedding tools so participants can inspect tokenization and prepare documentation for semantic search.

### Chunking keeps long documents within model limits
[1:03:24](https://www.youtube.com/watch?v=C0ZUdFg-iTo&t=3804s)
The embedding example scrapes MDN documentation, cleans the text, preserves source paths, counts tokens, and splits large pages into smaller chunks. Hein uses LangChain's RecursiveCharacterTextSplitter and a chunk size of 1,000 for the workshop data. Splitting matters because embedding models and language models have token limits. It also affects answer quality. A hard split in the middle of a sentence can remove meaning, while overlap gives neighboring chunks some shared context. Hein does not claim there is one best setting. The useful choice depends on the material, such as documentation, Wikipedia, or a novel.

### Retrieval-augmented generation filters context before the model answers
[1:03:41](https://www.youtube.com/watch?v=C0ZUdFg-iTo&t=3821s)
The question-answering project embeds the user's question, compares it with document embeddings using cosine distance, sorts the results, and keeps relevant material until a context limit is reached. The selected text becomes part of a prompt that says to answer from the supplied context and say 'I don't know' when the context is insufficient. This lets the application search a large documentation set without placing all of it into every request. Hein starts with a NumPy array because it runs locally and costs nothing, then mentions vector stores such as Pinecone and PGVector as options when performance becomes a problem.

### Few-shot prompts make code generation fit a project's conventions
[2:00:23](https://www.youtube.com/watch?v=C0ZUdFg-iTo&t=7223s)
The code-generation project upgrades from GPT-3.5 Turbo to GPT-4 and gives the model example input-output pairs. The examples establish conventions such as JSX and inline styles, so a request for a red box and blue button follows the demonstrated format. Hein says code is easier for models to predict than creative writing because it has strict, repeatable rules. He also describes using generated code to fill gaps in a developer's experience, including a Rails application he had not worked with before. The model can produce boilerplate, but the examples still need review.

### Image and speech projects expose both the reach and limits of AI APIs
[2:24:56](https://www.youtube.com/watch?v=C0ZUdFg-iTo&t=8696s)
The image project sends a prompt to DALL-E, downloads the returned image URL, and sends the image back through Telegram. Hein shows why prompting matters, while also naming weaknesses such as hands, counting people, and rendering text. He then adds Whisper speech-to-text. Telegram voice messages are downloaded, sent to the Whisper API, and returned as text. Whisper can also run locally, including through whisper.cpp, and WhisperX can add diarization to identify different speakers. These projects turn the original text bot into a multimodal application with relatively small additions.

## Notable quotes
- "You're not going to be an expert today. You will get a sampler of what we think is important and you can go home and go deeper on each of these topics." (02:45)
- "If it's not in that messages array, the LLM has no idea that it happened." (34:28)
- "We don't just give it the context of absolutely everything and ask it to filter through that. We do the filtering on our own." (1:48:08)
- "Code generation is very fast, it's very cheap, and is generally deterministic." (2:03:41)
- "If you have a podcast between me and my amazing helper Justin, diarization is the act of breaking down that MP3 file and labeling it, 'this is Noah talking' and 'this is Justin talking'." (2:25:20)

## Tools & references mentioned
- Latent Space University
- Python
- Telegram
- BotFather
- OpenAI
- GPT-3.5 Turbo
- GPT-4
- DALL-E
- DALL-E 2
- DALL-E 3
- Whisper
- Whisper.cpp
- WhisperX
- LangChain
- Pinecone
- PGVector
- NumPy
- TikToken
- MDN Web Docs
- Stability AI
- Stable Diffusion
- Midjourney
- Hugging Face Spaces
- Tortoise TTS
- Runway ML
- Cursor
- Replit
- Llama 2
- StarCoder
- Kenny Dodds
- Peter Levels
- RoomGPT

## Who should watch
- You are building your first AI-backed application and need a small project that connects model APIs to a real interface.
- You understand prompts but want to learn what tokens, embeddings, chunking, and retrieval do in code.
- You want practical examples of code generation, image generation, or speech recognition before choosing a larger production architecture.

## Related talks

- [AI Engineering 201: The Rest of the Owl](https://aietalks.com/talks/ai-engineering-201-the-rest-of-the-owl) (Charles Frye, Full Stack LLM Bootcamp, 56:57)
- [See, Hear, Speak, Draw](https://aietalks.com/talks/see-hear-speak-draw) (Logan Kilpatrick & Simón Fishman, OpenAI, 18:43)
- [From Software Developer to AI Engineer](https://aietalks.com/talks/from-software-developer-to-ai-engineer) (Antje Barth, AWS, 19:48)
- [AI Engineer Summit 2023, Day 2 Livestream](https://aietalks.com/talks/ai-engineer-summit-2023-day-2-livestream) (Mario Rodriguez, GitHub & Dedy Kredo, CodiumAI & Matt Welsh, Fixie.ai & Amelia Wattenberger, Adept & Samantha Whitmore & Jason Yuan, New Computer & Joseph Nelson, Roboflow & Hassan El Mghari, Vercel & Paul Copplestone, Supabase & Daniel Rosenwasser, Microsoft & Jason Liu, Fivesixseven & Anton Troynikov, Chroma & Jerry Liu, LlamaIndex & Mithun Hunsur, Ambient & Abi Aryan, O'Reilly & Simon Willison, Datasette & Benjamin Dunphy, Software 3.0 LLC & swyx, Latent.Space & Smol.ai, 7:30:56)
- [The 1,000x AI Engineer](https://aietalks.com/talks/the-1-000x-ai-engineer) (Swyx, AI Engineer Summit, latent.space, smol.ai, 09:27)