# Building AI For All

Amjad Masad & Michele Catasta, Replit | AI Engineer Summit 2023 | 25:13

Source: https://www.youtube.com/watch?v=ju73sWVtvU0
Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks.
Page: https://aietalks.com/talks/building-ai-for-all
Published: 2023-10-23
Tags: code-generation, developer-experience, inference, open-models

## TL;DR
- AI should be part of every programming interaction instead of an optional coding add-on.
- Replit Code V1.5 is a 3.3B-parameter code model trained on permissively licensed data and released open source.
- Replit optimized the model for fast inference, reaching more than 200 tokens per second on a single model without batching.

## Summary
Amjad Masad frames AI-assisted programming as the latest major change in software development, following the move from punch cards to text editors, compilers, and IDEs. He says Replit wants AI available to all of its users, including people coding on phones, and introduces Model Farm for calling language and image models from the IDE. Michele Catasta then explains Replit Code V1.5, a model trained from scratch on filtered, permissively licensed code across 30 programming languages. The team used a smaller model, more training data, a code-specific vocabulary, and current attention and optimization methods. Catasta compares its benchmark results with larger models, then focuses on practical performance: more than 200 tokens per second and a reduction in deployment time from 18 minutes to two. He also explains why Replit released the model and previews collaborations around instruction tuning, fill-in-the-middle generation, and hosted inference.

## Key ideas
### Programming has already changed through large jumps in abstraction
[00:33](https://www.youtube.com/watch?v=ju73sWVtvU0&t=33s)
Amjad Masad traces software development from ENIAC punch cards to assembly, compilers, C, JavaScript, and IDEs. Punch cards encoded binary instructions and were difficult enough that they did not create a software industry. Text-based programming was, in his estimate, at least a 10x improvement and possibly a 100x improvement. Replit later added Language Server Protocol support to every programming environment so users could access editor intelligence. Masad presents AI as another possible jump in how people create software, building on these earlier changes rather than appearing without precedent.

### AI should be built into programming instead of added as a separate feature
[02:22](https://www.youtube.com/watch?v=ju73sWVtvU0&t=142s)
Replit began exploring AI-assisted coding with GPT-2 and built Ghostwriter for autocomplete, chat, and other IDE features. Masad says AI tools have already improved productivity for some engineers by roughly 50 to 100 percent, while the larger gains are still ahead. He argues that products such as Copilot and Ghostwriter should not be the whole direction. AI needs to be part of the default programming experience, available during every programming interaction. That view leads to Replit's announcement that AI coding assistance will be provided to its millions of users, across devices including Android phones.

### Model Farm lets developers call AI models from the IDE
[05:05](https://www.youtube.com/watch?v=ju73sWVtvU0&t=305s)
Masad separates AI-assisted software creation from AI in the software's own call stack. Replit's Model Farm gives developers access to models directly inside the IDE, with three lines of code needed to start inference. The initial launch uses Google Cloud language models, with Llama and Stable Diffusion planned for addition. Replit also offers a free tier at least through the end of the year. The product is intended to let developers start building AI-based applications without leaving the environment where they write and deploy their code.

### Replit Code V1.5 uses more data to make a smaller model more capable
[08:39](https://www.youtube.com/watch?v=ju73sWVtvU0&t=519s)
Michele Catasta announces Replit Code V1.5, the successor to the 3B model released in May. The new model is 3.3B parameters with a 4K context, a new 32K code-specific vocabulary, and training on up to 1 trillion code tokens. Its data covers 30 programming languages and includes developer-oriented posts from Stack Exchange. Catasta connects this recipe to the Chinchilla paper's argument that models were being undertrained and could become more capable when trained on more high-quality data. The smaller model requires more training compute, then offers lower latency during inference.

### Data filtering and permissive licenses shape the model's training set
[10:46](https://www.youtube.com/watch?v=ju73sWVtvU0&t=646s)
The team starts from The Stack, a BigCode initiative that collects GitHub data, selects repositories, cleans the material, and retains code under permissive licenses such as MIT, BSD, and Apache 2. Catasta says data quality received much of the team's effort because the model was trained from scratch. The pipeline removes autogenerated and minified code, unparsable code, and toxic content. Replit also used Spark and open source components for the data pipeline. Catasta points to repeated passes over a small, high-quality dataset as a way to train to completion while keeping the licensing terms permissive.

### The model's practical target is fast code completion
[12:57](https://www.youtube.com/watch?v=ju73sWVtvU0&t=777s)
Replit Code V1.5 is tuned for inference rather than only for benchmark scores. Catasta describes a single model running without batching at more than 200 tokens per second. The architecture uses a small vocabulary, FlashAttention, grouped-query attention, and other current optimizations, with deployment targeted at the Triton inference server and related acceleration frameworks. Replit also shortened the first deployment from 18 minutes to two minutes. That change lets the service react to traffic spikes more quickly. Catasta says this fast deployment helped Replit put code completion in front of its users when Masad announced AI for all.

### The model compares closely with much larger code models
[15:37](https://www.youtube.com/watch?v=ju73sWVtvU0&t=937s)
On HumanEval and multilingual code evaluation, Catasta says Replit Code V1.5 was optimal across the listed languages at the 3B scale and performed better than StarCoder 3B. He is especially excited that it came close to Code Llama 7B, which had twice the parameter count, 2.5 times more data, and substantially more GPU compute. The reported HumanEval pass@1 result was 36 percent. Catasta compares that with an earlier model from his work that was almost 200 times larger and with Code-Davinci-001, which had the same reported score. These comparisons support Replit's choice to optimize a smaller model for serving.

### Replit released the model to encourage local use and further versions
[22:10](https://www.youtube.com/watch?v=ju73sWVtvU0&t=1330s)
Catasta gives adoption and collaboration as reasons for releasing Replit Code V1.5 under a commercially permissive license. After the first model was released, people made fine-tuned and instruction-tuned versions and ran it locally with tools such as llama.cpp on Apple silicon. Some users built privacy-aware alternatives to GitHub Copilot. Replit planned an instruction-tuned version with Glaive AI, a fill-in-the-middle format with Morph Labs that uses prefix and suffix context plus awareness of the abstract syntax tree, and hosted access through Perplexity AI. The team also planned to keep models current with recently popular libraries.

## Notable quotes
- "AI needs to be really infused in every programming interaction that you have and it needs to be part of the default experience of Replit." (04:15)
- "We really want to go as fast as possible with the most powerful small model we could train." (19:57)
- "The very first time we brought this infrastructure up it took 18 minutes to go from clicking until the model was deployed." (21:17)
- "The moment we did it we got a lot of adoption." (22:27)

## Tools & references mentioned
- Ghostwriter
- Model Farm
- GPT-2
- Llama
- Stable Diffusion
- The Chinchilla paper
- DeepMind
- The Stack
- BigCode
- HumanEval
- Multilingual Code Evaluation Harness
- StarCoder
- Code Llama
- Code-Davinci-001
- Glaive AI
- Morph Labs
- Perplexity AI
- MosaicML
- Spark
- Triton
- FlashAttention
- llama.cpp

## Who should watch
- You are building an AI coding product and need to decide whether to use a hosted general model or train a smaller model for latency.
- You want the concrete data, evaluation, and serving choices behind an open source code completion model.
- You maintain an IDE or developer platform and are considering how to make AI available across different devices and user skill levels.

## Related talks

- [The Rise of the AI Software Engineer](https://aietalks.com/talks/the-rise-of-the-ai-software-engineer) (Jesse Han, Morph Labs, 03:48)
- [The 1,000x AI Engineer](https://aietalks.com/talks/the-1-000x-ai-engineer) (Swyx, AI Engineer Summit, latent.space, smol.ai, 09:27)
- [AI Engineer Code Summit 2025](https://aietalks.com/talks/ai-engineer-code-summit-2025) (Jed Borovik, Google & Swyx, AI Engineer & Barry Zhang & Mahesh Murag, Anthropic & Dex Horthy, HumanLayer & Lee Robinson & Naman Jain, Cursor & Jacob Kahn, Meta & Rhythm Garg & Linden Li, Applied Compute & Will Brown, Prime Intellect & Will Hang & Cathy Zhou, OpenAI & Kitze, Independent & Kath Korevec, Google Labs & Eno Reyes, Factory AI & Beyang Liu, Amp Code / Sourcegraph & Natalie Serrino, Gimlet Labs & Jake Nations, Netflix & Eiso Kant & Jason Warner, Poolside & Aparna Dhinakaran, Arize & Nik Pash, Cline & Joel Becker, METR & Kevin Hou, Google DeepMind & Benjamin Dupy & Leah McBride, AI Engineer, 8:57:06)
- [Fast Models Need Slow Developers](https://aietalks.com/talks/fast-models-need-slow-developers) (Sarah Chieng, Cerebras, 18:02)
- [Accelerate your AI journey with Azure AI model catalog](https://aietalks.com/talks/accelerate-your-ai-journey-with-azure-ai-model-catalog) (Sharmila Chokalingam, Microsoft Azure AI, 23:14)