# Build enterprise generative AI apps using Llama 3 at 1,000 tokens/s on the SambaNova AI platform

Michelle Matern, Pedro Milan & Varon Krishna, SambaNova | AI Engineer World's Fair 2024 | 54:34

Source: https://www.youtube.com/watch?v=ksgLoPxEQzM
Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks.
Page: https://aietalks.com/talks/build-enterprise-generative-ai-apps-using-llama-3-at-1-000-tokens-s-on-the
Published: 2024-09-11
Tags: embeddings, enterprise, inference, rag

## TL;DR
- SambaNova combines its own hardware and software stack to run Llama 3 at more than 1,000 tokens per second.
- Samba-1 uses a composition of expert models behind one endpoint, with routing, fine-tuning schedules, model chaining, and model-level access controls.
- The workshop builds a RAG question-answering app with LangChain, document loaders, E5-large-v2 embeddings, ChromaDB, and a Llama 3 endpoint.

## Summary
This workshop introduces SambaNova's enterprise AI platform and then builds two applications. Michelle Matern explains the platform's full-stack design, including the SN40L reconfigurable data flow unit, its three-tier memory system, and the Samba-1 composition of expert models. The aim is to combine the broad coverage of large models with the control and privacy of smaller, specialized models. Pedro Milan then demonstrates Llama 3 through the SambaNova API and walks through a basic LangChain example. The main exercise builds a retrieval-augmented question-answering system for enterprise search. Documents are loaded and split, embedded with E5-large-v2, stored in ChromaDB, and retrieved to provide context to Llama 3. The workshop also covers practical setup issues, including Python environments, API keys, Tesseract, Poppler, notebooks, and Streamlit. The examples make the platform concrete, though the workshop spends more time on setup and demonstration than on evaluating the finished RAG system.

## Key ideas
### SambaNova builds the AI stack from its chip through the software layer
[04:18](https://www.youtube.com/watch?v=ksgLoPxEQzM&t=258s)
Michelle Matern presents SambaNova as a full-stack AI platform founded in 2017 out of Stanford University. The company builds its own chip, system software, and model-serving software. The stack supports fine-tuning, pre-training, and inference. Matern says the integration removes decisions that customers would otherwise have to make across chips, operating systems, compute, and models. The target customers include enterprises and government organizations that need private data handling, sovereign AI capabilities, and support for very large models.

### Samba-1 combines specialized models behind one enterprise endpoint
[10:58](https://www.youtube.com/watch?v=ksgLoPxEQzM&t=658s)
SambaNova's approach starts with expert models tuned for particular purposes, such as legal work, HR, or coding. Instead of making an application integrate each model separately, the platform places them behind a secure endpoint. A router chooses the expert that best fits an incoming prompt. The platform also supports scheduled fine-tuning, direct calls to a chosen model, model chaining for agentic applications, and model-level role-based access control. The pre-composed Samba-1 configuration contains 92 experts covering different languages, domains, and tasks.

### The composition of experts addresses the tradeoff between general and small models
[08:37](https://www.youtube.com/watch?v=ksgLoPxEQzM&t=517s)
Matern contrasts large monolithic models with smaller open-source expert models. Large models offer broad knowledge and simple integration, but enterprise users may worry about proprietary data, privacy, ownership, and rising per-token costs. Small models can be adapted to enterprise data and tasks, while preserving model and data ownership, but they create management and orchestration work. Samba-1 is intended to combine these properties by exposing specialized models through one interface while retaining control over which models each application or user can access.

### The SN40L memory hierarchy keeps many models available in one system
[18:52](https://www.youtube.com/watch?v=ksgLoPxEQzM&t=1132s)
SambaNova's SN40L chip uses three memory tiers: on-chip memory, high-bandwidth memory, and DDR. Models can move between these tiers as needed. Matern says the system can store up to five trillion parameters on DDR, with four gigabytes of on-chip SRAM, 512 gigabytes of high-bandwidth memory, and up to six terabytes of DDR. Her comparison with GPUs focuses on model hosting: a system with this much DDR can keep many models in one coupled system instead of distributing them across multiple GPU systems.

### The SambaNova endpoint exposes Llama 3 with low first-token and total latency
[22:39](https://www.youtube.com/watch?v=ksgLoPxEQzM&t=1359s)
Pedro Milan demonstrates SambaNova's Samba-1 Turbo interface with Llama 3 8B. He reports throughput of 1,000 tokens per second, a time to first token of 0.09 seconds, and an end-to-end inference time of 6.5 seconds for the demonstrated request. The interface includes a real-time mode that updates the response while a prompt is being written. Milan presents this as useful for drafting text and experimenting with prompts. Generation settings are fixed in the web interface, while the API allows callers to change model configuration.

### The basic LangChain example shows how to configure and call the model
[26:24](https://www.youtube.com/watch?v=ksgLoPxEQzM&t=1584s)
The first exercise uses the SambaNova repository and a Python environment. Participants clone the repository, create a .env file with a SambaStudio API key, install the requirements, and connect the environment to a notebook kernel. The code uses a LangChain wrapper, loads environment variables, and initializes the LLM with settings such as sampling, temperature, and maximum output tokens. A model endpoint selects Llama 3 8B Instruct. Milan first calls the model with LangChain's invoke method, then uses a prompt template and output parser to insert a country value dynamically.

### The RAG pipeline turns documents into searchable context for Llama 3
[38:25](https://www.youtube.com/watch?v=ksgLoPxEQzM&t=2305s)
The second exercise builds an enterprise question-answering app with retrieval augmented generation. Documents are loaded and parsed, split into smaller chunks, converted into vectors with the E5-large-v2 embedding model, and stored with their content and metadata in ChromaDB. A user question is embedded into the same vector space, and the retriever selects nearby chunks. The question and retrieved context are then passed to Llama 3 for a grounded answer. The exercise can run the embedding model on a CPU or on SambaNova hardware, while the LLM runs on SambaNova's hardware.

### Document parsing choices affect how well the RAG app handles PDFs and tables
[49:01](https://www.youtube.com/watch?v=ksgLoPxEQzM&t=2941s)
For the demonstration, Milan uses a 15-page SN40L paper, a chunk size of 1,200, and an overlap of 240, which produces 89 chunks. The vectors are indexed in ChromaDB, and the question-answering chain retrieves three chunks because no ranker is being used. The app answers a question about monolithic models and then retrieves information from a table about operation intensity and fusion level. Milan says that more complicated PDF tables may work better with the Unstructured data loader instead of PyPDF2. The repository includes both a notebook and a Streamlit version.

## Notable quotes
- "We are a full stack AI platform and we've existed since 2017." (04:18)
- "We put all of those behind a secure single endpoint." (12:42)
- "We can store up to five trillion parameters on DDR." (19:40)
- "RAG is a technique that we can use to supplement LLM with additional information from various sources to improve the model's response." (38:30)
- "The whole goal of this embedding is that it allows us to do semantic similarity and semantic search." (40:12)

## Tools & references mentioned
- SambaNova
- SN40L
- Samba-1
- Llama 3
- LangChain
- Unstructured
- E5-large-v2
- ChromaDB
- PyPDF2
- GPT-3.5 Turbo
- GPT-4
- OpenAI
- Gemini
- Claude
- Artificial Analysis
- Stanford University
- Snorkel
- Together AI
- Mistral
- Falcon
- BLOOM
- LLaVA
- CLIP
- DeepLab

## Who should watch
- You are building a Python LLM application and want a complete example that connects a hosted Llama 3 endpoint to LangChain.
- Your enterprise search prototype needs document parsing, embeddings, vector retrieval, and a grounded answer-generation step.
- You want to understand SambaNova's model-composition approach and the practical setup required to test its API and RAG examples.

## Related talks

- [Llama 3 at 1,000 tok/s on the SambaNova AI Platform](https://aietalks.com/talks/llama-3-at-1-000-tok-s-on-the-sambanova-ai-platform) (Rochelle Maturin, Petro Milan & Varun Krishna, SambaNova, 1:00:58)
- [The Future of Knowledge Assistants](https://aietalks.com/talks/the-future-of-knowledge-assistants) (Jerry Liu, LlamaIndex, 16:55)
- [Build an AI Research Agent](https://aietalks.com/talks/build-an-ai-research-agent) (Apoorva Joshi, MongoDB, 27:33)
- [RAG in 2025: State of the Art and the Road Forward](https://aietalks.com/talks/rag-in-2025-state-of-the-art-and-the-road-forward) (Tengyu Ma, MongoDB, 18:48)
- [Real ROI: Lessons from Enterprises That Have Already Succeeded with LLMs at Scale](https://aietalks.com/talks/real-roi-lessons-from-enterprises-that-have-already-succeeded-with-llms-at-scale) (Raza Habib, Humanloop, 20:01)