SambaNova combines its own RDU hardware, software stack, and model services to run Llama 3 at high throughput.
2
Samba-1 composes expert models behind one endpoint, with routing, dynamic fine-tuning, model chaining, and model-level access controls.
3
The workshop builds a RAG question-answering app with LangChain, document loaders, E5-large-v2 embeddings, ChromaDB, and Llama 3.
Summary
Rochelle Maturin introduces SambaNova's full-stack AI platform and explains how its SN40L reconfigurable data flow unit supports fast inference. She describes Samba-1 as a composition of expert models behind a secure endpoint. A router selects an expert for each prompt, while users can schedule fine-tuning and apply model-level access controls. Petro Milan then demonstrates Llama 3 8B Instruct through Samba-1 Turbo, reporting 1,000 tokens per second, 0.19 seconds to first token, and 0.65 seconds for total inference in the web interface. The hands-on work starts with a Python and LangChain inference example, including API setup, model configuration, sampling, temperature, and prompt formatting for Llama 3. The second exercise builds a RAG question-answering system. It loads and splits documents, embeds chunks with E5-large-v2, stores them in ChromaDB, retrieves relevant context, and sends that context to Llama 3. The workshop also compares PDF loaders and shows when Unstructured may work better for complex tables.
SambaNova builds the hardware and software stack together
Rochelle Maturin describes SambaNova as a full-stack AI platform founded in 2017 out of Stanford University. The company builds its own chip, system software, and software layer. The stack supports fine-tuning, pre-training, and high-performance inference. She says the goal is to reduce the number of separate integration decisions customers must make across chips, operating systems, compute, and models. SambaNova targets enterprise and government use cases, including proprietary data and sovereign AI requirements. Maturin also says the platform is intended to support models at the scale of a trillion parameters or more.
Samba-1 puts many expert models behind one endpoint
Samba-1 addresses the trade-offs between broad monolithic models and many smaller expert models. SambaNova can place models fine-tuned for areas such as legal work, HR, and coding behind a secure single endpoint. A router examines an incoming prompt and chooses the best-suited expert. The platform also supports direct model calls and model chaining, which Maturin connects to agentic AI use cases. Customers can schedule fine-tuning at different rates for different experts. Model-level role-based access control limits which applications, people, or groups can use particular models.
The pre-composed Samba-1 model contains 92 experts
SambaNova offers a flexible composition in which customers determine which models sit underneath the system, as well as a pre-composed Samba-1 version. Maturin says the pre-composed version has 92 underlying experts covering different languages, domains, and enterprise tasks. Those experts are supported by seven foundation model architectures, including Llama 2, Llama 3, Mistral, Falcon, and BLOOM, along with multimodal capabilities. SambaNova contributed or co-developed 12 of the experts, including language models for Thai, Japanese, and Hungarian, a text-to-SQL model, and work on BLOOM Chat.
SN40L uses three memory tiers to hold and swap models
Petro Milan explains that SambaNova's SN40L chip is an RDU, or reconfigurable data flow unit. It has three memory tiers: on-chip memory, high-bandwidth memory, and DDR. Models can move between these tiers as they are needed. Maturin says the system can store up to 5 trillion parameters on DDR. She gives the memory figures as 4 GB of on-chip SRAM, 512 GB of high-bandwidth memory, and up to 6 TB of DDR. Her comparison with GPUs focuses on hosting many models in one coupled system instead of distributing hundreds of models across separate systems.
Milan opens SambaNova's Samba-1 Turbo interface and selects Llama 3 8B. In the demonstration, he asks for a three-day-per-week workout schedule and shows the response arriving quickly. The displayed metrics report throughput of 1,000 tokens per second, time to first token of 0.19 seconds, and total inference time of 0.65 seconds. The interface also has a real-time mode that updates the response as the prompt changes. Milan suggests this can help with drafting emails and experimenting with prompts. He notes that generation settings are fixed in the web interface but can be changed when calling the endpoint through the API.
The first exercise teaches a basic LangChain endpoint call
The introductory exercise uses Python to load environment variables, set a SambaNova API key, initialize the LLM, and make an inference call. The setup uses the SambaNova starter kit, a .env file, a Python or conda environment, pip requirements, and a Jupyter kernel. Milan shows the Samba Studio LangChain wrapper and model configuration, including sampling, temperature, and maximum tokens. He selects the Meta Llama 3 8B Instruct expert. A direct prompt about the capital of France produces extra material until the prompt includes Llama 3's special beginning-of-text, user-query, and answer markers. He then shows how an LCEL chain connects a prompt template, the LLM, and an output parser.
The RAG exercise grounds answers in retrieved document chunks
The second exercise builds a question-answering system for enterprise search. Milan explains that RAG adds information from external sources so an off-the-shelf LLM can answer questions beyond its training data or use current information without retraining. The workflow loads and parses documents, splits them into chunks, converts chunks into vectors with an embedding model such as E5-large-v2, stores vectors and metadata in ChromaDB, retrieves chunks close to the embedded question, and gives the question plus retrieved context to the LLM. The exercise can run embeddings on a CPU or SambaNova hardware, while the LLM runs on SambaNova hardware.
Chunking and document loaders affect the RAG result
For a 15-page SN40L paper, Milan sets the chunk size to 1,200 and the overlap to 240, producing 89 chunks. The notebook embeds those chunks and indexes them in ChromaDB. At question time, the system retrieves the top three chunks because the demonstration does not use a re-ranker. Milan asks what a monolithic model is and receives an immediate grounded answer. He then tests a table in the PDF. When the PDF loader has difficulty with more complicated tables, he recommends switching from PyPDF to the Unstructured data loader through the configuration file. The repository includes both a Jupyter notebook and a Streamlit version of the application.
"RAG is a technique that you can use to supplement LLM with additional information from various sources to improve the model's response."44:52
Who should watch
You are building a Python LLM application and want a worked example of calling Llama 3 through a hosted API.
You need enterprise search over PDFs or other documents and want to see the data-loading, chunking, embedding, retrieval, and answer-generation steps connected in one app.
You are evaluating inference platforms or composition-of-experts systems and want SambaNova's explanation of its hardware, model routing, and access controls.
