# Harnessing the Power of LLMs Locally

Mithun Hunsur, Ambient | AI Engineer Summit 2023 | 17:09

Source: https://www.youtube.com/watch?v=MwqUYRQloGw
Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks.
Page: https://aietalks.com/talks/harnessing-the-power-of-llms-locally
Published: 2023-11-22
Tags: deployment, inference, open-models, privacy

## TL;DR
- Local models give developers control over cost, latency, privacy, model choice, and token generation.
- llm.rs brings local LLM inference into Rust as a customizable library that supports multiple model architectures, hardware backends, and operating systems.
- Running models locally still requires hardware trade-offs, and the fast-changing ecosystem can break workflows or impose restrictive model licenses.

## Summary
Mithun Hunsur presents llm.rs, a Rust library for running large language models on local hardware. He compares local inference with cloud APIs across model size, speed, latency, cost, customization, and privacy. Quantization makes larger models small enough for consumer hardware, while GPU support improves generation speed. The library is designed as a reusable, Rust-native component rather than an application tied to one model or backend. Hunsur shows how it supports Discord responses, system-wide code completion, and a Wikipedia date-extraction project using a fine-tuned model. He also explains why local deployment can simplify distribution by producing self-contained binaries and avoiding Python dependency problems. His account is practical about the limits: hardware still constrains performance, quality and speed involve trade-offs, model formats and workflows change quickly, and licenses can restrict commercial use.

## Key ideas
### Local models trade cloud scale for control over cost, latency, and privacy
[00:56](https://www.youtube.com/watch?v=MwqUYRQloGw&t=56s)
Hunsur contrasts cloud models such as GPT-3, GPT-4, and PaLM 2 with models that run on hardware the developer owns or rents. Local models usually have less capacity, although a smaller focused model can solve a specific problem better than a large general model. Local inference can start responding as the prompt arrives, instead of waiting for a complete request and a network response. Costs depend on the machine rather than a per-token cloud bill, and developers can choose from many custom models. Privacy also matters because some questions should not be sent to an online service.

### Quantization makes local inference practical on consumer hardware
[03:55](https://www.youtube.com/watch?v=MwqUYRQloGw&t=235s)
The obstacle is that models contain billions of parameters, each represented by numerical values, while consumer machines have limited memory. Hunsur explains quantization as a way to compress a model while retaining most of its ability. The smaller representation takes less storage and can run faster because the computer can process more of the model at once. He presents this compression as the technique that makes local models viable. It does not remove hardware limits, but it changes which models can fit and run on ordinary machines.

### llm.rs was designed as a reusable Rust library rather than a fixed application
[04:49](https://www.youtube.com/watch?v=MwqUYRQloGw&t=289s)
Hunsur says llm.rs began as a Rust reimplementation of llama.cpp, developed alongside another contributor before their efforts were combined. Its design goals were to make it a library, avoid coupling it to one application, allow extensive customization, support multiple model architectures, feel native to Rust, work with different compute backends, and run across platforms. The library now gives different architectures the same interface, so application code does not need to change when the model changes. Hunsur credits Lucas and Dan, along with other contributors, for this work.

### Sessions and callbacks let applications control inference details
[07:05](https://www.youtube.com/watch?v=MwqUYRQloGw&t=425s)
The sample API loads a model, creates one or more sessions, and passes prompts to generate the next tokens. Sessions can be reused for conversations, which avoids rebuilding the context for every request, although each session consumes memory. A callback delivers generated tokens. Hunsur explains that the convenience function is only a loop around the model with boundary conditions, so developers can replace its logic when they need different behavior. Defaults provide customization points for model loading, inference, and sampling. The library can run on a MacBook CPU, with GPU acceleration making the result more usable.

### Small Rust applications demonstrate local inference outside a chat window
[08:56](https://www.youtube.com/watch?v=MwqUYRQloGw&t=536s)
Hunsur names LocalAI, LM Chain, and flum as community projects using the library. He then describes LM Code, a Discord bot that sends prompts through a generation thread and response task. Sessions are discarded after each query in that example, but retaining them enables conversation. Alpa applies the same idea to system-wide code completion: it takes the text around the cursor, sends it to a model, and types the generated completion. These examples show that the library can fit ordinary request-response systems and interactive tools.

### A fine-tuned local model can turn messy dates into structured data
[11:34](https://www.youtube.com/watch?v=MwqUYRQloGw&t=694s)
Hunsur describes extracting dates from millions of Wikipedia pages to build a historical timeline. Regular expressions could find date strings, but they could not reliably capture context, and some dates were nonsensical. Cloud models also proved too expensive and slow at that scale, even after prompt engineering. He generated a representative dataset, built a tool to correct its errors, and trained a smaller model with Axel. The resulting model is fast and consistent. His Rust code treats it as a function: pass in dates and receive a structured representation that can be processed directly.

### Rust packaging can simplify distribution of local-model applications
[12:46](https://www.youtube.com/watch?v=MwqUYRQloGw&t=766s)
Hunsur argues that llm.rs avoids much of the Python dependency and deployment trouble associated with local machine-learning projects. Rust's cross-platform support and build system make it easier to ship self-contained binaries without asking users to install Torch. That opens the door to desktop applications that include local models. The Rust ecosystem also lets developers combine inference with other native libraries in one language. Hunsur gives examples including a Discord bot, a completion utility, a data-ingestion pipeline, and a utility explorer.

### Local inference gives developers direct control, with serious trade-offs
[13:46](https://www.youtube.com/watch?v=MwqUYRQloGw&t=826s)
With a local model, developers can control token sampling directly instead of repeatedly requesting limited probability information from a cloud service. They can also try new research papers before cloud providers add similar features. Hunsur is clear about the costs. Old computers, phones, and Raspberry Pis have limited power. Larger models can improve quality at the cost of speed, so each application must choose its compromise. Rapid ecosystem changes can break workflows, even with standard formats such as GGUF. Model licenses also need attention because free personal use does not always mean unrestricted commercial use.

## Notable quotes
- "Local models offer another way where you own the model and it runs on your computer." (00:56)
- "Quantization lets you compress a model while maintaining the majority of its intelligence." (03:55)
- "The entire point is you have the control you need to make the thing work." (08:08)
- "There ain't no such thing as a free lunch except if you're a conference." (14:40)

## Tools & references mentioned
- llm.rs
- Rust
- llama.cpp
- GPT-3
- GPT-4
- PaLM 2
- LLaMA
- Falcon
- Axel
- LocalAI
- LM Chain
- flum
- LM Code
- Alpa
- GGUF
- Mistral
- StableLM
- GitHub Copilot
- Wikipedia

## Who should watch
- You are building a Rust application that needs local text generation, code completion, or structured extraction without a cloud API.
- Your team wants to ship a self-contained desktop or native application and avoid Python dependency installation for inference.
- You need to weigh local hardware limits, model quality, latency, changing model formats, and licensing before moving a workload off a cloud service.

## Related talks

- [Navigating Challenges and Technical Debt in LLMs Deployment](https://aietalks.com/talks/navigating-challenges-and-technical-debt-in-llms-deployment) (Ahmed Menshawy, Mastercard, 16:15)
- [2025 in LLMs so far, illustrated by Pelicans on Bicycles](https://aietalks.com/talks/2025-in-llms-so-far-illustrated-by-pelicans-on-bicycles) (Simon Willison, 18:30)
- [Running LLMs Locally: Practical LLM Performance on DGX Spark](https://aietalks.com/talks/running-llms-locally-practical-llm-performance-on-dgx-spark) (Mozhgan Kabiri chimeh, NVIDIA, 10:16)
- [The LLM Triangle: Engineering Principles for Robust AI Applications](https://aietalks.com/talks/the-llm-triangle-engineering-principles-for-robust-ai-applications) (Almog Baku, 26:19)
- [Iterating on LLM apps at scale: Learnings from Discord](https://aietalks.com/talks/iterating-on-llm-apps-at-scale-learnings-from-discord) (Ian Webster, Discord, 18:26)