# The Hierarchy of Needs for Training Dataset Development

Chang She, LanceDB & Noah Shpak, Character AI | AI Engineer World's Fair 2024 | 16:32

Source: https://www.youtube.com/watch?v=i2vBaFzCEJw
Channel: AI Engineer (https://www.youtube.com/@aiDotEngineer). Summarised by AIE Talks.
Page: https://aietalks.com/talks/the-hierarchy-of-needs-for-training-dataset-development
Published: 2024-10-15
Tags: data-pipelines, embeddings, multimodal, synthetic-data

## TL;DR
- Training data needs to be understood at a deeper level as work moves from pre-training toward post-training.
- Character AI uses analytics, retrieval, clustering, classifiers, synthetic data, and human labeling to refine datasets through repeated experiments.
- Lance is designed for AI workloads that need fast scans, random access, and streaming of large text, image, video, or tensor data in one system.

## Summary
Noah Shpak describes dataset development at Character AI as an iterative process. Pre-training calls for broad decisions about domains and quantity, while post-training requires detailed information about task difficulty, data composition, and evaluation results. His team uses analytics, retrieval, clustering, classifiers, prompting, synthetic examples, and human labeling to understand and improve datasets. A materialization service separates dataset construction from training jobs and gives researchers a repeatable way to produce training files. Chang She then explains why AI data infrastructure has different requirements from ordinary warehouse workloads. Distributed training needs fast filtering, cheap random access for shuffling, and efficient streaming of large binary objects to GPUs. Lance combines columnar storage, random lookups, versioning, schema changes, time travel, and indexes for vectors, metadata, and text. The result is a single table that can support SQL analysis, search, retrieval, and model training across multimodal data.

## Key ideas
### Dataset work starts with clean data and ends with evaluations
[02:45](https://www.youtube.com/watch?v=i2vBaFzCEJw&t=165s)
Noah Shpak presents a hierarchy that begins with clean data and moves toward evaluations. He says evaluations provide a compass because it is hard to measure changes without knowing what outcome to compare against. Dataset mixtures need analysis beyond their names. Teams may need token counts, length information, and labels describing code difficulty, function counts, or expected classes. Reading data and inspecting model outputs have produced some of the team's largest gains because they expose what is actually happening in the training process.

### Pre-training needs broad coverage while post-training needs fine-grained detail
[01:52](https://www.youtube.com/watch?v=i2vBaFzCEJw&t=112s)
The talk separates pre-training from post-training. For pre-training, Noah says teams think about broad domains such as books or chat data, along with the amount of data needed for a model. Post-training focuses on particular tasks and asks more detailed questions, such as how difficult a math problem is or how easy a multiple-choice problem might be. Data-efficient learning, sampling, and diversity measurement sit between these concerns. The data platform has to support both broad collection and detailed analysis.

### Automated dataset improvement combines models with human review
[04:13](https://www.youtube.com/watch?v=i2vBaFzCEJw&t=253s)
Character AI uses language models to help improve other language-model datasets. Dataset selection can match the distribution of available data to the behavior the team wants from a model, using retrieval and clustering after embedding data. The team also builds classifiers for quality and other properties, while prompted classification can provide a simpler starting point. Synthetic data helps researchers explore preference-pair formats and other methods before investing in higher-quality examples. Human labeling then improves classifiers and rewrites synthetic or existing data that has problems.

### A materialization service keeps dataset construction separate from training
[06:03](https://www.youtube.com/watch?v=i2vBaFzCEJw&t=363s)
Noah describes a platform where researchers request a dataset and receive a list of files. The service separates how data is materialized from what the training job does with it. This matters because training formats such as TFRecords or JSON Lines can be error-prone, especially when researchers are changing data repeatedly and unexpected results make it unclear what was used. The platform combines large-data tools such as Spark and Trino with GPU-backed services for prompting, embedding, and classification.

### AI training needs fast filtering, random access, and large-object streaming together
[08:18](https://www.youtube.com/watch?v=i2vBaFzCEJw&t=498s)
Chang She describes a distributed training workload with three steps. It filters raw data to select samples, shuffles by drawing random rows, and streams text, images, or videos from object storage to GPUs. These steps require fast scans, fast random access, and support for very large binary objects. She argues that existing formats and infrastructure usually handle only one or two of these requirements well. Lance was designed around the combination.

### Multimodal data makes storage grow far faster than ordinary tabular data
[09:39](https://www.youtube.com/watch?v=i2vBaFzCEJw&t=579s)
She compares ordinary tabular rows with richer AI records. A scalar-only row is small, but embeddings make rows much larger, images add another large increase, and videos make the size much greater again. Generative AI also produces new observations at a rate that is not limited by manual data entry. As a result, even small teams may manage tens of terabytes or petabytes. The storage system therefore has to handle both scale and varied data types.

### Lance adds versioning and random access to columnar storage
[10:44](https://www.youtube.com/watch?v=i2vBaFzCEJw&t=644s)
Lance is a columnar format with scan performance similar to Parquet, while also supporting fast lookups. It removes the row-group limitation that makes it difficult to store and retrieve large blobs efficiently. The format versions data as it is added, allows columns to be added without copying the original dataset, and supports zero-copy schema evolution. Time travel lets a team roll back immediately after a mistake or bad data enters a dataset.

### One table can support analysis, search, retrieval, and training
[12:27](https://www.youtube.com/watch?v=i2vBaFzCEJw&t=747s)
Lance adds indexes for embeddings, scalar metadata, and full-text search. These allow vector search, filtering, keyword search, and fuzzy search directly against data in S3. The same table can plug into DuckDB, Trino, or Spark for SQL, into PyTorch for training, and into vector indexes for similarity search. She describes this as support for a complete AI workflow, from exploring data to searching it and fine-tuning models. Multimodal concerns include the data types, the workloads, and the production or research use case.

## Notable quotes
- "You should really care about what you're training on and you should care for it by giving it a nice format that does a lot of nice things for it." (01:35)
- "We always start with clean data and quickly go right up to evaluations." (02:45)
- "These three properties are required often in one AI workloads from training to search and retrieval." (09:05)
- "At the end of the day for us we just want to watch the GPUs go bur and the numbers go up." (07:18)

## Tools & references mentioned
- LanceDB
- Lance format
- Character AI
- pandas
- Spark
- Trino
- DuckDB
- PyTorch
- Parquet
- S3
- TFRecords
- JSON Lines

## Who should watch
- You are building a dataset platform for model training and need a repeatable way to materialize changing data.
- Your training pipeline mixes metadata with embeddings, text, images, audio, or video and ordinary storage formats are slowing iteration.
- You are evaluating LanceDB for a workflow that includes SQL analysis, vector search, full-text search, and model training.

## Related talks

- [The Messy Reality of Scale: Synthetic Data and Pre-Training](https://aietalks.com/talks/the-messy-reality-of-scale-synthetic-data-and-pre-training) (Marah Abdin & Robert McHardy, poolside, 17:31)
- [AI Engineer Paris 2025, Day 2](https://aietalks.com/talks/ai-engineer-paris-2025-day-2) (Emil Eifrem, Neo4j & Tushar Jain, Docker & Martin Woodward, GitHub & Yann Leger, Koyeb & Andreas Blattmann, Black Forest Labs & Laurent Sifre, H Company & Andreas Kollegger, Neo4j & Vaibhav Srivastav, Hugging Face & Aparna Dhinakaran, Arize AI & Steeve Morin, ZML & Tuana Çelik, LlamaIndex & Paige Bailey, Google DeepMind & Neil Zeghidour, Kyutai, 8:26:39)
- [Data is Your Differentiator: Building Secure and Tailored AI Systems](https://aietalks.com/talks/data-is-your-differentiator-building-secure-and-tailored-ai-systems) (Mani Khanuja, AWS, 20:10)
- [Data Quality Is the Compute Multiplier](https://aietalks.com/talks/data-quality-is-the-compute-multiplier) (Ari Morcos, DatologyAI, 19:05)
- [Robotics: why now?](https://aietalks.com/talks/robotics-why-now) (Quan Vuong & Jost Tobias Springenberg, Physical Intelligence, 18:07)