Training data needs to be understood at a deeper level as work moves from pre-training toward post-training.
2
Character AI uses analytics, retrieval, clustering, classifiers, synthetic data, and human labeling to refine datasets through repeated experiments.
3
Lance is designed for AI workloads that need fast scans, random access, and streaming of large text, image, video, or tensor data in one system.
Summary
Noah Shpak describes dataset development at Character AI as an iterative process. Pre-training calls for broad decisions about domains and quantity, while post-training requires detailed information about task difficulty, data composition, and evaluation results. His team uses analytics, retrieval, clustering, classifiers, prompting, synthetic examples, and human labeling to understand and improve datasets. A materialization service separates dataset construction from training jobs and gives researchers a repeatable way to produce training files. Chang She then explains why AI data infrastructure has different requirements from ordinary warehouse workloads. Distributed training needs fast filtering, cheap random access for shuffling, and efficient streaming of large binary objects to GPUs. Lance combines columnar storage, random lookups, versioning, schema changes, time travel, and indexes for vectors, metadata, and text. The result is a single table that can support SQL analysis, search, retrieval, and model training across multimodal data.
Dataset work starts with clean data and ends with evaluations
Noah Shpak presents a hierarchy that begins with clean data and moves toward evaluations. He says evaluations provide a compass because it is hard to measure changes without knowing what outcome to compare against. Dataset mixtures need analysis beyond their names. Teams may need token counts, length information, and labels describing code difficulty, function counts, or expected classes. Reading data and inspecting model outputs have produced some of the team's largest gains because they expose what is actually happening in the training process.
Pre-training needs broad coverage while post-training needs fine-grained detail
The talk separates pre-training from post-training. For pre-training, Noah says teams think about broad domains such as books or chat data, along with the amount of data needed for a model. Post-training focuses on particular tasks and asks more detailed questions, such as how difficult a math problem is or how easy a multiple-choice problem might be. Data-efficient learning, sampling, and diversity measurement sit between these concerns. The data platform has to support both broad collection and detailed analysis.
Automated dataset improvement combines models with human review
Character AI uses language models to help improve other language-model datasets. Dataset selection can match the distribution of available data to the behavior the team wants from a model, using retrieval and clustering after embedding data. The team also builds classifiers for quality and other properties, while prompted classification can provide a simpler starting point. Synthetic data helps researchers explore preference-pair formats and other methods before investing in higher-quality examples. Human labeling then improves classifiers and rewrites synthetic or existing data that has problems.
A materialization service keeps dataset construction separate from training
Noah describes a platform where researchers request a dataset and receive a list of files. The service separates how data is materialized from what the training job does with it. This matters because training formats such as TFRecords or JSON Lines can be error-prone, especially when researchers are changing data repeatedly and unexpected results make it unclear what was used. The platform combines large-data tools such as Spark and Trino with GPU-backed services for prompting, embedding, and classification.
AI training needs fast filtering, random access, and large-object streaming together
Chang She describes a distributed training workload with three steps. It filters raw data to select samples, shuffles by drawing random rows, and streams text, images, or videos from object storage to GPUs. These steps require fast scans, fast random access, and support for very large binary objects. She argues that existing formats and infrastructure usually handle only one or two of these requirements well. Lance was designed around the combination.
Multimodal data makes storage grow far faster than ordinary tabular data
She compares ordinary tabular rows with richer AI records. A scalar-only row is small, but embeddings make rows much larger, images add another large increase, and videos make the size much greater again. Generative AI also produces new observations at a rate that is not limited by manual data entry. As a result, even small teams may manage tens of terabytes or petabytes. The storage system therefore has to handle both scale and varied data types.
Lance adds versioning and random access to columnar storage
Lance is a columnar format with scan performance similar to Parquet, while also supporting fast lookups. It removes the row-group limitation that makes it difficult to store and retrieve large blobs efficiently. The format versions data as it is added, allows columns to be added without copying the original dataset, and supports zero-copy schema evolution. Time travel lets a team roll back immediately after a mistake or bad data enters a dataset.
One table can support analysis, search, retrieval, and training
Lance adds indexes for embeddings, scalar metadata, and full-text search. These allow vector search, filtering, keyword search, and fuzzy search directly against data in S3. The same table can plug into DuckDB, Trino, or Spark for SQL, into PyTorch for training, and into vector indexes for similarity search. She describes this as support for a complete AI workflow, from exploring data to searching it and fine-tuning models. Multimodal concerns include the data types, the workloads, and the production or research use case.
"You should really care about what you're training on and you should care for it by giving it a nice format that does a lot of nice things for it."01:35
Who should watch
You are building a dataset platform for model training and need a repeatable way to materialize changing data.
Your training pipeline mixes metadata with embeddings, text, images, audio, or video and ordinary storage formats are slowing iteration.
You are evaluating LanceDB for a workflow that includes SQL analysis, vector search, full-text search, and model training.
