MiniMax M3 uses an index branch to select relevant blocks and a sparse branch that attends only to those blocks.
2
MSA adapts sparse attention to grouped-query attention by keeping token selection different across KV heads and retrieving blocks instead of individual tokens.
3
Training with text and vision from the first step preserves text capability more reliably than adding multimodal data after text pretraining.
Summary
Olive Song explains how MiniMax M3 combines a 1 million token context window, agentic coding abilities, and native multimodality. Its MiniMax Sparse Attention system uses a lightweight index branch to find high-scoring tokens, then has the main sparse branch attend only to the selected blocks. MiniMax designed this around grouped-query attention and GPU memory access rather than directly applying DeepSeek's DSA design. Keeping selections separate across KV heads preserves their diversity, while block-level retrieval reduces top-k and memory-access overhead. Song then compares ways to add vision to a language model. Adding multimodal data late can damage text capability, while mixing text and vision from the first training step produced little text degradation under the same text-token budget. Attention maps showed stronger interaction between visual and text tokens in the native approach. The model also uses 3D attention in its vision transformer so patches from different video frames can interact directly. Four frames gave the best performance and efficiency trade-off in their exploration.
MiniMax M3 combines agentic coding, long context, and native multimodality
Olive Song describes M3 as an open-weight model that combines three capabilities. It supports coding and agentic tasks such as automatic task decomposition and use inside coding agents. It has a 1 million token context window for long user interactions and full code bases. MiniMax also trained it with native multimodality, so visual understanding and language generation were learned together from the start. Song connects this combination to applications such as computer-use agents. She says the model achieved nine times faster prefill and 15 times faster decoding than full attention through MiniMax Sparse Attention.
MSA uses a lightweight index branch before sparse attention
MSA has two branches. The index branch performs a lightweight full-attention computation with four query heads and one key head. It is trained with a KL loss to match the attention scores of the main branch. The index branch identifies tokens with the highest attention scores. The main sparse branch then attends only to the selected subset instead of every token. The selected blocks determine which keys and values the main attention computation needs. This lets the model reduce attention work while keeping the main branch responsible for the actual attention output.
DeepSeek's DSA did not directly fit MiniMax's grouped-query architecture
Song says DeepSeek's DSA was designed around DeepSeek's own architecture, especially multi-query attention and a large head dimension. Applying it directly to grouped-query attention creates two problems. Multiple KV heads should provide diversity, but sharing one token selection can make them attend to the same tokens. The design also creates awkward GPU reads because grouped-query attention splits the same total dimension across several KV heads. Song gives the comparison of one contiguous 1 by 512 layout versus four separate 128-dimensional heads. The latter requires multiple independent reads and lowers the compute-to-memory access ratio.
Block-level retrieval makes sparse attention easier to run on GPUs
MiniMax changed MSA in two main ways. It removed the aggregation that would merge index-branch results into one selected-token set, allowing different KV heads to choose different tokens. It also replaced token-level selection with block-level retrieval. Blocks reduce the number of candidates, which lowers the cost of top-k computation and result merging. They also make memory access more hardware-friendly and improve the compute-to-memory access ratio. Song says the change did not introduce a significant performance drop. MiniMax also built kernels for training and inference, which she describes as the system component that makes the design practical.
Adding vision late can interfere with learned text capability
M3 was trained with text and vision understanding from step zero. Song compares this with adding multimodal data after the model has mostly converged during the pretraining decay stage. That later approach can interfere with learned text capability and cause degradation. MiniMax also tried introducing multimodal training before the decay stage, after the model had already seen text tokens. Those results were highly sensitive to the data mixture and hyperparameters, including learning rate. A smaller learning rate helped in one setting but did not generalize to another run. Native multimodal training from scratch almost did not hurt text capability under the same text-token budget.
Native multimodal training produces more integrated attention maps
Song's attention-map analysis showed a difference between the training strategies. In models that received multimodal training in a CPT-style stage, text tokens often paid little attention to visual tokens, leaving visual regions mostly dark in the attention map. In models trained natively with text and vision, the map showed more interaction between text and visual tokens, without a clear separation between the two. Song interprets this as evidence that the model fuses visual and textual information more naturally when multimodality is present from the beginning. The goal is to add vision while preserving the model's text capability.
3D attention lets visual patches interact across video frames
M3 processes images or video with a vision transformer, then inserts the resulting visual tokens into the language-model input in place of image placeholders. Song says 3D attention inside the ViT improves visual understanding. Standard ViT attention operates within each image or frame, so patches from different video frames cannot interact at that stage. With 3D attention, patches across frames can attend to one another directly, giving the model stronger temporal and cross-frame visual modeling. The improvement also transferred to some image benchmarks. In MiniMax's exploration, four frames gave the best trade-off between performance and efficiency.
"Instead of simply borrowing an existing sparse attention design, we re-think the architecture under the constraints of GQA and real system efficiency."04:32
Who should watch
You are implementing long-context attention and need to understand why block selection and GPU memory layout matter together.
You are adding vision to a language model and want concrete comparisons between late multimodal training and training from step zero.
You work on multimodal video models and want to see why cross-frame attention inside the vision transformer can help.
