TL;DR
SEA-LION v4.8 adapts Nemotron models to Southeast Asian languages through continued pretraining, supervised fine-tuning, online on-policy distillation, and model merging. The report connects regional language performance to both data coverage and tokenizer constraints.
Why it matters
A concrete model-development report shows how OPD fits into a broader adaptation pipeline. Its language-specific results also illustrate why scaling training data alone may not resolve weak tokenization or missing language coverage.
Key findings
- 01
The family includes 30B-total/3B-active and 120B-total/12B-active models with base and post-trained checkpoints. Post-training interleaves SFT and online reverse-KL distillation from a Nemotron-3-Ultra-550B-A55B teacher, followed by merging with the reported Nemotron checkpoints.
- 02
The asynchronous infrastructure separates environments, serving, teacher scoring, and optimization. It captures exact generated tokens and masks tokens beyond its staleness window, connecting the learning objective to rollout and weight-update scheduling.
- 03
Table 5 reports a 30B SEA-HELM score of 51.57, compared with 46.89 for Nemotron 3 Nano and 46.06 for Nemotron 3.5 Lightning. At 120B, the score rises from Nemotron 3 Super’s 49.30 to 63.44. These are aggregate benchmark scores, not raw success percentages.
- 04
The evaluation covers seven Southeast Asian languages and uses eight generation runs per model, with prompt bootstrap confidence intervals. The larger model shows broader gains, but individual tasks and languages still exhibit trade-offs.
Evidence & limits
The pipeline combines continued pretraining, SFT, OPD, and merging. Its overall gains do not isolate the causal contribution or compute efficiency of OPD alone.
The retained Nemotron tokenizer is inefficient for several regional scripts, and the continued-pretraining mixture lacks a dedicated Filipino component. More total data does not automatically solve either limitation.
Scores are the report’s September 15 benchmark snapshot and cover seven languages, not the region’s full linguistic diversity. Improvement in the aggregate does not imply improvement on every capability.
The abstract’s 30B comparison starts at 46.06, while the main text also compares against 46.89. Table 5 assigns these to different baselines; this note names both explicitly rather than mixing them.
Scaling dimensions
Models, methods & benchmarks
- Models
- Nemotron-SEA-LION-v4.8-30B-A3B, Nemotron-SEA-LION-v4.8-120B-A12B, Nemotron-3-Ultra-550B-A55B
- Algorithms
- Continued pretraining, Supervised Fine-Tuning, On-Policy Distillation, Model merging
- Benchmarks
- SEA-HELM
Topics