TL;DR
Flat-RLVR applies periodic sharpness-aware minimization to group-relative RLVR to preserve low-probability correct outputs. Small-model experiments report higher average pass@256 with mixed single-attempt results. The evidence supports a coverage–accuracy trade-off under specific sampling budgets, rather than uniformly better reasoning or cost-free scaling.
Why it matters
Optimizing final-answer rewards can concentrate probability on a narrower set of outputs. Flat-RLVR tests whether parameter-space regularization can reduce that concentration, connecting optimization geometry to multi-attempt coverage while exposing the importance of reporting pass@1 alongside large-k metrics.
Key findings
- 01
The method adds a SAM ascent/descent step periodically to GRPO-style optimization, with gradient centralization and a prompt-difficulty-modulated perturbation radius. The default SAM period is ten steps; the radius is cosine-annealed from 0.01 to 0.002 with a post-modulation floor of 0.001. A lightweight perturbation approximation reduces memory storage.
- 02
The main backbones are Qwen2.5-Math-1.5B and Llama-3.2-1B-Instruct, trained on 7,500 MATH problems with group size six. The study also tests Qwen2.5-3B-Instruct and Qwen2.5-Coder-1.5B-Instruct, and applies the approach to GRPO, DAPO and CISPO.
- 03
On Qwen2.5-Math-1.5B, Table 2 reports mean benchmark pass@256 of 67.3% for Flat-GRPO versus 64.8% for GRPO, a 2.5-percentage-point increase. Mean pass@1 falls from 15.6% to 15.2%. For Llama-3.2-1B, mean pass@256 rises from 48.4% to 50.4%, while mean pass@1 rounds to 6.5% for both.
- 04
The five mathematical benchmarks are MATH500, AMC23 and AIME24/25/26. Evaluation generates 256 samples per problem and uses the combinatorial pass@k estimator. Pass@256 is success on at least one of 256 attempts, not all-attempt reliability or a complete characterization of possible reasoning paths.
- 05
The 3B experiment reports average pass@256 of 61.4% versus 59.4%, alongside pass@1 of 18.2% versus 18.5%. On MBPP with the coding backbone, Flat-GRPO reaches 89.6% pass@256 versus 85.6%, but pass@1 decreases from 56.7% to 54.5%; its coverage also remains below the base model’s 91.5%.
- 06
The reported 1.5B timing comparison is 16.91 hours for GRPO, 16.87 for SAM every ten steps and 17.00 for every five steps. Shortening the SAM interval improves high-k coverage in the shown ablations but causes a larger single-attempt decline. These timings describe the tested implementation rather than an inherent absence of extra computation.
Evidence & limits
The tables do not support uniform dominance: some benchmark-level pass@256 values decline, and pass@1 can fall. The paper’s broad claim of a consistent advantage at every k should be read against the reported numerical trade-offs.
Tables 2 and 3 label uncertainty as 95% confidence intervals over three seeds but give different interval widths for repeated GRPO/Flat-GRPO point estimates. This note reports point estimates without resolving that inconsistency; precise significance claims require clarification.
Finite-budget pass@256 depends on the sampler, output length and verifier. The mathematical evaluation allows 1,024 tokens for MATH500/AMC23 and 2,048 for AIME. Higher observed coverage does not prove an expansion of the model’s complete reasoning capabilities.
The theoretical comparison requires local approximations and a cumulative correction-dominance condition. It is not an unconditional theorem that flatter parameters improve coverage for every RLVR training run.
The implementation uses an approximate perturbation reconstruction rather than storing the full original perturbation. Its small-radius argument depends on local gradient/curvature behavior; a chosen radius alone does not guarantee uniformly negligible approximation error.
Near-equal wall-clock times do not eliminate the additional gradient computation required by SAM. No timing uncertainty or broad hardware sweep is reported, and the 1B–3B experiments do not establish frontier-scale efficiency.
The auxiliary flatness measurement uses a fixed batch from the training dataset and chosen parameter-noise normalization. It is a diagnostic of that loss geometry, not independent held-out evidence of causally improved generalization.
Scaling dimensions
Models, methods & benchmarks
- Models
- Qwen2.5-Math-1.5B, Llama-3.2-1B-Instruct, Qwen2.5-3B-Instruct, Qwen2.5-Coder-1.5B-Instruct
- Algorithms
- Flat-RLVR, SAM, GRPO, DAPO, CISPO
- Benchmarks
- MATH500, AMC23, AIME24, AIME25, AIME26, MBPP
Topics