RLVR that rewards red teaming the training environment

TL;DR

A LessWrong proposal to let policies report exploitable RLVR environment bugs after a rollout, reward high-quality verified reports, and use them to patch the training environment.

Why it matters

Reward hacking is partly an environment-maintenance problem. The proposal asks whether the policy's own search can be redirected from silently exploiting a bug toward making the bug legible and repairable.

Key findings

1. 01

   The proposal separates ordinary task reward from a post-task channel for submitting environment bug reports.

2. 02

   The discussion highlights hard design questions around cheaply verifying reports, balancing rewards, avoiding instability, and handling monitors weaker than the policy.

3. 03

   The author explicitly presents this as an untested idea rather than empirical evidence.

Scaling dimensions

Models, methods & benchmarks

Algorithms

RLVR, Environment red teaming

Topics