Back to the Neural Thickets website

Iterative RandOpt

July 2026  ·  Yulu Gan, Phillip Isola

tldr; The RandOpt algorithm introduced in the Neural Thickets paper is a fully parallelizable, iteration-free method. In this blog, we introduce Iterative RandOpt, which loops through random guessing → ensembling the top-k models → distilling them into one model. We run Iterative RandOpt with OLMo-3-7B on GSM8K, where it achieves 92.9% accuracy, outperforming PPO, GRPO, and ES under a matched compute budget, while still being deployed as a single model.

Code: https://github.com/sunrainyg/RandOpt/tree/iterative-randopt

What is RandOpt?

RandOpt is a fully parallelizable, iteration-free method for training LLMs. The algorithm is as follows:

1. Sample Gaussian noise and add it to the model weights.
2. Select top-k perturbed models that perform well on a given task.
3. Ensemble the top-k models by majority vote at inference.

RandOpt: Random Guessing in Weight Space + Ensemble

What is Iterative RandOpt, and why iterative?

One limitation of RandOpt for practical use is ensembling: voting over the top-k models requires k times more forward passes compared with GRPO/PPO/ES. Distill top-k models into single model fixes this (see Sec. 7 in the neural thickets paper). Since the distilled model is now a better base than the one you started with, you can search around it again. RandOpt becomes a loop:

1. Sample Gaussian noise and add it to the base model θ_t.
2. Select the top-k perturbed models that perform well on a given task.
3. Distill the top-k models (teachers) into the model θ_t (student) and get the distilled model θ_t+1.
4. Repeat, using θ_t+1 as the new base model.

How does it relate to PPO, ES, and on-policy distillation?

1. PPO/GRPO - sample completions from the current policy, score them with a reward function, and update the policy toward the high-scoring outputs.
2. On-policy distillation — sample completions from the current policy, let a stronger fixed teacher label or grade them, and distill those signals back into the same model.
3. Evolution strategies — sample perturbations in weight space, score the perturbed models, and move the weights toward the high-scoring perturbations.
4. Iterative RandOpt — sample perturbations in weight space, keep the top-k models, distill them back into one model, and repeat

All four methods above are combinations of the same two choices:

1. search in weight space vs search in action space
2. reward-weighted update vs distillation

Note that the distillation phase of iterative RandOpt is not a new objective. It is a policy-gradient update with a 0–1 reward: a trace gets reward 1 if it came from a selected model and 0 if it came from a rejected one, and we do maximum likelihood on the reward-1 traces. That family is well studied (rejection-sampling fine-tuning; RL against a binary verifier), and we don't mean to reinvent or rebrand it. What is new is that 1) the candidates come from weight-space search; 2) distillation supervises every step of the trace, whereas policy gradient update with 0-1 reward does not have stepwise supervision. In our experiments, this version outperforms standard PPO/GRPO/ES under a matched compute budget.

Experimental Results

Code

The code can be run on a single 8-GPU node with a pip install-able package that drops into existing verl / TRL pipelines without disturbing your pinned dependencies. Code, configs: github.com/sunrainyg/RandOpt @ iterative-randopt.

Citation

Gan, Yulu, and Phillip Isola. “Iterative RandOpt”. Neural Thickets (Jul 2026). https://thickets.mit.edu/blogs/iter_randopt.html

Or use the BibTex citation:

@article{gan2026iterativerandopt,
 title = {Iterative RandOpt},
 author = {Gan, Yulu and Isola, Phillip},
 journal = {github.com/sunrainyg/RandOpt},
 year = {2026},
 month = {July},
 url = "https://thickets.mit.edu/blogs/iter_randopt.html"
}

@misc{gan2026neuralthicketsdiversetask,
 title = {Neural Thickets: Diverse Task Experts Are Dense Around Pretrained Weights},
 author = {Yulu Gan and Phillip Isola},
 year = {2026},
 eprint = {2603.12228},
 archivePrefix = {arXiv},
 primaryClass = {cs.LG},
 url = "https://arxiv.org/abs/2603.12228"
}

References

[1] Williams, Ronald J. “Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning.” Machine Learning, 8:229–256, 1992.

[2] Schulman, et al. “Proximal Policy Optimization Algorithms.” arXiv preprint arXiv:1707.06347, 2017.

[3] Shao, et al. “DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models.” arXiv preprint arXiv:2402.03300, 2024.

[4] Agarwal, et al. “On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes.” ICLR 2024.

[5] Yang, et al. “Qwen3 Technical Report.” arXiv preprint arXiv:2505.09388, 2025.

[6] Thinking Machines Lab. “On-Policy Distillation.” Thinking Machines Lab: Connectionism, 2025.

[7] Shenfeld, et al. “Self-Distillation Enables Continual Learning.” arXiv preprint arXiv:2601.19897, 2026.

[8] Zelikman, et al. “STaR: Bootstrapping Reasoning With Reasoning.” NeurIPS 2022.

[9] Gulcehre, et al. “Reinforced Self-Training (ReST) for Language Modeling.” arXiv preprint arXiv:2308.08998, 2023.

[10] Dong, et al. “RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment.” TMLR 2023.

[11] Singh, et al. “Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Models.” TMLR 2024.

[12] Xiong, et al. “A Minimalist Approach to LLM Reasoning: from Rejection Sampling to Reinforce.” arXiv preprint arXiv:2504.11343, 2025.

[13] Sehnke, et al. “Parameter-Exploring Policy Gradients.” Neural Networks, 23(4):551–559, 2010.

[14] Salimans, et al. “Evolution Strategies as a Scalable Alternative to Reinforcement Learning.” arXiv preprint arXiv:1703.03864, 2017.

[15] Qiu, et al. “Evolution Strategies at Scale: LLM Fine-Tuning Beyond Reinforcement Learning.” arXiv preprint arXiv:2509.24372, 2025.

[16] Gan and Isola. “Neural Thickets: Diverse Task Experts Are Dense Around Pretrained Weights.” arXiv preprint arXiv:2603.12228, 2026.