A long-term research initiative at MIT

Neural Thickets Mapping the Dense Landscape of Task Experts in Pretrained Models

Neural Thickets reframes pretraining as a local distribution over weights: in large, well-pretrained models, many nearby parameter vectors are already task-improving experts, so simple random sampling, top-K selection, and majority-vote ensembling can rival post-training methods like PPO and GRPO.

Founding Paper arXiv GitHub Get Involved

θ̃ = θ + σε ε ~ N(0, I) select top-k θ̃ᵢ output = vote(θ̃₁ … θ̃ₖ)

Small model: Needle in a Haystack — Small Model — "Needle in a Haystack" Gemini

Large model: Neural Thicket — Large Model — "Neural Thicket" Gemini

Core Method

RandOpt Pipeline

The foundational algorithm behind Neural Thickets: a zero-training approach that discovers task-specific experts by perturbing pretrained weights and ensembling the best candidates.

Perturb

Add Gaussian noise: θ̃ = θ + σε

Evaluate

Score each candidate in parallel across GPUs

Select

Keep top-k by task performance

Ensemble

Aggregate via majority vote

Research Thrusts

Exploring the Thicket

Neural Thickets opens multiple interconnected research directions—from theoretical foundations to practical applications. Each thrust feeds back into the others.

Thrust 1 · Foundations

Geometry of the Loss Landscape

Why are task experts so densely distributed near pretrained weights? We characterize the local geometry, connectivity, and mode structure of the weight-space thicket.

Theory Empirics Visualization

Thrust 2 · Scaling

Scaling Laws for Thicket Density

How does thicket density change with model size, data volume, and pretraining compute? We derive and validate scaling laws that predict when thickets become exploitable.

Paper Benchmarks

Thrust 3 · Applications

Training-Free Adaptation

Deploying models to new tasks without any gradient updates. We explore applications in low-resource languages, medical imaging, and on-device personalization.

NLP Vision Multimodal

Thrust 4 · Efficiency

Efficient Search Strategies

Random search is just the beginning. We are developing evolutionary, gradient-guided, and structured perturbation methods to navigate thickets orders of magnitude faster.

Algorithms Systems

Thrust 5 · Theory

Connections to Ensemble & Bayesian Methods

Thicket ensembles resemble posterior sampling. We formalize the link to Bayesian deep learning, uncertainty quantification, and model diversity theory.

Theory Uncertainty