(づ•ᴥ•)づ┬─┬

Notes on math and machine learning.

Local theory for neural thickets

Cross-posted this to alphaxiv.

I read the Gan, Isola 2026 paper on neural thickets. If we start from a pre-trained model, we can use fine-tuning to convert the model into a task-expert, e.g., a model better at math questions, by following gradients w.r.t some training set. What Gan and Isola claimed is that some large models have some of those task experts relatively close in the weight space -- close enough that they can be reached through random search!

image Image is from the Gan, Isola paper.

What is of interest here:

In this post, I wanted to explore how a local theory of thickets would look like and how much it would predict:

What we show is that in the local regime, we can predict the answers by using the per-task gradient and Hessian. On six small LLMs, this prediction holds for small σ, although the gains are small.

Local improvement theory

We will denote the parameters of a model by θ∈ℝD.

We have T tasks, defined in terms of losses lt(θ), where lt(θ):=lt*+(θ−θt)THt(θ−θt)/2,where Ht positive semi-definite and symmetric and θt is a minimum of lt. Next, we also define the combined loss L(θ):=1T∑ilt(θ).

At θ0, gt:=∇lt(θ0)=Ht(θ0−θt)=Htdt. This is not zero at θ0 except if θt=θ0. Typically θ0 represents a compromise between tasks, so this is expected.

Perturbations: we assume θ=θ0+σu, where σ>0 and u~N(0,I).

What we are interested in is the difference in loss if we use θ instead of θ0 for task t:

Δlt=lt(θ)−lt(θ0)=σgt⊤u+12σ2u⊤Htu,

or we may talk about the "improvement", i.e., Δit(u):=−Δlt(u).

The average perturbation hurts: 𝔼[Δlt]=1/2σ2tr(Ht)>0 (here we used 𝔼[uTHtu]=tr(Ht)).

Best-of-N improvement

The idea is that we sample u~N(0,I) and we want to pick the best perturbations with respect to −Δlt(u). This corresponds to tracking what the best expert can do.

Inner product with MV IID Gaussian gives univariate Gaussian: u~N(0,I) and gt is a constant vector, gt⊤u=‖gt‖z,z~N(0,1).

Trace approximation: 𝔼[uTHtu]=tr(Ht). Assuming D is large enough and the columns of Ht are spread out enough, a plausible approximation is uTHtu≈tr(Ht).

Using the trace approximation decouples the second term from u! We can write the improvement as:

Δit(z)=−σ‖gt‖z−12σ2tr(Ht).

To get the best improvement, we sample N Z~N(0,1) and pick the one that gives the best improvement, i.e., the largest negative z in the sample. We can ballpark that as −maxiZi≈−2logN to get:

σ‖gt‖2logN−12σ2tr(Ht).

This also allows us to solve for the best σ:

σ*=‖gt‖tr(Ht)2logN,

which finally gives the best improvement:

Δit=logN‖gt‖2tr(Ht).

As a reminder, N is the number of samples we get from N(0,I), gt=∇lt(θ0) and Ht is the Hessian at θ0. In practice, we can estimate tr(Ht) by using an estimator of 𝔼[uTHtu].

Experiments

The best-of-N improvement depends on the model only through two numbers, ‖g‖ and tr(H). Those don't depend on the perturbations.

Setup.

The plot. The six models differ a lot:

To compare them on one plot, we can rescale each point.

image

For σ≤σ*/2, we see close agreement with the curve. This includes models whose σ* differ by more than a factor of 10, so it's not one lucky fit. Beyond that we lose agreement as higher order terms start to matter more.

To be clear, the gains are small. The best perturbation raises the probability the model gives to each correct token by about 2% (Qwen2.5-0.5B), and by up to about 10% for the other models.

Does it carry over to new problems? For Qwen2.5-0.5B I repeated the experiment with 128 selection problems and 128 fresh held-out problems. The perturbation that won on the selection set keeps 84–92% of its improvement on the held-out set. The reason is that the gradients of the two sets point almost the same way: cos(gA,gB)=0.99.

What about accuracy? The loss here is NLL. As it is smooth, it is expected that for small enough σ local theory applies well. Measuring accuracy under each perturbation means generating fresh answers from every perturbed model. I didn't do this with LLMs due to cost + noise, but on a toy problem accuracy was also predicted.

Caveats.

Neural Thickets

Gan and Isola's paper got a fair amount of follow-up. The two I found most useful:

Evolution strategies

The local theory in this post is not new. The expected improvement from picking the best of N random perturbations on a quadratic was worked out in the evolution strategies literature, most generally by Akimoto, Auger & Hansen, building on Beyer's The Theory of Evolution Strategies (2001). What I add is measuring the two things that theory needs, the gradient norm and the Hessian trace, on actual LLMs.

For ES on LLMs, see Qiu et al. and Liang et al.. Liang et al. argue that the curvature of LLM fine-tuning landscapes lives in a few directions, which is why small ES populations work.

  1. see https://www.ethanepperly.com/index.php/2024/01/28/dont-use-gaussians-in-stochastic-trace-estimation/↩

  2. For comparison, Gan & Isola use σ ∈ {1,2,3}×10⁻³; at their population size our σ* for Qwen2.5-0.5B and 1.5B is 1.3–2×10⁻³.↩