Scale and Direction: Understanding Homogeneous Functions

30 Dec, 2025

TL;DR: This post collects useful facts about homogeneous functions—functions satisfying $f(ax)=a^{r}f(x)$. Key insight: they decompose into spherical behavior plus radial scaling. I also connect to two references in ML.

A function $f$ is (positively) r-homogeneous if $f(ax)=a^{r}f(x)$ for all $a\ge 0$ and $x\in X$, e.g., $X\subset {ℝ}^n$. We will use $H_r(X,Y)$ to denote such functions from $X$ to $Y$ and $H_r(X)$ for functions from $X$ to $ℝ$. We will be using $H_r$ when context is sufficient and $H$ when we are talking about homogeneous functions in general.

Supposing we work in a normed space with norm $\Vert .\Vert$, a fun little substitution is $x=\Vert x\Vert ·x/\Vert x\Vert$ which gives

In other words, an $H_r$ function can do interesting stuff on the unit sphere and then we apply a scale term that is independent of $f$ to get the value at $x$. Especially if $f\in H_0$ we get $f(x)=f(x/\Vert x\Vert ),$ i.e., $f$ is constant on rays.

If we are on ${ℝ}_{>0}\to ℝ$, then $f\in H_r$ means $f(x)=k{x}^r$. There’s also Euler’s theorem which gives some implications about the derivatives of $f$.

Another interesting property: if $x,y\in X$, $a\in {ℝ}^{*}$, and $f\in H_0$ and continuous, then

There are various ways to show this, e.g., as $f\in H_0$, we have

if $a$ is large, and continuity does the rest.

As building blocks

One may want to construct more complex functions by using $H$. There are some ways to do this:

Composition: If $f\in H_0$ and we get some generic $g$, then $g\circ f(ax)=g(f(ax))=g(f(x))$, so $g\circ f\in H_0$. However, for $f\circ g(ax)=f\circ g(x)$ we need $g\in H_1\cup H_0$.

Multiplication¹: If $f\in H_r$ and $g\in H_k$, then $f·g\in H_{r+k}$.

Activations and Linear Maps: If $A$ is some matrix, $c$ is a positive scalar, and $\sigma$ is ReLU, then:

and so we have shown that $f$ is positively 1-homogeneous. Accordingly, and as long as we use appropriate activation functions, all NNs without bias² are $H_1$.

Averages: All sorts of averages are also $H_1$. For example, if $c>0$, the arithmetic mean satisfies $g(c{x}_1,\dots ,c{x}_n)=c(x_1+\dots +x_n)/n=cg(x_1,\dots ,x_n)$.

These building blocks appear throughout deep learning. As a practical example:

Normalization and scale separation: Batch normalization provides a practical example of scale-direction separation. By normalizing inputs to unit variance (after centering), it removes scale information—similar to our decomposition $f(x)=\Vert x{\Vert }^{r}f(x/\Vert x\Vert )$ where scale ($\Vert x{\Vert }^r$) and direction ($f(x/\Vert x\Vert )$) are separated. The $ϵ$ stabilization term and learnable parameters mean BatchNorm is only approximately homogeneous, but it demonstrates how normalization relates to homogeneity in practice.

Related work

I haven’t done a particularly deep dive in the references for this, but one work that I enjoyed reading is from Merrill, W., et al.³. Among other things, the authors look into the approximate homogeneity of transformers with respect to their parameters, i.e., $f(x;c\theta )\approx c^{k}f(x;\theta )$. Transformers without bias terms are shown to be approximately $H_1$.