The Change Of Variable Formula and the Gaussian Integral

November 30, 2022

In this post, I present a simple way to calculate the Gaussian Integral, that is a very appealing application of the change of variable formula.

The Gaussian Integral

The Gaussian Integral is a very important result that gives the value of the integral $$I=\int\limits_{-\infty}^{\infty} e^{-t^2} dt=\sqrt{\pi}$$.

This integral is famous since it relates two important constants $e$ and $\pi$. Moreover, it is widely used for instance in probability theory, as it is the partition function of the standard Normal Distribution $\mathcal{N}(0,1)$.

However, it can be shown that it is impossible to integrate this using simple functions. In this article I will describe how a simple trick allows to calculate it in a few lines of derivations.

The Jacobian and the Jacobian Matrix

Let $\mathcal{U}$ and $\mathcal{V}$ be two open subsets of $\mathbb{R}^n$ and $\mathbb{R}^m$ respectively and $\phi:\mathcal{V}\rightarrow\mathcal{U}$ be a differentiable map between them.

$\phi$ maps every vector $v=(v_1,…,v_m)\in \mathcal{V}$ to a unique vector $\phi(v)=u=(u_1,…,u_n)\in \mathcal{U}$.

The Jacobian Matrix of $\phi$ at point $v$, denoted $J_{\phi}(v)$ is the matrix whose entries $J_{\phi}^{i,j}(v)$, is the infinitesimal difference on the $i$-th output $u_i$ of $\phi$ obtained by applying an infinitesimal change to the $j$-th output $v_j$.

In mathematical terms we have $J_{\phi}(v) = (\frac{\partial u_i}{\partial v_j})_{i,j}$.

The Jacobian is the determinant of the Jacobian Matrix, i.e. $Jac_\phi(v) =det(J_{\phi}(v))$. The Jacobian has an important geometrical interpretation.

Indeed, the $j$-th column of the Jacobian Matrix contains the coordinates of the image of the canonical basis vectors $e_1,…,e_j$ of $\mathbb{R}^m$ through a first order approximation of $\phi$ around $v$. The images of these basis vector form a signed hypervolume in the space $\mathbb{R}^n$, whose value is given specifically by $det(J_{\phi}(v))$.

Thus, the Jacobian tells us by how much the unit volume of hypercube in $\mathbb{R}^m$ is stretched when passed through the linear transformation $J_{\phi}(v)$.

The Change of Variable Formula.

Let’s suppose that we want to cal.culate an integral of a real-valued continuous function $f$ over an open set $\mathcal{U}$ of a measured space $(\mathcal{X}, \mu)$ : $$\int\limits_{u\in\mathcal{U}} f(u)\mu(du),$$ but that there’s no easy way to do this due to a discrepancy between the structure of $\mathcal{U}$ and the form of $f$.

Suppose moreover that there exists another open set $\mathcal{V}$ on a space $(\mathcal{Z}, \nu)$ and a continuously differentiable map $\phi:\mathcal{V}\rightarrow\mathcal{U}$ between the two sets.

Then we have

$$ \int\limits_{u\in\mathcal{U}} f(u)\mu(du) = \int\limits_{v\in\mathcal{V}} f\circ\phi(v) Jac_\phi(v) \nu(dv) $$

In this expression, $Jac_{\phi}(v)$ is the Jacobian of $\phi$ around $v$.

This tells us that we can compute the integral in the space $\mathcal{Z}$ if it is simpler, provided that we rescale the differential by the Jacobian.

Example: the polar coordinates

The polar coordinates system allows describing points in $\mathbb{R}^2$ not as a pair of Cartesian coordinates but instead using a radius and an angle.

It is represented by the transformation $\phi:(u,v)\mapsto (u.cos(\theta), v.sin(\theta))$, or its inverse transformation $\psi:(x,y)\mapsto(\sqrt{x^2+y^2},arctan(\vert\frac{y}{x}\vert))$

The Jacobian of $\phi$ has the very simple form $Jac_{\phi}(u,\theta) = u$.

For any integrable function $f :\mathbb{R}^2 \rightarrow \mathbb{R}$ such that the integral $\int\limits f(x,y)dxdy$ exists, the following holds:

$$ \int f(x,y)dxdy =\int f\circ \phi(u,\theta) u du d\theta $$

Intuitively, if we want to use the polar coordinates instead of the Cartesian ones, we just need to scale the differential $dxdy$ by a factor $u$.

One can think of $dxdy$ (resp. $du d\theta$) as the volume of the parallelogram obtained from the pair of vectors $(dx \vec{e_x}, dy \vec{e_y})$ (respectively $(du \vec{e_u}, d\theta \vec{e_\theta})$), where $\vec{e_{.}}$ are the unit vectors of the canonical basis of each coordinate spaces.

Virtually, the change of variable formula tells us how these volumes relate to each other, provided that we know a differentiable transformation going from one space to the other.

Application to the Gaussian Integral.

To apply the previous result to the Gaussian Integral $I=\int\limits_{-\infty}^{\infty} e^{-t^2} dt$, we will look instead at a similar integral but in dimension 2:

$$ J=\int\limits_{-\infty}^{\infty}\int\limits_{-\infty}^{\infty} e^{-(x^2+y^2)} dx dy. $$

Surprisingly, this integral will show to be easier to calculate. Moreover, using Fubini’s theorem, we can easily show that

$$ J =\int\limits_{-\infty}^{\infty}\int\limits_{-\infty}^{\infty} e^{-(x^2+y^2)} dx dy =\int\limits_{-\infty}^{\infty} e^{-x^2} dx \int\limits_{-\infty}^{\infty}e^{-y^2}dy = I^2 $$

Now in order to calculate it, we first realize that if we denote $S_R =[-R,R]^2$ the square of size $R$ for any $R>0$, and $J_R= \int\limits_{S_R} e^{-(x^2+y^2)} dx dy$, we have:

$$ J= lim_{R \rightarrow \infty} J_R $$

Moreover, since the diagonal of the square $S_R$ has length $\sqrt{2} R$, if we denote $D_R = \{ (x,y)\vert x^2+y^2 \leq R^2\}$ the disk of radius $R$, we have that $D_R \subset S_R \subset D_{\sqrt{2}R}$.

Hence, denoting $K_R= \int\limits_{D_R} e^{-(x^2+y^2)} dx dy$, we have

$$K_R \leq J_{R} \leq K_{\sqrt{2}R}.$$

Now we will show that $K_R \rightarrow \pi$ when $R \rightarrow \infty$, as a consequence, $J_R$ will also tend to $\pi$ since it is surrounded by expressions that tend to $\pi$.

This is where the change of variable formula comes at play. Indeed, we will express the integral in polar coordinates. Indeed, denoting $(x,y)=\phi(u,\theta) = (u.cos(\theta), u.sin(\theta))$ and $f(x,y)=e^{-(x^2+y^2)}$ we have that $$\int f(x,y) dxdy = \int f\circ \phi (u,\theta) Jac_{\phi}(u,\theta) du d\theta$$

Thus:

$$ \begin{aligned}K_R &= \int\limits_{D_R} e^{-(x^2+y^2)} dx dy \\ &= \int\limits_{[0,R[\times [0,2\pi[} e^{-r^2} r dr d\theta \\ &= \frac{1}{2}\int\limits_{[0,\sqrt{R}[\times [0,2\pi[} e^{-s} ds d\theta\\ &= \pi (1-e^{-R})
\end{aligned} $$

So $K_{R}\rightarrow \pi$ when $R \rightarrow \infty$.

This shows that $I=\sqrt{\pi}$

Conclusion

We have seen that the change of variable formula is a powerful tool to calculate integrals that don’t have closed forms. The Jacobian allows to express the ratio between the infinitesimal volume used in the integrals. This technique can be applied to calculate seemingly complicated integrals in a few lines. This change of variable formula has been used extensively in Machine Learning, especially in the reparametrization of distributions in Representation Learning.

For instance Normalizing Flows or Variational Autoencoders rely heavily on it.