Suppose we have random variables $X_1, \cdots, X_n$, and a real-valued function $U(X_1, \cdots, X_n)$. In this chapter, we're only going to do one thing: introduce multiple methods to find the probability distribution of $U$. Any one of these methods can be employed to find the distribution , but usually one of the methods leads to a simpler derivation than the others. The "best" method varies from one application to another.

## 5.1 The method of distribution functions

The first way of finding the distribution of $U$ is using the definition directly.

$F_U(a) = P\{U \leq a\} = P\{ U(X_1, \cdots, X_n) \leq a \}$

We can work this out in the following steps:

1. Write out the distribution function of $U$: $F_U(a) = P\{ U(X_1, \cdots, X_n) \leq a \}$.
2. Find the region of $X_1, \cdots, X_n$ such that $U(X_1, \cdots, X_n) \leq a$ and denote the region as $D$.
3. Integrate $f(X_1, \cdots, X_n)$ over $D$.

$F_U(a) = \int\int\limits_D \cdots \int f(x_1, \cdots, x_n)dx_1 \cdots dx_n$

The hardest part of this method is finding the set $D$. We'll gain some insight with some examples.

### Example 5.1.1

A company is selling sugar online. Suppose the amount of sugar it can sell per day is $Y$ tons, which is a continuous random variable with density function defined as

$f_Y(y) = \begin{cases} 2y, & 0 \leq y \leq 1 \\ 0, & \text{otherwise}\end{cases}$

For each ton of sugar sold, the company can earn $\$300$. The daily opeartion cost is$\$100$. Find the probability distribution of the daily income of this company.

Solution: Let the random variable $X$ denote the daily profit in hundred dollars. We want to write $X$ as $g(Y)$ and find $F_X(a)$.

\begin{aligned} X &= 3Y - 1 \\ F_X(a) &= P\{X \leq a\} \\ &= P\{ 3Y - 1 \leq a \} \\ D &= \{Y: 3Y - 1 \leq a\} \\ &= \left\{Y \leq \frac{a+1}{3}\right\} \\ F_X(a) &= \int\limits_{Y \leq \frac{a+1}{3}}f_Y(y)dy \\ &= \int_{-\infty}^\frac{a+1}{3} f_Y(y)dy\end{aligned}

If $\frac{a+1}{3} < 0$, $f_Y(y) = 0 \Rightarrow F_X(a) = 0$. If $\frac{a+1}{3} > 1$, $f_Y(y) = 1 \Rightarrow F_X(a) = 1$. When $0 \leq \frac{a+1}{3} \leq 1$,

\begin{aligned} F_X(a) &= \int_0^\frac{a+1}{3} f_Y(y)dy \\ &= \int_0^\frac{a+1}{3} 2ydy \\ &= y^2 \bigg|_0^\frac{a+1}{3} \\ &= \frac{(a+1)^2}{9}\end{aligned}

Note that as $y$ ranges from $0$ to $1$, $u$ ranges from $-1$ to $2$, so the distribution function of $X$ is

$F_X(a) = \begin{cases} 0, & a < -1 \\ \frac{(a+1)^2}{9}, & -1 \leq a \leq 2 \\ 1, & a > 2\end{cases}$

The density function can also be calculated:

$f_X(x) = \frac{dF_X(a)}{da} = \begin{cases} 0, & a < -1 \text{ or } a > 2 \\ \frac{2}{9}(a+1), & -1 \leq a \leq 2\end{cases}$

### Example 5.1.2

Suppose $Y_1$ and $Y_2$ are two continuous random variables with joint density function

$f(y_1, y_2) = \begin{cases} 3y_1, & 0 \leq y_2 \leq y_1 \leq 1 \\ 0, & \text{otherwise}\end{cases}$

Find the density function of $U = Y_1 - Y_2$. Also use the density function of $U$ to find $E[U]$.

Solution: We first need to find $F_U(a)$ and use it to obtain the density function $f_U(u)$.

$F_U(a) = P\{U \leq a\} = P\{Y_1 - Y_2 \leq a\}$

Now we need to find the region of $y_1$ and $y_2$. We know that $y_1 - y_2 \leq a$ and $0 \leq y_2 \leq y_1 \leq 1$, and $a$ is bounded between $0$ and $1$ due to the latter condition.

We can find the integral by subtracting the lower-right red triangle region ($P\{U \geq a\}$) from the entire red triangle.

\begin{aligned} D &= \{y_1, y_2: 0 \leq y_1 \leq 1, 0 \leq y_2 \leq y_1\} \\ F &= \{y_1, y_2: a \leq y_1 \leq 1, 0 \leq y_2 \leq y_1 - a \} \\ F_U(a) &= \iint_D f(y_1, y_2)dy_1dy_2 - \iint_F f(y_1, y_2)dy_1dy_2 \\ &= 1 - \int_a^1 \int_0^{y_1 - a} f(y_1, y_2)dy_2 dy_1 \\ &= 1 - \int_a^1 dy_1 \int_0^{y_1 - a}3y_1 dy_2 \\ &= 1- \int_a^1 3y_1(y_1 - a) dy_1 \\ &= 1 - \left( y_1^3 - \frac{3}{2}ay_1^2 \right)\bigg|_a^1 \\ &= 1 - \left( 1 - \frac{3}{2}a - a^3 + \frac{3}{2}a^3 \right) \\ &= \frac{a}{2}\left(3-a^2\right),\qquad 0 \leq a \leq 1\end{aligned}

So the distribution function of $U$ is

$F_U(a) = \begin{cases} 0, & a < 0 \\ \frac{a}{2}\left(3-a^2\right), & 0 \leq a \leq 1 \\ 1, & a > 1\end{cases}$

Calculating the density function of $U$ is now straightforward:

\begin{aligned} f_U(a) &= \frac{dF_U(a)}{da} = \frac{d\left[ \frac{3}{2}a - \frac{1}{2}a^3 \right]}{da} \\ &= \frac{3}{2} - \frac{3}{2}a^2 \\ &= \frac{3}{2}(1-a^2), \qquad 0 \leq a \leq 1\end{aligned}

$f_U(u) = \begin{cases} \frac{3}{2}\left(1-u^2\right), & 0 \leq u \leq 1 \\ 0, & \text{otherwise}\end{cases}$

The expectation of $U$ can be calculated as

\begin{aligned} E[U] &= \int_0^1 u \cdot \frac{3}{2}(1-u^2)du \\ &= \int_0^1 \frac{3}{2}(u - u^3)du \\ &= \frac{3}{2}\left[ \frac{u^2}{2} - \frac{u^4}{4} \right] \bigg|_0^1 \\ &= \frac{3}{2} \cdot \frac{1}{4} = \frac{3}{8}\end{aligned}

## 5.2 Sum of independent random variables

An important application of the method of distribution functions is to calculate the distribution of $Z = X + Y$ from the distributions of $X$ and $Y$ when they are independent, continuous random variables.

The first few steps are the same as what's described above:

\begin{aligned} F_{X+Y}(a) &= P\{X+Y \leq a\} \\ &= \iint\limits_{X+Y \leq a} f_{X,Y}(x, y)dxdy \\ D &= \{x, y: -\infty < y < \infty, -\infty < x < a-y \} \\ F_{X+Y}(a) &= \iint_D f_{X, Y}(x, y)dxdy \\ &= \int_{-\infty}^\infty \int_{-\infty}^{a-y}f(x, y)dxdy\end{aligned}

Now the independence comes into place:

\begin{aligned} F_{X+Y}(a) &= \int_{-\infty}^\infty f_Y(y)dy \underbrace{\int_{-\infty}^{a-y}f_X(x)dx}_{P\{X \leq a-Y\}} \\ &= \int_{-\infty}^\infty F_X(a-y)f_Y(y)dy\end{aligned}

The distribution function of $F_{X+Y}(a)$ is called the convolution of $F_X(a)$ and $F_Y(a)$. By differentiating the above distribution function, we can find the density function of $X+Y$.

\begin{aligned} f_{X+Y}(a) &= \frac{d}{da}\left\{ \int_{-\infty}^\infty F_X(a-y)f_Y(y)dy \right\} \\ &= \int_{-\infty}^\infty \frac{d}{da}F_X(a-y)f_Y(y)dy \\ &= \int_{-\infty}^\infty f_Y(y)dy \left[ \frac{d}{da}F_X(a-y) \right] \\ &= \int_{-\infty}^\infty f_Y(y)dy \left[ \frac{dF_X(a-y)}{d(a-y)}\frac{d(a-y)}{da} \right] \\ &= \int_{-\infty}^\infty f_Y(y) f_X(a-y)dy\end{aligned}

### Example 5.2.1

If $X$ and $Y$ are two independent random variables both uniformly distributed on $(0, 1)$, calculate the probability density of $X+Y$.

Solution: We can directly apply the equation above

\begin{aligned} f_{X+Y}(a) &= \int_{-\infty}^\infty f_Y(y)f_X(a-y)dy \\ f_Y(y) &= f_X(x) = \begin{cases} 1, & 0 \leq x \leq 1 \text{ or } 0 \leq y \leq 1 \\ 0, & \text{otherwise} \end{cases} \\ f_{X+Y}(a) &= \int_0^1 1 \cdot f_X(a-y)dy\end{aligned}

We know that $0 \leq y \leq 1$ and $0 \leq a-y \leq 1$.

1. If $a \leq 0$, $a-y \leq 0$, then $f_X(a-y) = 0$.
2. If $0 < a \leq 1$, we also need $a-y \geq 0 \Rightarrow 0 \leq y \leq a$.
3. If $1 < a \leq 2$, we also need $a-y \leq 1 \Rightarrow y \geq a-1$.
4. If $a > 2$, $f_X(a-y) = 0$.

When $0 \leq a \leq 1$,

$f_{X+Y}(a) = \int_0^a f_X(a-y)dy = \int_0^ady = a$

When $1 \leq a \leq 2$,

$f_{X+Y}(a) = \int_{a-1}^1 f_X(a-y)dy = 2-a$

In summary,

$f_{X+Y}(a) = \begin{cases} a, & 0 < a \leq 1 \\ 2-a, & 1 < a \leq 2 \\ 0, & \text{otherwise}\end{cases}$

The sum of two uniform random variables is called a triangular random variable because of the shape of the density function above.

### Example 5.2.2

Suppose $X$ and $Y$ are two independent standard normal random variables. Find the density of $Z = X + Y$.

Solution: We have $X \sim N(0, 1) \perp Y \sim N(0, 1)$.

$f_X(x) = \frac{1}{\sqrt{2\pi}} e^{-\frac{x^2}{2}}, f_Y(y) = \frac{1}{\sqrt{2\pi}} e^{-\frac{y^2}{2}}$

\begin{aligned} f_{X+Y}(a) &= \int_{-\infty}^\infty f_X(a-y)f_Y(y)dy \\ f_X(a-y)f_Y(y) &= \frac{1}{2\pi}e^{-\frac{(a-y)^2}{2} - \frac{y^2}{2}} \\ &= \frac{1}{2\pi} \exp\left\{ -\frac{a^2 - 2ay + 2y^2}{2} \right\} \\ &= \frac{1}{2\pi} \exp\left\{ -\frac{4y^2 - 4ay + 2a^2}{4} \right\} \\ &= \frac{1}{2\pi} \exp\left\{ -\frac{(2y-a)^2 + a^2}{4} \right\} \\ &= \underbrace{\frac{1}{2\pi} \exp\left\{ -\frac{a^2}{4} \right\}}_{\text{constant w.r.t. }y} \exp\left\{ -\frac{(2y-a)^2}{4} \right\} \\ f_{X+Y}(a) &= \frac{1}{2\pi}\exp\left\{-\frac{a^2}{4}\right\} \int_{-\infty}^\infty \exp\left\{ -\frac{(2y-a)^2}{4} \right\}dy \\ &= C \cdot \exp\left\{-\frac{a^2}{4}\right\}\end{aligned}

Therefore $X+Y$ is a normal random variable with mean $0$ and variance $2$. Similar results of this example can be obtained for a more general case, which is a very important property of normal random variables.

#### Theorem

Let $X_i, i = 1, \cdots, n$ be a sequence of independent normal random variables with parameters $\mu_i$ and $\sigma_i^2$, then $\sum_{i=1}^n X_i$ is normally distributed with parameters $\sum_{i=1}^n \mu_i$ and $\sum_{i=1}^n \sigma_i^2$.

$\sum_{i=1}^n X_i \sim N\left( \sum_{i=1}^n \mu_i, \sum_{i=1}^n \sigma_i^2 \right)$

## 5.3 The method of transformations

We first consider the univariate case. Suppose $X$ is a continuous random variable with density function $f_X(x)$. To find

$Y = g(X) \Rightarrow F_Y(a),$

what we can do is to transform the condition back to a condition of $X$. We know that $g^{-1}(x)$ exists when the function $g(\cdot)$ is monotonic, i.e. $\forall x_1 \neq x_2, g(x_1) \neq g(x_2)$.

Given $g(x)$ is a monotonic function of $X$, $g(\cdot)$ maps every distinct value of $X$ to a dinstinct value of $y$. If $g(\cdot)$ is a monotonically increasing function,

\begin{aligned} F_Y(a) &= P\{g(X) \leq a\} = P\left\{ X \leq g^{-1}(a) \right\} = F_X\left( g^{-1}(a) \right) \\ f_Y(a) &= \frac{dF_Y(a)}{da} = \frac{dF_X(g^{-1}(a))}{dg^{-1}(a)} \cdot \frac{dg^{-1}(a)}{da} = f_X(g^{-1}(a))\frac{dg^{-1}(a)}{da}\end{aligned}

If $g(\cdot)$ is a monotonically decreasing function,

\begin{aligned} F_Y(a) &= P\{g(X) \leq a\} = P\left\{ X \geq g^{-1}(a) \right\} = 1- F_X\left( g^{-1}(a) \right) \\ f_Y(a) &= \frac{d[1-F_X(g^{-1}(a))]}{da} -\frac{dF_Y(a)}{da} = f_X(g^{-1}(a))\left[-\frac{dg^{-1}(a)}{da}\right]\end{aligned}

These two cases can be unified as follows using the difference in the sign of $\frac{dg^{-1}(a)}{da}$. If $g(X)$ is a monotonic function for all $x$ such that $f_X(x) > 0$,

$f_Y(a) = f_X\left( g^{-1}(a) \right) \left| \frac{dg^{-1}(a)}{da} \right|$

### Example 5.3.1

In Example 5.1.1, we defined the density function of $Y$ as

$f_Y(y) = \begin{cases} 2y, & 0 \leq y \leq 1 \\ 0, & \text{otherwise}\end{cases}$

and $U = g(Y) = 3Y-1$. Find $f_U(u)$ with the method of transformations.

Solution: When $y_1 < y_2$, $g(y_1) - g(y_2) = 3(y_1 - y_2) < 0$, so $g(\cdot)$ is a monotonically decreasing function. The inverse function is

$Y = g^{-1}(U) = \frac{U+1}{3}$

and the first-order derivative is

$\frac{dg^{-1}(u)}{du} = \frac{d\frac{u+1}{3}}{du} = \frac{1}{3}$

Now we can find the density function using the formula

\begin{aligned} f_U(u) &= f_Y(g^{-1}(u)) \left| \frac{dg^{-1}(u)}{du} \right| \\ &= f_Y\left( \frac{u+1}{3} \right) \cdot \frac{1}{3} \\ &= 2\left(\frac{u+1}{3}\right) \cdot \frac{1}{3}, \quad 0 \leq \frac{u+1}{3} \leq 1\end{aligned}

Clean this up a bit and we get

$f_U(u) = \begin{cases} \frac{2}{9}(u+1), & -1 \leq u \leq 2 \\ 0, & \text{otherwise}\end{cases}$

### Example 5.3.2

The transformation method can also be used in multivariate situations. Let random variables $Y_1$ and $Y_2$ have a joint density function

$f(y_1, y_2) = \begin{cases} e^{-(y_1 + y_2)}, & y_1 \geq 0, y_2 \geq 0 \\ 0, & \text{otherwise}\end{cases}$

Find the density function for $U = Y_1 + Y_2$.

Solution: We can prove that $Y_1 \perp Y_2$ and use the method described earlier to find the distribution for the sum of two independent random variables. We can also apply the method of transformations here.

If we fix $Y_1 = y_1$, we have $U = g(Y_1, Y_2) = y_1 + Y_2 = h(Y_2)$. From here we can consider this as a one-dimensional transformation problem.

\begin{aligned} Y_2 &= U - y_1 = h^{-1}(u) \\ f_{Y_1, U}(y_1, u) &= f_{Y_1, Y_2}\left[y_1, h^{-1}(u) \right] \left| \frac{dg^{-1}(u)}{du} \right| \\ &= e^{-(y_1 + u - y_1)} \cdot 1 \\ &= e^{-u}, \quad y_1 \geq 0, y_2 = u - y_1 \geq 0 \end{aligned}

$f_{U, Y_1}(u, y_1) = \begin{cases} e^{-u}, & 0 \leq y_1 \leq u \\ 0, & \text{otherwise}\end{cases}$

Using the joint density function of $Y_1$ and $U$, we can obtain the marginal density function of $U$:

\begin{aligned} f_U(u) &= \int_{-\infty}^\infty f_{U, Y_1}(u, y_1)dy_1 \\ &= \int_0^u e^{-u}dy_1 \\ &= ue^{-u}, \quad u \geq 0\end{aligned}

So our final answer is

$f_U(u) = \begin{cases} ue^{-u}, & u \geq 0 \\ 0, & \text{otherwise}\end{cases}$

### The multivariate case

As shown in the example above, when our problem is $U = g(X_1, X_2)$ and we want to find $f_U(u)$, the procedure is

1. Fix $X_1 = x_1$, and denote $U|_{X_1 = x_1} = h(X_2) = g(X_1 = x_1, X_2)$.
2. Calculate the joint density function of $X_1$ and $U$ using the formula $f_{X_1, U}(x_1, u) = f_{X_1, X_2}(x_1, h^{-1}(u))\left| \frac{dh^{-1}(u)}{du} \right|$.
3. Find the maarginal density of $U$ with $f_U(u) = \int_{-\infty}^\infty f_{X_1, U}(x_1, u)dx_1$.

### Example 5.3.3

Random variables $Y_1$ and $Y_2$ have a joint density function

$f(y_1, y_2) = \begin{cases} 2(1-y_1), & 0 \leq y_1 \leq 1, 0 \leq y_2 \leq 1 \\ 0, & \text{otherwise}\end{cases}$

Find the density function of $U = Y_1Y_2$.

Solution: We first fix $Y_1 = y_1$ for some $0 \leq y_1 \leq 1$. Then we consider the univariate transformation $U = h(Y_2) = y_1Y_2$ and get the joint density function for $Y_1$ and $U$.

\begin{aligned} U|_{Y_1 = y_1} &= g(Y_1 = y_1, Y_2) = y_1Y_2 = h(Y_2) \\ Y_2 &= h^{-1}(U) = \frac{u}{y_1}, \frac{dh^{-1}(u)}{du} = \frac{1}{y_1} \\ f_{Y_1, U}(y_1, u) &= f_{Y_1, Y_2}(y_1, h^{-1}(u))\left| \frac{dh^{-1}(u)}{du} \right| \\ &= 2(1-y_1) \cdot \frac{1}{y_1}, \quad 0 < y_1 \leq 1, 0 \leq y_2 = \frac{u}{y_1} \leq 1\end{aligned}

The joint density function of $Y_1$ and $U$ is

$f_{Y_1, U} = \begin{cases} \frac{2}{y_1}(1-y_1), & 0 < y_1 \leq 1, 0 \leq u \leq y_1 \\ 0, & \text{otherwise}\end{cases}$

and now we can obtain the marginal density of $U$.

\begin{aligned} f_U(u) &= \int_{-\infty}^\infty f_{Y_1, U}(y_1, u)dy_1 \\ &= \int_u^1 \frac{2}{y_1}(1-y_1) dy_1 \\ &= 2\int_u^1 \left( \frac{1}{y_1} - 1 \right)dy_1 \\ &= 2[\ln(y_1) - y_1]\bigg|_u^1 \\ &= 2[(0 - 1) - (\ln(u) - u)] \\ &= 2[u - \ln(u) - 1], \quad 0 \leq u \leq 1\end{aligned}

## 5.4 The method of moment generating functions

We know that for a random variable $X$, its associated moment generating functions $M_X(t)$ is given by $E\left[e^{tX}\right]$. The $k$-th moment can help us find $E[X^k] = M_X^{(k)}(0)$.

For random variables $X$ and $Y$, if both moment generating functions exist and $M_X(t) = M_Y(t)$, we have $F_X(a) = F_Y(a)$, i.e. $X$ and $Y$ have the same probability distribution.

### Example 5.4.1

Let $Z \sim N(0, 1)$ and $X = \sigma Z + \mu$. Show that $X \sim N(\mu, \sigma^2)$ using the method of moment generating functions.

Solution: Define $Y \sim N(\mu, \sigma^2)$. If we can show that $M_X(t) = M_Y(t)$, then the distribution of $X$ is identical to the distribution of $Y$.

\begin{aligned} M_Y(t) &= E\left[ e^{tY} \right] = \int_{-\infty}^\infty e^{ty}f_Y(y)dy \\ &= \exp\left\{ \frac{\sigma^2 t^2}{2} + \mu t \right\} \\ M_X(t) &= E\left[ e^{tX} \right] = E\left[ e^{t(\sigma Z + \mu)} \right] \\ &= E\left[ e^{t\mu}e^{(t\sigma)Z} \right] \\ &= e^{t\mu} E\left[ e^{(t\sigma)Z} \right] \\ &= e^{t\mu} M_Z(t\sigma)\end{aligned}

We know that $M_Z(t) = \exp\left\{ \frac{t^2}{2} \right\}$, so

\begin{aligned} M_X(t) &= e^{t\mu} e^{\frac{\sigma^2t^2}{2}} \\ &= \exp\left\{ \frac{\sigma^2t^2}{2} + \mu t \right\} = M_Y(t)\end{aligned}

The method of moment generating functions is also very useful for calculating the probability distributions for the sum of independent random variables.

Suppose $X$ and $Y$ are independent with moment generating functions $M_X(t)$ and $M_Y(t)$. If $U = X + Y$, then

\begin{aligned} M_U(t) &= E\left[e^{tU}\right] = E\left[e^{t(X+Y)}\right] \\ &= E\left[\underbrace{e^{tX}}_{g(X)} \cdot \underbrace{e^{tY}}_{h(Y)}\right] \\ &= E\left[e^{tX}\right]E\left[e^{tY}\right] = M_X(t) \cdot M_Y(t)\end{aligned}

In this special case, $X \perp Y \Leftrightarrow M_{X+Y}(t) = M_X(t) M_Y(t)$.

### Example 5.4.2

Find the distribution of $X+Y$ if $X$ and $Y$ are

1. independent Binomial random variables with parameters $(n, p)$ and $(m, p)$, respectively.
2. independent Poisson random variables with parameters $\lambda_1$ and $\lambda_2$, respectively.

Solution: In the first scenerio we have $X \sim Binomial(n, p)$ and $Y \sim Binomial(m, p)$. $M_{X+Y}(t) = M_X(t)M_Y(t)$ since $X \perp Y$. Recall that

$M_X(t) = \left(pe^t + 1 - p \right)^n,$

so the MGF of $X+Y$ is

$M_{X+Y}(t) = \left(pe^t + 1 - p \right)^{m+n}$

which tells us $X+Y \sim Binomial(m+n, p)$.

In the second case, we have $X \sim Poisson(\lambda_1)$ and $Y \sim Poisson(\lambda_2)$. Since

$M_X(t) = \exp\left\{ \lambda(e^t - 1) \right\},$

the MGF of $X+Y$ can be written as

\begin{aligned}M_{X+Y}(t) &= \exp\{\lambda_1(e^t-1)\} \cdot \exp\{\lambda_2(e^t-1)\} \\&= \exp\{(\lambda_1 + \lambda_2)(e^t-1)\}\end{aligned}

So $X+Y \sim Poisson(\lambda_1 + \lambda_2)$.