Measure theoretic basics of Probability

Imre Péter Tóth

September 24, 2013

Abstract

For the “Probability 1” course at CEU, Fall 2013. Occasionally changing draft, with many typos, mistakes, errors and inconsistencies. These notes are mostly copied from my earlier lecture notes on Mathematical Statistical Physics.

Contents

1 Probability warming-up 1 1.1 Measure, measure space, probability, probability space …. 1 1.2 Measurable functions, random variables and their distributions 7 1.3 Integral and expectation . . . . . . . . . . . . . . . . . . . . . 9

1.3.1 Integral, integrability . . . . . . . . . . . . . . . . . . . 9 1.3.2 Exchanging limits with integrals . . . . . . . . . . . . . 12

Introduction

Nothing of the material presented is my own result. In fact, it is by now considered “classical” or “standard knowledge” in Mathematics, and I will not attempt to give references to the original papers of the true authors. Instead, I will (when I get to it) cite some textbooks and lecture notes where the material is easily accessible, and which I myself use to reduce the number of false statements written.

1 Probability warming-up

1.1 Measure, measure space, probability, probability space

Definition 1.1 (σ-algebra). For a nonempty set Ω, a family F of subsets of ω (i.e. F ⊂ 2^{Ω}, where 2^{Ω} := {A : A ⊂ Ω} is the power set of Ω) is called a σ-algebra over Ω if

• ∅ ∈ F

• if A ∈ F, then A^{C} := Ω\A ∈ F (that is, F is closed under complement taking)

• if A_{1}, A_{2}, ⋯ ∈ F, then (∪_{i=1}^{∞}A_{i}) ∈ F (that is, F is closed under countable union).

Definition 1.2 (measurable space, measurable set). If Ω is a nonempty set and F is a σ-algebra over Ω, then the pair (Ω, F) is called a measurable space. The elements of F arer called measurable subsets of Ω.

Lemma 1.3. A σ-algerba is closed under finite intersection, countable union and finite union.

Proof. Homework.

Note that a σ-algebra is in general not closed under arbitrary intersection and union. For example, the Borel σ-algebra on the set ℝ of real numbers (see later) contains every 1-element subset of ℝ, but it does not contain every subset (a fact we will not prove).

Two trivial examples of σ-algebra:

Definition 1.4 (indiscrete σ-algebra). For a nonempty set Ω, the family of subsets F_{ind} = {∅, Ω} is called the indiscrete or trivial σ-algebra over Ω.

Definition 1.5 (discrete σ-algebra). For a nonempty set Ω, the family of subsets F_{discr} = 2^{Ω} (the entire power set) is called the discrete σ-algebra over Ω.

It is immediate from the definition that these are indeed σ-algebras over Ω.

Lemma 1.6. The intersection of any (nonempty) family of σ-algebras over the same Ω is also a σ-algerba over Ω. That is, if Ω is a nonempty set and F_{i} is a σ-algerba over Ω for every i ∈ I where I is any nonempty index set, then F := ∩_{i∈I}F_{i} is also a σ-algebra over Ω.

Proof. (trivial set algebra) By definition, ∅ ∈ F_{i} for every i ∈ I, and I is nonempty, so ∅ ∈ ∩_{i∈I}F_{i} = F. Similarly, if A ∈ F = ∩_{i∈I}F_{i}, then A ∈ F_{i} for every i ∈ I, so by definition Ω\A ∈ F_{i} for every i ∈ I, so Ω\A ∈ F. Finally, if A_{1}, A_{2}, ⋯ ∈ F, then A_{1}, A_{2}, ⋯ ∈ F_{i} for every i ∈ I, so by definition (∪_{k=1}^{∞}A_{k}) ∈ F_{i} for every i ∈ I, which means that (∪_{k=1}^{∞}A_{k}) ∈ F.

It is important to note that I being any nonempty set means in particular that it can well be a large set, having infinitely many, or even uncountably many, or possibly much more elements.

Corollary 1.7. If Ω is a nonempty set and H ⊂ 2^{Ω} is any family of subsets, then there exists a unique σ-algebra σ(H) over Ω, which is the smallest σalgebra containig H in the following sense:

• H ⊂ σ(H)

• If F is any σ-algebra over Ω with H ⊂ F, then F ⊂ σ(H).

Proof. The family

{F : F is a σ-algebra over Ω and H ⊂ F}

is nonempty, since it contains at least the discrete σ-algebra 2^{Ω}. Thus by the above lemma,

σ(H) := ∩{F : F is a σ-algebra over Ω and H ⊂ F}

will do. Uniqueness also follows from the lemma: if there were two diferent such minimal σ-algebras, their intersection would also be a σ-algebra, but it would not contain them – a contradiction.

Definition 1.8 (σ-algebra generated by a family of sets). The above σ(H) is called the σ-algebra generated by H.

Definition 1.9 (Borel σ-algebra). If (Ω, T ) is a topological space (which means that it makes sense to talk about open subsets of Ω, and T is the set of these open subsets), then B(Ω) := σ(T ) is called the Borel σ-algebra on Ω.

Remark 1.10. The set T is called the topology, so the Borel σ-algebra is the σ-algebra generated by the topology. For those, who haven’t heard, but are interested: A collection T ⊂ 2^{Ω} of subsets of Ω is called (by definition) a topology over Ω if it contains ∅ and Ω, and it is closed under finite intersection and arbitrary union. Then the elements of T are called open sets, and the pair (Ω, T ) is called a topological space. (So the definition only says that ∅ and Ω are open, the intersection of finitely many open set is open, and that the union of any family of open sets is open.)

When we talk about the Borel sets on ℝ or ℝ^{n}, we always think of the usual notion of open sets on these spaces.

Remark 1.11. Not every subset of [0, 1] is Borel. In fact, a non-Borel subset can be constructed (and not only the existence can be proven). We don’t go into that.

Notation 1.12. We denote by ℝ^{+} the set of nonnegative real numbers – that is, ℝ^{+} = [0, ∞). In particular, ℝ^{+} includes zero.

Definition 1.13 (measure space, measure). Let (Ω, F) be a measurable space. The nonnegative extended real valued function μ on F (that is, μ : F → ℝ^{+} ∪ {∞}) is called a measure on Ω, if

• μ(∅) = 0,

• μ is σ-additive, meaning that if {A_{i}}_{i∈I} is a countable family of pairwise disjoint measureable sets (with formulas: A_{i} ∈ F for every i ∈ I where I is a countable index set, and A_{i} ∩ A_{j} = ∅ for every i ≠ j, i, j ∈ I), then

μ (∪_{i∈I}A_{i}) = ∑_{i∈I} μ(A_{i}).

Then the triple (Ω, F, μ) is called a measure space and μ(A) is called the measure of the set A.

Remark 1.14. It is absolutely important that in the definition of σ-additivity I is countably infinite: σ-additivity is more than finite additivity and less than arbitrary additivity. For example, if μ is the Lebesgue measure on ℝ (see later), every 1-element set {x} has μ({x}) = 0, which implies that every countable set has zero measure, but of course

1 = μ([0, 1]) = μ(∩_{x∈[0,1]}{x}) ≠ ∑_{x∈[0,1]} μ({0}) = ∑_{x∈[0,1]} 0 = 0.

(Whatever the sum of uncountably many real numbers could mean.)

Once it came up, we mention the following, absolutely non-important definition:

Definition 1.15 (sum of many nonnegative extended real numbers). If a_{i} ∈ ℝ^{+} ∪ {∞} for every i ∈ I where I is any index set, then we define the sum of all a_{i} as

∑_{i∈I} a_{i} := sup {∑_{i∈J} a_{i} : J ⊂ I and J is finite} .

Note that it is important that the a_{i} are nonnegative. This definition obviously coincides with the usual sum of (an arbitrarily reorderable) series if I is countable. This new notion of an infinite sum is no serious extention of the well known notion of a countable series: it is easy to see that if the sum is finite, then at most countably many terms can be nonzero.

Remark 1.16. In the definition of the measure, the first requirement μ(∅) = 0 is almost automatic from σ-additivity: it’s only there to rule out the trivial nonsense μ(∅) = ∞. In fact it would be enough to require that at least one measurable set A has finite measure: σ-additivity implies

μ(A) = μ(A∪∅∪∅∪∅∪…) = μ(A)+μ(∅)+μ(∅)+μ(∅)+⋯ = μ(A)+∑_{i=1}^{∞} μ(∅).

If μ(A) < ∞, then this implies 0 = ∑_{i=1}^{∞} μ(∅), so μ(∅) = 0.

Definition 1.17. The measure χ on a nonempty set Ω equipped with the discrete σ-algebra 2^{Ω} defined as

χ(A) := ♯A := \begin{cases}number of elements in A, & if A is finite \\ ∞, & if A is infinite \end{cases}

is called the counting measure on Ω. The restriction of χ to any σ-algebra F ⊂ 2^{Ω} is still called a counting measure (on F).

One of the most important examples of a measure is the Lebesgue measure on ℝ or on ℝ^{d}

Definition 1.18 (Lebesgue measure vaguely). Consider the set ℝ with the Borel σ-algebra B. The measure Leb : B → ℝ^{+} ∪ {∞} is called the Lebesgue measure on ℝ, if it assigns to every interval its length – that is, for every a, b ∈ ℝ, a ≤ b we have

Leb((a, b)) = Leb((a, b]) = Leb([a, b)) = Leb([a, b]) = b − a.

The restriction of Leb to a Borel subset of ℝ (e.g. an interval [c, d] or (c, ∞)) is still called Lebesgue measure and is still denoted by Leb. (More precisely, if (ℝ, B, Leb) is the Lebesgue measure space on ℝ, and I ∈ B, than one can define the “restriction of Leb to I” as the measure space (I, B_{I}, Leb_{I}) where B_{I} := {A ∩ I : A ∈ B} = {B : B ∈ B, B ⊂ I} ⊂ B and Leb_{I} := Leb|_{B_{I}} is the restriction of Leb to B_{I}.)

Similarly, the “Lebesgue measure on ℝ^{d}” is the measure on Borel subsets of ℝ^{d} which assigns to every box its d-dimensional volume, i.e. for every a_{1} ≤ b_{1}, a_{2} ≤ b_{2}, … a_{d} ≤ b_{d} ∈ ℝ we have

Leb_{d}([a_{1}, b_{1}] × [a_{2}, b_{2}] × ⋯ × [a_{d}, b_{d}]) = ∏_{i=1}^{d}(b_{i} − a_{i}).

Restrictions to Borel subsets of ℝ^{d} are still called Lebesgue measure, and denoted by Leb_{d} or just Leb.

Remark 1.19. The above “definition” of the Lebesgue measure is far from being complete, and is not the usual definition – it’s actually a characterization of the Lebesgue measure which shows its essence. It can be (and needs to be) shown that such a measure indeed exists, since in the “definition” we only gave the value of Leb for a few very special sets, and not every Borel set. Also uniqueness can and needs to be shown. These questions lead to the construction of measures based on their pre-known values on certain pre-chosen “to-be-measurable” sets, which can sometimes be of crucial importance, but we don’t go into that here.

Remark 1.20. In the measure theory literature, Lebesgue measure is defined on a σ-algerba F which is larger than the Borel σ-algebra (i.e. B ⫋ F), called the “set of Lebesgue measurable sets”. In particular, F has the property that if B ∈ F, Leb(B) = 0 and A ⊂ B, then A ∈ F, which is not true for Borel sets. However, in probability theory it is usual to consider Lebesgue measure restricted to Borel sets only (as in the above definition).

The following definition shows that the basic object of probability theory, called “the probability” is in fact a measure.

Definition 1.21 (Kolmogorov probability space). The triple (Ω, F, ℙ) is called a Kolmogorov probability space (or just probability space) if it is a measure space and ℙ(Ω) = 1. Then ℙ is called the probability or a “probability measure”, elements of F are called events, and elements of Ω are the elementary events. For A ∈ F, ℙ(A) ∈ [0, 1] is called the probability of the event A.

Picture ω ∈ Ω as possible outcomes of an experiment, so an event A ∈ F often consists of many possible outcomes of that experiment, which have some common property that we are interested in. By definition, an “event” is something which has a probability.

1.2 Measurable functions, random variables and their distributions

Notation 1.22. For a function f : Ω → Ω′ and a set A′ ⊂ Ω′, let f^{−1}(A′) denote, as usual, the complete inverse image of A′ defined as f^{−1}(A′) := {ω ∈ Ω : f(ω) ∈ A′}. Note that this makes sense for any function and any A′ – in particular, f need not be invertable.

Definition 1.23 (measureable function, observable, random variable). Let (Ω, F) and (Ω′, F′) be measurable spaces. The function X : Ω → Ω′ is called measurable or (F, F′)-measurable, if for every A′ ∈ F′ we have X^{−1}(A′) ∈ F. (That is, if the inverse image of any measurable set is also measurable.) In physical applications, when Ω is the (possibly complicated) phase space of a system and Ω′ is a (usually simple) set of possible measurement results (e.g. Ω′ = ℝ), the same X is called an observable. In the context of probability theory, when (Ω, F, ℙ) is a probability space, X is called a (Ω′-valued) random variable.

Note that the notion of measurability of a function depends on the choice of the σ-algebras F and F′. However, in many cases when this choice is clear from the context, we simply say “measurable” instead of “(F, F′)- measurable”. When we talk about a random variable, and do not specify the range, usually (Ω′, F′) = (ℝ, B) is understood.

Remark 1.24. If X : Ω → Ω′ is a random variable and x ∈ Ω′, then we denote by {X = x} the set of elementary events where X takes the value x – that is,

{X = x} := {ω ∈ Ω : X(ω) = x} = X^{−1}({x}).

Similarly, if A′ ⊂ Ω′, then {X ∈ A′} denotes the set of elementary events where X takes values in A′:

{X ∈ A′} := {ω ∈ Ω : X(ω) ∈ A′} = X^{−1}(A).

With this in mind, the definition of a random variable as a measurable function is very natural. The definition says exactly that is A′ is a measurable subset of the range Ω′, then the set {X ∈ A} is indeed an event – that is, it makes sense to talk about its probability.

Example 1.25. coordinate, number rolled, sum of these

Random variables are the central objects of study in probability theory. In a typical situation they extract fairly little information (e.g. a single number) from a big probability space containing many complicated possible outcomes of an experiment. So to “understand” a random variable X : Ω → Ω′ on the probability space (Ω, F, ℙ) well, we need less information than the what ℙ and X contain. From another point of view: when we consider a random variable, Ω is often not needed, or not even known. All we need to know is the possible values (in Ω′) X can take, and the probability of these being taken. This information is contained exactly in a measure on Ω′, as the following definition shows.

Definition 1.26 (distribution of a random variable). Let (Ω, F, ℙ) be a probability space, (Ω′, F′) a measurable space and X : Ω → Ω′ a random variable. Then the distribution of X is the measure μ on (Ω′, F′) which is defined as

μ(A) := ℙ({X ∈ A}) = ℙ(X^{−1}(A)) for every A ∈ F′.

This can be written in short as

μ := ℙ ∘ X^{−1}.

μ is nothing else than the “push-forward” of the probability ℙ by X to Ω′.

In the special case when Ω′ = ℝ and X is discrete (meaning that it can take finitely many, or at most countably many values), there is a convenient alternative way to “desribe the distribution of X”, by simply listing the possible values x_{k} and their probabilities p_{k} := ℙ(X = x_{k}) := ℙ({X = x_{k}}). Then the information contained in the sequence of pairs {(x_{k}, p_{k})}_{k=1}^{N} (with possibly N = ∞) is called the discrete probability distribution. Having this information, one can calculate probabilities of events by summation:

μ(A) = ℙ({X ∈ A}) = ∑_{k:x_{k}∈A} p_{k}.

Similarly, in the special case when Ω′ = ℝ and X is absolutely continuous (see later), there is convenient alternative “description of the distribution” by a density function f : ℝ → ℝ^{+} from which one can calculate probabilities of events by integration:

μ(A) = ℙ({X ∈ A}) = ∫_{A} f(x) dx.

The above notion of a probability distribution is a far-reaching generalization of both notions.

Example 1.27. sum of two rolled numbers real number generated by a sequence of fair coin tosses real number generated by a sequence of biased coin tosses

1.3 Integral and expectation

1.3.1 Integral, integrability

For an (extended) real-valued measurable functions X : Ω → ℝ on a measure space (Ω, F, μ) it makes sense to talk about the integral ∫_{Ω} X dμ. This is an essential tool, and also an important object of study both in measure theory and in probability theory. We don’t go deep into the definition and properties of the integral here – we don’t want to, and we can’t substitute a measure theory course now. I just give very briefly one of the shortest possible definitions, and point out a few main feaures.

Since we don’t want to exclude the case when either a function or a measure takes the value ∞, we work with extended real numbers, and use the convention

0 · ∞ := ∞ · 0 := 0.

We start by defining the integral of nonnegative functions.

Definition 1.28 (integral of non-negative extended real valued functions). If (Ω, F, μ) is a measure space and X : Ω → ℝ^{+} ∪ {∞} is measurable, we introduce a sequence X_{n} : Ω → ℝ^{+} of simple functions (i.e. taking only finitely many values) which approximate X (from below) as

X_{n}(ω) := max\{x ∈ \{0, \frac{1}{2^{n}} , \frac{2}{2^{n}} , …, 2^{n} − \frac{1}{2^{n}} , 2^{n}\} : x ≤ X(ω)\}.

Than we define the n-th integral-approximating sum as

I_{n} := ∑_{x∈\{0, \frac{1}{2^{n}} , \frac{2}{2^{n}} ,...,2^{n}− \frac{1}{2^{n}} ,2^{n}\}} xμ(X_{n}^{−1}(\{x\})),

and the integral of X as

∫_{Ω} X dμ := lim_{n→∞} I_{n}.

The sets X_{n}^{−1}({x}) ⊂ Ω in the definition of I_{n} are ensured to be Fmeasurable by the assumption that X is measurable (and the fact that {x} is Borel-measurable in ℝ), so μ(X_{n}^{−1}({x})) makes sense. The sequence X_{n} of functions is cleverly defined to be increasing, and so is the sequence I_{n}, so the limit in the above definition exists, but is possibly infinite.

We can now go on to the general definition of the integral for extended real valued functions:

Definition 1.29 (integral). If (Ω, F, μ) is a measure space and X : Ω → ℝ∪ {−∞, ∞} is measurable, we introduce the positive part X_{+} and the negative part X_{−} of X as

X_{+}(ω) := \begin{cases}X(ω), & if X(ω) > 0, \\ 0, & if not \end{cases} , X_{−}(ω) := \begin{cases}−X(ω), & if X(ω) < 0, \\ 0, & if not \end{cases} .

Note that both X_{+} and X_{−} are nonnegative and X = X_{+} − X_{−}.

Now

• If either ∫_{Ω} X_{+} dμ < ∞ or ∫_{Ω} X_{−} dμ < ∞, then we define the integral of X as

∫_{Ω} X dμ := ∫_{Ω} X_{+} dμ − ∫_{Ω} X_{−} dμ,

which can possibly be ∞ or −∞.

• If both ∫_{Ω} X_{+} dμ = ∞ and ∫_{Ω} X_{−} dμ = ∞, then we say that the integral of X doesn’t exist (or that it is undefined).

In the usual mathematical language, there is an important distinction between the existence of an integral and the integrability of a function. We emphasize this in the following definition:

Definition 1.30 (integrability). Let (Ω, F, μ) be a measure space and X : Ω → ℝ∪{−∞, ∞} measurable. If the integral ∫_{Ω} X dμ exists and it is finite, then we say that X is integrable (with respect to μ).

So integrability of X means −∞ < ∫_{Ω} X dμ < ∞, which is equivalent to both ∫_{Ω} X_{+} dμ and ∫_{Ω} X_{−} dμ being finite.

Remark 1.31 (comparison to the Riemann integral). The above definition of the integral is similar to the construction of the good old Riemann integral: in both cases the domain of integration, Ω, is chopped up into pieces, on each of which the function X takes nearly constant values. Than the “size” of each piece is multiplied by the approximate value of the function there, and these products are added up to obtain an integral approximating sum. The crucial difference is that in the case of Riemann integral, these small pieces of Ω had to be intervals, while here they can be any measurable subset of Ω. In particular, the points in X_{n}^{−1}({x}) don’t need to be “close” to each other in any sense, thus Ω absolutely doesn’t need to be the real line or anything similar. It doesn’t need to have any additional structure that would give sense to the words “distance” or “being close”. Really, any measure space will do.

A trivial but important example of integrable functions:

Lemma 1.32 (bounded functions on finite measure spaces are integrable). If (Ω, F, μ) is a finite measure space (meaning that μ(Ω) < ∞) and X : Ω → ℝ is measurable and bounded – meaning that there exists an M ∈ ℝ such that −M ≤ X(ω) ≤ M for every ω ∈ Ω, then X is integrable w.r.t. μ.

Proof. X being bounded by M implies that 0 ≤ X_{+}(ω), X_{−}(ω) ≤ M. when calculating the integral of, say, X_{+}, we have X_{n}(ω) ≤ X_{+}(ω) ≤ M, and thus

I_{n} := ∑_{x∈\{0, \frac{1}{2^{n}} , \frac{2}{2^{n}} ,...,2^{n}− \frac{1}{2^{n}} ,2^{n}\}} xμ(X_{n}^{−1}(\{x\}))

≤ M ∑_{x∈\{0, \frac{1}{2^{n}} , \frac{2}{2^{n}} ,...,2^{n}− \frac{1}{2^{n}} ,2^{n}\}} μ(X_{n}^{−1}(\{x\}))
= Mμ(∪_{x∈\{0, \frac{1}{2^{n}} , \frac{2}{2^{n}} ,...,2^{n}− \frac{1}{2^{n}} ,2^{n}\}}X_{n}^{−1}(\{x\}))
= Mμ(Ω).

So we get ∫_{Ω} X_{+} dμ < Mμ(Ω). Similarly, ∫_{Ω} X_{−} dμ < Mμ(Ω), so

−∞ < −Mμ(Ω) ≤ ∫_{Ω} X dμ ≤ Mμ(Ω) < ∞.

A basic property of the integral with a huge importance is linearity in the function integrated (the integrand):

Theorem 1.33 (linearity of integrability and the integral). Let X_{1} and X_{2} be real valued measurable functions on the same probability space (Ω, F, μ), and let α_{1}, α_{2} ∈ ℝ. If both I_{1} := ∫_{Ω} X_{1} dμ and I_{2} := ∫_{Ω} X_{2} dμ exist and α_{1}I_{1} + α_{2}I_{2} is not of the form ∞ − ∞, then ∫_{Ω}(α_{1}X_{1} + α_{2}X_{2}) dμ exists and

∫_{Ω}(α_{1}X_{1} + α_{2}X_{2}) dμ = α_{1} ∫_{Ω} X_{1} dμ + α_{2} ∫_{Ω} X_{2} dμ.

As a consequence, if X_{1} and X_{2} are both integrable, then so is α_{1}X_{1} + α_{2}X_{2}.

The proof is easy from the definition, but we don’t discuss it here. See any measure theory book. It is useful to note that the integral is linear not only in the integral, but also in the measure:

Theorem 1.34 (linearity of the integral II.). Let (Ω, F) be a measurable space, μ_{1} and μ_{2} measures on it, X a real valued measurable function and 0 ≤ α_{1}, α_{2} ∈ ℝ. If both I_{1} := ∫_{Ω} X dμ_{1} and I_{2} := ∫_{Ω} X dμ_{2} exist and α_{1}I_{1} + α_{2}I_{2} is not of the form ∞ − ∞, then ∫_{Ω} X d(α_{1}μ_{1} + α_{2}μ_{2}) exists and

∫_{Ω} X d(α_{1}μ_{1} + α_{2}μ_{2}) = α_{1} ∫_{Ω} X dμ_{1} + α_{2} ∫_{Ω} X dμ_{2}.

As a consequence, if X is integrable w.r.t. both μ_{1} and μ_{2}, then so it is w.r.t. α_{1}μ_{1} + α_{2}μ_{2}.

Remark 1.35 (bilinearity of the integral). In the last theorem, we required that α_{1} and α_{2} be nonnegative – otherwise α_{1}μ_{1}+α_{2}μ_{2} may not be a measure, since in our definition a measure has to be nonnegative. For the same reason, the measures on a measurable space do not form a linear space. In functional analysis, to overcome that limitation, it is common to introduce the notion of “signed measures” (say, as a difference of two measures), which already form a linear space (with the usual notion of addition and multiplication). Then the two-variable real-valued mapping

⟨μ, X⟩ := ∫ X dμ

can be defined on suitably chosen linear spaces of measures and functions (e.g. μ ∈ {signed finite measures}, X ∈ {bounded measurable functions}.) The above two theorems show that this mapping is bilinear, which is the property where functional analysis starts.

1.3.2 Exchanging limits with integrals

It is utmost important that Theorem 1.33 is about the linear combination of two integrable functions. Of course, it immediately implies linearity of the integral for finite linear combinations, but does not say anything about infinite sums. Indeed, linearity of the integral for infinite sums is not at all true in general. In fact, it is an important issue, in which cases exchanging an integral with a limit is possible – one has to be at least always careful. In the following we state (without proof) three theorems, which are the most frequently (and almost exclusively) used tools in checking exchangeability. In a situation where none of them works, exchangeability is usually hard to prove, and may very well not be true.

The first and most used tool is the Lebesgue dominated convergence theorem:

Theorem 1.36 (dominated convergence). Let (Ω, F, μ) be a measure space and f_{1}, f_{2}, … measurable real valued functions on Ω which converge to the limit function pointwise, μ-almost everywehere. (That is, lim_{n→∞} f_{n}(x) = f(x) for every x ∈ Ω, except possibly for a set of x-es with μ-measure zero.) Assume furthermore that the f_{n} admit a common integrable dominating function: there exists a g : Ω → ℝ such that |f_{n}(x)| ≤ g(x) for every x ∈ Ω and n ∈ ℕ, and ∫_{Ω} g dμ < ∞. Then (all the f_{n} and also f are integrable and)

lim_{n→∞} ∫_{Ω} f_{n} dμ = ∫_{Ω} f dμ.

As you see, it is enough to require “almost everywhere” convergence, which is no surprise, since changing a function on a set of measure zero doesn’t change the integral. In fact, it would be enough to require that g dominates the f_{n} almost everywhere – moreover, it would be enough to require that the f_{n} and g be extended real valued and defined almost everywhere. This is not a serious generalization, so I decided to rather keep the formulation simple. In the literature usually the most general form is given.

The second and easiest tool is Beppo Levi’s monotone convergence theorem:

Theorem 1.37 (monotone convergence). Let (Ω, F, μ) be a measure space and f_{1}, f_{2}, … a sequence of measurable nonnegative real valued functions on Ω which is pointwise increasing. (That is, 0 ≤ f_{n}(x) ≤ f_{n+1}(x) for every n ∈ ℕ and x ∈ Ω.) Then the pointwise limit function f defined by f(x) := lim_{n→∞} f_{n}(x) is also measurable and

lim_{n→∞} ∫_{Ω} f_{n} dμ = ∫_{Ω} f dμ.

The third and trickiest tool is the Fatou lemma:

Theorem 1.38 (Fatou lemma). Let (Ω, F, μ) be a measure space and f_{1}, f_{2}, … a sequence of measureabale functions f_{n} : Ω → ℝ, which are nonneagtive, e.g. f_{n}(x) ≥ 0 for every n = 1, 2, … and every x ∈ Ω. Then

∫_{Ω} lim inf_{n→∞} f_{n}(x) dμ(x) ≤ lim inf_{n→∞} ∫_{Ω} f_{n}(x) dμ(x)

(and both sides make sense).

Definition 1.39 (Absolute continuity of measures, singularity of measures). Let μ and ν be two measures on the same measurable space Ω, F. We say that ν is absolute continuous with respect to μ (notation: ν ≪ μ) if for every A ∈ F for which μ(A) = 0, also ν(A) = 0. We say that ν and μ are singular (with respect to each other) (notation: ν ⊥ μ) if there exists and A ∈ F for which ν(A) = 0 and μ(Ω \ A) = 0.

The best known probability distributions are all examples of either one or the other of these: the “discrete” probability distributions are singular w.r.t. Lebesgue measurer on ℝ, while the ones that are loosely called “continuous” are actually absolutely continuous w.r.t. Lebesgue measurer on ℝ.

Theorem 1.40 (Radon-Nykodim). If μ and ν are two measures on the same measurable space Ω, F and ν ≪ μ, then there exists a measurable f : Ω → ℝ, called the density of ν w.r.t. μ, which satisfies ν(A) = ∫_{A} f dμ.

absolute continuity w.r.t. counting measure absolute continuity w.r.t. Lebesgue measure integration w.r.t counting measure integration w.r.t. Lebesgue measure integration w.r.t. absolutely continouos measures density function, distribution function continuity of measures expectation if it exists - real-valued - complex or R^{d} -valued - remark

about more complicated spaces integration by substitution

Theorem 1.41. Let (Ω, F) and (Ω′, F′) be measurable spaces, μ a measure on Ω, X : Ω → Ω′ measurable and ν := μ ∘ X^{−1} the push-forward of μ by X (a measure on Ω′) (the distribution of X, if μ is a probability). Then for any measurable g : Ω′ → ℝ

∫_{Ω} g(X) dμ = ∫_{Ω′} g dν.

expectation of a distribution linearity of expectation variance, standard deviation moments, centered moments, moment-generating function, characteristic

function

product measure space, product measure - be careful with infinite products

pairwise independence, marginal distributions mutual independence variance of linear combination of independent random variables Markov inequality weak law of large numbers central limit theorem weak convergence of measures weak convergence of random variables continuity of the characteristic function