Probability Review I: Foundations & Expectation

Vocabulary Check: Probability Basics

what do these terms mean to you?

1. Experiment
2. Outcome
3. Sample Space (\Omega)
4. Probability Measure (\Pr)
5. Event (A)
6. Independence
7. Random Variable
8. Expectation

Probability Spaces & Axioms

• Sample Space \Omega: The set of all possible outcomes of an experiment.
• Event A \subseteq \Omega: A subset of outcomes to which we assign a probability.
• Probability Measure \Pr: A function mapping events to [0,1] satisfying Kolmogorov’s Axioms:
  1. Non-negativity: \Pr(A) \ge 0 for all A.
  2. Normalization: \Pr(\Omega) = 1.
  3. Additivity: For disjoint events A, B, \Pr(A \cup B) = \Pr(A) + \Pr(B).
• Probability Distribution p(\omega): For a discrete space, \sum_{\omega \in \Omega} p(\omega) = 1.

Events & Set Operations

• Event: a subset of the sample space.
• Union A \cup B: at least one happens.
• Intersection A \cap B: both happen.
• Two coins: \Omega = \{HH, HT, TH, TT\}, each with \Pr = 1/4.

Inclusion–Exclusion & Union Bound

• Inclusion–Exclusion: \Pr(A \cup B) = \Pr(A) + \Pr(B) - \Pr(A \cap B).
  • Extends to n events: alternates adding and subtracting intersection probabilities.
• Union Bound (Boole’s Inequality): For any events A_1, A_2, \dots: \Pr\Big(\bigcup_{i} A_i\Big) \le \sum_{i} \Pr(A_i)
  • Crucial: Needs no independence whatsoever. Often a key first step in algorithm analysis.

Random Variables & Expectation

• A random variable (RV) is a function X: \Omega \to \mathbb{R}.
• The expected value of a discrete RV:

E[X] = \sum_{x} x \cdot \Pr(X = x)

The average value of X over many repetitions of the experiment.

Indicator Random Variables

• For an event A, the indicator I_A = 1 if A occurs, else 0.
• Fundamental lemma: E[I_A] = \Pr(A).

Linearity of Expectation

For any random variables X and Y — even dependent ones:

E[X + Y] = E[X] + E[Y]

Worked Example 1: Expected Number of Heads

• Flip a fair coin n times; X = number of heads.
• Write X = \sum_{k=1}^n I_k where I_k is the indicator that flip k is heads.
• E[I_k] = \Pr(\text{heads}) = \frac{1}{2}.
• By linearity, E[X] = \sum_{k=1}^n E[I_k] = \frac{n}{2}.

Worked Example 2: Fixed Points

• A random permutation of n elements; X = number of fixed points.
• Let I_i indicate that element i maps to itself: \Pr(\pi(i) = i) = \frac{1}{n}, so E[I_i] = \frac{1}{n}.
• By linearity, E[X] = \sum_{i=1}^n \frac{1}{n} = 1.
• Surprise: the expected number of fixed points is exactly 1 — regardless of n, and the I_i are highly dependent.

Independence: Pairwise vs. Mutual

• Pairwise Independence: Events A and B are independent if \Pr(A \cap B) = \Pr(A)\Pr(B).
• Mutual Independence: \{A_1, \dots, A_n\} are mutually independent if for any subset J: \Pr\Big(\bigcap_{j \in J} A_j\Big) = \prod_{j \in J} \Pr(A_j)
• Pairwise \ne Mutual (XOR Counterexample): Flip two independent fair coins (C_1, C_2).
  • A: C_1 is H. B: C_2 is H. C: coins match (C_1 \oplus C_2 = 0).
  • Any pair is independent, but they are not mutually independent: \Pr(A \cap B \cap C) = \frac{1}{4} \ne \Pr(A)\Pr(B)\Pr(C) = \frac{1}{8}.

Conditional Probability & Bayes

• Conditional probability: \Pr(A \mid B) = \dfrac{\Pr(A \cap B)}{\Pr(B)}.
• Law of total probability: \Pr(A) = \sum_i \Pr(A \mid B_i)\Pr(B_i) over a partition.
• Bayes’ theorem: \Pr(B_i \mid A) = \dfrac{\Pr(A \mid B_i)\Pr(B_i)}{\Pr(A)}.

Variance & Moments

• Variance: Measures spread from the mean: \mathrm{Var}(X) = E\big[(X - E[X])^2\big] = E[X^2] - E[X]^2.
• Standard Deviation: \sigma = \sqrt{\mathrm{Var}(X)} (shares units with X).
• Additivity: If X and Y are independent: \mathrm{Var}(X + Y) = \mathrm{Var}(X) + \mathrm{Var}(Y).
• Moments & MGF: The k-th moment is E[X^k]. The Moment Generating Function (MGF) is M_X(\lambda) = E[e^{\lambda X}] (central to Chernoff bounds in Lecture 3).

Standard Distributions

• Bernoulli(p): Indicator of success. X \in \{0,1\}, \Pr(X=1) = p.
  • E[X] = p, \mathrm{Var}(X) = p(1-p).
• Binomial(n, p): Number of successes in n independent trials. X \in \{0, \dots, n\}.
  • \Pr(X=k) = \binom{n}{k} p^k (1-p)^{n-k}, E[X] = np, \mathrm{Var}(X) = np(1-p).
• Geometric(p): Number of independent trials until first success. X \in \{1, 2, \dots\}.
  • \Pr(X=k) = (1-p)^{k-1}p, E[X] = 1/p, \mathrm{Var}(X) = \frac{1-p}{p^2}.

Concept Check

After throwing n balls into n bins independently and uniformly at random, what is the expected number of empty bins?

Next Lecture

Probability Review II: Tail Bounds — moments & generating functions, then Markov, Chebyshev, and Chernoff bounds with proofs.