Why Average-Case Analysis?

Worst-case analysis is the gold standard, but sometimes too pessimistic:

Average Case Analysis

• Worst-case: upper bound on running time for any input of size n.
• Average-case: expected running time over a distribution of inputs.

Formally, for input distribution p_n over \mathcal{I}_n:

T_{\text{avg}}(n) = \sum_{I \in \mathcal{I}_n} T(I) \cdot p_n(I)

Worst vs Average vs Best

\begin{array}{lccc} \textbf{Algorithm} & \textbf{Worst} & \textbf{Average} & \textbf{Best} \\ \hline \text{Insertion Sort} & \Theta(n^2) & \Theta(n^2) & \Theta(n) \\ \text{Merge Sort} & \Theta(n \log n) & \Theta(n \log n) & \Theta(n \log n) \\ \text{Quicksort (randomized)} & \Theta(n^2) & \Theta(n \log n) & \Theta(n \log n) \\ \text{Linear Search} & \Theta(n) & \Theta(n) & \Theta(1) \end{array}

Insertion Sort: Average Case Analysis

Insertion Sort Algorithm

Input distribution assumption: Random permutation model — all n! orderings equally likely. The new element a_i is equally likely to be k^{\text{th}} largest among a_1,\dots,a_i for any k = 1,\dots,i.

Computing Expected Comparisons

Let X = total comparisons. Let X_i = comparisons to insert a_i.

• If a_i is k^{\text{th}} largest: exactly k comparisons needed
• \Pr(a_i \text{ is } k^{\text{th}} \text{ largest}) = 1/i

\mathbb{E}[X_i] = \sum_{k=1}^{i} \frac{1}{i} \cdot k = \frac{i+1}{2}

By linearity of expectation (X = X_2 + \cdots + X_n):

\mathbb{E}[X] = \sum_{i=2}^{n} \frac{i+1}{2} = \frac{n^2 + 3n - 4}{4} = \Theta(n^2)

Insertion Sort: Result

Why? Even on average, a new element compares with half the sorted prefix. Linear cost per insertion \to quadratic total.

The Probabilistic Method

Recipe: 1. Design a random experiment 2. Compute \mathbb{E}[\text{relevant quantity}] 3. Conclude existence (\mathbb{E}[Z] \ge c \;\Rightarrow\; \exists outcome with Z \ge c)

Pioneered by Paul Erdős.

Lower Bound on Independent Sets

Proof: Pick each vertex independently with probability p. Let X = |S|, Y = |\{\text{edges inside } S\}|.

• \mathbb{E}[X] = np, \mathbb{E}[Y] = \frac{nd}{2}p^2
• \mathbb{E}[X - Y] = np - \frac{nd}{2}p^2

Maximize over p: \frac{d}{dp} = n - ndp = 0 \;\Rightarrow\; p = 1/d.

Independent Set Bound (contd.)

Plugging p = 1/d:

\mathbb{E}[X - Y] = \frac{n}{d} - \frac{n}{2d} = \frac{n}{2d}

Since \mathbb{E}[X - Y] > 0, there exists a specific set S with X - Y \ge \frac{n}{2d}.

Cleanup: Delete one endpoint from each edge inside S. At most Y vertices removed, leaving S^* with:

|S^*| \ge X - Y \ge \frac{n}{2d}

S^* has no internal edges \Rightarrow independent set. \square

Key Takeaways

Next lecture: Amortized Analysis

Thank You