BITS Pilani
BITS Logo
CS G526: Advanced Algorithms and Complexity

Online Algorithms II: Paging & Caching

Lecture 22 |2026-09-29
Tulasimohan Molli
BITS Pilani, Hyderabad Campus

Agenda

  1. Real-World Systems Motivation: The Memory Hierarchy Pyramid (Registers \to L1/L2/L3 \to RAM \to SSD \to Disk/Cloud).
  2. The Paging Model: Fast cache of size k vs. backing store of N pages (k \ll N).
  3. The Offline Benchmark: Belady’s \text{MIN} (Longest Forward Distance).
  4. The Sleator–Tarjan \Omega(k) Lower Bound: Adversarial cyclic sequences (k+1 pages).
  5. Deterministic LRU & Phase Partitioning: Proving the k-competitive upper bound.
  6. The Randomized Marking Scheme: Intuitive mechanism, toy trace, and the O(\log k) landscape.

The Memory Hierarchy Pyramid

Why Caching Matters in Practice

Across all computing systems, memory follows a simple trade-off:

The Speed–Size Trade-off

  • Fast memory (registers, cache) is small and expensive.
  • Slow storage (RAM, disk) is huge and cheap.
  • Accessing slow storage causes a massive delay (often 100\times to 10,000\times slower).

The Core Algorithmic Question

  • We have room for only k items in fast storage.

  • When a new item is needed and the cache is full: \textbf{Which item do we evict?}

  • Good eviction decisions hide the speed difference!

Our Goal Today: Analyze eviction rules (like LRU) mathematically without knowing the future, and prove rigorous bounds against the best possible offline strategy.

The Paging Abstraction (k \ll N)

The Paging Model Formulation

Modern computing systems rely on two-tier storage hierarchies:

Formal Model

  • Fast Cache: Holds at most k pages simultaneously.
  • Slow Backing Store: Holds N total pages (k \ll N).
  • Request Sequence: \sigma = \langle r_1, r_2, \dots, r_m \rangle where each r_t \in \{1, \dots, N\}.
  • Cost Structure:
    • Hit (r_t \in \text{Cache}): \text{Cost} = 0.
    • Page Fault / Miss (r_t \notin \text{Cache}): \text{Cost} = 1. Must evict some page currently in cache to load r_t.
  • Goal: Minimize the total number of page faults.

The Offline Optimal: Belady’s \text{MIN} Rule

If an omniscient algorithm knows the entire future sequence \sigma in advance:

Belady’s \text{MIN} Algorithm (1966)

On a page fault when the cache is full: \textbf{Evict the page whose next request occurs furthest in the future.} (If a page is never requested again, its next request is at \infty).

Theorem (Belady 1966)

Belady’s \text{MIN} achieves the minimum possible number of page faults on any input sequence \sigma: \text{cost}(\text{MIN}(\sigma)) = \text{cost}(\text{OPT}(\sigma))

The Online Dilemma & Standard Heuristics

In real operating systems and hardware caches, the future is completely unknown:

The Core Question

How well can any deterministic online heuristic (like LRU) perform compared to the omniscient Belady \text{MIN}? Can we achieve a constant competitive ratio independent of cache size k?

The Sleator–Tarjan Lower Bound

Theorem (Sleator & Tarjan 1985)

Let \mathcal{A} be any deterministic online paging algorithm for a cache of size k. Then the competitive ratio of \mathcal{A} is at least k: \text{Competitive Ratio } c \ge k

Even for a modest cache (k = 1000 pages), an online deterministic algorithm can be forced to fault 1000\times more often than \text{OPT}!

Let us see why through a simple adversarial example with k=3.

Proof by Toy Example: k = 3 Cache Slots

Consider a slow memory with just k + 1 = 4 pages: \{p_1, p_2, p_3, p_4\}. Cache capacity k = 3.

  1. Initial cache state holds \{p_1, p_2, p_3\}. Page p_4 is missing.
  2. Adversary rule: Always request whichever page is currently missing from cache!
    • Step 1: Adversary requests p_4 \implies Fault! Algorithm evicts p_1. Cache = \{p_2, p_3, p_4\}.
    • Step 2: Adversary requests p_1 \implies Fault! Algorithm evicts p_2. Cache = \{p_1, p_3, p_4\}.
    • Step 3: Adversary requests p_2 \implies Fault! Algorithm evicts p_3. Cache = \{p_1, p_2, p_4\}.
  3. Deterministic Cost: On m requests, algorithm faults on every single step: \text{cost}(\text{ALG}) = m

How Belady’s \text{MIN} Beats the Adversary

What does offline \text{OPT} do on that exact same adversarial sequence \langle p_4, p_1, p_2, p_3, p_4, p_1, \dots \rangle?

  • Initial cache: \{p_1, p_2, p_3\}.
  • Request p_4 arrives: \text{OPT} faults once.
  • \text{OPT} looks ahead into the future:
    • Next requests are p_1, then p_2, then p_3.
    • \text{OPT} evicts p_3 (furthest in future)!
  • Now \text{OPT} has \{p_1, p_2, p_4\} in cache.
  • The next two requests (p_1, p_2) are both instant hits (Cost = 0)!
  • \text{OPT} only faults once every k requests: \text{cost}(\text{OPT}) \le \frac{m}{k}

\text{Competitive Ratio} = \frac{\text{cost}(\text{ALG})}{\text{cost}(\text{OPT})} \ge \frac{m}{m/k} = \mathbf{k}

Can LRU Match the Lower Bound k?

The lower bound is k. Can common algorithms like LRU actually guarantee c = k?

The Phase Partitioning Technique

To prove LRU is k-competitive, partition the request sequence \sigma into phases:

  • Phase 1: Starts at the first request; ends just before the (k+1)-th distinct page is requested.
  • Phase i: Contiguous block containing requests to at most k distinct pages.

LRU Performance in a Single Phase

Consider any single Phase i under LRU:

  1. Phase i contains requests to at most k distinct pages.
  2. The first time a page p is requested in Phase i, it might cause a page fault (if not already in cache).
  3. Once p is brought into the cache, can it ever be evicted during Phase i?
    • Under LRU, a page is only evicted if k other distinct pages are requested after it.
    • But there are at most k distinct pages in the entire phase!
    • Therefore, p can never be evicted during Phase i.
  4. Conclusion: Every distinct page faults at most once per phase: \text{cost}(\text{LRU}, \text{Phase } i) \le k

\text{OPT} Performance Across Phases

How many faults must offline \text{OPT} incur across two consecutive phases?

  • Phase i-1 ended because a new distinct page arrived.
  • Phase i accesses k distinct pages.
  • Together, Phase i-1 \cup \text{Phase } i contains at least k+1 distinct pages!
  • But \text{OPT}’s cache can hold at most k pages at any moment.
  • By the Pigeonhole Principle, \text{OPT} must fault at least once between the start of Phase i-1 and the end of Phase i: \text{cost}(\text{OPT}) \ge \frac{P - 1}{2} (With exact boundary accounting, \text{cost}(\text{OPT}) \ge P - 1).

\text{cost}(\text{LRU}) \le k \cdot P \le k \cdot (\text{OPT} + 1) \implies \mathbf{\text{LRU is } k\text{-competitive!}}

Breaking the k Barrier via Randomization

Why did deterministic algorithms hit a hard barrier of k?

  • The adversary knew exactly which page was evicted at every step.
  • What if the algorithm evicts a page at random from an eligible pool?

The Vision

Can randomization drop the competitive ratio from k \to O(\log k)?

  • For k = 1000 slots:
    • Deterministic ratio = 1000
    • Randomized ratio \approx 2 \ln(1000) \approx \mathbf{13.8}
    • Over a 70\times theoretical improvement!

The Randomized Marking Algorithm (Fiat et al. 1991)

Maintain a 1-bit “mark” (0 or 1) on every cache slot:

Algorithm Rules

  1. Initial State: All pages currently in cache are unmarked (\text{mark} = 0).
  2. On Request r_t:
    • Hit (r_t is in cache):
      • Simply set \text{mark}(r_t) = 1.
    • Page Fault / Miss (r_t is not in cache):
      • Case 1: If there are unmarked pages in cache:
        • Evict a page chosen uniformly at random from the UNMARKED pages.
      • Case 2: If ALL k pages are already marked (Phase ends):
        • Unmark all pages in the cache (a new phase begins).
        • Evict a page chosen uniformly at random (all are now unmarked).
      • Fetch requested page r_t into the freed slot and set \text{mark}(r_t) = 1.

Toy Example: Marking Algorithm (k = 3)

Suppose cache has 3 slots, currently holding \{p_1, p_2, p_3\}, all unmarked:

  • Step 1: Request p_1: Present! Mark p_1. Cache state: \{p_1^*, p_2, p_3\}.
  • Step 2: Request p_4 (Miss):
    • Unmarked pages are \{p_2, p_3\}.
    • Flip a coin: evict p_2 with prob 1/2, or p_3 with prob 1/2.
    • Suppose p_2 is evicted. Bring p_4 and mark it.
    • Cache state: \{p_1^*, p_4^*, p_3\}.
  • Step 3: Request p_1 again: Instant hit! p_1 is marked, so it was protected from eviction.
  • Key Insight: Marked pages are completely shielded. Only unreferenced pages face random eviction!

The Complete Landscape of Online Paging

Algorithm Type Competitive Ratio Practical Status
Deterministic Lower Bound Theoretical Limit \ge k Hard barrier for any det. alg
FIFO Deterministic k Simple, sub-optimal in practice
LRU Deterministic k Industry gold standard
Randomized Marking Randomized 2 H_k \approx 2 \ln k Simple 1-bit randomized mark
Equitable / Partition Randomized (Optimal) H_k \approx \ln k Matches randomized lower bound

Lecture Takeaway

Randomization conceals internal cache choices from the adversary, dropping competitive vulnerability from a linear penalty k to an exponential improvement \Theta(\log k).