Data Compression Problem

Suppose we have a text file. We want to represent the same information using fewer bits.

Fixed-length encoding: All characters use the same number of bits.

• ASCII: 7 bits per character
• 8-bit extended ASCII: 8 bits per character

Variable-length encoding: Frequent characters get shorter codes, infrequent characters get longer codes.

• Like Morse code: E (most common) = ., Q = --.-

What are Prefix codes?

A prefix code is a variable-length code where no codeword is a prefix of any other codeword.

This ensures unambiguous decoding — we can parse a bitstream uniquely left-to-right.

Variable code uses 0.60 \times 1 + 0.25 \times 2 + 0.10 \times 3 + 0.05 \times 3 = 1.55 bits/char on average vs 2 bits/char fixed.

Binary Tree Representation

Every prefix code corresponds to a binary tree:

• Leaves represent characters, labeled with their frequency.
• Left edge = 0, Right edge = 1.
• Codeword = path from root to leaf.

graph TD
    Root["1.00"] -->|"0"| A["0.60<br/>A: 0"]
    Root -->|"1"| N1["0.40"]
    N1 -->|"0"| B["0.25<br/>B: 10"]
    N1 -->|"1"| N2["0.15"]
    N2 -->|"0"| C["0.10<br/>C: 110"]
    N2 -->|"1"| D["0.05<br/>D: 111"]

Optimal prefix code: the tree that minimizes \sum f(c) \cdot \text{depth}(c).

Problem Formulation

Input: A set of characters C, each with frequency f(c).

Output: A prefix code (binary tree) that minimizes the total cost:

B(T) = \sum_{c \in C} f(c) \cdot d_T(c)

where d_T(c) is the depth of character c in the tree T.

This is exactly the optimal prefix code or minimum-cost binary tree problem.

Key Insight

In an optimal prefix-code tree:

• Every internal node has exactly two children.
• If a node has only one child, we can eliminate it and shorten the codes — would not be optimal.

Observation: The two characters with the smallest frequencies should be at the bottom of the tree, as siblings. This is the greedy choice.

Algorithm

Huffman(C):
    // C is a set of n characters with frequencies f(c)
    n = |C|
    Q = C    // min-priority queue keyed by f(c)
    
    for i = 1 to n - 1:
        x = Extract-Min(Q)   // smallest frequency
        y = Extract-Min(Q)   // second smallest
        z = new node
        f(z) = f(x) + f(y)
        z.left = x
        z.right = y
        Insert(Q, z)
    
    return Extract-Min(Q)    // root of Huffman tree

• Uses a min-heap (priority queue) to repeatedly merge the two least frequent nodes.

Worked Example

Characters: a: 45, b: 13, c: 12, d: 16, e: 9, f: 5

Step 1

Merge e (9) and f (5) → internal node with frequency 14.

graph TD
    A["a: 45"]
    B["b: 13"]
    C["c: 12"]
    D["d: 16"]
    N14["14"] --> e["e: 9"]
    N14 --> f["f: 5"]
    style N14 fill:#e74c3c,color:#fff,stroke-width:2px

Step 2

Merge c (12) and b (13) → node (25).

graph TD
    A["a: 45"]
    D["d: 16"]
    N14["14"] --> e["e: 9"]
    N14 --> f["f: 5"]
    N25["25"] --> c["c: 12"]
    N25 --> b["b: 13"]
    style N25 fill:#e74c3c,color:#fff,stroke-width:2px

Step 3

Merge node (14) and d (16) → node (30).

graph TD
    A["a: 45"]
    N25["25"] --> c["c: 12"]
    N25 --> b["b: 13"]
    N30["30"] --> N14["14"]
    N30 --> d["d: 16"]
    N14 --> e["e: 9"]
    N14 --> f["f: 5"]
    style N30 fill:#e74c3c,color:#fff,stroke-width:2px

Step 4

Merge node (25) and node (30) → node (55).

graph TD
    A["a: 45"]
    N55["55"] --> N25["25"]
    N55 --> N30["30"]
    N25 --> c["c: 12"]
    N25 --> b["b: 13"]
    N30 --> N14["14"]
    N30 --> d["d: 16"]
    N14 --> e["e: 9"]
    N14 --> f["f: 5"]
    style N55 fill:#e74c3c,color:#fff,stroke-width:2px

Step 5

Merge a (45) and node (55) → root (100).

graph TD
    Root["100"] --> a["a: 45"]
    Root --> N55["55"]
    N55 --> N25["25"]
    N55 --> N30["30"]
    N25 --> c["c: 12"]
    N25 --> b["b: 13"]
    N30 --> N14["14"]
    N30 --> d["d: 16"]
    N14 --> e["e: 9"]
    N14 --> f["f: 5"]
    style Root fill:#e74c3c,color:#fff,stroke-width:2px

Final Tree

graph TD
    Root["100"] -->|"0"| a["a: 45"]
    Root -->|"1"| N55["55"]
    N55 -->|"0"| N25["25"]
    N55 -->|"1"| N30["30"]
    N25 -->|"0"| c["c: 12"]
    N25 -->|"1"| b["b: 13"]
    N30 -->|"0"| N14["14"]
    N30 -->|"1"| d["d: 16"]
    N14 -->|"0"| e["e: 9"]
    N14 -->|"1"| f["f: 5"]

Total: 45\!\times\!1 + 13\!\times\!3 + 12\!\times\!3 + 16\!\times\!3 + 9\!\times\!4 + 5\!\times\!4 = 224 bits.

Fixed-length (3 bits × 100 chars) = 300 bits. 25% savings.

Complexity

• Building initial heap: O(n)
• Each iteration performs two Extract-Min + one Insert: O(\log n)
• n - 1 iterations total: O(n \log n)

Overall: \Theta(n \log n) time, O(n) space.

Greedy Choice Property

Optimal Substructure