BITS Pilani
BITS Logo
CS G526: Advanced Algorithms and Complexity

Number Theory & Randomized Primality Testing

Lectures 19–20 |2026-09-24
Tulasimohan Molli
BITS Pilani, Hyderabad Campus

Agenda

Part I: Primality & The Fermat Filter

  • Arithmetic Complexity: Bit operations on k-bit integers.
  • Density: Infinitude & Prime Number Theorem.
  • The Euclidean Algorithm for \gcd: O(\log n) efficiency.
  • Fermat’s Test & The Carmichael Flaw: Why Fermat fails.

Part II: Group Theory & Miller–Rabin

  • Group Theory Primer: Groups, (\mathbb{Z}_n^*, \cdot), and Lagrange’s Theorem.
  • The CRT Bijection: \mathbb{Z}_n^* \cong \mathbb{Z}_p^* \times \mathbb{Z}_q^* & square roots of 1.
  • The Miller–Rabin Test: Squaring chain & live trace on n = 65.
  • Analysis: Rabin’s Subgroup Theorem (\le 1/4 error).
  • Complexity: \text{co-RP}, \text{BPP}, and AKS in \text{P}.

Part I: Number-Theoretic Foundations

  • Part I: Number-Theoretic Foundations
  • Part II: Group Theory & The Miller–Rabin Test

What is a Prime Number?

Primes & Composites

  • An integer p > 1 is prime if its only positive divisors are 1 and p.
  • An integer n > 1 is composite if n = a \cdot b for some integers 1 < a, b < n.
  • Two integers a, b are coprime (or relatively prime) if \gcd(a, b) = 1.
  • The Input Size Representation:

    • For a number n, the input length is its number of bits: k = \lceil \log_2(n+1) \rceil

    • An algorithm running in time polynomial in n is exponential in the input size k (O(n) = O(2^k)).

    • An efficient algorithm must run in time \text{poly}(k) = \text{poly}(\log n)!

Bit Complexity of Basic Operations

Let a, b be k-bit integers (k \approx \log_2 n):

Operation Naive Cost Best Known Cost
Addition / Subtraction (a \pm b) O(k) O(k)
Multiplication (a \cdot b) O(k^2) O(k \log k)
(Harvey–Hoeven 2019)
Division (a = qb + r) O(k^2) O(k \log k)
Modular Exponentiation (a^b \bmod m) O(k^3)
(Repeated squaring: O(k) mults)		 \tilde{O}(k^2)
GCD (\gcd(a, b)) O(k^2)
(Euclid: O(k) divisions)		 \tilde{O}(k)
(Half-GCD)

Subquadratic arithmetic is what makes 2048-bit modular exponentiation in RSA practical in milliseconds!

Can We Improve Integer Multiplication from n^2?

  • The Goal: Multiply two n-digit numbers faster than grade-school O(n^2).
  • Karatsuba’s Insight (1960):

    • Split each number into halves: x = x_1 2^{n/2} + x_0, y = y_1 2^{n/2} + y_0.

    • Standard recursion needs 4 multiplications: x_1 y_1, \; x_1 y_0, \; x_0 y_1, \; x_0 y_0

    • Only recurse three times instead of four!

Karatsuba Algorithm: 3 Multiplications

Recurrence & Depth

  • T(n) = 3T(n/2) + O(n)
  • Cutting in half \log_2 n times yields: 3^{\log_2 n} = n^{\log_2 3} \approx \mathbf{O(n^{1.585})}

. . .

  • Relevance to Primality Testing:
    • Testing primality requires modular exponentiation (a^{n-1} \bmod n).
    • Repeated squaring takes O(\log n) multiplications of k-bit numbers.
    • Faster multiplication makes 2048-bit arithmetic practical!

How Many Primes Are There? Density

Euclid’s Theorem (~300 BCE)

There exist infinitely many primes.
Proof: If \{p_1, \dots, p_k\} were all primes, N = p_1 \cdots p_k + 1 has a prime factor not in the list.

  • Prime Number Theorem (Hadamard & de la Vallée Poussin, 1896): \pi(x) \sim \frac{x}{\ln x} \implies \Pr_{n \in [1, x]}[n \text{ is prime}] \approx \frac{1}{\ln x}

  • For a 2048-bit integer (x \approx 2^{2048}): \Pr[n \text{ is prime}] \approx \frac{1}{2048 \ln 2} \approx \frac{1}{1419}

  • Takeaway for Key Generation: A random 2048-bit odd number has a 1 / 710 chance of being prime. We only need \approx 700 trials! Crucial requirement: We need an ultrafast primality test!

The Euclidean Algorithm for \gcd

Invariant of the Euclidean Algorithm

For integers a \ge b > 0: \gcd(a, b) = \gcd(b, a \bmod b)

def gcd(a, b):
  while b != 0:
    a, b = b, a % b
  return a
  • Why does it terminate fast?
    • When a \ge b, a \bmod b < a/2.
    • The product a \cdot b drops by at least a factor of 2 every step.
    • Number of steps \le 2 \log_2 b = O(k).
  • Lamé’s Theorem (1844): The worst case occurs on consecutive Fibonacci numbers (F_{k+1}, F_k).

Fermat’s Little Theorem: The First Primality Filter

Fermat’s Little Theorem (1640)

If p is prime and \gcd(a, p) = 1, then: a^{p-1} \equiv 1 \pmod p

The Fermat Test for n:

  1. Pick a random base a \in \{2, \dots, n-2\}.
  2. Compute x = a^{n-1} \bmod n.
  3. If x \ne 1, n is definitely composite! (a is a Fermat witness).
  4. If x = 1, declare n “probably prime”.

The Fatal Flaw

Are there composite numbers where a^{n-1} \equiv 1 \pmod n for all coprime bases a?

YES: Carmichael Numbers!

Carmichael Numbers: The Impostors

Definition: Carmichael Numbers

A composite integer n is a Carmichael number if for every base a with \gcd(a, n) = 1: a^{n-1} \equiv 1 \pmod n

  • The Smallest Impostor: 561 = 3 \times 11 \times 17
    • 561 - 1 = 560.
    • Notice: (3 - 1) \mid 560, (11 - 1) \mid 560, and (17 - 1) \mid 560!
    • Thus a^{560} - 1 is simultaneously divisible by 3, 11, and 17, forcing: a^{560} \equiv 1 \pmod{561} \quad \text{for all } a \text{ with } \gcd(a, 561) = 1!
  • Alford, Granville, Pomerance (1994): There exist infinitely many Carmichael numbers!
  • Verdict: Fermat’s test can be fooled infinitely often. We need a deeper algebraic invariant.

Part II: Group Theory & The Miller–Rabin Test

  • Part I: Number-Theoretic Foundations
  • Part II: Group Theory & The Miller–Rabin Test

Primer: What is a Group?

Definition: Group (G, \circ)

A set G equipped with a binary operation \circ : G \times G \to G is a group if it satisfies four axioms:

  1. Closure: For all a, b \in G, a \circ b \in G.

  2. Associativity: For all a, b, c \in G, (a \circ b) \circ c = a \circ (b \circ c).

  3. Identity: There exists e \in G such that for all a \in G: a \circ e = e \circ a = a

  4. Inverse: For each a \in G, there exists a^{-1} \in G such that: a \circ a^{-1} = a^{-1} \circ a = e

If a \circ b = b \circ a for all a, b \in G, the group is called Abelian (commutative).

Two Fundamental Examples: (\mathbb{Z}_n, +) and (\mathbb{Z}_n^*, \cdot)

1. Additive Group (\mathbb{Z}_n, +)

  • Set: \{0, 1, 2, \dots, n-1\}.

  • Operation: Addition modulo n.

  • Identity: e = 0.

  • Inverse: -a = (n - a) \bmod n.

  • Group Size (Order): |(\mathbb{Z}_n, +)| = n

  • Holds for any integer n \ge 1.

2. Multiplicative Group (\mathbb{Z}_n^*, \cdot)

  • Set: Elements coprime to n: \mathbb{Z}_n^* = \{a \in \{1, \dots, n-1\} : \gcd(a, n) = 1\}

  • Operation: Multiplication modulo n.

  • Identity: e = 1.

  • Inverse: Modular inverse via Euclid!

  • Group Size: Euler’s totient \phi(n).

    • If n = p is prime: |\mathbb{Z}_p^*| = \mathbf{p - 1}.

Subgroups & Lagrange’s Theorem

Definition: Subgroup

A subset H \subseteq G is a subgroup (written H \le G) if H is itself a group under the same operation \circ:

  1. e \in H (contains identity)
  2. Closed under operation: a, b \in H \implies a \circ b \in H
  3. Closed under inverses: a \in H \implies a^{-1} \in H

Lagrange’s Theorem (1777)

If G is a finite group and H \le G is a subgroup, then the size of H divides the size of G: \frac{|G|}{|H|} = [G : H] \in \mathbb{Z}^+ \implies |H| \text{ divides } |G|

  • Immediate Consequence: If H is a proper subgroup (H \ne G), then: |H| \le \frac{1}{2}|G|

  • Why this powers randomized primality: If “bad” witness bases form a proper subgroup of \mathbb{Z}_n^*, then at most half the elements can fool us!

The Chinese Remainder Theorem: Group Bijection

How does the multiplicative group \mathbb{Z}_n^* behave when n = p \cdot q is composite (\gcd(p, q) = 1)?

The CRT Bijection

The mapping: f : \mathbb{Z}_n^* \longrightarrow \mathbb{Z}_p^* \times \mathbb{Z}_q^* \quad \text{defined by} \quad f(x) = (x \bmod p, \; x \bmod q) is a bijective group homomorphism (an isomorphism): \mathbb{Z}_n^* \cong \mathbb{Z}_p^* \times \mathbb{Z}_q^*

  • One-to-One Correspondence:

    • For every pair (u, v) \in \mathbb{Z}_p^* \times \mathbb{Z}_q^*, there is a unique x \in \mathbb{Z}_n^* such that: x \equiv u \pmod p \quad \text{and} \quad x \equiv v \pmod q

  • Multiplication is Component-Wise: f(x \cdot y) = (xy \bmod p, \; xy \bmod q) = (x \bmod p, \; x \bmod q) \cdot (y \bmod p, \; y \bmod q)

  • Why This Unlocks Non-Trivial Square Roots:

    • To solve x^2 \equiv 1 \pmod n, we independently solve u^2 \equiv 1 \pmod p and v^2 \equiv 1 \pmod q.
    • Since u \in \{\pm 1\} and v \in \{\pm 1\}, the combinations yield 4 distinct square roots of 1 in \mathbb{Z}_n^*!

The Missing Invariant: Square Roots of 1

What happens when we take the square root of 1 modulo n?

x^2 \equiv 1 \pmod n \iff x^2 - 1 \equiv (x - 1)(x + 1) \equiv 0 \pmod n

If n = p is Prime:

\mathbb{Z}_p is a field. A polynomial of degree 2 has at most 2 roots: x \equiv +1 \quad \text{or} \quad x \equiv -1 \pmod p There are no other roots!

If n = p \cdot q is Composite:

By the CRT bijection \mathbb{Z}_n^* \cong \mathbb{Z}_p^* \times \mathbb{Z}_q^*: (x \bmod p, \; x \bmod q) \in \{(\pm 1, \; \pm 1)\}

  • (+1, +1) \leftrightarrow \mathbf{+1} and (-1, -1) \leftrightarrow \mathbf{-1} (trivial).
  • (+1, -1) and (-1, +1) produce nontrivial roots: x \not\equiv \pm 1 \pmod n \quad \text{yet} \quad x^2 \equiv 1 \pmod n

A Live Example of Nontrivial Roots: n = 65

Consider n = 65 = 5 \times 13:

x = 14 \pmod{65} x^2 = 14^2 = 196 = 3 \times 65 + 1 \equiv \mathbf{1 \pmod{65}}

The Discovery

14 \not\equiv \pm 1 \pmod{65}, yet 14^2 \equiv 1 \pmod{65}!
14 is an irrefutable proof that 65 is composite.

  • Bonus: Direct Factor Extraction via Euclid! \gcd(x - 1, n) = \gcd(14 - 1, 65) = \gcd(13, 65) = \mathbf{13} \gcd(x + 1, n) = \gcd(14 + 1, 65) = \gcd(15, 65) = \mathbf{5} Any non-trivial square root of 1 cracks the factorization of n!

The Miller–Rabin Squaring Chain

How do we systematically search for nontrivial square roots of 1?

Write n - 1 as: n - 1 = 2^s \cdot d \quad (d \text{ odd})

For any base a \in \mathbb{Z}_n^*, consider the sequence obtained by repeated squaring: \langle x_0, \; x_1, \; x_2, \; \dots, \; x_s \rangle x_0 = a^d \bmod n, \quad x_1 = a^{2d} \bmod n, \quad x_2 = a^{4d} \bmod n, \quad \dots, \quad x_s = a^{2^s d} = a^{n-1} \bmod n

The Prime Invariant

If n is prime, then:

  1. The last element must be x_s = a^{n-1} \equiv 1 \pmod n.
  2. The first time 1 appears in the sequence, the preceding value must be -1 \equiv n - 1 (unless x_0 = 1 from the start)!

The Miller–Rabin Algorithm

def miller_rabin(n, k=40):
  # Write n - 1 = 2^s * d with d odd
  s, d = 0, n - 1
  while d % 2 == 0:
    s += 1
    d //= 2

  for _ in range(k):
    a = random.randint(2, n - 2)
    x = pow(a, d, n)
    if x == 1 or x == n - 1:
      continue  # Passes this round

    for r in range(s - 1):
      x = pow(x, 2, n)
      if x == n - 1:
        break  # Safely reached -1 before 1
      if x == 1:
        return "COMPOSITE"  # Caught nontrivial square root of 1!
    else:
      return "COMPOSITE"  # Never reached -1; a^(n-1) != 1

  return "PROBABLY PRIME"

Trace on n = 65 = 5 \times 13

Factor n - 1 = 64 = 2^6 \cdot 1 \implies s = 6, \; d = 1.

Base a = 8 (Fails to catch 65)

  • x_0 = 8^1 = 8 \pmod{65}

  • x_1 = 8^2 = 64 \equiv \mathbf{-1} \pmod{65}

  • Hits -1 immediately!

  • Subsequent terms will be: (-1)^2 = 1, \; 1, \; 1, \dots

  • a = 8 is a liar (does not witness compositeness).

Base a = 12 (Catches 65!)

  • x_0 = 12^1 = 12 \pmod{65}

  • x_1 = 12^2 = 144 \equiv 14 \pmod{65}

  • x_2 = 14^2 = 196 \equiv \mathbf{1} \pmod{65}

  • x_2 = 1, but predecessor was: x_1 = 14 \not\equiv -1 \pmod{65}

  • Nontrivial root of 1 detected!

  • Outputs: COMPOSITE.

Can Composites Hide? Rabin’s Theorem

Could there be a composite number where 99.9\% of bases are liars?

Rabin’s Subgroup Theorem (1980)

If n is an odd composite number, then at most \frac{1}{4} of the bases a \in \{1, 2, \dots, n-1\} fail to witness the compositeness of n: \Pr_{a \sim \text{Unif}}[a \text{ is a liar for } n] \le \frac{1}{4}

  • Amplification via k Independent Rounds: \Pr[\text{Error on composite } n] \le \left(\frac{1}{4}\right)^k = 2^{-2k}

  • For k = 40 rounds: \text{Error} \le 2^{-80} \approx 10^{-24} (Far lower than the probability of cosmic ray hardware memory corruption!)

Carmichael Numbers & CRT: Independent Squaring

For any Carmichael number n = p \cdot q (\gcd(p, q) = 1), every base satisfies a^{n-1} \equiv 1 \pmod n. Why does repeated squaring catch them so easily?

The Component View via CRT

By the isomorphism \mathbb{Z}_n^* \cong \mathbb{Z}_p^* \times \mathbb{Z}_q^*, each element in the chain is a pair: x_t \longleftrightarrow (x_t \bmod p, \; x_t \bmod q) Both components square independently at every step!

  • What can precede +1 \equiv (1, 1)? Just before hitting (1, 1), each component must be either +1 or -1: (+1, +1) \leftrightarrow \mathbf{+1} \quad \text{and} \quad (-1, -1) \leftrightarrow \mathbf{-1} \quad \text{— Only 2 Trivial Roots!}

  • What happens when components split signs? (+1, -1)^2 = (1, 1) \equiv \mathbf{1} \quad \text{and} \quad (-1, +1)^2 = (1, 1) \equiv \mathbf{1 \pmod n} Squares to +1, but the predecessor is neither +1 nor -1 \pmod n! \mathbf{A \; non\text{-}trivial \; square \; root \; of \; 1 \; is \; exposed!}

Sign-Splitting: The Probability Bound

Why can at most a small fraction of bases be liars on Carmichael numbers?

Trivial Roots (Liars)

Signs match across all components: (+1, +1, \dots, +1) \leftrightarrow \mathbf{+1} (-1, -1, \dots, -1) \leftrightarrow \mathbf{-1} Only 2 combinations can safely hit +1 without revealing a root!

Split Signs (Witnesses)

Signs differ across components: (+1, -1, \dots), \; (-1, +1, \dots) Every split sign combination exposes a non-trivial square root of 1!

The Math

  • For n with r \ge 3 prime factors, there are 2^r \ge 8 sign patterns, all equally distributed by CRT.
  • Only the 2 uniform patterns fail to expose a non-trivial root: \Pr[a \text{ is a liar}] \le \frac{2}{2^r} \le \frac{2}{8} = \mathbf{\frac{1}{4}}

Complexity Classification of PRIMES

Where does Miller–Rabin situate the computational complexity of primality?

Asymmetric One-Sided Error

  • If n \in \text{PRIMES} \implies \Pr[\text{Outputs "PROBABLY PRIME"}] = 1 (Zero error!).
  • If n \in \text{COMPOSITES} \implies \Pr[\text{Outputs "COMPOSITE"}] \ge 1 - 2^{-2k}.

  • Therefore, the language \text{COMPOSITES} \in \mathbf{RP}.

  • By complement: \mathbf{PRIMES \in co\text{-}RP \subseteq BPP}

  • Pratt’s Certificate (1975): \text{PRIMES} \in \mathbf{NP} \cap \mathbf{co\text{-}NP}.

  • Agrawal, Kayal, Saxena (AKS, 2002): Proved that \mathbf{PRIMES \in P} deterministically in \tilde{O}(\log^6 n) time!

  • Practical Reality: AKS runtime is \approx O(\log^6 n) vs Miller–Rabin’s O(\log^3 n). For 2048-bit keys, Miller–Rabin takes fractions of a millisecond; AKS takes minutes. Miller–Rabin powers all modern cryptography.

Summary & Bridge to Online Algorithms

What We Achieved Today:

  • Number-theoretic machinery: Euclid, CRT, Fermat.
  • Uncovered why Carmichael numbers fool Fermat’s test.
  • Turned non-trivial square roots of 1 into an algorithmic test.
  • Proved that witnesses outnumber liars \ge 3:1 via subgroups.

Coming Up Next:

  • Online Algorithms & Competitive Analysis:
    • When inputs arrive dynamically one by one.
    • How to make optimal decisions under total uncertainty.
    • The Ski Rental Problem and Competitive Ratios.