How numbers are represented

A floating-point number stores a sign, a significand, and an exponent, like scientific notation with a fixed number of digits.

In a binary format, x≈(−1)ˢ×1.f×2ᵉ for normal numbers. The exponent controls scale and the fraction controls precision.

Machine epsilon describes the gap from 1 to the next representable number. Relative gaps are roughly stable for normal values, while absolute gaps grow with magnitude.

Overflow produces values too large for the format; underflow reaches tiny subnormal values or zero. Neither is the same as ordinary rounding.

A decimal such as 0.1 repeats in base two for the same reason one third repeats in base ten: the denominator has prime factors outside the base.

An ulp is the gap between neighbouring representable numbers near a value, making it a local measure of absolute precision that grows with the exponent.

Rounding mode decides how an exact result between representable numbers is chosen; round-to-nearest is common, but directed modes are useful for rigorous bounds.

Roundoff, cancellation, and summation

Subtracting two nearly equal rounded numbers can erase their shared leading digits and expose noisy leftovers.

Catastrophic cancellation is loss of relative accuracy after subtracting close quantities. Rewrite formulas to avoid it when possible.

Floating-point addition is not associative: (a+b)+c may differ from a+(b+c) because each intermediate result rounds.

Summing from small magnitude to large, pairwise summation, or Kahan compensated summation can reduce lost low-order contributions.

Cancellation does not create absolute error from nothing; it turns earlier small absolute rounding errors into a large relative error when the true difference is tiny.

Summation itself can be ill-conditioned when large positive and negative terms nearly cancel, so even a stable summing method cannot promise small relative error to a near-zero total.

A stable reformulation should be algebraically checked and tested across ordinary, tiny, huge, and sign-changing inputs because one branch may not be safe everywhere.

Practical deep dive for this lesson

Why a computer can lose a small but real difference

Floating-point numbers store a fixed number of significant binary digits across a huge range. Rounding is usually tiny, but subtraction or scale imbalance can expose and magnify what was discarded.

A computer does not store every real number. It stores sign, exponent, and a finite significand, much like scientific notation with a fixed number of digits.

Values such as decimal 0.1 usually have repeating binary expansions, so the stored value is the nearest representable neighbour rather than exact 0.1.

Machine epsilon describes spacing near one, but absolute spacing grows with magnitude. Near 10¹⁶ in binary64, neighbouring representable numbers are farther apart than one.

Adding a tiny number to a huge one may therefore leave the stored huge number unchanged. The small contribution falls below available resolution.

Catastrophic cancellation happens when two close rounded numbers are subtracted. Leading digits cancel and the remaining result consists largely of earlier rounding error.

Algebraic rewriting can avoid this. For small x, √(1+x)−1 is better computed as x/[√(1+x)+1], which replaces dangerous subtraction with stable operations.

Overflow means a magnitude exceeds the format's range; underflow makes tiny values round toward zero or subnormal values. Both can break later formulas.

Stable log-sum-exp, compensated summation, wider accumulators, and careful scaling are algorithmic responses to these representation limits.

Tests should compare within tolerances tied to the calculation's scale and real requirement, not demand exact equality from rounded approximations.

Numerical literacy separates harmless last-bit differences from errors that reverse a decision, violate conservation, or accumulate into a material total.