Confidence Intervals: Embracing Uncertainty

When we estimate a number (like the average height of a population), stating a single number (“The average is 170cm”) is almost certainly wrong. It’s an overconfident Point Estimate.

Instead, we should provide a range: “The average is likely between 168cm and 172cm”. This is a Confidence Interval (CI).

1. The Fishing Net Analogy

Imagine you are trying to catch a fish (the true population mean) in a murky lake.

Spear

Point Estimate

Trying to hit the fish with a single spear point.

Outcome: Likely to Miss

Net

Confidence Interval

Throwing a wide net to capture the fish.

Outcome: Likely to Catch

[!NOTE] Confidence Level The Confidence Level (e.g., 95%) is the size of the net. A 99% net is wider (less precise) but catches the fish more often. A 90% net is narrower (more precise) but might miss more often.

2. Interactive: The CI “Catcher”

Let’s visualize what “95% Confidence” actually means. It does not mean “there is a 95% probability the true mean is in this specific interval”.

It means: If we repeated this experiment 100 times, we would expect 95 of our calculated intervals to contain the true population mean.

Click “Draw 20 Samples” below. The green line is the True Population Mean. The bars are the confidence intervals calculated from random samples.

  • Blue: The interval “caught” the true mean.
  • Red: The interval “missed” the true mean.
Success Rate: -

3. Hardware Reality: Sensor Fusion

In robotics and self-driving cars, “Confidence Intervals” are a matter of life and death.

A LIDAR sensor might report an obstacle at 10.5m ± 0.2m. A Camera might report the same obstacle at 10.8m ± 0.5m.

How do we combine these? We use a Kalman Filter.

  • The Kalman Filter treats each measurement as a Gaussian distribution (Mean + Variance).
  • It multiplies the distributions together.
  • The result is a new distribution with a lower variance (narrower confidence interval) than either sensor alone.

This is mathematically why 5 cheap sensors can be better than 1 expensive sensor. The “Intersection” of their confidence intervals pinpoints the truth.

4. The Computational Way: Bootstrapping

The formula Mean ± Z * SE assumes the data is Normal. What if your data is skewed, like website latency or income?

We use Bootstrapping:

  1. Take your original sample (e.g., 100 data points).
  2. Resample from it with replacement 10,000 times.
  3. Calculate the mean for each of these 10,000 “fake” samples.
  4. Cut off the top 2.5% and bottom 2.5% of these means.
  5. The remaining range is your 95% Confidence Interval.

Python Implementation

import numpy as np
import matplotlib.pyplot as plt

# 1. Our single observed sample (e.g., 50 server latencies)
# Highly skewed (Exponential)
data = np.random.exponential(scale=200, size=50)

print(f"Observed Mean: {np.mean(data):.2f}")

# 2. Bootstrapping
# Resample with replacement 10,000 times
bootstrap_means = []
n_iterations = 10000

for _ in range(n_iterations):
    # The Magic: Resample from our own data!
    sample = np.random.choice(data, size=len(data), replace=True)
    bootstrap_means.append(np.mean(sample))

# 3. Calculate 95% CI using Percentiles
lower_bound = np.percentile(bootstrap_means, 2.5)
upper_bound = np.percentile(bootstrap_means, 97.5)

print(f"95% Confidence Interval: [{lower_bound:.2f}, {upper_bound:.2f}]")

# 4. Visualization
plt.figure(figsize=(10, 5))
plt.hist(bootstrap_means, bins=50, color='skyblue', edgecolor='black', alpha=0.7)
plt.axvline(lower_bound, color='red', linestyle='--', label='95% CI')
plt.axvline(upper_bound, color='red', linestyle='--')
plt.title('Bootstrap Distribution of Sample Means')
plt.legend()
plt.show()

Why this works: By resampling with replacement, we are treating our sample as a “mini-population”. The variation in the resampled means approximates the sampling distribution of the true mean.

5. Summary

  • Point Estimates are risky; Confidence Intervals provide context.
  • 95% Confidence means the method works 95% of the time, not that the specific interval has a 95% probability.
  • Bootstrapping is a powerful modern technique to calculate CIs for any data distribution without memorizing formulas.
  • Sensor Fusion (Kalman Filters) is essentially the mathematical intersection of multiple confidence intervals to reduce uncertainty.