Bootstrap Method

In classical statistics, we often assume data follows a specific distribution (like the Normal distribution) to calculate confidence intervals or test hypotheses. But what if your data is skewed, has outliers, or the sample size is too small for the Central Limit Theorem to kick in?

Enter the Bootstrap Method.

Invented by Bradley Efron in 1979, bootstrapping is a powerful computer-intensive technique that allows you to estimate the sampling distribution of almost any statistic using only your available data. It requires no assumptions about the underlying population distribution.

[!NOTE] The name comes from the phrase “pulling oneself up by one’s bootstraps,” referring to the seemingly impossible task of estimating the entire population’s properties from just a single sample.

1. The Core Concept: Resampling with Replacement

The fundamental idea is deceptively simple: treat your sample as if it were the population.

  1. Original Sample: You have a dataset S of size n.
  2. Resampling: Create a “bootstrap sample” S* by drawing n observations from S with replacement.
    • Some observations from S will appear multiple times in S*.
    • Some will not appear at all.
  3. Statistic: Calculate your statistic of interest (mean, median, variance, etc.) on S*. Let’s call this θ*.
  4. Repeat: Do this B times (typically B</i ≥ 1000 or more).
  5. Inference: The distribution of these B statistics (θ*1, &dots;, θ*B) approximates the true sampling distribution of the statistic.

2. Interactive: The Bootstrapper

Let’s visualize how resampling works. Below is a small dataset. Click “Generate Bootstrap Sample” to create a new sample of the same size (with replacement) and calculate its mean. Watch how the distribution of bootstrap means builds up to approximate the sampling distribution.

Original Sample

Original Mean: --

Current Bootstrap Sample

Click generate...

Sample Mean: --

Distribution of Bootstrap Means

Bootstrap Samples: 0

Standard Error (Est): 0.00

95% Confidence Interval: [ -- , -- ]

3. Implementation Examples

We can easily implement the Bootstrap method using Python, Java, or Go. Here’s how to calculate the 95% confidence interval for the mean.

Python (NumPy)

import numpy as np

# 1. Our sample data
data = np.array([12, 15, 18, 22, 25, 29, 35, 42])

# 2. Parameters
n_iterations = 10000
n_size = len(data)

# 3. Bootstrap Resampling
bootstrap_means = np.empty(n_iterations)

for i in range(n_iterations):
    # Resample with replacement
    resampled_data = np.random.choice(data, size=n_size, replace=True)
    # Calculate statistic (mean)
    bootstrap_means[i] = np.mean(resampled_data)

# 4. Calculate Confidence Interval
lower_bound = np.percentile(bootstrap_means, 2.5)
upper_bound = np.percentile(bootstrap_means, 97.5)
std_error = np.std(bootstrap_means)

print(f"95% CI: [{lower_bound:.2f}, {upper_bound:.2f}]")

Java

import java.util.Arrays;
import java.util.Random;

public class Bootstrap {
    public static void main(String[] args) {
        double[] data = {12, 15, 18, 22, 25, 29, 35, 42};
        int nObj = data.length;
        int nIter = 10000;
        double[] means = new double[nIter];
        Random rand = new Random();

        // Bootstrap Resampling
        for (int i = 0; i < nIter; i++) {
            double sum = 0;
            for (int j = 0; j < nObj; j++) {
                // Random index with replacement
                sum += data[rand.nextInt(nObj)];
            }
            means[i] = sum / nObj;
        }

        // Calculate Percentiles
        Arrays.sort(means);
        double lower = means[(int)(0.025 * nIter)];
        double upper = means[(int)(0.975 * nIter)];

        System.out.printf("95%% CI: [%.2f, %.2f]%n", lower, upper);
    }
}

Go

package main

import (
	"fmt"
	"math/rand"
	"sort"
	"time"
)

func main() {
	data := []float64{12, 15, 18, 22, 25, 29, 35, 42}
	n := len(data)
	nIter := 10000
	means := make([]float64, nIter)

	// Seed random number generator
	rand.Seed(time.Now().UnixNano())

	// Bootstrap Resampling
	for i := 0; i < nIter; i++ {
		sum := 0.0
		for j := 0; j < n; j++ {
			sum += data[rand.Intn(n)]
		}
		means[i] = sum / float64(n)
	}

	// Calculate Percentiles
	sort.Float64s(means)
	lower := means[int(0.025*float64(nIter))]
	upper := means[int(0.975*float64(nIter))]

	fmt.Printf("95%% CI: [%.2f, %.2f]\n", lower, upper)
}

Why does this work?

The logic relies on the Plug-in Principle:

  • The Population is to the Sample…
  • …as the Sample is to the Bootstrap Samples.

By repeatedly sampling from our data, we simulate the process of sampling from the population. This allows us to “see” the variability of our statistic without needing to collect more data.

4. When to Use Bootstrap

Scenario Use Bootstrap? Reason
Unknown Distribution ✅ Yes No need to assume Normality.
Complex Statistic ✅ Yes Standard error formulas for median or ratio are hard/impossible to derive analytically.
Small Sample Size ✅ Yes (With caution) Better than asymptotic assumptions that haven’t kicked in yet.
Extreme Outliers ⚠️ Be Careful Bootstrap is sensitive to outliers in small samples.
Dependent Data ❌ No Standard bootstrap assumes i.i.d. (independent and identically distributed) data. Use Block Bootstrap for time series.

[!WARNING] Sample Size Matters: Bootstrapping cannot create information that isn’t there. If your original sample is biased or too small to represent the population structure, your bootstrap estimates will also be biased. It estimates the sampling distribution of the statistic, it doesn’t fix a bad sample.

5. Advanced: The Bootstrap Confidence Intervals

There are multiple ways to calculate the confidence interval (CI) from the bootstrap distribution:

  1. Percentile Method (Used above): Take the 2.5th and 97.5th percentiles. Simple and intuitive.
  2. Basic (Reverse Percentile) Method: Corrects for bias in the bootstrap distribution.
  3. BCa (Bias-Corrected and Accelerated): The gold standard. Corrects for both bias and skewness, but computationally more intensive.

For most practical data science applications, the Percentile Method is a sufficient approximation, provided the bootstrap distribution is roughly symmetric.