PYTHON / SCIPY
Statistical distributions and hypothesis tests
Query SciPy's distribution objects (pdf, cdf, ppf, fit) and run t-, chi-square and rank tests, reading a p-value as a tail area under the null distribution.
What you will learn
- Freeze a distribution with stats.norm(loc, scale) and query cdf, sf, ppf and interval
- Read any p-value as the tail area of the test statistic under its null distribution
- Match the test to the design: ttest_1samp, ttest_rel, chisquare, mannwhitneyu
- Reproduce a t-test p-value by hand with 2 * stats.t.sf(abs(t), df=n-1)
Understanding Statistical distributions and hypothesis tests
Every distribution in scipy.stats is an object exposing the same handful of methods, which is what makes the module worth learning once: pdf (or pmf for discrete ones) for density, cdf for the left tail, sf for the right tail, ppf for quantiles, rvs for samples, fit for parameter estimation. Continuous distributions are all built from one standard shape that you shift with loc and stretch with scale, so stats.norm(loc=10, scale=2) and stats.gamma(a=2, scale=3) are parameterised the same way even though only one of them has a shape parameter. Calling the class with arguments freezes it: the parameters are stored once and every later method call reuses them. Prefer sf(x) over 1 - cdf(x) deep in the tail, because 1 - cdf loses all its significant digits as soon as cdf rounds to 1.0.
A hypothesis test function is a thin wrapper over those same objects. ttest_1samp computes t = (mean - mu0) / (s / sqrt(n)), then asks the t distribution with n - 1 degrees of freedom how much probability lies beyond |t| in both tails. That is why the main example below reproduces the reported p-value with 2 * stats.t.sf(abs(t), df) to the last digit: nothing else is happening. Once you see a p-value as a tail area computed under an assumed null distribution, the degrees of freedom stop being magic and you can always check a test result yourself.
Choosing a test is choosing which assumptions you are willing to make. ttest_ind assumes roughly normal data and (by default) equal variances; mannwhitneyu only compares ranks, so it survives skew and outliers, but with tiny samples its null distribution is discrete and coarse: with 3 observations per group there are only 20 rank orderings, so the smallest possible two-sided p-value is 2/20 = 0.1. Also keep the statistic and an interval, not just the p-value, because a p-value mixes effect size with sample size: a difference of 0.01 units becomes 'significant' once n is large enough.
from scipy import stats
# 1. Distributions are objects; calling the class "freezes" the parameters.
z = stats.norm(loc=0, scale=1)
print(f"cdf(1.96) = {z.cdf(1.96):.4f}")
print(f"ppf(0.975) = {z.ppf(0.975):.4f}")
# 2. A test = a statistic + the tail area of its null distribution.
sample = [1.0, 2.0, 3.0]
res = stats.ttest_1samp(sample, popmean=0.0)
df = len(sample) - 1
print(f"t = {res.statistic:.4f} df = {df} p = {res.pvalue:.4f}")
# The p-value is nothing more than this:
by_hand = 2 * stats.t.sf(abs(res.statistic), df)
print(f"2 * t.sf(|t|, df) = {by_hand:.4f}")
print(f"reject H0 at alpha=0.05? {bool(res.pvalue < 0.05)}")A hypothesis test is a statistic plus the tail probability of that statistic under an explicitly named null distribution, and that distribution is an object you can query yourself.
Worked examples
Chi-square goodness of fit on die rolls
Tests whether six observed face counts are consistent with a fair die, using the chi2 null distribution.
from scipy import stats
observed = [8, 12, 9, 11, 14, 6] # 60 rolls of a six-sided die
res = stats.chisquare(observed) # H0: all six faces equally likely
print("expected per face:", sum(observed) / 6)
print(f"chi2 = {res.statistic:.2f}")
print(f"p = {res.pvalue:.4f}")
print(f"same as chi2.sf: {stats.chi2.sf(res.statistic, df=5):.4f}")Example explained
Line 1With no f_exp argument, chisquare assumes a uniform expectation, so each expected count is 60 / 6 = 10.
Line 2The statistic is sum((o - e)**2 / e) = (4 + 4 + 1 + 1 + 16 + 16) / 10 = 4.2.
Line 3Degrees of freedom are 6 - 1 = 5 because the total count is fixed, and the p-value is exactly stats.chi2.sf(4.2, 5).
Line 4p = 0.52 means deviations this large happen about half the time with a fair die, so there is no evidence of bias.
Fitting a distribution, then using it
Estimates normal parameters by maximum likelihood and queries the fitted frozen distribution.
from scipy import stats
data = [2, 4, 4, 4, 5, 5, 7, 9]
loc, scale = stats.norm.fit(data) # MLE: mean, and std with ddof=0
print(f"loc = {loc:.1f} scale = {scale:.1f}")
fitted = stats.norm(loc, scale)
low, high = fitted.interval(0.95)
print(f"central 95% of the fitted model: {low:.3f} to {high:.3f}")
print(f"P(X > 9) under the fit = {fitted.sf(9):.4f}")Example explained
Line 1norm.fit returns the maximum-likelihood estimates, which for the normal are the sample mean and the standard deviation with ddof=0, not the ddof=1 version numpy's np.std default and pandas disagree about.
Line 2Passing loc and scale back into stats.norm produces a frozen fitted model you can query like any built-in distribution.
Line 3interval(0.95) is loc +/- 1.959964 * scale: a statement about where 95% of the modelled values lie, not a confidence interval for the mean.
Line 4sf(9) is the upper tail; 9 is exactly two scales above loc, hence the familiar 0.0228.
Parametric versus rank-based test on the same data
Shows how a t-test and Mann-Whitney U disagree at alpha = 0.05 because of the rank test's discrete null distribution.
from scipy import stats
a = [1.0, 2.0, 3.0]
b = [4.0, 5.0, 6.0]
t = stats.ttest_ind(a, b)
u = stats.mannwhitneyu(a, b) # exact null distribution for small samples
print(f"t-test: t = {t.statistic:.4f} p = {t.pvalue:.4f}")
print(f"Mann-Whitney: U = {u.statistic:.1f} p = {u.pvalue:.4f}")Example explained
Line 1ttest_ind pools the two sample variances and refers t = -3.6742 to a t distribution with 3 + 3 - 2 = 4 degrees of freedom.
Line 2mannwhitneyu ranks all six values; U = 0 because every value in a is below every value in b, the most extreme result possible.
Line 3There are only C(6, 3) = 20 equally likely rank splits, so the smallest achievable two-sided exact p-value is 2/20 = 0.1000.
Line 4Same data, opposite verdicts at alpha = 0.05: the t-test's smaller p-value is bought entirely with its normality assumption.
Important notes
Continuous distributions expose .pdf, discrete ones .pmf; stats.norm.pmf and stats.binom.pdf both fail. And pdf values are densities, not probabilities: stats.norm(0, 0.1).pdf(0) is about 3.99.
Test functions are two-sided by default; pass alternative='greater' or 'less' for a directional hypothesis, but choosing the direction after seeing the data invalidates the p-value.
Common mistakes
Reading p = 0.03 as 'a 3% chance the null is true'. The p-value is computed assuming the null holds, so it cannot be a probability about the null; and with n = 100000 a p of 0.001 can accompany a difference too small to care about.
Feeding proportions instead of raw counts to stats.chisquare. The statistic scales with the total, so [0.13, 0.20, 0.17, 0.50] instead of [13, 20, 17, 50] divides chi2 by 100 and returns a p-value near 1, hiding a real imbalance.
Using ttest_ind on before/after measurements from the same subjects. Ignoring the pairing inflates the standard error and can hide an effect that ttest_rel, which tests the within-subject differences, would find.
Try it yourself
Change, predict, then run
Fit a normal to [98, 102, 105, 97, 101, 99, 103, 96] with stats.norm.fit and print loc and scale, then run stats.ttest_1samp against popmean=100 and assert that its p-value equals 2 * stats.t.sf(abs(statistic), df=7) to within 1e-12.
Open the Python workspaceCheck your understanding
A one-sample t-test on 8 measurements reports statistic = 2.4 and pvalue = 0.047. Which statement is true?
- The p-value equals 2 * stats.t.sf(2.4, df=7), the two-tailed area under the t distribution with 7 degrees of freedom
- There is a 4.7% probability that the null hypothesis is true
- The p-value equals 2 * stats.t.sf(2.4, df=8), because there are 8 observations
- A p-value of 0.047 shows the sample mean is far from the null value
Show answer
ttest_1samp is exactly a statistic plus the two-tailed area under a t distribution with n - 1 = 7 degrees of freedom, one degree lost to estimating the mean from the same data. Option 3 is the tempting near-miss: t with 8 df has slightly thinner tails and returns a different number. The p-value is computed under the assumption that the null is true, so it is not the probability of the null, and it says nothing on its own about how large the difference is.