You simulate an AR(1) process with parameter ρ = 0.7 and compute its sample autocorrelation at lag 1. What value do you expect? The theoretical answer is simple: the lag-1 autocorrelation of an AR(1) process is exactly ρ. But how quickly does the sample autocorrelation converge to this value — and how noisy is the estimator at realistic sample sizes?
In this post, we derive the theoretical autocorrelation function of an AR(1) process, simulate it in NumPy, and verify both the point estimate and its dispersion against theory with a Monte-Carlo experiment.
The AR(1) process definition #
An autoregressive process of order 1 (AR(1)) is defined by:
X_t = c + ρ·X_{t-1} + ε_tWhere:
cis a constant (drift term)ρis the autoregressive coefficient, with|ρ| < 1for stationarityε_tis white noise:ε_t ~ N(0, σ²)i.i.d.
For a stationary AR(1) process (|ρ| < 1), the mean, variance, and autocorrelation function all have closed forms.
Theoretical properties #
Mean and variance #
The stationary mean is:
E[X_t] = c / (1 - ρ)The stationary variance is:
Var(X_t) = σ² / (1 - ρ²)Autocorrelation function (ACF) #
The key result we’ll verify: the autocorrelation at lag k is:
ρ_k = ρ^kIn particular, the lag-1 autocorrelation is simply ρ_1 = ρ — regardless of the noise variance σ² or the constant c.
Simulation setup #
import numpy as np
rng = np.random.default_rng(42)
# AR(1) parameters
c = 0.5 # drift term
rho_true = 0.7 # true AR coefficient
sigma = 1.0 # noise standard deviation
def simulate_ar1(n, c, rho, sigma, rng, burn_in=100):
"""Simulate an AR(1) process with burn-in to reach stationarity."""
x = np.zeros(burn_in)
for t in range(1, burn_in):
x[t] = c + rho * x[t-1] + rng.normal(0, sigma)
x_full = np.zeros(n + burn_in)
x_full[:burn_in] = x
for t in range(burn_in, n + burn_in):
x_full[t] = c + rho * x_full[t-1] + rng.normal(0, sigma)
return x_full[burn_in:] # stationary portion only
def sample_rho1(x):
x_centered = x - np.mean(x)
auto_cov_1 = np.mean(x_centered[:-1] * x_centered[1:])
auto_var = np.mean(x_centered ** 2)
return auto_cov_1 / auto_var
for n in [100, 1000, 10000]:
x = simulate_ar1(n, c, rho_true, sigma, rng)
print(f"n={n:6d}: sample rho1 = {sample_rho1(x):.6f} (true = {rho_true})")Output:
n= 100: sample rho1 = 0.672382 (true = 0.7)
n= 1000: sample rho1 = 0.740188 (true = 0.7)
n= 10000: sample rho1 = 0.696657 (true = 0.7)Notice the n = 1000 result: a miss of 0.04 in the wrong direction, and a bigger miss than at n = 100. That is not a bug — it is exactly what the sampling variance of the estimator predicts, as we verify next.
Dispersion: Bartlett’s formula, verified by Monte Carlo #
For a stationary AR(1), the large-sample (Bartlett) approximate variance of the lag-1 sample autocorrelation is:
Var(ρ̂_1) ≈ (1 − ρ²) / nFor ρ = 0.7:
n = 100: theoretical sd≈ √(0.51/100) = 0.071; observed deviation|0.672 − 0.7| = 0.028— a 0.4σ eventn = 1000: theoretical sd≈ 0.023; observed deviation0.040— a 1.7σ event, unremarkablen = 10000: theoretical sd≈ 0.0071; observed deviation0.0033— a 0.5σ event
One path is anecdote, so let’s verify the distribution itself over many replications:
from scipy.signal import lfilter
def rho1_estimates(n, reps=2000, rho=0.7, c=0.5, sigma=1.0, rng=rng, burn_in=100):
"""Vectorized AR(1) simulation via a linear filter — returns `reps` lag-1 ACF estimates."""
eps = rng.normal(0.0, sigma, (reps, n + burn_in))
x = lfilter([1.0], [1.0, -rho], eps, axis=1) + c / (1.0 - rho)
x = x[:, burn_in:]
xc = x - x.mean(axis=1, keepdims=True)
return (xc[:, :-1] * xc[:, 1:]).mean(axis=1) / (xc ** 2).mean(axis=1)
print(f"{'n':>7} | {'emp mean':>9} | {'emp sd':>7} | {'theory sd':>9} | {'-1/n':>7}")
for n in [100, 1000, 10000]:
r1 = rho1_estimates(n)
theo = np.sqrt((1 - rho_true ** 2) / n)
print(f"{n:>7} | {r1.mean():>9.4f} | {r1.std():>7.4f} | {theo:>9.4f} | {-1.0/n:>7.4f}")Output:
n | emp mean | emp sd | theory sd | -1/n
100 | 0.6684 | 0.0770 | 0.0714 | -0.0100
1000 | 0.6969 | 0.0232 | 0.0226 | -0.0010
10000 | 0.6998 | 0.0070 | 0.0071 | -0.0001Reading the table #
- Dispersion matches theory. The empirical standard deviations (0.077, 0.023, 0.0070) sit within ~8% of Bartlett’s formula (√((1−ρ²)/n) = 0.071, 0.023, 0.0071) and converge to it as n grows.
- There is a small-sample bias, not just noise. The empirical mean sits below
ρat every n — about−3.1/n(i.e.−(1 + 3ρ)/n): −0.031 at n = 100, −0.003 at n = 1000. At short samples the sample autocorrelation is downward-biased, which is exactly why ARIMA identification on short series needs care. - So the honest statement is bivariate: the estimator converges to
ρand needs(1 − ρ²)/n + (1 + 3ρ)²/n²-scale error bars around it.
Conclusion #
The AR(1) process demonstrates a clean connection between theory and simulation — but only if you track both moments of the estimator:
- For an AR(1),
ρ_k = ρ^k, so the lag-1 sample autocorrelation estimates the AR coefficient directly. - The estimator’s dispersion follows Bartlett’s formula
(1 − ρ²)/n— verified here over 2000 replications and three sample sizes. - At short samples there is a systematic downward bias of order
(1 + 3ρ)/n— a reminder that point estimates on ~100 observations carry more structure than “close to truth”.
When building time series models, use these error bars: an AR(1) coefficient estimated as 0.67 on 100 observations is fully consistent with a true value anywhere in roughly [0.53, 0.81] — which matters when the next decision, like an ARIMA order, is based on whether the coefficient is “significant”.
This post includes fully reproducible code (fixed seeds throughout). Run it yourself and vary ρ, n, and σ — the Monte-Carlo block will confirm the same two laws.