Ergodic Theory: What the Long Run Can Teach Us

What ergodicity means

An ergodic system is one in which a sufficiently long journey through time can reveal the same statistical structure we would see by observing many copies of the system at once. The shorthand is:

Time averages equal ensemble averages.

Imagine a gas in a sealed box. We could photograph every molecule at one instant and estimate how matter is distributed across the box. Or we could follow one molecule for a very long time and record where it goes. If the system is ergodic, those two views converge to the same statistical picture.

For an observable (f) of a process (X_t), the discrete-time idea is:

(1/T) Σ[t=1…T] f(X_t) → Eπ[f(X)]

Here, π is the long-run distribution of states. The statement is not that the trajectory visits every imaginable state, nor that every system settles into equilibrium. It is that, under the right conditions, one long realization becomes representative of the distribution being studied. The Birkhoff ergodic theorem gives this intuition a precise mathematical form.

A non-ergodic system behaves differently. It may become trapped in one region of its state space, retain path-dependent memory, or split into regimes that a single trajectory never crosses. Watching it longer then produces more observations, but not necessarily a more complete view.

That distinction—between more observations and more information—is where ergodic theory becomes practical.

The law of large numbers, with memory

The ordinary law of large numbers concerns repeated samples (X_1, X_2, …, X_n) that are independent and identically distributed. Under familiar conditions, their sample average converges toward the true expectation:

(1/n) Σ[i=1…n] X_i → E[X]

This result supports much of the statistical logic behind machine learning. A model is usually trained by minimizing empirical risk:

R̂(f) = (1/n) Σ[i=1…n] L(f(X_i), Y_i)

The hope is that this measured loss approximates the loss the model will face in the larger population:

R(f) = E[L(f(X), Y)]

If training examples are representative samples from a stable distribution, the law of large numbers is part of why that hope is reasonable.

But many important datasets are not independent draws from a bag. Telemetry, market prices, sensor readings, language, and user behavior arrive as sequences:

X₁ → X₂ → X₃ → …

The present depends on the past. In that setting, an ergodic theorem can play a role analogous to the law of large numbers. Temporal dependence does not automatically prevent long-run averages from becoming informative. If the process is sufficiently stationary and ergodic—and mixes on a useful timescale—a long trajectory may still reveal stable expectations.

This is the bridge:

• The law of large numbers explains convergence across many representative samples.
• Ergodic theory explains when convergence may be possible along one dependent trajectory.

The word may matters. Ergodicity is a property to justify or test, not a permission slip to treat every long sequence as representative.

One server, observed for a long time

Suppose we want to learn the normal behavior of a Linux web server. A telemetry stream might contain a connection accepted by Nginx, a configuration read, a database query, a DNS lookup, and another connection. These events are causally and temporally related. They are the unfolding behavior of one system, not independent specimens.

If the host is approximately stationary and ergodic over the observation window, then one long trace might approximate many samples from its normal behavioral distribution:

One host observed for a long time ≈ many observations of the host’s normal states.

This question sits near the center of my current thesis work: Dendroaspis. Dendroaspis collects and normalizes Linux behavioral telemetry and is being developed toward a Mamba-based sequential anomaly detector. Mamba is a selective state-space model designed to process long sequences efficiently. The research question is not merely whether the model can fit a stream. It is whether a state-space model can learn enough of the stable structure in a host’s evolving behavior to distinguish meaningful deviations from ordinary variation.

The public Dendroaspis v0.1 repository preserves an earlier project view; recent telemetry design, parser work, and evaluation planning live in the private development repository.

Ergodicity supplies a rigorous way to frame the assumption underneath the experiment. Can telemetry from one or a few long host trajectories adequately characterize normal behavior? Under what operating conditions? Over what collection period? And how quickly does the process mix—how long must we observe before averages become useful?

The work does not assume that a production server is globally ergodic. In fact, the difficult and interesting case is that it probably is not.

Real systems change regimes

A server can occupy many legitimate operating regimes:

• ordinary daytime traffic
• nightly backups
• software deployments
• maintenance windows
• kernel upgrades
• incident response
• attacks

The distribution at time (t) may be better written as:

P(X_t | time, workload, configuration, users, software version)

rather than as one fixed P(X).

Suppose we collect two weeks of benign data but never perform a kernel upgrade. The resulting dataset may contain millions of events while omitting an entire legitimate region of the state space. When an upgrade finally occurs, the detector may assign it an extreme anomaly score. The problem is not a shortage of rows. The trajectory lacked the relevant regime.

More samples are not more information when the system remains in the same corner of its state space.

This is also why stationarity and ergodicity should not be used as synonyms. Stationarity asks whether a distribution stays unchanged over time. Ergodicity asks whether one long realization exposes the statistical properties of that distribution.

A process can be stationary and still be non-ergodic. Imagine that each new server is assigned permanently to be either a web server or a database server, with equal probability. The population remains 50/50 over time. But observing one machine forever reveals only its assigned type; it cannot recover the population mixture.

For machine learning, raw volume therefore matters less than coverage. A useful telemetry corpus needs temporal diversity, representative workloads, configuration changes, seasonal cycles, and explicit attention to concept drift. Benign collection duration should be chosen around regime coverage and mixing behavior, not a target event count alone.

The systems-engineering consequence

Ergodic thinking changes how we design observability and validation.

First, it asks us to identify regimes explicitly. Backups, deployments, maintenance, failover, and recovery are not inconvenient noise around “normal” operation. They are part of normal operation.

Second, it turns coverage into an engineering artifact. A test plan should record which states and transitions were exercised, not just how many events were captured. Canaries, synthetic transactions, game days, and controlled change windows can expose legitimate behavior that passive collection might miss.

Third, it makes drift visible. When software, users, or workloads change, yesterday’s empirical distribution may no longer describe today’s system. Monitoring a model therefore requires monitoring the data-generating process around it.

Finally, it encourages local claims. A system may be approximately ergodic within a stable regime and non-ergodic across the full span of its life. A detector can be useful without solving the global problem if it knows which regime it is in, recognizes transitions, and communicates uncertainty.

The engineering lesson is precise: do not ask only, “Do we have enough data?” Ask, “What process produced these data, and what parts of its state space did we actually see?”

Humans are learning systems, not theorems

The same idea is tempting to apply to human lives: keep making attempts, collect more evidence, and eventually converge on the desired goal or the “real” state.

There is something valuable in that analogy, but it needs a boundary. Human beings are not independent samples, fixed distributions, or globally ergodic systems. We change as we learn. Our bodies age, our environments shift, our preferences develop, and our goals can move. An attempt does not merely measure the person who made it; it can change that person.

Neuroscience nevertheless gives the analogy some substance. Learning often depends on differences between what was predicted and what actually occurred. Human studies have found prediction-error signals in dopaminergic and striatal systems during reward learning; recent direct measurements show that dopamine can encode reward and punishment prediction errors on distinct timescales. This does not make dopamine a “goal-convergence chemical.” It shows that outcomes can update future behavior when the nervous system can detect and use the discrepancy.

Practice also contains structured variation. Research on human motor learning suggests that the temporal structure of movement variability can predict learning ability and that the motor system can reshape variability to explore useful alternatives. Repetition is not valuable only because it increases the count. It can be valuable because it searches the neighborhood around the current solution.

That is the deeper connection to ergodicity: progress depends on whether repeated experience explores relevant possibilities and produces informative feedback.

Ten identical attempts made under identical conditions may confirm the same limitation ten times. A smaller number of deliberately varied attempts—different strategy, environment, feedback, or difficulty—may reveal far more of the practical state space. At the same time, indiscriminate variability can become noise. Exploration must remain coupled to a stable enough objective that improvement can be recognized.

A practical law for long goals

For a long-term goal, it helps to treat each attempt as an observation in a changing system:

1. Name the observable. Decide what outcome actually indicates progress. Effort, time, and activity are not always the target variable.
2. Preserve feedback. Record enough about the attempt and its conditions to distinguish signal from mood or memory.
3. Vary the trajectory. If repeated attempts occupy the same regime, change the method, environment, sequence, or constraint.
4. Allow adaptation. Use prediction error to update the next attempt rather than treating failure as a fixed statement about identity.
5. Recheck the distribution—and the goal. A changing life can make an old baseline or objective obsolete.

More attempts can improve an estimate. More varied, feedback-rich attempts can improve a person. Neither guarantees convergence, because the learner, environment, and goal are all allowed to move.

The honest version of the principle is therefore not “keep trying and success becomes inevitable.” It is:

Repeated experience converges toward truth only when the process keeps revealing truth.

For a model, that means representative data, adequate coverage, and assumptions that survive contact with the system. For an engineer, it means designing observations around regimes and transitions. For a person, it means making attempts that teach—then being willing to become different because of what they taught.

Further reading

• Birkhoff ergodic theorem
• Mamba: Linear-Time Sequence Modeling with Selective State Spaces
• Temporal structure of motor variability predicts motor learning ability
• Human dopamine encodes reward and punishment prediction errors