Infrastructure · Primitives
ShannonEntropy#a177
The average surprise of a distribution: how many bits it takes to predict an outcome
Mechanism
For a discrete distribution P = {p_i} over states i, H(P) = -sum_i p_i log p_i, the expected self-information -log p_i of an outcome. Rare outcomes carry much information, likely ones little. H is maximal (log m for m states) when all states are equally likely and zero when all probability sits on one state. Applied to a system's possible configurations, high H means many weakly constrained configurations (variability, flexibility, exploration) and low H means strong constraints (regularity, coordination). Use log base 2 to measure in bits.
Invariants
- H is non-negative and zero only for a distribution concentrated on one state
- H is maximal for the uniform distribution over the state space
- H depends only on the probabilities, not on what the states are or how they are ordered
Failure modes
- State-space dependence: the same system yields different entropies under different state definitions or binnings, so values are comparable only within one state space
- Finite-sample bias: entropy estimated from few samples is biased low
- Reading low entropy as 'good order' without asking what constrains it
Where this came from
- Thesis Kevin Do, PhD thesis on ant collective behaviour (NCSU, 2026), Ch. 1.3.1, Eqs. 1.3-1.5
Used by
Identity
The name is the hash of the definition. Change a word and it becomes a different pattern.
- Sema ID
sema:ShannonEntropy#mh:SHA-256:a1774e05a3085b62748089790da6da7581e12e1144cab18adf622cb9059b1142- Reference
ShannonEntropy#a177- Ring
- 0
- Tier
- 0