PRB-06

Binary entropy

Max at p=½.

Reading speed

Probabilityentropy

Governing equation

H=-p\\log_2 p-(1-p)\\log_2(1-p)

$p \in \left[0.01, 0.99\right]$

where

p: p (—)
H: H (bit)

The figure, part by part

Each numbered marker wears the color of its card. Hover a card to light the marker. Tap a marker to jump to that card. The slider walks a red point along the curve; Animate drawing traces it like a pen.

00:0 / 00:08

①

Governing law — red

H=-p\\log_2 p-(1-p)\\log_2(1-p)

$p \in \left[0.01, 0.99\right]$

$p = 0.01$ $H = 0.081$

Along the curve

②

Domain of the variable — blue

p_{\min} = 0.01,\quad p_{\max} = 0.99

$p \in \left[0.01, 0.99\right]$

$p = 0.01$ $p = 0.01$

Along the curve

③

Response curve — green

H=-p\\log_2 p-(1-p)\\log_2(1-p)

$p \in \left[0.01, 0.99\right]$

$p = 0.01$ $H = 0.081$

Along the curve

④

Rate of change — amber

\frac{\mathrm{d}H}{\mathrm{d}p}

$p \in \left[0.01, 0.99\right]$

$p = 0.016$ $H^{\prime} = 5.964$

Along the curve

Lecture brief

Historical brief

Bernoulli trials, Poisson, Gauss, Bayes and Shannon entropy turned chance into a calculus of belief and noise. The lab computes likelihood, interval and information in SI-free counts. This sheet (PRB-06 — Binary entropy) is the form associated with entropy. Working symbols: $p$ $\rightarrow$ $H$ . Max at p=½.

Purpose

Purpose: compute

H

from

p

in Probability & statistics via

H=-p\\log_2 p-(1-p)\\log_2(1-p)

Max at p=½. Use it when a real probability & statistics question must be answered in SI before a code check.

Live realistic example

In symbols

Live case. Given

p = 0.300\,\mathrm{—}

, the governing relation

H=-p\\log_2 p-(1-p)\\log_2(1-p)

yields

H = 0.8813\,\mathrm{bit}

. Max at p=½. Move a slider: the numbers are this situation, not a canned story.

Calculator

Inputs

p p0.3 —

Outputs

H H0.8813 bit

Reading speed

Watch on YouTube

Free library

Full library

Free PDF / open book

YouTube channels

00:0 / 00:08

Narration of this film

Max at p=½.

Reading speed

Watch on YouTube