What is the entropy of a communication system that consists of six messages with probabilities 1/8, 1/8, 1/8, 1/8, 1/4, and 1/4 respectively ?

Concept:

The entropy of a probability distribution is the average or the amount of information when drawing from a probability distribution.

It is calculated as:

\(H=\underset{i=1}{\overset{n}{\mathop \sum }}\,{{p}_{i}}{{\log }_{2}}\left( \frac{1}{{{p}_{i}}} \right)bits/symbol\)

pi is the probability of the occurrence of a symbol.

Calculation:

The entropy will be:

\(H=4\times\frac{1}{8}log_2(8)+2\times\frac{1}{4}log_2(4)\)

\(H=\frac{4\times3}{8}+\frac{2\times2}{4}\)

H = 2.5 bits/message