Re: Shannon's information theory

• Previous message: Ralph Hartley: "Re: Zenkin's paper on Cantor (reply of Dr. Zenkin)"
• In reply to: Thomas B.: "Shannon's information theory"
• Next in thread: Dr Chaos: "Re: Shannon's information theory"
• Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]

Date: Wed, 22 Dec 2004 01:50:59 GMT

"Thomas B." <thetom@unixisnot4dummies.org> wrote ...

> I have a question about calculating the entropy of an integer
> value (32 bit).

Shannon's entropy is a function whose value is determined
by a probability *distribution*. So if there is an integer
about whose value you have some subjective uncertainty
described by a (subjective) probability distribution, the
integer could be said to "have" that amount of entropy,
although the entropy is associated with the distribution.

> Let's call the value x. x's range is 0-(2^32-1).
> I make a meassure of x. I got 5 samples and x is
> 100 everytime.
> I repeat this experiment to verify my results,
> everytime x is 5 times 100.
>
> Therefore my mind tells me: no entropy.

So you're not talking about *an* x, but about a sequence
x(1), x(2), ... of integers. Given x(1)=...=x(5)=100,
you now have some conditional probability distribution
on the set of possible values for x(6),
i.e. p_0, p_1, ... p_(2^32-1),
where p_i = pr(x(6)=i|x(1)=...=x(5)=100). Finding the
p_i typically involves Bayesian statistics.

When you say "my mind tells me: no entropy", don't you
actually mean that, subjectively, most of the p_i's are
much smaller than p_100? The more concentrated the
probability distribution is (i.e. the more sure you
become of what x(6) is), the closer the entropy is to
H(p_0,...,p_(2^32-1)) = H(0,0,0,0,0,1,0,...,0) = 0
-- consistent with "what your mind was telling you".

> But what about the formula? How should I set
> the probability?
>
> If I set p to 5/5 = 1 then entropy is 0.
> (5 times occurence, 5 samples)
> But this looks wrong, because x can be
> every value from 0 to 2^32-1.
>
> Therefore p = 5/2^32 which leads to an entropy > 0.

There isn't just one p --the entropy value depends on
the probability *distribution* -- i.e. on all 2^32
probability values. If p_100 ~ 1, then all the other
p_i ~ 0, giving entropy ~ 0.

> So what does matter:
> a) the number of theoretically possible values
> x can have?
> b) the number of different values that really
> occur?

All that matters for the value of Shannon's entropy
is the probability distribution in question, which
typically depends on both (a) and (b).

--r.e.s.

• Previous message: Ralph Hartley: "Re: Zenkin's paper on Cantor (reply of Dr. Zenkin)"
• In reply to: Thomas B.: "Shannon's information theory"
• Next in thread: Dr Chaos: "Re: Shannon's information theory"
• Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]