Re: Shannon's information theory

From: Erik Trulsson (ertr1013_at_student.uu.se)
Date: 12/21/04 

Date: Tue, 21 Dec 2004 13:25:10 GMT

Thomas B. <thetom@unixisnot4dummies.org> wrote:
> Hello.
> I have a question about calculating the entropy of an integer
> value (32 bit).
>
> Let's call the value x. x's range is 0-(2^32-1).
> I make a meassure of x. I got 5 samples and x is
> 100 everytime.
> I repeat this experiment to verify my results,
> everytime x is 5 times 100.
>
> Therefore my mind tells me: no entropy.
>
> But what about the formula? How should I set
> the probability?
>
> If I set p to 5/5 = 1 then entropy is 0.
> (5 times occurence, 5 samples)
> But this looks wrong, because x can be
> every value from 0 to 2^32-1.

But is every value equally probable?
>From your description above it seems unlikely.

> Therefore p = 5/2^32 which leads to an entropy > 0.
>
> So what does matter:
> a) the number of theoretically possible values
> x can have?
> b) the number of different values that really
> occur?

Both. What actually matters is the probability for each possible
value to occur.

(Note: Just taking a small number of samples from some random
distribution of values will not enable you to make an accurate
calculation of the entropy of the distribution, only to make a
reasonable guess. To calculate the entropy you need to know what the
distribution looks like.) 

-- 
<Insert your favourite quote here.>
Erik Trulsson
ertr1013@student.uu.se

Relevant Pages

  • Re: behavior as mapping
    ... estimating a probability distribution, the distribution ... sequence with equal probability - since you have microsecond temporal ... reduction of the entropy Pto the entropy P ... If there were 4 genes we would need 2 bits of binding site info. ...
    (comp.ai.philosophy)
  • Re: behavior as mapping
    ... response, we generally do not call this an instance of operant conditioning. ... that a single temporal conjunction changes the estimate of the probability ... estimating a probability distribution, the distribution ... reduction of the entropy Pto the entropy P ...
    (comp.ai.philosophy)
  • Re: behavior as mapping
    ... estimating a probability distribution, the distribution ... sequence with equal probability - since you have microsecond temporal ... reduction of the entropy Pto the entropy P ... If there were 4 genes we would need 2 bits of binding site info. ...
    (comp.ai.philosophy)
  • Re: Shannons information theory
    ... Shannon's entropy is a function whose value is determined ... described by a probability distribution, ... When you say "my mind tells me: ...
    (comp.theory)
  • Re: Gian-Carlo Rotas "Twelve problems in probability no one likes to bring up."
    ... > tweleve problems are: ... > Work on entropy and algebra ... > Conditional probability, Bayes' Law ... > Multivariate Normal Distribution and the Clifford Distribution ...
    (sci.stat.math)
Loading