Re: Shannon's information theory

From: Amir massoud Farahmand (sologen_at_yahoo.com)
Date: 12/24/04 

Date: 24 Dec 2004 10:55:52 -0800

Hi! (:

The actual problem is due to the difference between the estimated
probability distribution and the actual one. Your estimation indicates
that p(x=100) = 1 and p(x=otherwise)=0. However, note that this is only
an estimation and can be wrong. You may provide a bound on the accuracy
of your results and ... and ... ! (:

Thomas B. wrote:
> Hello.
> I have a question about calculating the entropy of an integer
> value (32 bit).
>
> Let's call the value x. x's range is 0-(2^32-1).
> I make a meassure of x. I got 5 samples and x is
> 100 everytime.
> I repeat this experiment to verify my results,
> everytime x is 5 times 100.
>
> Therefore my mind tells me: no entropy.
>
> But what about the formula? How should I set
> the probability?
>
> If I set p to 5/5 = 1 then entropy is 0.
> (5 times occurence, 5 samples)
> But this looks wrong, because x can be
> every value from 0 to 2^32-1.
> Therefore p = 5/2^32 which leads to an entropy > 0.
>
> So what does matter:
> a) the number of theoretically possible values
> x can have?
> b) the number of different values that really
> occur?
>

(a) matters but you should use "theoretically possible" phrase
considering the pdf of your event; having an integer value does not
lead to having a uniform distribution in it.
> Thanks for your help.
>
> Thomas

Amir massoud

Relevant Pages

  • Re: diehard and ent results quesion
    ... The estimation is not made on the basis of the data, ... "absolute" entropy would be related to the length ... where by "absolute" I mean ...
    (sci.crypt)
  • Re: Calculating joint entropy
    ... Entropy is a property of a probability distribution. ... cells, instead having to fill entries of full joint table with 32 ... results in low accuracy in estimation. ...
    (sci.stat.math)
  • Re: ESTIMATING MA COEFFICIENT FROM ARMA
    ... just keep in mind the order estimation ... It's not sufficient to know that a system is ARMA. ... can require substantial amounts of work, ...
    (comp.dsp)
  • Re: Thousands of KISS Army Members Converge
    ... the power of estimation still boggles your mind. ...
    (rec.music.artists.kiss)
  • Re: Entropy testing utility
    ... > I need some feedback on my entropy tester, it will test how many bits an ... > entropy source will produce. ... estimation. ...
    (sci.crypt)