CompSci 267P Homework #1

  1. [Sayood p.38#1] Suppose X is a random variable that takes on values from an M-letter alphabet. Show that 0 < H(X) < lg M.
     
  2. [Sayood p.38#3] Given an alphabet {a,b,c,d}, find the first-order entropy in the following cases:
    (a) P(a)=P(b)=P(c)=P(d) = 1/4
    (b) P(a)= 1/2, P(b)= 1/4, P(c)=P(d)= 1/8
    (c) P(a)=0.505, P(b)=1/4, P(c)=1/8, P(d)=.12
     
  3. [Sayood p.39#6abcd] Conduct an experiment to see how well a model can describe a source.
    (a) Write a program that randomly selects letters from the 26-letter alphabet and forms four-letter words. Form 100 such words and see how many of these words make sense. Do this several times and determine the approximate expected number of sensible words.

    (b) File http://www.ics.uci.edu/~dan/class/267P/datasets/text/4letter.words    contains a list of four-letter words. Using this file, and remembering to fold upper- and lower-case letters, obtain a probability model for the alphabet.
    Repeat part (a) generating words using the probability model. (You may use the random number generator located in file http://www.ics.uci.edu/~dan/class/267P/programs/random.c   )  Compare your results with part (a).

    (c) Repeat part (b) using a single-letter context.

    (d) Repeat part (b) using a two-letter context.
     
  4. (a) Find the entropy of a source with 6 symbols having probabilities .5, .2, .1, .1, .05, and .05.
     
    (b) An information source has 128 equally probable symbols. How long is a message from the source whose entropy is 56 bits?
     
    (c) What is the entropy of a message of 32 symbols from a source with three symbols if
    • each symbol is equally likely
    • the symbol probabilities are .6, .3, and .1.