Calculation of mutual information in R

Question

I'm having trouble interpreting the results of the mi.plugin() (or mi.empirical() ) function from the entropy package. As far as I understand, MI = 0 tells you that the two variables you are comparing are completely independent; and as MI increases, the relationship between these two variables becomes more and more random.

Why then do I get the value 0 when I run the following in R (using the {entropy} package):

mi.plugin( rbind( c(1, 2, 3), c(1, 2, 3) ) )

when do i compare two vectors that are exactly the same?

I suppose my confusion is based on a theoretical misunderstanding on my part, can someone tell me where I was wrong?

Thanks in advance.

Answer 1

Use mutinformation(x,y) from the infotheo package.

 > mutinformation(c(1, 2, 3), c(1, 2, 3) ) [1] 1.098612 > mutinformation(seq(1:5),seq(1:5)) [1] 1.609438 

and the normalized mutual information will be 1.

Answer 2

The mi.plugin function works on a joint frequency matrix of two random variables. The joint frequency matrix indicates the number of times that X and Y get specific results x and y. In your example, you would like X to have 3 possible results - x = 1, x = 2, x = 3, and Y should also have 3 possible results, y = 1, y = 2, y = 3. Skip your example and calculate the general frequency matrix:

 > X=c(1, 2, 3) > Y=c(1, 2, 3) > freqs=matrix(sapply(seq(max(X)*max(Y)), function(x) length(which(((X-1)*max(Y)+Y)==x))),ncol=max(X)) > freqs [,1] [,2] [,3] [1,] 1 0 0 [2,] 0 1 0 [3,] 0 0 1 

This matrix shows the number of occurrences X = x and Y = y. For example, there was one observation for which X = 1 and Y = 1. 0 cases were found for which X = 2 and Y = 1. Now you can use the mi.plugin function:

 > mi.plugin(freqs) [1] 1.098612