How to create a matrix of confusion containing many judgments in R?

Question

I have a data set from two evaluators judging a set of videos by several (binary) criteria. I would like to build a matrix of confusion in order to better understand their agreement / disagreement. But all the examples that I have found so far relate to cases where each judge evaluates only one criterion per clip. In my case, the judges evaluate each criterion for each clip.

Let's say I have 4 binary criteria (A_Con..A_Mod), evaluated by two evaluators (A and B) for a set of videos (in this case 80):

str (mydata) 'data.frame': 160 obs. of 6 variables: $ A_Con: int 0 0 0 0 0 0 0 0 0 0 ... $ A_Dom: int 0 0 0 1 0 0 0 0 0 0 ... $ A_Met: int 0 0 0 0 0 0 1 0 0 1 ... $ A_Mod: int 0 0 0 1 0 1 0 0 0 1 ... $ Rater: Factor w/ 2 levels "A","B": 2 2 2 2 2 2 2 2 2 2 ... $ Clip : int 1 2 3 4 5 6 7 8 9 10 ... 

I can melt this into:

 > str(mymolten) 'data.frame': 640 obs. of 4 variables: $ Rater : Factor w/ 2 levels "A","B": 2 2 2 2 2 2 2 2 2 2 ... $ Clip : int 1 2 3 4 5 6 7 8 9 10 ... $ variable: Factor w/ 4 levels "A_Con","A_Dom",..: 1 1 1 1 1 1 1 1 1 1 ... $ value : int 0 0 0 0 0 0 0 0 0 0 ... 

But I can't figure out how to turn it into a matrix of confusion that would take into account combinations (which are not as good as this):

  Rater B A_Con A_Dom A_Met A_Mod A_Con 19 1 0 0 Rater A A_Dom 1 20 0 0 A_Met 0 0 20 5 A_Mod 0 2 0 20 

The table () function seems to be a way, but how to format the data?

Answer 1

This may not be the easiest solution. You can split the data for two evaluators, and merge resulting data.frames.

 # Sample data n <- 80 d0 <- data.frame( A_Con = round(runif(2*n)), A_Dom = round(runif(2*n)), A_Met = round(runif(2*n)), A_Mod = round(runif(2*n)), Rater = rep(c("A","B"), n), Clip = rep(1:n,each=2) ) library(reshape2) library(plyr) d <- melt(d0, id.vars=c("Rater","Clip")) d <- d[ d$value==1, ] A <- d[d$Rater=="A",] B <- d[d$Rater=="B",] A <- data.frame( Clip=A$Clip, A=A$variable ) B <- data.frame( Clip=B$Clip, B=B$variable ) d <- merge(A, B, all=FALSE) d <- ddply( d, c("A", "B"), summarize, n=length(Clip) ) dcast( d, A ~ B )