R - data frame - conversion to sparse matrix

Question

I have a data frame that basically has zeros (sparse data frame?), Something similar to

name,factor_1,factor_2,factor_3 ABC,1,0,0 DEF,0,1,0 GHI,0,0,1 

The actual data is about 90,000 rows with 10,000 functions. Can I convert this to a sparse matrix? I expect to get time and space savings by using a sparse matrix instead of a data frame.

Any help would be appreciated

Update # 1: Here is the code to create the data frame. Thanks to Richard for providing this.

 x <- structure(list(name = structure(1:3, .Label = c("ABC", "DEF", "GHI"), class = "factor"), factor_1 = c(1L, 0L, 0L), factor_2 = c(0L,1L, 0L), factor_3 = c(0L, 0L, 1L)), .Names = c("name", "factor_1","factor_2", "factor_3"), class = "data.frame", row.names = c(NA,-3L)) 

Answer 1

This may be slightly more efficient with memory (but slower) to avoid copying all the data into a dense matrix:

 y <- Reduce(cbind2, lapply(x[,-1], Matrix, sparse = TRUE)) rownames(y) <- x[,1] #3 x 3 sparse Matrix of class "dgCMatrix" # #ABC 1 . . #DEF . 1 . #GHI . . 1 

If you have enough memory, you should use Richard's answer, i.e. turn your data.frame into a dense matrix and use Matrix .

Answer 2

How resolved is your matrix? This determines how to improve size.

Your example matrix has 3 1 and 6 0 s. With this ratio, the smallest space saving is least used by the Matrix.

 > library('pryr') # for object_size > library('Matrix') > m <- matrix(rbinom(9e4*1e4, 1, 1/3), ncol = 1e4) > object_size(m) 3.6 GB > object_size(Matrix(m, sparse = T)) 3.6 GB 

Answer 3

You can make the first column in row names, and then use the Matrix from the Matrix package.

 rownames(x) <- x$name x <- x[-1] library(Matrix) Matrix(as.matrix(x), sparse = TRUE) # 3 x 3 sparse Matrix of class "dtCMatrix" # factor_1 factor_2 factor_3 # ABC 1 . . # DEF . 1 . # GHI . . 1 

where the original data frame x is

 x <- structure(list(name = structure(1:3, .Label = c("ABC", "DEF", "GHI"), class = "factor"), factor_1 = c(1L, 0L, 0L), factor_2 = c(0L, 1L, 0L), factor_3 = c(0L, 0L, 1L)), .Names = c("name", "factor_1", "factor_2", "factor_3"), class = "data.frame", row.names = c(NA, -3L)) 

Answer 4

I do this all the time, and it’s a pain in the butt, so I wrote a method for it called sparsify () in my R package - mltools . It works on data.table , which is just fancy data.frames .

---

To solve your specific problem ...

Install mltools (or just copy sparsify () into your environment)

Download Packages

 library(data.table) library(Matrix) library(mltools) 

Sparsify

 x <- data.table(x) # convert x to a data.table sparseM <- sparsify(x[, !"name"]) # sparsify everything except the name column rownames(sparseM) <- x$name # set the rownames > sparseM 3 x 3 sparse Matrix of class "dgCMatrix" factor_1 factor_2 factor_3 ABC 1 . . DEF . 1 . GHI . . 1 

---

In general, the sparsify () method is quite flexible. Here are some examples of how you can use it:

Make some data. Pay attention to data types and unused factor levels.

 dt <- data.table( intCol=c(1L, NA_integer_, 3L, 0L), realCol=c(NA, 2, NA, NA), logCol=c(TRUE, FALSE, TRUE, FALSE), ofCol=factor(c("a", "b", NA, "b"), levels=c("a", "b", "c"), ordered=TRUE), ufCol=factor(c("a", NA, "c", "b"), ordered=FALSE) ) > dt intCol realCol logCol ofCol ufCol 1: 1 NA TRUE aa 2: NA 2 FALSE b NA 3: 3 NA TRUE NA c 4: 0 NA FALSE bb 

Out-of-Box Use

 > sparsify(dt) 4 x 7 sparse Matrix of class "dgCMatrix" intCol realCol logCol ofCol ufCol_a ufCol_b ufCol_c [1,] 1 NA 1 1 1 . . [2,] NA 2 . 2 NA NA NA [3,] 3 NA 1 NA . . 1 [4,] . NA . 2 . 1 . 

Convert NA to 0s and Sparsify Them

 > sparsify(dt, sparsifyNAs=TRUE) 4 x 7 sparse Matrix of class "dgCMatrix" intCol realCol logCol ofCol ufCol_a ufCol_b ufCol_c [1,] 1 . 1 1 1 . . [2,] . 2 . 2 . . . [3,] 3 . 1 . . . 1 [4,] . . . 2 . 1 . 

Generate columns defining NA values

 > sparsify(dt[, list(realCol)], naCols="identify") 4 x 2 sparse Matrix of class "dgCMatrix" realCol_NA realCol [1,] 1 NA [2,] . 2 [3,] 1 NA [4,] 1 NA 

Generate columns defining NA values ​​in the most efficient memory operation

 > sparsify(dt[, list(realCol)], naCols="efficient") 4 x 2 sparse Matrix of class "dgCMatrix" realCol_NotNA realCol [1,] . NA [2,] 1 2 [3,] . NA [4,] . NA