R: error using randomForest on a large data set (RAM 48-512 GB)

Question

I am trying to conduct an R-analysis of randomization in R on a wide genetic dataset (662 x 35350). All variables, except the result, are numerical, and 99% of them are binary 0/1. I am very familiar with R randomForest (), but only worked with datasets with 5000-10000 variables before. The next planned analysis step will consist of an extremely large data set with millions of variables, so I am motivated to find a solution to this problem.

I understand that R randomForest does not have an inherent restriction on the number of variables, and I know that I read a published work with variables of 100,000. When I try to analyze the complete data set (setting ntree = 100, I get: "Error: protect ( ): overflow protection stack "

This is true whether the dataset is a data framework (as it was originally provided) or when I convert it to a matrix. When I send the launch to the cluster for parallel processing, I see that all my kernels work as soon as I run the code. I also see that in no case does my use of RAM fit the machine limit (48 GB). In the best case, this takes up about 16% of RAM during the attempted execution. (I also had the same problem on my 512 GB of RAM in the office, where it never used more than about 5%).

I tried several solutions found on the Internet, including one in the previous postoverflow post ( Increase (or decrease) memory available for R processes ). I tried the instructions provided by BobbyShaftoe in 2009 (adding - max-mem-size = 49000M and -max-vsize = 49000M in the shortcut tab properties), but this prevented R from opening correctly. I also tried to run these commands on the command line, but they are generated: '--max-ppsize' / '- max-vsize = 5000M "is not recognized as an internal or external command, operating program or batch file.

I also read the suggestions made in this post: How to improve RandomForest performance? . I cannot reduce the number of functions until I have at least one full-featured launch. (Also, I'm not sure if the problem is in RAM, as such.)

I am on Windows 7 running Revolution R 7.2 (64-bit). My memory limit is set at 49807 MB, but I'm not sure that memory.limit specifically allows for the size of the security stack.

Destroying the data set into smaller pieces of variables (which solves the problem) does not solve the analytical problem. Are there any suggestions regarding R settings that may allow analysis of the full data set?

########################################## # input DF ########################################## object.size(inputDF) # 191083664 bytes (as matrix, size=189391088 bytes, not much smaller) dim(inputDF) # 662 x 35350 ########################################## #Load necessary packages into R memory ########################################## require(iterators) require(foreach) require(parallel) require(doParallel) require(randomForest) ########################################### # Get the number of available logical cores ########################################### cores <- detectCores() cores #12 ########################################### # Print info on computer, OS, cores ########################################### print(paste('Processor: ', Sys.getenv('PROCESSOR_IDENTIFIER')), sep='') print(paste('OS: ', Sys.getenv('OS')), sep='') print(paste('Cores: ', cores, sep='')) ########################################################################### # Setup clusters via parallel/DoParallel ########################################################################### cl.spec <- rep("localhost", 10) cl <- makeCluster(cl.spec, type="SOCK") registerDoParallel(cl, cores=10) ########################################################################### # RUN RANDOM FOREST ########################################################################### system.time(forestOUT<- randomForest(as.factor(Dx01) ~ ., data=inputDF, do.trace = 10, ntree=100, mtry = sqrt(ncol(inputDF)), nodesize = 0.1*nrow(inputDF), importance=T, proximity=F, replace=TRUE, keep.forest=TRUE)) stopCluster(cl) 

See sessionInfo ()

 #Language: R #OS: Windows 7 sessionInfo() #R version 3.0.3 (2014-03-06) #Platform: x86_64-w64-mingw32/x64 (64-bit) # #locale: #[1] LC_COLLATE=English_Canada.1252 LC_CTYPE=English_Canada.1252 LC_MONETARY=English_Canada.1252 LC_NUMERIC=C #[5] LC_TIME=English_Canada.1252 # #attached base packages: #[1] stats4 parallel splines grid stats graphics grDevices utils datasets methods base # #other attached packages: #[1] QuantPsyc_1.5 boot_1.3-13 perturb_2.05 RCurl_1.95-4.5 bitops_1.0-6 car_2.0-22 #[7] reprtree_0.6 plotrix_3.5-10 rpart.plot_1.4-5 sqldf_0.4-7.1 RSQLite.extfuns_0.0.1 RSQLite_1.0.0 #[13] gsubfn_0.6-6 proto_0.3-10 XML_3.98-1.1 RMySQL_0.9-3 DBI_0.3.1 mlbench_2.1-1 #[19] polycor_0.7-8 sfsmisc_1.0-26 quantregForest_0.2-3 tree_1.0-35 maptree_1.4-7 cluster_1.15.3 #[25] mice_2.22 VIM_4.0.0 colorspace_1.2-4 randomForest_4.6-10 ROCR_1.0-5 gplots_2.15.0 #[31] caret_6.0-37 partykit_0.8-0 biomaRt_2.18.0 NCBI2R_1.4.6 snpStats_1.12.0 betareg_3.0-5 #[37] arm_1.7-07 lme4_1.1-7 Rcpp_0.11.3 Matrix_1.1-4 nlme_3.1-118 mvtnorm_1.0-1 #[43] taRifx_1.0.6 sos_1.3-8 brew_1.0-6 R.utils_1.34.0 R.oo_1.18.0 R.methodsS3_1.6.1 #[49] rattle_3.3.0 jsonlite_0.9.13 httpuv_1.3.2 httr_0.5 gmodels_2.15.4.1 ggplot2_1.0.0 #[55] JGR_1.7-16 iplots_1.1-7 JavaGD_0.6-1 party_1.0-18 modeltools_0.2-21 strucchange_1.5-0 #[61] sandwich_2.3-2 zoo_1.7-11 pROC_1.7.3 e1071_1.6-4 psych_1.4.8.11 gtools_3.4.1 #[67] functional_0.6 modeest_2.1 stringi_0.3-1 languageR_1.4.1 utility_1.3 data.table_1.9.4 #[73] xlsx_0.5.7 xlsxjars_0.6.1 rJava_0.9-6 snow_0.3-13 doParallel_1.0.8 iterators_1.0.7 #[79] foreach_1.4.2 reshape2_1.4 reshape_0.8.5 plyr_1.8.1 xtable_1.7-4 stringr_0.6.2 #[85] foreign_0.8-61 Hmisc_3.14-6 Formula_1.1-2 survival_2.37-7 class_7.3-11 MASS_7.3-35 #[91] nnet_7.3-8 Revobase_7.2.0 RevoMods_7.2.0 RevoScaleR_7.2.0 lattice_0.20-27 rpart_4.1-5 # #loaded via a namespace (and not attached): #[1] abind_1.4-0 acepack_1.3-3.3 BiocGenerics_0.8.0 BradleyTerry2_1.0-5 brglm_0.5-9 caTools_1.17.1 chron_2.3-45 #[8] coda_0.16-1 codetools_0.2-9 coin_1.0-24 DEoptimR_1.0-2 digest_0.6.4 flexmix_2.3-12 gdata_2.13.3 #[15] glmnet_1.9-8 gtable_0.1.2 KernSmooth_2.23-13 latticeExtra_0.6-26 lmtest_0.9-33 minqa_1.2.4 munsell_0.4.2 #[22] nloptr_1.0.4 pkgXMLBuilder_1.0 png_0.1-7 RColorBrewer_1.0-5 revoIpe_1.0 robustbase_0.92-2 scales_0.2.4 #[29] sp_1.0-16 tcltk_3.0.3 tools_3.0.3 vcd_1.3-2