I can run code in parallel in R Studio using the ForEach and DoSnow packages. I can easily run a 32-core AWS EC2 instance with R Studio Server and run the code in parallel using the following code:
cl<-makeCluster(32) registerDoSNOW(cl) foreach () %dopar% {}
However, the maximum number of cores per instance of AWS EC2 is 32. How can I use the same exact code for multiple AWS EC2 nodes and use all the cores?
For example, let's say I want to run the same code in parallel through 4 AWS EC2, which will use 128 cores, for example:
cl<-makeCluster(128) registerDoSNOW(cl) foreach () %dopar% {}
Is there an easy way to “link” multiple EC2 instances together so that R Studio Server, ForEach, and DoSnow see these 128 cores as if it were a single EC2 instance?
source share