R Studio Server: Multiple AWS EC2 Nodes

I can run code in parallel in R Studio using the ForEach and DoSnow packages. I can easily run a 32-core AWS EC2 instance with R Studio Server and run the code in parallel using the following code:

cl<-makeCluster(32) registerDoSNOW(cl) foreach () %dopar% {} 

However, the maximum number of cores per instance of AWS EC2 is 32. How can I use the same exact code for multiple AWS EC2 nodes and use all the cores?

For example, let's say I want to run the same code in parallel through 4 AWS EC2, which will use 128 cores, for example:

 cl<-makeCluster(128) registerDoSNOW(cl) foreach () %dopar% {} 

Is there an easy way to “link” multiple EC2 instances together so that R Studio Server, ForEach, and DoSnow see these 128 cores as if it were a single EC2 instance?

+5
source share

Source: https://habr.com/ru/post/1203164/


All Articles