Use detectCores function in R to indicate the number of cores for parallel processing?

Question

The help for detectCores() says:

This is not suitable for use directly for the mc.cores argument mclapply and specifying the number of cores in makeCluster . The first because it can return NA, and the second because it does not give the number of allowed cores.

However, I saw quite a lot of sample code, for example:

 library(parallel) k <- 1000 m <- lapply(1:7, function(X) matrix(rnorm(k^2), nrow=k)) cl <- makeCluster(detectCores() - 1, type = "FORK") test <- parLapply(cl, m, solve) stopCluster(cl) 

where detectCores() used to indicate the number of cores in makeCluster .

My use cases include parallel processing both on my own multi-core laptop (OSX) and on its launch on different multi-core servers (Linux). So, I was not sure if there was a better way to indicate the number of cores, or perhaps the advice on not using detectCores was more for package developers, where the code is designed to work in a wide range of hardware and operating environments,

So in short:

• Should I use the detectCores function in R to indicate the number of cores for parallel processing?
• What does the difference between detected and resolved kernels mean and when is it relevant?

Answer 1

I find it wise to use detectCores as a starting point for the number of workers / processes when calling mclapply or makeCluster . However, there are many reasons why you may need or need to start fewer workers, and even in some cases when you can reasonably start more.

On some hypersurface machines, for example, it is not recommended to set mc.cores=detectCores() . Or, if your script runs on an HPC cluster, you should not use more resources than the job scheduler assigned to your work. You should also be careful in nested parallel situations, for example, when your code can be executed in parallel with a calling function or you are running a multi-threaded function in parallel. In general, it is a good idea to run some preliminary tests before starting a long job to determine the maximum number of workers. I usually track the benchmark with top to find out if the number of processes and threads makes sense, and to make sure memory usage is reasonable.

The tip you quote is especially suitable for package developers. Of course, the idea of ​​a package developer is to always start with detectCores() when calling mclapply or makeCluster , so it's best to leave the solution to the end user. At the very least, the package should allow the user to specify the number of workers to run, but perhaps detectCores() is not even a good default value. Therefore, the default value of mc.cores for mc.cores changed from detectCores() to getOptions("mc.cores", 2L) when mclapply was included in the parallel package.

I think the real warning point that you quoted is that the R functions should not assume that they belong to the whole machine, or that they are the only function in your script that uses multiple cores. If you call mclapply with mc.cores=detectCores() in the package that you send to CRAN, I expect your package to be rejected until you change it. But if you are an end user, running a parallel script on your own machine, then you decide how many script cores are allowed to use.

Answer 2

In my case, it is better (I use mac) future::availableCores() , because detectCores() shows 160, which is obviously wrong.

Answer 3

By future here: The future::availableCores() function confirms various HPC environment variables (for example, NSLOTS , PBS_NUM_PPN and SLURM_CPUS_PER_TASK ), as well as system and R settings that are used to indicate the number of cores available for the process, and if they are not specified , it will return to parallel::detectCores() . When I or others find out about advanced settings, I will be happy to add automatic support for them; There is always an open GitHub issue for this at https://github.com/HenrikBengtsson/future/issues/22 (there are several open requests for help).

In addition, if sysadm sets the environment variable R_FUTURE_AVAILABLECORES_FALLBACK=1 sitewide, then future::availableCores() will return 1 if not explicitly overridden by other means (task scheduler, user settings, ...). This additionally protects against software that uses all cores by default.

In other words, if you use future::availableCores() rather than parallel::detectCores() , you can be sure that your code will work well in multi-user environments (if it turns out that this is not enough, let us know in the above GitHub ) and that any end user can control the number of cores without changing the code.