How clusterExport function without evaluation environment

Question

I am trying to use parLapply inside another function not defined in the global environment. The worker function uses a list of other functions that I want clusterExport in advance that are also not defined in the global environment. My problem is that both functions export their evaluation environments to clusters that are huge and not needed.

We call the working function workerFunction and the list of functions functionList .

  workerFunction <- function(i) { intermediateOutput <- functionList[[i]](y) result <- otherCalculations(intermediateOutput) return(result) } library(parallel) cl <- makeCluster(detectCores()) environment(workerFunction) <- .GlobalEnv environment(functionList) <- .GlobalEnv clusterExport(cl, varlist=c("functionList", "y"), envir=.GlobalEnv) output <- parLapply(cl, inputVector, workerFunction) 

I get:

 Error in get(name, envir = envir) (from <text>#53) : object 'functionList' not found 

If I do not set environment(functionList) <- .GlobalEnv , then the huge environment functionList exported to clusters. Why can't R find functionList in global environment?

Answer 1

It is impossible to guess the problem without a complete example, but I wonder if the error message from clusterExport , and not parLapply . This will happen if functionList was defined in the function, and not in the global environment, because the clusterExport envir argument indicates the environment from which variables can be exported.

To export variables defined in a function from the same function, you should use:

 clusterExport(cl, varlist=c("functionList", "y"), envir=environment()) 

I just assume this might be a problem for you, since I don't know how and where you defined the functionList . Note that clusterExport always assigns variables to the global cluster work environment.

I am also suspicious of how you are apparently setting up a list environment: this seems legal, but I don't think it will change the environment of functions in this list. In fact, I suspect that exporting functions for workers on the list may have other problems that you have not yet encountered. I would use something like this:

 mainFunction <- function(cl) { fa <- function(x) fb(x) fb <- function(x) fc(x) fc <- function(x) x y <- 7 workerFunction <- function(i) { do.call(functionNames[[i]], list(y)) } environment(workerFunction) <- .GlobalEnv environment(fa) <- .GlobalEnv environment(fb) <- .GlobalEnv environment(fc) <- .GlobalEnv functionNames <- c("fa", "fb", "fc") clusterExport(cl, varlist=c("functionNames", functionNames, "y"), envir=environment()) parLapply(cl, seq_along(functionNames), workerFunction) } library(parallel) cl <- makeCluster(detectCores()) mainFunction(cl) stopCluster(cl) 

Please note that I took liberties with your example, so I'm not sure how this fits your problem.