R mlr package - is it possible to save all models from parameter settings?

Question

I wanted to ask if it is possible to save all the models created during the settings, for example. with function tuneParams. I would like to save models from each cross-validation check for each set of hyperparameters.

I see that there is a parameter modelsfor functions resampleand benchmark, but I cannot find it in tuneParamsor a similar function, and I cannot find a way to simulate this behavior using other functions (I am new to mlr).

Is there any way to do this?

PS I know this may seem crazy, however I need it for some internal verification.

PS2 Unfortunately, there is no tag "mlr" yet, and I do not have enough rep to create it.

Answer 1

I guess there are shorter solutions, but the following is not so hacky. We use Wrapper to get the model so we can save it to a list in a global environment. Alternatively, you can change this line to something more complex and save it to your hard drive. This can be useful because models can become quite large.

library(mlr)

# Define the tuning problem
ps = makeParamSet(
  makeDiscreteParam("C", values = 2^(-2:2)),
  makeDiscreteParam("sigma", values = 2^(-2:2))
)
ctrl = makeTuneControlGrid()
rdesc = makeResampleDesc("Holdout")
lrn = makeLearner("classif.ksvm")


# Define a wrapper to save all models that were trained with it
makeSaveWrapper = function(learner) {
  mlr:::makeBaseWrapper(
    id = paste0(learner$id, "save", sep = "."),
    type = learner$type,
    next.learner = learner,
    par.set = makeParamSet(),
    par.vals = list(),
    learner.subclass = "SaveWrapper",
    model.subclass = "SaveModel")
}

trainLearner.SaveWrapper = function(.learner, .task, .subset, ...) {
  m = train(.learner$next.learner, task = .task, subset = .subset)
  stored.models <<- c(stored.models, list(m)) # not very efficient, maybe you want to save on hard disk here?
  mlr:::makeChainModel(next.model = m, cl = "SaveModel")
}

predictLearner.SaveWrapper = function(.learner, .model, .newdata, ...) {
  NextMethod(.newdata = .newdata)
}

stored.models = list() # initialize empty list to store results
lrn.saver = makeSaveWrapper(lrn)

res = tuneParams(lrn.saver, task = iris.task, resampling = rdesc, par.set = ps, control = ctrl)

stored.models[[1]] # the normal mlr trained model
stored.models[[1]]$learner.model # the underlying model
getLearnerParVals(stored.models[[1]]$learner) # the hyper parameter settings
stored.models[[1]]$subset # the indices used to train the model