Context:
I have a Spark ML pipeline that contains VectorAssembler, StringIndexer, and DecisionTreeClassifier. Using this pipeline, I can successfully fit the model and convert my data frame. I would like to keep this model for future use, but I keep getting the following error:
Pipeline write will fail on this Pipeline because it contains a stage which does not implement Writable.
Non-Writable stage: dtc_9c04161ed2d1 of type class org.apache.spark.ml.classification.DecisionTreeClassificationModel
What I tried :
val pipeline = new Pipeline().setStages(Array(assembler, labelIndexer, dt))
val model = pipeline.fit(dfIndexed)
model.write.overwrite().save("test/model/pipeline")
This works correctly when I remove the classifier (i.e. dt). Is there a way to save the DecisionTreeClassifier model?
My data consists of some indexed categorical values that I have to return to the original form (I know that this will require the use of IndexToString). I am using Spark 1.6.