How to access individual forecasts in Spark RandomForest?

Question

I want to use the pyspark.mllib.tree.RandomForest module to get the approximation matrix for my observations.

So far, my data has been small enough to load directly into memory. So I used sklearn.ensemble.RandomForestClassifier to get the proximity matrix as follows: suppose X is a matrix containing functions and Y is a vector containing labels. I trained a random forest in order to distinguish between objects labeled “0” and “1”. Having a trained random forest, I wanted to get a measure of the proximity between each pair of observations in my data set, referring to how many decision trees for both cases received the same final node (= sheet). Thus, for 100 decision trees, the proximity between two observations can vary from 0 (never fall into the same final sheet) and 100 (dropped to the same final sheet in all decision trees). Python implementation:

import numpy
from sklearn import ensemble

## data
print X.shape, Y.shape # X is a matrix that holds the 4281 features and contains 8562 observations and Y contains 8562 labels
>> (8562, 4281) (8562,)

## train the tree
n_trees = 100
rand_tree = sklearn.ensemble.RandomForestClassifier(n_estimators=n_tress)
rand_tree.fit(X, Y)

## get proximity matrix
apply_mat = rand_tree.apply(X)
obs_num = len(apply_mat)
sim_mat = numpy.eye(obs_num) * len(apply_mat[0]) # max values that they can be similar at = N estimators

for i in xrange(obs_num):
    for j in xrange(i, obs_num):
        vec_i = apply_mat[i]
        vec_j = apply_mat[j]
        sim_val = len(vec_i[vec_i==vec_j])
        sim_mat[i][j] = sim_val
        sim_mat[j][i] = sim_val

sim_mat_norm = sim_mat / len(apply_mat[0])
print sim_mat_norm.shape
>> (8562, 8562)

, , , Spark. , "" , . ? ( , Spark: https://spark.apache.org/docs/1.2.0/mllib-ensembles.html#classification):

from pyspark.mllib.tree import RandomForest
from pyspark.mllib.util import MLUtils

# Load and parse the data file into an RDD of LabeledPoint.
data = MLUtils.loadLibSVMFile(sc, 'data/mllib/sample_libsvm_data.txt')
# Split the data into training and test sets (30% held out for testing)
(trainingData, testData) = data.randomSplit([0.7, 0.3])

model = RandomForest.trainClassifier(trainingData, numClasses=2, categoricalFeaturesInfo={},
                                 numTrees=3, featureSubsetStrategy="auto",
                                 impurity='gini', maxDepth=4, maxBins=32)

, . !

Answer 1

PySpark MLlib . :

from pyspark.mllib.tree import DecisionTreeMode

numTrees = 3
trees = [DecisionTreeModel(model._java_model.trees()[i])
    for i in range(numTrees)]

predictions = [t.predict(testData) for t in trees]

, , ML:

from pyspark.ml.feature import StringIndexer
from pyspark.ml.classification import RandomForestClassifier

df = sqlContext.read.format("libsvm").load("data/mllib/sample_libsvm_data.txt")

indexer = StringIndexer(inputCol="label", outputCol="indexed").fit(df)
df_indexed = indexer.transform(df)

model = RandomForestClassifier(
    numTrees=3, maxDepth=2, labelCol="indexed", seed=42
).fit(df_indexed)

rawPrediction probability:

model.transform(df).select("rawPrediction", "probability").show(5, False)

## +---------------------------------------+-----------------------------------------+
## |rawPrediction                          |probability                              |
## +---------------------------------------+-----------------------------------------+
## |[0.0,3.0]                              |[0.0,1.0]                                |
## |[2.979591836734694,0.02040816326530612]|[0.9931972789115647,0.006802721088435374]|
## |[2.979591836734694,0.02040816326530612]|[0.9931972789115647,0.006802721088435374]|
## |[2.979591836734694,0.02040816326530612]|[0.9931972789115647,0.006802721088435374]|
## |[2.979591836734694,0.02040816326530612]|[0.9931972789115647,0.006802721088435374]|
## +---------------------------------------+-----------------------------------------+

. , Spark, / . .