How is scikit-learn cross_val_predict calculated?

Question

Is there a cross_val_predict (see doc , v0.18) using the k-fold method, as shown in the code below, to calculate the accuracy for each short-term and assimilate them completely or not?

 cv = KFold(len(labels), n_folds=20) clf = SVC() ypred = cross_val_predict(clf, td, labels, cv=cv) accuracy = accuracy_score(labels, ypred) print accuracy 

Answer 1

No!

According to the cross-validation document page, cross_val_predict does not return any ratings, but only labels based on the specific strategy described here:

The cross_val_predict function has an interface similar to cross_val_score, but for each input element it returns the forecast obtained for this element when it was in the test suite . You can use only cross-validation strategies that assign all elements to the test case exactly once (otherwise, an exception occurs).

And therefore, by calling accuracy_score(labels, ypred) you simply calculate the accuracy points of the labels predicted by the aforementioned specific strategy compared to true labels. This is again indicated on the same documentation page:

These forecasts can be used to evaluate the classifier:

 predicted = cross_val_predict(clf, iris.data, iris.target, cv=10) metrics.accuracy_score(iris.target, predicted) 

Please note that the result of this calculation may differ slightly from the results obtained using cross_val_score, since the elements are grouped differently.

If you need precision estimates for various bends, you should try:

 >>> scores = cross_val_score(clf, X, y, cv=cv) >>> scores array([ 0.96..., 1. ..., 0.96..., 0.96..., 1. ]) 

and then for average accuracy of all folds use scores.mean() :

 >>> print("Accuracy: %0.2f (+/- %0.2f)" % (scores.mean(), scores.std() * 2)) Accuracy: 0.98 (+/- 0.03) 

---

How to calculate Cohen Kapp coefficient and confusion matrix for each bend?

To calculate the Cohen Kappa coefficient matrix and the confusion matrix, I suggested that you mean the Kappa coefficient and the confusion matrix between the true labels and the labels predicted in each case

 from sklearn.model_selection import KFold from sklearn.svm.classes import SVC from sklearn.metrics.classification import cohen_kappa_score from sklearn.metrics import confusion_matrix cv = KFold(len(labels), n_folds=20) clf = SVC() for train_index, test_index in cv.split(X): clf.fit(X[train_index], labels[train_index]) ypred = clf.predict(X[test_index]) kappa_score = cohen_kappa_score(labels[test_index], ypred) confusion_matrix = confusion_matrix(labels[test_index], ypred) 

---

What does cross_val_predict return?

It uses KFold to split the data into k parts, and then for i=1..k iterations:

• takes i'th part as test data, and all the rest - training data
• trains the model with training data (all parts except i'th )
• then, using this trained model, predicts labels for the i'th part (test data)

In each iteration, the label i'th of the data part is predicted. At the end, cross_val_predict combines all the partially predicted labels and returns them as the final result.

This code shows this process step by step:

 X = np.array([[0], [1], [2], [3], [4], [5]]) labels = np.array(['a', 'a', 'a', 'b', 'b', 'b']) cv = KFold(len(labels), n_folds=3) clf = SVC() ypred_all = np.chararray((labels.shape)) i = 1 for train_index, test_index in cv.split(X): print("iteration", i, ":") print("train indices:", train_index) print("train data:", X[train_index]) print("test indices:", test_index) print("test data:", X[test_index]) clf.fit(X[train_index], labels[train_index]) ypred = clf.predict(X[test_index]) print("predicted labels for data of indices", test_index, "are:", ypred) ypred_all[test_index] = ypred print("merged predicted labels:", ypred_all) i = i+1 print("=====================================") y_cross_val_predict = cross_val_predict(clf, X, labels, cv=cv) print("predicted labels by cross_val_predict:", y_cross_val_predict) 

Result:

 iteration 1 : train indices: [2 3 4 5] train data: [[2] [3] [4] [5]] test indices: [0 1] test data: [[0] [1]] predicted labels for data of indices [0 1] are: ['b' 'b'] merged predicted labels: ['b' 'b' '' '' '' ''] ===================================== iteration 2 : train indices: [0 1 4 5] train data: [[0] [1] [4] [5]] test indices: [2 3] test data: [[2] [3]] predicted labels for data of indices [2 3] are: ['a' 'b'] merged predicted labels: ['b' 'b' 'a' 'b' '' ''] ===================================== iteration 3 : train indices: [0 1 2 3] train data: [[0] [1] [2] [3]] test indices: [4 5] test data: [[4] [5]] predicted labels for data of indices [4 5] are: ['a' 'a'] merged predicted labels: ['b' 'b' 'a' 'b' 'a' 'a'] ===================================== predicted labels by cross_val_predict: ['b' 'b' 'a' 'b' 'a' 'a'] 

Answer 2

As you can see from the cross_val_predict code on github , the function calculates forecasts for each fold and combines them. Predictions are based on a model learned from other folds.

Here is a combination of your code and an example in the code

 from sklearn import datasets, linear_model from sklearn.model_selection import cross_val_predict, KFold from sklearn.metrics import accuracy_score diabetes = datasets.load_diabetes() X = diabetes.data[:400] y = diabetes.target[:400] cv = KFold(n_splits=20) lasso = linear_model.Lasso() y_pred = cross_val_predict(lasso, X, y, cv=cv) accuracy = accuracy_score(y_pred.astype(int), y.astype(int)) print(accuracy) # >>> 0.0075 

Finally, to answer your question: "No, the accuracy is not averaged for each time"

Answer 3

As written in the sklearn.model_selection.cross_val_predict documentation:

It is impractical to transfer these forecasts to the assessment metric. Use cross_validate to measure generalization errors.

Answer 4

I would like to add an option for a quick and easy answer, in addition to what the previous developers did.

If you take the micro-average value of F1, you get a measure of accuracy. So, for example, it will be:

 from sklearn.model_selection import cross_val_score, cross_val_predict from sklearn.metrics import precision_recall_fscore_support as score y_pred = cross_val_predict(lm,df,y,cv=5) precision, recall, fscore, support = score(y, y_pred, average='micro') print(fscore) 

This works mathematically since micro-average gives a weighted average of the confusion matrix.

Good luck.