How to get the classifier's confidence rating for the forecast in sklearn?

Question

I would like to receive an assessment of the confidence of each of the forecasts that it makes, showing how confident that the classifier is correct in its prediction.

I need something like this:

How confident is the classifier in its prediction?

Class 1: 81%, which is class 1
Class 2: 10%
Class 3: 6%
Class 4: 3%

Samples of my code:

features_train, features_test, labels_train, labels_test = cross_validation.train_test_split(main, target, test_size = 0.4) # Determine amount of time to train t0 = time() model = SVC() #model = SVC(kernel='poly') #model = GaussianNB() model.fit(features_train, labels_train) print 'training time: ', round(time()-t0, 3), 's' # Determine amount of time to predict t1 = time() pred = model.predict(features_test) print 'predicting time: ', round(time()-t1, 3), 's' accuracy = accuracy_score(labels_test, pred) print 'Confusion Matrix: ' print confusion_matrix(labels_test, pred) # Accuracy in the 0.9333, 9.6667, 1.0 range print accuracy model.predict(sub_main) # Determine amount of time to predict t1 = time() pred = model.predict(sub_main) print 'predicting time: ', round(time()-t1, 3), 's' print '' print 'Prediction: ' print pred 

I suspect that I will use the score () function, but I seem to continue to execute it correctly. I do not know if this function is correct or not, but how to get the percentage of confidence from the forecast of the classifier?

Answer 1

In the SVC documentation , it looks like you need to change the way you build SVC:

 model = SVC(probability=True) 

and then use the pred_proba method:

 class_probabilities = model.predict_proba(sub_main) 

Answer 2

For those estimates that implement the predict_proba() method, as Justin Peel suggested, you can simply use predict_proba() to get the probability of your prediction.

For those estimates that do not implement the predict_proba() method, you can build the confidence interval yourself using the bootstrap concept (repeatedly calculate your point estimates in many subsamples).

Let me know if you need detailed examples to demonstrate either of these two cases.