Grid Search with Weighted AUC

Question

So, I use the sample weights function, and I don't want to measure performance using the default sclearn scoring function.

It looks like he says here that I can pass the GridSearchCVstring 'roc_auc' and it should calculate auc for me, but will auc calculate be weighted auc or just vanilla auc?

Answer 1

Thanks for the snipe nerd.

I created a binary classification dataset to test this problem.

x y weight 
0 0   1
1 1   1
<repeated 25 times>
0 1   0
1 0   0
<repeated 25 times>

Using python:

X = np.array([[0], [1]] * 25 +  [[0], [1]] * 25)
y = np.array([ 0 ,  1 ] * 25 +  [ 1 ,  0 ] * 25)
w = np.array([ 1 ,  1 ] * 25 +  [ 0 ,  0 ] * 25)

, , . , .

GridSearchCV, , .

clf = LogisticRegression(solver='newton-cg', C=100)
gs = GridSearchCV(clf, {},
                  fit_params={"sample_weight": w}, 
                  scoring="log_loss", cv=KFold(y.shape[0],10, shuffle=True))
gs.fit(X,y)
gs.grid_scores_

[mean: -2.68562, std: 0.68038, params: {}]

, , , .

scikit-learn, . , . https://github.com/scikit-learn/scikit-learn/compare/master...dmaust:master

score_sample_weight, , .

gs.score_sample_weight=True
gs.fit(X,y)
gs.grid_scores_

[mean: -0.00486, std: 0.00016, params: {}]