How to measure accuracy of predictions using Python / Pandas?

Question

I used Elo and Glicko rating systems along with the results for matches to create ratings for players. Before each match, I can generate a mathematical expectation (float between 0 and 1) for each player based on their respective ratings. I would like to check how accurate this expectation is for two reasons:

• To compare difference rating systems
• To set up variables (e.g. kfactor in Elo) used to calculate grades

There are several differences from what you need to know about chess:

• Possible results are wins (which I consider as 1.0), losses (0.0), with very rare (<5%) draws (0.5 each). Each individual match is evaluated, not a series, as in chess.
• Players have fewer matches - many have less than 10, few have more than 25, and maximum is 75.

I think that the corresponding function is a “correlation”, I tried to create a DataFrame containing the forecast in one column (float between 0, 1) and the result in another (1 | 0.5 | 0) and using corr(), but based on the result, I do not sure that is correct.

If I create a DataFrame containing expectations and results only for the first player in the match (the results will always be 1.0 or 0.5, because due to my data source, the losers are never displayed in the first place), corr () returns very low: <0.05 . However, if I create a series that has two lines for each match and contains both the expectation and the result for each player (or, alternatively, randomly chooses which player to add, so the results will be either 0, 0.5, either 1), corr () is much higher: from 0.15 to 0.30. I don’t understand why this would affect me, which makes me wonder if I am abusing a function or not using the wrong function.

If this helps, here are some real (non-random) sample data: http://pastebin.com/eUzAdNij

Answer 1

, , , . , , ! , , 1.0, .

- (, , 0.5), .

, .

, Elo , . : . . : , . :

• , . 0...0.4 , 0.4..0.6 - 0.6...1.0 - , .
• |predicted_score-score|/number_of_games . , .
• : x, x, 1-x, (, (1-x)*x/4 - , 0.5 1). . , .

Answer 2

(ROC). sklearn matplotlib .

ROC . , , , . Area Under Curve (AUC) - : , .

import pandas as pd

# read data
df = pd.read_csv('sample_data.csv', header=None, names=['classifier','category'])

# remove values that are not 0 or 1 (two of those)
df = df.loc[(df.category==1.0) | (df.category==0.0),:]

# examine data frame
df.head()

from matplotlib import pyplot as plt
# add this magic if you're in a notebook
# %matplotlib inline

from sklearn.metrics import roc_curve, auc
# matplot figure
figure, ax1 = plt.subplots(figsize=(8,8))

# create ROC itself
fpr,tpr,_ = roc_curve(df.category,df.classifier)

# compute AUC
roc_auc = auc(fpr,tpr)

# plotting bells and whistles
ax1.plot(fpr,tpr, label='%s (area = %0.2f)' % ('Classifier',roc_auc))
ax1.plot([0, 1], [0, 1], 'k--')
ax1.set_xlim([0.0, 1.0])
ax1.set_ylim([0.0, 1.0])
ax1.set_xlabel('False Positive Rate', fontsize=18)
ax1.set_ylabel('True Positive Rate', fontsize=18)
ax1.set_title("Receiver Operating Characteristic", fontsize=18)
plt.tick_params(axis='both', labelsize=18)
ax1.legend(loc="lower right", fontsize=14)
plt.grid(True)
figure.show()

, :