Up / cumulative growth graphs are not a good way to evaluate a model (because it cannot be used to compare between models) and instead are a means of evaluating results in which your resources are finite. Either because there are costs for the action of each result (in the marketing scenario), or you want to ignore a certain number of guaranteed voters and only the action of those who are on the fence. If your model is very good and has a high classification accuracy for all results, you will not be able to significantly increase the efficiency of ordering your results.
import sklearn.metrics import pandas as pd def calc_cumulative_gains(df: pd.DataFrame, actual_col: str, predicted_col:str, probability_col:str):
The method is as follows: first sort the data in the bins and order with confidence. The method returns a data frame that will be used for visualization.
df.sort_values(by=probability_col, ascending=False, inplace=True) subset = df[df[predicted_col] == True] rows = [] for group in np.array_split(subset, 10): score = sklearn.metrics.accuracy_score(group[actual_col].tolist(), group[predicted_col].tolist(), normalize=False) rows.append({'NumCases': len(group), 'NumCorrectPredictions': score}) lift = pd.DataFrame(rows) #Cumulative Gains Calculation lift['RunningCorrect'] = lift['NumCorrectPredictions'].cumsum() lift['PercentCorrect'] = lift.apply( lambda x: (100 / lift['NumCorrectPredictions'].sum()) * x['RunningCorrect'], axis=1) lift['CumulativeCorrectBestCase'] = lift['NumCases'].cumsum() lift['PercentCorrectBestCase'] = lift['CumulativeCorrectBestCase'].apply( lambda x: 100 if (100 / lift['NumCorrectPredictions'].sum()) * x > 100 else (100 / lift[ 'NumCorrectPredictions'].sum()) * x) lift['AvgCase'] = lift['NumCorrectPredictions'].sum() / len(lift) lift['CumulativeAvgCase'] = lift['AvgCase'].cumsum() lift['PercentAvgCase'] = lift['CumulativeAvgCase'].apply( lambda x: (100 / lift['NumCorrectPredictions'].sum()) * x) #Lift Chart lift['NormalisedPercentAvg'] = 1 lift['NormalisedPercentWithModel'] = lift['PercentCorrect'] / lift['PercentAvgCase'] return lift
To plot the cumulative gain, you can use this code below.
import matplotlib.pyplot as plt def plot_cumulative_gains(lift: pd.DataFrame): fig, ax = plt.subplots() fig.canvas.draw() handles = [] handles.append(ax.plot(lift['PercentCorrect'], 'r-', label='Percent Correct Predictions')) handles.append(ax.plot(lift['PercentCorrectBestCase'], 'g-', label='Best Case (for current model)')) handles.append(ax.plot(lift['PercentAvgCase'], 'b-', label='Average Case (for current model)')) ax.set_xlabel('Total Population (%)') ax.set_ylabel('Number of Respondents (%)') ax.set_xlim([0, 9]) ax.set_ylim([10, 100]) labels = [int((label+1)*10) for label in [float(item.get_text()) for item in ax.get_xticklabels()]] ax.set_xticklabels(labels) fig.legend(handles, labels=[h[0].get_label() for h in handles]) fig.show()
And visualize the rise:
def plot_lift_chart(lift: pd.DataFrame): plt.figure() plt.plot(lift['NormalisedPercentAvg'], 'r-', label='Normalised \'response rate\' with no model') plt.plot(lift['NormalisedPercentWithModel'], 'g-', label='Normalised \'response rate\' with using model') plt.legend() plt.show()
The result looks like this:
I found these sites useful for reference:
Edit:
I found that the MS link is somewhat misleading in my descriptions, but the Paul Te Braak link is very informative. To respond to a comment,
@Tanguy for the accumulated winnings chart above, all calculations are based on accuracy for this particular model. As the Paul Te Braak link notes, how does my model prediction accuracy reach 100% (red line on the graph)? The best scenario (green line) is how quickly we can achieve the same accuracy as the red line throughout the population (for example, our optimal scenario for cumulative income). Blue is if we simply randomly select a classification for each sample in a population. Thus, the cumulative profits and elevator schedules are intended solely to understand how this model (and only this model) will give me more influence in the scenario when I will not interact with the entire population.
One scenario that I used in the cumulative profit diagram is fraud cases where I want to know how many applications we can ignore or prioritize (because I know that the model predicts them, and also can) for the top X percentage. In this case, for the “middle model”, I instead chose a classification from a real disordered dataset (to show how existing applications are processed and how - using the model - we could instead prioritize application types).
So, to compare models, just stick to the ROC / AUC, and once you are happy with the model you choose, use the cumulative profit / rise chart to see how it reacts to the data.