Is there any good McNemar test implemented in Python? I don't see it anywhere in Scipy.stats or Scikit-Learn. I might have missed some other good packages. Please advise.
The McNemar test is almost a test for comparing two classification algorithms / models based on a set of retention tests (not through the K-fold method or oversampling to simulate a test set). Two common alternatives are: a t-test to compare directly the true positive proportions p_A and p_B from two algorithms and models A and B by 1), assuming that the deviations follow the binomial distributions or 2) evaluate the deviations using repeated oversampling and test sets.
However, the last two were shown with high errors of type 1 (claim models are statistically different, but essentially the same). The McNemar test is still considered the best when comparing two classification algorithms or models. See
Dietterich10 .
Or as an alternative, how do people statistically compare two classification models in practice, if not using the McNemar test?
source share