I am trying to use Spearman's correlation wrapped in a user-defined metric to find the nearest neighbors in Scikit-learn. For some reason, it only works when the number of columns in my training data is 5 and k = 5. For any other combination (for example, the number of columns = 8 and k = 6) this would give me the following error. (Here the train and test set have 4 columns and k = 4). It works fine if I use Pearson for correlation. Does anyone know why this might happen or how to fix it? Thank.
from scipy.stats import spearmanr
def spearmancorr(x,y):
rho, pval = spearmanr(x,y, axis=0)
return rho * (-1)
from sklearn.neighbors import NearestNeighbors
nbrs = NearestNeighbors(n_neighbors=4, algorithm='ball_tree', metric=spearmancorr)
nbrs.fit(train)
dist, ind = nbrs.kneighbors(test)
SystemError Traceback (most recent call last)
<ipython-input-11-f04b508b1263> in <module>()
5 for i in range(1):
6 nbrs = NearestNeighbors(n_neighbors=4, algorithm='ball_tree', metric=spearmancorr)
----> 7 nbrs.fit(train)
8 dist, ind = nbrs.kneighbors(test)
9 print "for: " + funcs[i]
C:\Users\AppData\Local\Enthought\Canopy\User\lib\site-packages\sklearn\neighbors\base.pyc in fit(self, X, y)
797 or [n_samples, n_samples] if metric='precomputed'.
798 """
--> 799 return self._fit(X)
C:\Users\AppData\Local\Enthought\Canopy\User\lib\site-packages\sklearn\neighbors\base.pyc in _fit(self, X)
238 self._tree = BallTree(X, self.leaf_size,
239 metric=self.effective_metric_,
--> 240 **self.effective_metric_params_)
241 elif self._fit_method == 'kd_tree':
242 self._tree = KDTree(X, self.leaf_size,
SystemError: NULL result without error in PyObject_Call
source
share