Ridge regression: Scikit-learn versus direct calculation does not match alpha> 0

Question

In Ridge Regression, we solve Ax=bwith L2Regularization. Direct calculation is defined as follows:

x = (A ^T A + alpha * I) ^-1 A ^T b

I looked at the scikit-learn code and they do the same calculation. But I can’t get the same results foralpha > 0

The minimum code to play.

import numpy as np
A = np.asmatrix(np.c_[np.ones((10,1)),np.random.rand(10,3)])
b = np.asmatrix(np.random.rand(10,1))
I = np.identity(A.shape[1])
alpha = 1
x = np.linalg.inv(A.T*A + alpha * I)*A.T*b
print(x.T)
>>> [[ 0.37371021  0.19558433  0.06065241  0.17030177]]

from sklearn.linear_model import Ridge
model = Ridge(alpha = alpha).fit(A[:,1:],b)
print(np.c_[model.intercept_, model.coef_])
>>> [[ 0.61241566  0.02727579 -0.06363385  0.05303027]]

Any suggestions on what I can do to resolve this inconsistency?

Answer 1

This modification seems to give the same result for the direct version and the numpy version:

import numpy as np
A = np.asmatrix(np.random.rand(10,3))
b = np.asmatrix(np.random.rand(10,1))
I = np.identity(A.shape[1])
alpha = 1
x = np.linalg.inv(A.T*A + alpha * I)*A.T*b
print (x.T)


from sklearn.linear_model import Ridge
model = Ridge(alpha = alpha, tol=0.1, fit_intercept=False).fit(A ,b)

print model.coef_
print model.intercept_

, Ridge fit_intercept=True ( _BaseRidge) (source)

_solve_cholesky.

ridge.py,

        X, y, X_mean, y_mean, X_std = self._center_data(
        X, y, self.fit_intercept, self.normalize, self.copy_X,
        sample_weight=sample_weight)

, , , 1. , , fit_intercept=False

: Ridge ?

solver.

, solver Ridge, solver='auto' ( solver='cholesky'). .

, _solve_cholesky numpy.linalg.solve numpy.inv. ,

np.linalg.solve(A.T*A + alpha * I, A.T*b)

,

np.linalg.inv(A.T*A + alpha * I)*A.T*b