Logistic Regression

Question

I use sklearn's logistic regression function and wonder what each of the solvers actually does behind the scenes to solve the optimization problem.

Can someone briefly describe what "newton-cg", "sag", "lbfgs" and "liblinear" do? If not, any links or reading material are also greatly appreciated.

Thank you very much in advance.

Answer 1

Well, I hope I'm not late for the party! Let me first try to establish some intuition before digging into loads of information ( warning : this is not a short comparison)

---

h(x) .

, , , (.. , ..).

- (aka Thetas Weights), . , , , - , !

, (.. ) (. . , , ).

, , (.. ) , , , :

(.. ), = 0 ( , ).

, , , , Local Optima; :

, , : Deravative, Tangent Line, Cost Function, Hypothesis..etc.

: (. ).

---

:

f(x) x=a. L (x) : L(x)=f(a)+f′(a)(x−a).

:

, x=a . L(x) f(x) x=a. x=a.

:

, , , , 0, .

( , ), :

, , , , ,... ( ): Qa(x) = f(a) + f'(a)(xa) + f''(a)(xa)2/2

.

---

1.

x: (.. ).

. , (.. ).

( - nxn).

, , f(x) xn / ( ). , f(x) , .

:

1. - (.. ).

2. , (.. , , !).

2. . ---. :

, , , ( ). , .

" " , , .

, , L-BFGS , , "" , , , ,

3. :

, ( , , ).

(CD), , .

LIBLINEAR ICML 2008. (aka L1 Regularization), , ( )

:

1. (.. ), .

2. .

3. () ; "-vs-rest", .

: Scikit: "liblinear" .

4. :

SAG . (SG), SAG . SAG , SG .

, , .

:

1. L2.

2. O(N), N ( ).

5. SAGA:

SAGA - SAG, = l1 ( L1-). , , .

: Scikit: SAGA .

---

Scikit