What is the fastest way to minimize function in python?

Question

Therefore, I have the following problem to minimize. I have a vector wthat I need to find in order to minimize the following function:

import numpy as np
from scipy.optimize import minimize

matrix = np.array([[1.0, 1.5, -2.],
                   [0.5, 3.0, 2.5],
                   [1.0, 0.25, 0.75]])

def fct(x):
    return x.dot(matrix).dot(x)

x0 = np.ones(3) / 3
cons = ({'type': 'eq', 'fun': lambda x: x.sum() - 1.0})
bnds = [(0, 1)] * 3

w = minimize(fct, x0, method='SLSQP', bounds=bnds, constraints=cons)['x']

I chose method='SLSQP'because it seems that he is the only one that allows you to use boundsand constraints. My problem is that I will have to loop my solution into several options, so I'm trying to get some speed here. Is my solution the fastest with the optimizer or will there be any other faster solutions? Thank.

Answer 1

Introduction

, .

scipy.minimize , , . , .

, SLSQP, , , , ( , ).

: SLSQP ( , , , LinearProgramming SemidefiniteProgramming).

, M ( , ), , ( , ).

:

• scipy, , -, -

: M -/-

, M , , , . , , .

:

• SLSQP -
• ,
  • : Gurobi, CPLEX, Mosek
  • : ECOS, SCS

Python + cvxpy + ecos/scs

, linprog, .

, , , .

:

• cvxpy
  • , !
    • ( )
• ecos
  • • Inside-Point
• scs
  • • (ADMM)
  • :
    • ( )
    • ( )
      • GPU -
  • ECOS,

:

• :
  • 1000x1000
  • : SCS
    • (, BLAS)

:

import time
import numpy as np
from cvxpy import *                 # Convex-Opt

""" Create some random pos-def matrix """
N = 1000
matrix_ = np.random.normal(size=(N,N))
matrix = np.dot(matrix_, matrix_.T)

""" CVXPY-based Convex-Opt """
print('\ncvxpy\n')
x = Variable(N)
constraints = [x >= 0, x <= 1, sum(x) == 1]
objective = Minimize(quad_form(x, matrix))
problem = Problem(objective, constraints)
time_start = time.perf_counter()
problem.solve(solver=SCS, use_indirect=True, verbose=True)  # or: solver=ECOS
time_end = time.perf_counter()
print(problem.value)
print('cvxpy (modelling) + ecos/scs (solving) used (secs): ', time_end - time_start)

:

cvxpy

----------------------------------------------------------------------------
    SCS v1.2.6 - Splitting Conic Solver
    (c) Brendan O'Donoghue, Stanford University, 2012-2016
----------------------------------------------------------------------------
Lin-sys: sparse-indirect, nnz in A = 1003002, CG tol ~ 1/iter^(2.00)
eps = 1.00e-03, alpha = 1.50, max_iters = 2500, normalize = 1, scale = 1.00
Variables n = 1001, constraints m = 3003
Cones:  primal zero / dual free vars: 1
    linear vars: 2000
    soc vars: 1002, soc blks: 1
Setup time: 6.76e-02s
----------------------------------------------------------------------------
 Iter | pri res | dua res | rel gap | pri obj | dua obj | kap/tau | time (s)
----------------------------------------------------------------------------
     0|      inf       inf      -nan      -inf      -inf       inf  1.32e-01 
   100| 1.54e-02  1.48e-04  7.63e-01 -5.31e+00 -4.28e+01  1.10e-11  1.15e+00 
   200| 1.53e-02  1.10e-04  7.61e-01 -3.87e+00 -3.17e+01  1.08e-11  1.95e+00 
   300| 1.53e-02  7.25e-05  7.55e-01 -2.47e+00 -2.08e+01  1.07e-11  2.79e+00 
   400| 1.53e-02  3.61e-05  7.39e-01 -1.11e+00 -1.03e+01  1.06e-11  3.61e+00 
   500| 7.64e-03  2.55e-04  1.09e-01 -2.01e-01 -6.32e-02  1.05e-11  4.64e+00 
   560| 7.71e-06  4.24e-06  8.61e-04  2.17e-01  2.16e-01  1.05e-11  5.70e+00 
----------------------------------------------------------------------------
Status: Solved
Timing: Solve time: 5.70e+00s
    Lin-sys: avg # CG iterations: 1.71, avg solve time: 9.98e-03s
    Cones: avg projection time: 3.97e-06s
----------------------------------------------------------------------------
Error metrics:
dist(s, K) = 5.1560e-16, dist(y, K*) = 0.0000e+00, s'y/|s||y| = 2.4992e-17
|Ax + s - b|_2 / (1 + |b|_2) = 7.7108e-06
|A'y + c|_2 / (1 + |c|_2) = 4.2390e-06
|c'x + b'y| / (1 + |c'x| + |b'y|) = 8.6091e-04
----------------------------------------------------------------------------
c'x = 0.2169, -b'y = 0.2157
============================================================================
0.21689554805292935
cvxpy (modelling) + ecos/scs (solving) used (secs):  7.105745473999832

-: 5000x5000 ~ 9 ( ).

:

• SCS , ECOS ()
• SCS/ECOS , ( jacobi-matrix) SLSQP ( , , N >= 100); = N
• SLSQP

Answer 2

pylang, jacobian , :

def fct_deriv(x):
    return 2 * matrix.dot(x)

minimize(fct, x0, method='SLSQP', jac=fct_deriv, bounds=bnds, constraints=cons)['x']

, SLSQP . , SLSQP ( ).

:

import numpy as np
from scipy.optimize import minimize

matrix = np.array([[1.0, 1.5, -2.],
                   [0.5, 3.0, 2.5],
                   [1.0, 0.25, 0.75]])

def fct(x):
    return x.dot(matrix).dot(x)

def fct_deriv(x):
    return 2 * matrix.dot(x)

x0 = np.ones(3) / 3
cons = ({'type': 'eq', 'fun': lambda x: x.sum() - 1.0})
bnds = [(0, 1)] * 3

w = minimize(fct, x0, method='SLSQP', jac=fct_deriv, bounds=bnds, constraints=cons)['x']

( jacobian ):

cons2 = ({'type': 'eq', 'fun': lambda x: x.sum() - 1.0, 'jac': lambda x: np.ones_like(x)})

w = minimize(fct, x0, method='SLSQP', jac=fct_deriv, bounds=bnds, constraints=cons2)['x']