All geek questions in one place

Scikit-learn GridSearchCV stops working when n_jobs> 1

I previously set the following lines of code here:

parameters = [{'weights': ['uniform'], 'n_neighbors': [5, 10, 20, 30, 40, 50, 60, 70, 80, 90, 100]}] clf = GridSearchCV(neighbors.KNeighborsRegressor(), parameters, n_jobs=4) clf.fit(features, rewards)

But when I started it, there was another problem that was not related to a previously asked question. Python ends with the following OS error message:

 Process: Python [1327] Path: /Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python Identifier: Python Version: 2.7.2.5 (2.7.2.5.r64662-trunk) Code Type: X86-64 (Native) Parent Process: Python [1316] Responsible: Sublime Text 2 [308] User ID: 501 Date/Time: 2014-08-12 10:27:24.640 +0200 OS Version: Mac OS X 10.9.4 (13E28) Report Version: 11 Anonymous UUID: D10CD8B7-221F-B121-98D4-4574A1F2189F Sleep/Wake UUID: 0B9C4AE0-26E6-4DE8-B751-665791968115 Crashed Thread: 0 Dispatch queue: com.apple.main-thread Exception Type: EXC_BAD_ACCESS (SIGSEGV) Exception Codes: KERN_INVALID_ADDRESS at 0x0000000000000110 VM Regions Near 0x110: --> __TEXT 0000000100000000-0000000100001000 [ 4K] rx/rwx SM=COW /Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python Application Specific Information: *** multi-threaded process forked *** crashed on child side of fork pre-exec Thread 0 Crashed:: Dispatch queue: com.apple.main-thread 0 libdispatch.dylib 0x00007fff91534c90 dispatch_group_async_f + 141 1 libBLAS.dylib 0x00007fff9413f791 APL_sgemm + 1061 2 libBLAS.dylib 0x00007fff9413cb3f cblas_sgemm + 1267 3 _dotblas.so 0x0000000102b0236e dotblas_matrixproduct + 5934 4 org.activestate.ActivePython27 0x00000001000c552d PyEval_EvalFrameEx + 23949 5 org.activestate.ActivePython27 0x00000001000c7ad6 PyEval_EvalCodeEx + 2118 6 org.activestate.ActivePython27 0x00000001000c5d10 PyEval_EvalFrameEx + 25968 7 org.activestate.ActivePython27 0x00000001000c7ad6 PyEval_EvalCodeEx + 2118 8 org.activestate.ActivePython27 0x000000010003d390 function_call + 176 9 org.activestate.ActivePython27 0x000000010000be12 PyObject_Call + 98 10 org.activestate.ActivePython27 0x00000001000c098a PyEval_EvalFrameEx + 4586 11 org.activestate.ActivePython27 0x00000001000c7ad6 PyEval_EvalCodeEx + 2118 12 org.activestate.ActivePython27 0x00000001000c5d10 PyEval_EvalFrameEx + 25968 13 org.activestate.ActivePython27 0x00000001000c7ad6 PyEval_EvalCodeEx + 2118 14 org.activestate.ActivePython27 0x00000001000c5d10 PyEval_EvalFrameEx + 25968 15 org.activestate.ActivePython27 0x00000001000c7137 PyEval_EvalFrameEx + 31127 16 org.activestate.ActivePython27 0x00000001000c7137 PyEval_EvalFrameEx + 31127 17 org.activestate.ActivePython27 0x00000001000c7ad6 PyEval_EvalCodeEx + 2118 18 org.activestate.ActivePython27 0x000000010003d390 function_call + 176 19 org.activestate.ActivePython27 0x000000010000be12 PyObject_Call + 98 20 org.activestate.ActivePython27 0x00000001000c098a PyEval_EvalFrameEx + 4586 21 org.activestate.ActivePython27 0x00000001000c7ad6 PyEval_EvalCodeEx + 2118 22 org.activestate.ActivePython27 0x000000010003d390 function_call + 176 23 org.activestate.ActivePython27 0x000000010000be12 PyObject_Call + 98 24 org.activestate.ActivePython27 0x000000010001d36d instancemethod_call + 365 25 org.activestate.ActivePython27 0x000000010000be12 PyObject_Call + 98 26 org.activestate.ActivePython27 0x0000000100077dfa slot_tp_call + 74 27 org.activestate.ActivePython27 0x000000010000be12 PyObject_Call + 98 28 org.activestate.ActivePython27 0x00000001000c098a PyEval_EvalFrameEx + 4586 29 org.activestate.ActivePython27 0x00000001000c7ad6 PyEval_EvalCodeEx + 2118 30 org.activestate.ActivePython27 0x000000010003d390 function_call + 176 31 org.activestate.ActivePython27 0x000000010000be12 PyObject_Call + 98 32 org.activestate.ActivePython27 0x00000001000c098a PyEval_EvalFrameEx + 4586 33 org.activestate.ActivePython27 0x00000001000c7137 PyEval_EvalFrameEx + 31127 34 org.activestate.ActivePython27 0x00000001000c7137 PyEval_EvalFrameEx + 31127 35 org.activestate.ActivePython27 0x00000001000c7ad6 PyEval_EvalCodeEx + 2118 36 org.activestate.ActivePython27 0x000000010003d390 function_call + 176 37 org.activestate.ActivePython27 0x000000010000be12 PyObject_Call + 98 38 org.activestate.ActivePython27 0x000000010001d36d instancemethod_call + 365 39 org.activestate.ActivePython27 0x000000010000be12 PyObject_Call + 98 40 org.activestate.ActivePython27 0x0000000100077a28 slot_tp_init + 88 41 org.activestate.ActivePython27 0x0000000100074e25 type_call + 245 42 org.activestate.ActivePython27 0x000000010000be12 PyObject_Call + 98 43 org.activestate.ActivePython27 0x00000001000c267d PyEval_EvalFrameEx + 11997 44 org.activestate.ActivePython27 0x00000001000c7137 PyEval_EvalFrameEx + 31127 45 org.activestate.ActivePython27 0x00000001000c7137 PyEval_EvalFrameEx + 31127 46 org.activestate.ActivePython27 0x00000001000c7ad6 PyEval_EvalCodeEx + 2118 47 org.activestate.ActivePython27 0x000000010003d390 function_call + 176 48 org.activestate.ActivePython27 0x000000010000be12 PyObject_Call + 98 49 org.activestate.ActivePython27 0x000000010001d36d instancemethod_call + 365 50 org.activestate.ActivePython27 0x000000010000be12 PyObject_Call + 98 51 org.activestate.ActivePython27 0x0000000100077a28 slot_tp_init + 88 52 org.activestate.ActivePython27 0x0000000100074e25 type_call + 245 53 org.activestate.ActivePython27 0x000000010000be12 PyObject_Call + 98 54 org.activestate.ActivePython27 0x00000001000c267d PyEval_EvalFrameEx + 11997 55 org.activestate.ActivePython27 0x00000001000c7ad6 PyEval_EvalCodeEx + 2118 56 org.activestate.ActivePython27 0x00000001000c5d10 PyEval_EvalFrameEx + 25968 57 org.activestate.ActivePython27 0x00000001000c7ad6 PyEval_EvalCodeEx + 2118 58 org.activestate.ActivePython27 0x000000010003d390 function_call + 176 59 org.activestate.ActivePython27 0x000000010000be12 PyObject_Call + 98 60 org.activestate.ActivePython27 0x000000010001d36d instancemethod_call + 365 61 org.activestate.ActivePython27 0x000000010000be12 PyObject_Call + 98 62 org.activestate.ActivePython27 0x0000000100077dfa slot_tp_call + 74 63 org.activestate.ActivePython27 0x000000010000be12 PyObject_Call + 98 64 org.activestate.ActivePython27 0x00000001000c267d PyEval_EvalFrameEx + 11997 65 org.activestate.ActivePython27 0x00000001000c7ad6 PyEval_EvalCodeEx + 2118 66 org.activestate.ActivePython27 0x00000001000c5d10 PyEval_EvalFrameEx + 25968 67 org.activestate.ActivePython27 0x00000001000c7ad6 PyEval_EvalCodeEx + 2118 68 org.activestate.ActivePython27 0x00000001000c5d10 PyEval_EvalFrameEx + 25968 69 org.activestate.ActivePython27 0x00000001000c7ad6 PyEval_EvalCodeEx + 2118 70 org.activestate.ActivePython27 0x00000001000c5d10 PyEval_EvalFrameEx + 25968 71 org.activestate.ActivePython27 0x00000001000c7ad6 PyEval_EvalCodeEx + 2118 72 org.activestate.ActivePython27 0x00000001000c7bf6 PyEval_EvalCode + 54 73 org.activestate.ActivePython27 0x00000001000ed31e PyRun_FileExFlags + 174 74 org.activestate.ActivePython27 0x00000001000ed5d9 PyRun_SimpleFileExFlags + 489 75 org.activestate.ActivePython27 0x00000001001041dc Py_Main + 2940 76 org.activestate.ActivePython27.app 0x0000000100000ed4 0x100000000 + 3796 Thread 0 crashed with X86 Thread State (64-bit): rax: 0x0000000000000100 rbx: 0x00007fff7cd43640 rcx: 0x0000000000000000 rdx: 0x0000000105e00000 rdi: 0x0000000000000008 rsi: 0x0000000105e01000 rbp: 0x00007fff5fbfa370 rsp: 0x00007fff5fbfa350 r8: 0x0000000000000001 r9: 0x0000000105e00000 r10: 0x0000000105e01000 r11: 0x0000000000000000 r12: 0x000000010ba10530 r13: 0x000000010b000000 r14: 0x00000001066d1970 r15: 0x00007fff915311af rip: 0x00007fff91534c90 rfl: 0x0000000000010206 cr2: 0x0000000000000110 Logical CPU: 2 Error Code: 0x00000006 Trap Number: 14 ......... VM Region Summary: ReadOnly portion of Libraries: Total=183.7M resident=97.0M(53%) swapped_out_or_unallocated=86.7M(47%) Writable regions: Total=1.3G written=142.8M(11%) resident=503.6M(39%) swapped_out=0K(0%) unallocated=791.7M(61%)

When I replaced the second line in my code with:

 clf = GridSearchCV(neighbors.KNeighborsRegressor(), parameters, n_jobs=1)

Then everything works fine, except that I do not use multiple threads.

My operating system is OSX 10.9.4

My python version is 2.7.8 | Anaconda 2.0.1 (x86_64) | (default, July 2, 2014 15:36:00) [GCC 4.2.1 (Apple Inc. build 5577)]

My version of scikit-lern is 0.14.1

My version of numpy 1.8.1

And my scipy version 0.14.0

My question is, does anyone know how to make GridSearchCV more than one thread?

EDIT:

I realized that this error actually only occurs for some of my input datasets. Unfortunately, the problematic datasets (its X) are too large, so they cannot be copied here. The input function data is mainly tf-idf vectors, and y vectors are floats> 0, in particular:

 [60.0, 7.0, 12.0, 21.0, 5.5, 3.0, 0.0, 2.5, 11.0, 3.0, 16.0, 2.0, 0.0, 4.5, 2.5, 6.0, 9.5, 2.5, 15.0, 7.0, 8.0, 13.0, 14.0, 8.0, 3.5, 6.0, 22.5, 7.0, 4.0, 3.5, 4.5, 6.0, 5.5, 7.0, 2.0, 0.0, 0.0, 0.0, 14.5, 8.0, 7.5, 2.5, 11.5, 1.0, 3.0, 14.5, 10.0, 14.5, 8.0, 8.0, 7.0, 2.5, 3.5, 3.0, 13.5, 7.0, 6.5, 2.5, 9.0, 8.0, 11.0, 17.5, 12.5, 4.5, 5.5, 8.0, 2.0, 7.0, 4.0, 1.5, 3.0, 21.5, 4.5, 4.0, 7.0, 9.0, 13.5, 8.0, 10.5, 4.5, 1.5, 11.5, 7.5, 11.5, 4.5, 5.0, 7.0, 9.5, 4.0, 4.0, 6.0, 3.5, 4.5, 7.5, 3.5, 3.5, 3.5, 6.0, 5.0, 5.5, 25.0, 6.5, 5.0, 2.0, 2.0, 10.5, 0.0, 6.5, 19.0, 9.0, 1.0, 1.5, 1.0, 0.0, 1.0, 4.5, 2.5, 17.5, 39.5, 7.5, 5.5, 8.0, 1.0, 6.0, 12.0, 10.0, 5.5, 19.0, 4.5, 1.5, 25.5, 4.0, 10.0, 18.5, 9.5, 10.5, 2.5, 6.0, 1.0, 10.0, 8.5, 12.5, 13.5, 5.0, 6.5, 11.0, 4.5, 8.0, 7.5, 11.5, 14.5, 9.0, 3.0, 1.5, 3.5, 5.5, 2.5, 12.5, 6.5, 5.5, 5.0, 0.0, 8.0, 3.0, 14.5, 5.0, 14.0, 7.0, 13.5, 12.5, 4.0, 1.5, 6.5, 10.5, 9.0, 16.5, 4.0, 4.0, 15.0, 11.5, 2.5, 8.5, 3.0, 5.0, 4.0, 8.5, 6.0, 5.0, 5.0, 5.0, 5.5, 8.0, 11.0, 4.0, 0.0, 5.5, 0.0, 4.5, 1.5, 0.0, 6.5, 11.0, 2.5, 8.0, 15.5, 5.5, 4.5, 5.0, 4.0, 5.5, 10.5, 7.5, 6.5, 8.5, 2.5, 1.5, 1.5, 18.0, 15.0, 14.0, 9.5, 5.5, 7.5, 14.5, 2.5, 5.0, 60.0, 6.5, 14.5, 6.5, 4.0, 1.5, 2.0, 4.0, 27.0, 3.0, 5.0, 4.0, 2.5, 1.0, 1.5, 1.5, 9.0, 4.0, 8.5, 4.0, 4.0, 0.0, 1.5, 7.5, 1.5, 7.5, 1.0, 28.5, 15.5, 7.5, 1.0, 2.5, 2.5, 2.5, 16.0, 5.5, 8.5, 4.0, 2.5, 5.0, 2.5, 6.0, 11.0, 10.0, 4.5, 6.5, 8.0, 6.0, 4.5, 15.5, 4.0, 5.0]

The 1-task version works for all of my input sets, even for that.

+5
source share
2 answers

libdispatch.dylib from Grand Central Dispatch is used internally by OSX with an embedded BLAS implementation called Accelerate when numpy.dot called. The GCD runtime does not work when programs call Syscall POSIX fork without using syscall exec after that and therefore do all Python programs that use the multiprocessing module, which is prone to crash. sklearn GridsearchCV uses the Python multiprocessing module to parallelize.

In Python 3.4 and later, you can force Python multiprocessing to use the forkserver startup method instead of the default fork mode to work around this problem, for example, at the beginning of your program’s main file:

 if __name__ == "__main__": import multiprocessing as mp; mp.set_start_method('forkserver')

Alternatively, you can rebuild numpy from the source and associate it with ATLAS or OpenBLAS instead of OSX Accelerate. Numpy developers work on binary distributions that include ATLAS or OpenBLAS by default.

+3
source

This worked fine for me (the upgrade was a bit draggable, but that was the only fix that many tried to use in my case). For any other ipython laptop users, the best way to do this is to add it to your laptop configuration (you will receive an error message to run it directly in the laptop). Commands can be added as follows:

 # in ipython_notebook_config.py c.IPKernelApp.exec_lines = ['import multiprocessing', 'multiprocessing.set_start_method("forkserver")']
+1
source

Source: https://habr.com/ru/post/1200096/

More articles:


All Articles