Why is there no good acceleration of parallel testing of Astropy with pytest-xdist?

I run Astropy tests in parallel using the option python setup.py test --parallel Non my Macbook (4 real cores, solid state drive) that pytest-xdist uses to run ~ 8000 tests in parallel.

I tried different Nin the range from 1 to 10, but in all cases I can only get accelerations of about 2, but I expected to get acceleration in the range of 3 to 4 (since the launch of tests should be limited by the CPU).

Why is speed decreasing and how can I get good accelerations (using several cores on one computer)?

Update

I tried the ramdisk suggestion from @Iguananaut: diskutil erasevolume HFS+ 'ramdisk' hdiutil attach -nomount ram://8388608 mkdir//ramdisk/tmp python setup.py test -a '--basetemp =//ramdisk/tmp' --parallel 8 > ~ 2.2 ~ 2.0 SSD. , - 3 4. , - .

+4
1

, SSD . , , - . , , . , . . 6x, .

, , ramdisk temp. OSX diskutil. Google, , . ./setup.py test -A '--basetemp=path/to/ramdisk'. Astropy , . , , , - .

, , . ramdisk - RAM , I/O. , , CPU, , I/O , ( ). - , , , , , . , , , , . , , , , , .

TL; DR , , , - .

+2

Source: https://habr.com/ru/post/1537208/


All Articles