All geek questions in one place

How to use InfiniBand network with Dask?

I have a cluster with a high-performance network (InfiniBand). However, when I set up the Dasks and Work Scheduler, the performance does not seem as fast as I expected. How can I tell Dask to use this network?

Disclaimer: I just ask this question to answer it. This is a frequently asked question.

+6
source share
1 answer

With dask.distributed version 1.16.3 you can specify the network interface for executable files dask-schedulerand dask-workerusing the keyword --interfaceas shown below:

dask-scheduler --interface ib0 --scheduler-file ~/my.cluster.yaml
dask-worker --interface ib0 --scheudler-file ~/my.cluster.yaml

, ib0. , - ifconfig

$ ifconfig
lo          Link encap:Local Loopback                       # Localhost
            inet addr:127.0.0.1  Mask:255.0.0.0
            inet6 addr: ::1/128 Scope:Host
eth0        Link encap:Ethernet  HWaddr XX:XX:XX:XX:XX:XX   # Ethernet
            inet addr:192.168.0.101
            ...
ib0         Link encap:Infiniband                           # Fast InfiniBand
            inet addr:172.42.0.101
+9

Source: https://habr.com/ru/post/1017121/

More articles:


All Articles