Yes
The largest Dask.distributed cluster I've seen is about a thousand nodes. We could theoretically go more, but not for a huge amount.
The current limit is that the scheduler is taken at 200 microseconds per task. This means about 5,000 tasks per second. If each of your tasks takes about one second, then the scheduler can saturate about 5,000 cores.
Historically, we have encountered other restrictions, such as file access restrictions and the like. They were all cleaned up to the scale we saw (1000 nodes), and in general, everything is fine on Linux or OSX. Dask schedulers on Windows stop scaling at low hundreds of nodes (although you can use the Linux scheduler with working Windows). I would not be surprised if other problems arise, as we scale to 10 thousand. Nodes.
In short, you probably don't want to use Dask to replace MPI workloads on your millionth Big Iron Super supercomputer or Google Scale. Otherwise, you are probably fine.
source share