How to convert from dask data frame to futures list?

Question

I have a dask dataframe as shown below:

import dask.dataframe as dd
df = dd.read_csv('s3://...')

How do I get a list of futures from this frame?

Answer 1

You can use the .to_delayed method to convert from a dask data frame to a dask.delayed objects list

L = df.to_delayed()

You can then convert these lagging objects to dask futures using the method client.compute.

from dask.distributed import Client
client = Client()

futures = client.compute(L)