Join limited concurrency futures

Question

I have a large vector of Hyper HTTP HTTP request futures and you want to allow them to be a result vector. Since there is a limit on the maximum open files, I want to limit the number of frames to the amount of concurrency.

I experimented with Stream::buffer_unordered, but it looks like he executed futures one by one.

Answer 1

We used code like this in the project to avoid opening too many TCP sockets. These futures have Hyper futures inside, so this is similar to the same case.

// Convert the iterator into a `Stream`. We will process
// `PARALLELISM` futures at the same time, but with no specified
// order.
let all_done =
    futures::stream::iter(iterator_of_futures.map(Ok))
    .buffer_unordered(PARALLELISM);

// Everything after here is just using the stream in
// some manner, not directly related

let mut successes = Vec::with_capacity(LIMIT);
let mut failures = Vec::with_capacity(LIMIT);

// Pull values off the stream, dividing them into success and
// failure buckets.
let mut all_done = all_done.into_future();
loop {
    match core.run(all_done) {
        Ok((None, _)) => break,
        Ok((Some(v), next_all_done)) => {
            successes.push(v);
            all_done = next_all_done.into_future();
        }
        Err((v, next_all_done)) => {
            failures.push(v);
            all_done = next_all_done.into_future();
        }
    }
}

, (core) . , , , . , , , , .