Combine enumerated lists lazily

Question

I would like to write a function similar to List.concat / 1, which enumerates lists and emits concatenated lists as a continuous stream.

It will work as follows:

iex> 1..3 |> Stream.map(&([&1])) |> Enum.to_list [[1], [2], [3]] iex> 1..3 |> Stream.map(&([&1])) |> MyStream.concat |> Enum.to_list [1, 2, 3] 

So far I have come to the following:

 defmodule MyStream do def concat(lists) do Enumerable.reduce(lists, [], fn(x, acc) -> acc ++ x end) end end 

This gives the correct result, but obviously not lazy.

I tried unsuccessfully to use Stream.Lazy , but did not really understand its inner workings. Any explanation on Stream.Lazy would be Stream.Lazy appreciated!

Answer 1

Enums in Elixir are represented using abbreviations. We can display any structure while you tell us how to reduce it.

The whole idea of ​​Stream is that you can create those reduction functions. Take the map as an example:

 def map(enumerable, f) do Lazy[enumerable: enumerable, fun: fn(f1) -> fn(entry, acc) -> f1.(f.(entry), acc) end end] end 

You get an enumerated value, and you want to match each element using the f function. The tape version receives the actual reduction function f1 and returns a new function that receives entry and acc (the same arguments f1 will receive) and then calls f.(entry) , effectively matching the element before calling f1 (reduction function). Notice how we match the elements one at a time.

A flat map version of this is likely to be something like this:

 def flat_map(enumerable, f) do Lazy[enumerable: enumerable, fun: fn(f1) -> fn(entry, acc) -> Enumerable.reduce(f.(entry), acc, f1) end end] end 

Now, every time you call f.(entry) , you get the list back and you want to iterate over each element of this new list, rather than iterating over the list as a whole.

I have not tried the code above (and maybe I missed some details), but that Streams work in general.

Answer 2

With the help of Jose Valim, it was just a small step from his code to what I was looking for. I probably asked this question pretty poorly, but what I was really looking for was equivalent to the Python function itertools.chain .

 def chain(enumerable) do Stream.Lazy[enumerable: enumerable, fun: fn(f1) -> fn(entry, acc) -> Enumerable.reduce(entry, acc, f1) end end] end 

This allows you to link potentially endless enumerations of both streams and lists.

 iex> 1..1000000 |> Stream.map(&(1..(&1))) |> MyModule.chain |> Enum.take(20) [1, 1, 2, 1, 2, 3, 1, 2, 3, 4, 1, 2, 3, 4, 5, 1, 2, 3, 4, 5]