How to smooth nested lists in PySpark?

Question

I have an RDD structure like:

rdd = [[[1],[2],[3]], [[4],[5]], [[6]], [[7],[8],[9],[10]]]

and I want this to become:

rdd = [1,2,3,4,5,6,7,8,9,10]

How do I write a map or reduce a function to make it work?

Answer 1

You can, for example, flatMapuse lists:

rdd.flatMap(lambda xs: [x[0] for x in xs])

or make it a little more general:

from itertools import chain

rdd.flatMap(lambda xs: chain(*xs)).collect()