How to get a String Iterator using a Dataframe in SparkSQL

Question

I have an application in SparkSQL that returns a large number of rows that are very difficult to fit in memory, so I cannot use the collection function in the DataFrame, there is a way by which I can get all these rows as an Iterable installation of all rows as a list .

Note. I am running this SparkSQL application using a yarn client

Answer 1

Generally speaking, transferring all the data to the driver seems like a pretty bad idea, and most of the time there is a better solution, but if you really want to do this, you can use the method toLocalIteratoron RDD:

val df: org.apache.spark.sql.DataFrame = ???
df.cache // Optional, to avoid repeated computation, see docs for details
val iter: Iterator[org.apache.spark.sql.Row]  = df.rdd.toLocalIterator 

Answer 2

In fact, you can simply use:, df.toLocalIteratorhere is the link in the Spark source code:

/**
 * Return an iterator that contains all of [[Row]]s in this Dataset.
 *
 * The iterator will consume as much memory as the largest partition in this Dataset.
 *
 * Note: this results in multiple Spark jobs, and if the input Dataset is the result
 * of a wide transformation (e.g. join with different partitioners), to avoid
 * recomputing the input Dataset should be cached first.
 *
 * @group action
 * @since 2.0.0
 */
def toLocalIterator(): java.util.Iterator[T] = withCallback("toLocalIterator", toDF()) { _ =>
withNewExecutionId {
  queryExecution.executedPlan.executeToIterator().map(boundEnc.fromRow).asJava
  }
}