How do I collect a list of rows from a spark column in a DataFrame after a GroupBy operation?

Question

The solution described here (at zero 323) is very close to what I want with two twists:

• How do I do this in Java?
• What if the column had a list of rows instead of one String, and I want to collect all such lists into one list after GroupBy (another column)?

I am using Spark 1.6 and trying to use

org.apache.spark.sql.functions.collect_list(Column col)as described in solving this issue, but received the following error:

An exception in the stream "main" org.apache.spark.sql.AnalysisException: undefined function collect_list; at org.apache.spark.sql.catalyst.analysis.SimpleFunctionRegistry $$ anonfun $ 2.apply (FunctionRegistry.scala: 65) at org.apache.spark.sql.catalyst.analysis.SimpleFunctionRegistry $$ anonfun $ 2.apply (FunctionRegistry. scala: 65) on scala.Option.getOrElse (Option.scala: 121)

Answer 1

The error you see suggests using a simple SQLContextnot HiveContext. collect_listis UDF bush and as such requires HiveContext. It also does not support complex columns, so the only option is explode:

import org.apache.spark.api.java.*;
import org.apache.spark.SparkConf;
import org.apache.spark.sql.SQLContext;
import org.apache.spark.sql.hive.HiveContext;
import java.util.*;
import org.apache.spark.sql.DataFrame;
import static org.apache.spark.sql.functions.*;

public class App {
  public static void main(String[] args) {
    JavaSparkContext sc = new JavaSparkContext(new SparkConf());
    SQLContext sqlContext = new HiveContext(sc);
    List<String> data = Arrays.asList(
            "{\"id\": 1, \"vs\": [\"a\", \"b\"]}",
            "{\"id\": 1, \"vs\": [\"c\", \"d\"]}",
            "{\"id\": 2, \"vs\": [\"e\", \"f\"]}",
            "{\"id\": 2, \"vs\": [\"g\", \"h\"]}"
    );
    DataFrame df = sqlContext.read().json(sc.parallelize(data));
    df.withColumn("vs", explode(col("vs")))
           .groupBy(col("id"))
           .agg(collect_list(col("vs")))
           .show();
  }
}

, .