How to rotate multiple columns on a Spark Dataframe

I am trying to rotate the Spark dataframe into multiple columns, I use the Pivot function, but after adding two columns it gives an error as an overloaded parameter.

This is the error I get after adding a third column that overloaded the pivot parameter value with alternatives: (pivotColumn: String, values: java.util.List [Any]) org.apache.spark.sql.RelationalGroupedDa taset (pivotColumn: String, values : Seq [Any]) org.apache.spark.sql.RelationalGroupedDataset (pivotColumn: String) org.apache.spark.sql.RelationalGroupedDataset cannot be applied to (String, String, String)

Here is my job:

 val df_new=df.join(df1, df("Col1")<=>df1("col1") && df1("col2")<=> df("col2")).groupBy(df("Col6"))
                             .agg(
                                 sum(df("Col1")).alias("Col1"), 
                                 sum(df("Col2")).alias("Col2")  ,
                                 sum(df("Col3")).alias("Col3")  ,
                                 sum(df("Col4")).alias("Col4")  ,
                                 sum(df("Col5")).alias("Col5")  
                                 ).select(
                                         Amount,'Col1, 'Col2,'Col3,'Col4,'Col5
                                          )

- pivot

val pivotdf=df_new.groupBy($"Col1").
                  pivot("Col1","Col2","Col3","col4")

I need to turn on col1, Col2, col3, col4 and col5. Please explain to me how I can do this.

+4

Source: https://habr.com/ru/post/1672335/


All Articles