What is the difference between explosion function and operator?

Question

What is the difference between a function explodeand an operator explode?

Answer 1

spark.sql.functions.explode

Function

explode creates a new row for each element in the specified column or column of the map (in the DataFrame).

val signals: DataFrame = spark.read.json(signalsJson)
signals.withColumn("element", explode($"data.datapayload"))

explodecreates a Column .

See functions and an example in How to unwind an array in a DataFrame (from JSON)?

Dataset<Row> explode/ flatMapoperator (method)

Operator

explodeis almost a function explode.

From the skaladok:

explode , . LATERAL VIEW HiveQL. , .

ds.flatMap(_.words.split(" "))

, ( scaladoc):

( 2.0.0) functions.explode() flatMap() select()

. API- ?

---

, explode ( explode flatMap), , , . , . , .

, flatMap (.. explode) Scala - , flatMap Scala ( , ).