Generating code for the entire scene in Spark 2.0

Question

I heard about Whole-Stage Code Generationfor sql to optimize queries. via p539-neumann.pdf and sparksql-sql-codegen-is-not-giving-any-improvemnt

But, unfortunately, no one answered this question.

Curious to know what are the usage scenarios for this Spark 2.0 feature. But did not get proper use after googling.

Whenever we use sql, can we use this function? if so, is there any proper use case to see this working?

Answer 1

Spark 2.0, . DataFrame . , UDF Python, .

Catalyst Spark SQL. , Catalyst Optimizer : (1) , (2) (3) (4)

-

• Spark SQLs Catalyst

• Apache Spark :

!