Spark 2.0 Scala - RDD.toDF ()

Question

I work with Spark 2.0 Scala. I can convert RDD to DataFrame using toDF () method.

val rdd = sc.textFile("/pathtologfile/logfile.txt")
val df = rdd.toDF()

But for life, I cannot find where this is in the API docs. This is not under RDD. But it is under the DataSet ( link 1 ). However, I have an RDD, not a DataSet.

Also, I cannot see this under implications ( ref. 2 ).

So please help me understand why toDF () can be called for my RDD. Where does this method inherit from?

Answer 1

This comes from here:

API Spark 2

: sqlContext.implicits._, RDD DataSetHolder (rddToDataSetHolder), toDF DataSetHolder

Answer 2

, , , . , , - . case, List, , , , DF. , , Spark 2.1.1 toDF, . , , , .read. 5 .

, .

import spark.sqlContext.implicits._
val df = <mutable list object>.toDF 
df.show()

Answer 3

, sqlContext implicits :

val sqlContext = //create sqlContext

import sqlContext.implicits._

val df = RDD.toDF()

"toDF" RDD

.