Why can't IDEA find toDS () and toDF () functions?

Question

My code works well in the spark shell:

scala> case class Person(name:String,age:Int)
defined class Person

scala> val person = Seq(Person("ppopo",23)).toDS()
person: org.apache.spark.sql.Dataset[Person] = [name: string, age: int]

scala> person.show()
+-----+---+
| name|age|
+-----+---+
|ppopo| 23|
+-----+---+

but wrong in IDEA:

I imported all the jars into "spark-2.0.0-bin-hadoop2.7 / jars /" but still cannot find this function.

Answer 1

I find the problem, add the dependency before the toDS () user:

val ss = SparkSession.builder().appName("DataSet Test")
  .master("local[*]").getOrCreate()

// This import is needed
import ss.implicits._
val simpleDS = Seq(Person("po",12)).toDS()
simpleDS.show()

Answer 2

if you are working with spark version 1.6, then use this code to convert rdd to df

from pyspark.sql import SQLContext, Row
sqlContext = SQLContext(sc)
df = sqlContext.createDataFrame(rdd)

if you want to assign a title to the lines use this

df= rdd.map(lambda p: Row(ip=p[0], time=p[1], zone=p[2]))

ip, time, zone are the line headers in this example.