I downloaded the excel file from S3 using the syntax below, but I am wondering what parameters need to be set here.
Why is it necessary to install all of the options below to download an excel file? None of these options are required to download other types of files, such as csv, del, json, avro, etc.
val data = sqlContext.read.
format("com.crealytics.spark.excel").
option("location", s3path).
option("useHeader", "true").
option("treatEmptyValuesAsNulls", "true").
option("inferSchema","true").
option("addColorColumns", "true").
load(path)
I get the following error if any of the above parameters (except the location) is not set:
sqlContext.read.format("com.crealytics.spark.excel").option("location", s3path).load(s3path)
Error message:
Name: java.lang.IllegalArgumentException
Message: Parameter "useHeader" is missing in options.
StackTrace: at com.crealytics.spark.excel.DefaultSource.checkParameter(DefaultSource.scala:37)
at com.crealytics.spark.excel.DefaultSource.createRelation(DefaultSource.scala:19)
at com.crealytics.spark.excel.DefaultSource.createRelation(DefaultSource.scala:7)
at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:345)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:149)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:132)
at $anonfun$1.apply(<console>:47)
at $anonfun$1.apply(<console>:47)
at time(<console>:36)
source
share