Point Spark Column Name

Question

I am trying to take columns from a DataFrame and convert it to RDD[Vector] .

The problem is that I have columns with a dot in their name as the following dataset:

 "col0.1","col1.2","col2.3","col3.4" 1,2,3,4 10,12,15,3 1,12,10,5 

This is what I do:

 val df = spark.read.format("csv").options(Map("header" -> "true", "inferSchema" -> "true")).load("C:/Users/mhattabi/Desktop/donnee/test.txt") val column=df.columns.map(c=>s"`${c}`") val rows = new VectorAssembler().setInputCols(column).setOutputCol("vs") .transform(df) .select("vs") .rdd val data =rows.map(_.getAs[org.apache.spark.ml.linalg.Vector](0)) .map(org.apache.spark.mllib.linalg.Vectors.fromML) val mat: RowMatrix = new RowMatrix(data) //// Compute the top 5 singular values and corresponding singular vectors. val svd: SingularValueDecomposition[RowMatrix, Matrix] = mat.computeSVD(mat.numCols().toInt, computeU = true) val U: RowMatrix = svd.U // The U factor is a RowMatrix. val s: Vector = svd.s // The singular values are stored in a local dense vector. val V: Matrix = svd.V // The V factor is a local dense matrix. println(V) 

Please help me consider the columns with a dot in their names. thanks

Answer 1

The problem here is the implementation of VectorAssembler , not the columns as such. You can, for example, skip the header:

 val df = spark.read.format("csv") .options(Map("inferSchema" -> "true", "comment" -> "\"")) .load(path) new VectorAssembler() .setInputCols(df.columns) .setOutputCol("vs") .transform(df) 

or rename columns before moving on to VectorAssembler :

 val renamed = df.toDF(df.columns.map(_.replace(".", "_")): _*) new VectorAssembler() .setInputCols(renamed.columns) .setOutputCol("vs") .transform(renamed) 

Finally, a better approach is to explicitly provide a schema:

 import org.apache.spark.sql.types._ val schema = StructType((0 until 4).map(i => StructField(s"_$i", DoubleType))) val dfExplicit = spark.read.format("csv") .options(Map("header" -> "true")) .schema(schema) .load(path) new VectorAssembler() .setInputCols(dfExplicit.columns) .setOutputCol("vs") .transform(dfExplicit)