I am doing data preparation on a Spark DataFrame with categorical data. I need to do one-hot coding by categorical data, and I tried it on spark 1.6
sqlContext = SQLContext(sc)
df = sqlContext.createDataFrame([
(0, "a"),
(1, "b"),
(2, "c"),
(3, "a"),
(4, "a"),
(5, "c")
], ["id", "category"])
stringIndexer = StringIndexer(inputCol="category", outputCol="categoryIndex")
model = stringIndexer.fit(df)
indexed = model.transform(df)
encoder = OneHotEncoder(dropLast=False, inputCol="categoryIndex", outputCol="categoryVec")
encoded = encoder.transform(indexed)
encoded.select("id", "categoryVec").show()
This piece of code resulted in hot encoded data in this format.
+---+-------------+
| id| categoryVec|
+---+-------------+
| 0|(3,[0],[1.0])|
| 1|(3,[2],[1.0])|
| 2|(3,[1],[1.0])|
| 3|(3,[0],[1.0])|
| 4|(3,[0],[1.0])|
| 5|(3,[1],[1.0])|
+---+-------------+
Typically, what I expect from the One-Hot Encoding method is each column for each category and 0.1 corresponding values. How can I get this kind of data?