Pyspark Dataframe Single Line Encoding

I am doing data preparation on a Spark DataFrame with categorical data. I need to do one-hot coding by categorical data, and I tried it on spark 1.6

sqlContext = SQLContext(sc)
df = sqlContext.createDataFrame([
    (0, "a"),
    (1, "b"),
    (2, "c"),
    (3, "a"),
    (4, "a"),
    (5, "c")
], ["id", "category"])

stringIndexer = StringIndexer(inputCol="category", outputCol="categoryIndex")
model = stringIndexer.fit(df)
indexed = model.transform(df)
encoder = OneHotEncoder(dropLast=False, inputCol="categoryIndex", outputCol="categoryVec")
encoded = encoder.transform(indexed)
encoded.select("id", "categoryVec").show()

This piece of code resulted in hot encoded data in this format.

+---+-------------+
| id|  categoryVec|
+---+-------------+
|  0|(3,[0],[1.0])|
|  1|(3,[2],[1.0])|
|  2|(3,[1],[1.0])|
|  3|(3,[0],[1.0])|
|  4|(3,[0],[1.0])|
|  5|(3,[1],[1.0])|
+---+-------------+

Typically, what I expect from the One-Hot Encoding method is each column for each category and 0.1 corresponding values. How can I get this kind of data?

+4
source share

Source: https://habr.com/ru/post/1680711/


All Articles