I just tried using the Apache Spark ml library for logistic regression, but whenever I tried it, an error message appeared, for example
"ERROR OWLQN: Failed! Reset History: breeze.optimize.NaNHistory:"
An example dataset for logistic regression is as follows:
+-----+---------+---------+---------+--------+-------------+
|state|dayOfWeek|hourOfDay|minOfHour|secOfMin| features|
+-----+---------+---------+---------+--------+-------------+
| 1.0| 7.0| 0.0| 0.0| 0.0|(4,[0],[7.0])|
And for logistic regression there is a code:
StructType schema = new StructType(
new StructField[]{
new StructField("state", DataTypes.DoubleType, false, Metadata.empty()),
new StructField("dayOfWeek", DataTypes.DoubleType, false, Metadata.empty()),
new StructField("hourOfDay", DataTypes.DoubleType, false, Metadata.empty()),
new StructField("minOfHour", DataTypes.DoubleType, false, Metadata.empty()),
new StructField("secOfMin", DataTypes.DoubleType, false, Metadata.empty())
});
List<Row> dataFromRDD = bucketsForMLs.map(p -> {
return RowFactory.create(p.label(), p.features().apply(0), p.features().apply(1), p.features().apply(2), p.features().apply(3));
}).collect();
Dataset<Row> stateDF = sparkSession.createDataFrame(dataFromRDD, schema);
String[] featureCols = new String[]{"dayOfWeek", "hourOfDay", "minOfHour", "secOfMin"};
VectorAssembler vectorAssembler = new VectorAssembler().setInputCols(featureCols).setOutputCol("features");
Dataset<Row> stateDFWithFeatures = vectorAssembler.transform(stateDF);
StringIndexer labelIndexer = new StringIndexer().setInputCol("state").setOutputCol("label");
Dataset<Row> stateDFWithLabelAndFeatures = labelIndexer.fit(stateDFWithFeatures).transform(stateDFWithFeatures);
MLRExecutionForDF mlrExe = new MLRExecutionForDF(javaSparkContext);
mlrExe.execute(stateDFWithLabelAndFeatures);
LogisticRegressionModel lrModel = new LogisticRegression().setMaxIter(maxItr).setRegParam(regParam).setElasticNetParam(elasticNetParam)
.fit(stateDFWithLabelAndFeatures);