Spark 2.0, DataFrame, row column filter, unequal operator (! ==) is deprecated

I am trying to filter a DataFrame by storing only those rows for which a particular column of the row is not empty.

The operation is as follows:

df.filter($"stringColumn" !== "")

My compiler shows that! == deprecated since I upgraded to Spark 2.0.1

How to check if row column value is empty in Spark> 2.0?

+4
source share
1 answer

Use =!=as a replacement:

df.filter($"stringColumn" =!= "")
+10
source

Source: https://habr.com/ru/post/1658319/


All Articles