How to check isEmpty on Column Data Spark scala

Question

My data is as follows:

[null,223433,WrappedArray(),null,460036382,0,home,home,home]

How to check if col3 is empty in a query in spark sql? I tried to explode, but when I do this, empty array lines disappear. Can anyone suggest me a way to do this.

I tried:

val homeSet = result.withColumn("subscriptionProvider", explode($"subscriptionProvider"))

where subscriptionProvider(WrappedArray())is a column with an array of values, but some arrays may be empty. I need to get a Provider subscriber with null values, and in the subscriptionProvider array - "Comcast"

Answer 1

Try:

import org.apache.spark.sql.functions._

val tmp = df.withColumn("subscriptionProvider", 
  when(size($"subscriptionProvider") !== 0, $"subscriptionProvider").otherwise(array(lit(null).cast("string"))))

tmp.withColumn("subscriptionProvider", explode($"subscriptionProvider"))

Answer 2

LostInOverflow dataframe. size. , :

val dfSchema = df.schema
val filtered = df.rdd.filter(!_.getList[String](2).isEmpty)
sqlContext.createDataFrame(filtered, dfSchema)