Pyspark "IS IN IN IN" Data Operator

Question

I would like to rewrite this from R to Pyspark, any nice suggestions?

array <- c(1,2,3) dataset <- filter(!(column %in% array)) 

Answer 1

In pyspark you can do it like this:

 array = [1, 2, 3] dataframe.filter(dataframe.column.isin(*array) == False) 

Answer 2

 df_result = df[df.column_name.isin([1, 2, 3]) == False] 

Answer 3

Take the ~ operator, which means the opposite:

 df_filtered = df.filter(~df["column_name"].isin([1, 2, 3])) 

Answer 4

slightly different syntax and dataset:

 toGetDates={'2017-11-09', '2017-11-11', '2017-11-12'} df= df.filter(df['DATE'].isin(toGetDates) == False) 

Answer 5

You can also loop the array and filter:

 array = [1, 2, 3] for i in array: df = df.filter(df["column"] != i)