DataFrameGroupBy diff () provided

Question

Suppose I have a DataFrame:

df = pd.DataFrame({'CATEGORY':['a','b','c','b','b','a','b'],
                   'VALUE':[pd.np.NaN,1,0,0,5,0,4]})

which looks like

    CATEGORY    VALUE
0      a         NaN
1      b         1
2      c         0
3      b         0
4      b         5
5      a         0
6      b         4

I am grouping it:

df = df.groupby(by='CATEGORY')

And now, let me show what I want with an example in one group "b":

df.get_group('b')

group b:

    CATEGORY    VALUE
1      b          1
3      b          0
4      b          5
6      b          4

I need to . Within each group, count diff () between values VALUE, skipping all NaNand 0s. Thus, the result should be:

    CATEGORY    VALUE  DIFF
1      b          1      - 
3      b          0      -
4      b          5      4
6      b          4     -1

Answer 1

You can use diffto subtract values ​​after deleting the values 0and NaN:

df = pd.DataFrame({'CATEGORY':['a','b','c','b','b','a','b'],
               'VALUE':[pd.np.NaN,1,0,0,5,0,4]})

grouped = df.groupby("CATEGORY")

# define diff func
diff = lambda x: x["VALUE"].replace(0, np.NaN).dropna().diff()
df["DIFF"] = grouped.apply(diff).reset_index(0, drop=True)

print(df)

  CATEGORY  VALUE  DIFF
0        a    NaN   NaN
1        b    1.0   NaN
2        c    0.0   NaN
3        b    0.0   NaN
4        b    5.0   4.0
5        a    0.0   NaN
6        b    4.0  -1.0

Answer 2

Sounds like a job for an operation pd.Series.shift()with a mask notnull.

,

nonull_df = df[(df['VALUE'] != 0) & df['VALUE'].notnull()]
groups = nonull_df.groupby(by='CATEGORY')

diff

nonull_df['next_value'] = groups['VALUE'].shift(1)
nonull_df['diff'] = nonull_df['VALUE'] - nonull_df['next_value']

,

df.loc[nonull_df.index] = nonull_df

df
  CATEGORY  VALUE  next_value  diff
0        a    NaN         NaN   NaN
1        b    1.0         NaN   NaN
2        c    0.0         NaN   NaN
3        b    0.0         1.0  -1.0
4        b    5.0         1.0   4.0
5        a    0.0         NaN   NaN
6        b    4.0         5.0  -1.0