Pandas concat generates nan values

Question

I am curious why the simple concatenation of two data frames in pandas:

shape: (66441, 1) dtypes: prediction int64 dtype: object isnull().sum(): prediction 0 dtype: int64 shape: (66441, 1) CUSTOMER_ID int64 dtype: object isnull().sum() CUSTOMER_ID 0 dtype: int64 

the same form and both without NaN values

 foo = pd.concat([initId, ypred], join='outer', axis=1) print(foo.shape) print(foo.isnull().sum()) 

can lead to many NaN values ​​if combined.

 (83384, 2) CUSTOMER_ID 16943 prediction 16943 

How can I fix this problem and prevent the input of NaN values?

Trying to play it as

 aaa = pd.DataFrame([0,1,0,1,0,0], columns=['prediction']) print(aaa) bbb = pd.DataFrame([0,0,1,0,1,1], columns=['groundTruth']) print(bbb) pd.concat([aaa, bbb], axis=1) 

failed for example. It worked just fine, since no NaN values ​​were entered.

Answer 1

I think there is a problem with different index values, so when concat cannot align get NaN :

 aaa = pd.DataFrame([0,1,0,1,0,0], columns=['prediction'], index=[4,5,8,7,10,12]) print(aaa) prediction 4 0 5 1 8 0 7 1 10 0 12 0 bbb = pd.DataFrame([0,0,1,0,1,1], columns=['groundTruth']) print(bbb) groundTruth 0 0 1 0 2 1 3 0 4 1 5 1 print (pd.concat([aaa, bbb], axis=1)) prediction groundTruth 0 NaN 0.0 1 NaN 0.0 2 NaN 1.0 3 NaN 0.0 4 0.0 1.0 5 1.0 1.0 7 1.0 NaN 8 0.0 NaN 10 0.0 NaN 12 0.0 NaN 

reset_index solution if index values ​​are not needed:

 aaa.reset_index(drop=True, inplace=True) bbb.reset_index(drop=True, inplace=True) print(aaa) prediction 0 0 1 1 2 0 3 1 4 0 5 0 print(bbb) groundTruth 0 0 1 0 2 1 3 0 4 1 5 1 print (pd.concat([aaa, bbb], axis=1)) prediction groundTruth 0 0 0 1 1 0 2 0 1 3 1 0 4 0 1 5 0 1