Align python arrays with missing data

Question

I have time series data, say:

# [ [time] [ data ] ]
a = [[0,1,2,3,4],['a','b','c','d','e']]
b = [[0,3,4]['f','g','h']]

and I would like to get a result with some filler value, now say “No”:

a_new = [[0,1,2,3,4],['a','b','c','d','e']]
b_new = [[0,1,2,3,4],['f',None,None,'g','h']]

Is there a built-in function in python / numpy for this (or something like this)? Basically, I would like to have all my vectors of equal size so that I can compute statistics (np.mean) and process the missing data accordingly.

Answer 1

smarx has an excellent answer, but pandas was made specifically for such things.

# your data
a = [[0,1,2,3,4],['a','b','c','d','e']]
b = [[0,3,4],['f','g','h']]

# make an empty DataFrame (can do this faster but I'm going slow so you see how it works)
df_a = pd.DataFrame()
df_a['time'] = a[0]
df_a['A'] = a[1]
df_a.set_index('time',inplace=True)

# same for b (a faster way this time)
df_b = pd.DataFrame({'B':b[1]}, index=b[0]) 

# now merge the two Series together (the NaNs are in the right place)
df = pd.merge(df_a, df_b, left_index=True, right_index=True, how='outer') 

In [28]: df
Out[28]: 
     A    B
0    a    f
1    b  NaN
2    c  NaN
3    d    g
4    e    h

Now the fun is just beginning. Inside the DataFrame you can

• calculate all summary statistics (e.g. df.mean())

• (, df.plot())

• / , (, df.groupby())

• (, df.fillna()),

• (, df.resample()) .

( , ), 10 pandas .

Answer 2

? ( , b , , , .)

>>> b = [[0,3,4], ['f','g','h']]
>>> b_new = [list(range(5)), [None] * 5]
>>> for index, value in zip(*b): b_new[1][index] = value
>>> b_new
[[0, 1, 2, 3, 4], ['f', None, None, 'g', 'h']]

Answer 3

NumPythonic -

def align_arrays(A):
    time, data = A

    time_new = np.arange(np.max(time)+1)

    data_new = np.full(time_new.size, None, dtype=object)
    data_new[np.in1d(time_new,time)] = data

    return time_new, data_new

-

In [113]: a = [[0,1,2,3,4],['a','b','c','d','e']]

In [114]: align_arrays(a)
Out[114]: (array([0, 1, 2, 3, 4]), array(['a', 'b', 'c', 'd', 'e'], dtype=object))

In [115]: b = [[0,3,4],['f','g','h']]

In [116]: align_arrays(b)
Out[116]: (array([0, 1, 2, 3, 4]),array(['f', None, None, 'g', 'h'],dtype=object))