Python pandas / numpy quick way to replace all values ​​according to mapping scheme

Question

let's say I have a huge panda / numpy dataset, where each element is a list of ordered values:

sequences = np.array([12431253, 123412531, 12341234,12431253, 145345],
                     [5463456, 1244562, 23452],
                     [243524, 141234,12431253, 456367], 
                     [456345, 253451], 
                     [75635, 14145, 12346,12431253])

or,

sequences = pd.DataFrame({'sequence': [[12431253, 123412531, 12341234,12431253, 145345],
                                      [5463456, 1244562, 23452],
                                      [243524, 141234, 456367,12431253],
                                      [456345, 253451],
                                      [75635, 14145, 12346,12431253]]})

and I want to replace them with a different set of identifiers that start with 0, so I create this mapping:

from compiler.ast import flatten
from sets import Set
mapping = pd.DataFrame({'v0': list(Set(flatten(sequences['sequence']))), 'v1': range(len(Set(flatten(sequences['sequence'])))})

......

so the result i was looking for:

sequences = np.array([1, 2, 3,1, 4], [5, 6, 7], [8, 9, 10,1], [11, 12], [13, 14, 15,1])

How can I scale this to a huge data frame / number of sequences?

Thanks so much for any guidance! Very grateful!

Answer 1

, 1D, np.unique , -

lens = np.array(map(len,sequences))
seq_arr = np.concatenate(sequences)
ids = np.unique(seq_arr,return_inverse=1)[1]
out = np.split(ids,lens[:-1].cumsum())

-

In [391]: sequences = np.array([[12431253, 123412531, 12341234,12431253, 145345],
     ...:                      [5463456, 1244562, 23452],
     ...:                      [243524, 141234,12431253, 456367], 
     ...:                      [456345, 12431253], 
     ...:                      [75635, 14145, 12346,12431253]])

In [392]: out
Out[392]: 
[array([12, 13, 11, 12,  5]),
 array([10,  9,  2]),
 array([ 6,  4, 12,  8]),
 array([ 7, 12]),
 array([ 3,  1,  0, 12])]

In [393]: np.array(map(list,out)) # If you need NumPy array as final o/p
Out[393]: 
array([[12, 13, 11, 12, 5], [10, 9, 2], [6, 4, 12, 8], [7, 12],
       [3, 1, 0, 12]], dtype=object)

Answer 2

1

stop = sequences.sequence.apply(np.size).cumsum()
start = end.shift().fillna(0).astype(int)
params = pd.concat([start, stop], axis=1, keys=['start', 'stop'])
params.apply(lambda x: list(np.arange(**x)), axis=1)

0    [0, 1, 2, 3]
1       [4, 5, 6]
2       [7, 8, 9]
3        [10, 11]
4    [12, 13, 14]
dtype: object