How to assemble a DataFrame column into key value pairs as a string in python

Question

I am trying to assemble a pandas DataFrame column into key value pairs and list it as a string in python. If we take the following DataFrame example, I want to go from here:

import pandas as pd
from collections import OrderedDict

df = pd.DataFrame({'value_2016': [200],
                   'value_2017': [300],
                   'value_2018': [float('NaN')]})
print(df)

     value_2016  value_2017  value_2018
0         200         300         NaN

at

df_result = pd.DataFrame(OrderedDict({'year': [2016, 2017],
                                      'value': [200, 300]}))

print(df_result)

   year  value
0  2016    200
1  2017    300

If you are familiar with R, the equivalent would be something like this:

require("plyr"); require("dplyr"); require(tidyr)

df <- data.frame(value_2016 = 200,
                 value_2017 = 300,
                 value_2018 = NA)

df %>% 
   gather(year, value, value_2016:value_2018) %>% 
   mutate(year = gsub(x = .$year, replacement = "", "value_")) %>% 
   na.exclude

     year value
   1 2016   200
   2 2017   300

Any help would be very cool!

Answer 1

You can create MultiIndex splitand then change stack:

df.columns = df.columns.str.split('_', expand=True)
df = df.stack().reset_index(level=0, drop=True).rename_axis('year').reset_index()
#if necessary convert float to int
df.value = df.value.astype(int)
print (df)
   year  value
0  2016    200
1  2017    300

If you want to use the constructor DataFrame, use get_level_values:

df.columns = df.columns.str.split('_', expand=True)
df = df.stack()

df_result = pd.DataFrame(OrderedDict({'year': df.index.get_level_values(1),
                                      'value': df['value'].astype(int).values}))

print(df_result)
   year  value
0  2016    200
1  2017    300

Answer 2

You can use rename, stackandreset_index

In [4912]: (df.rename(columns=lambda x: x.split('_')[-1]).stack()
              .reset_index(level=0, drop=True)
              .rename_axis('year')
              .reset_index(name='value'))
Out[4912]:
   year  value
0  2016  200.0
1  2017  300.0