Pandas dataframe with an extra “layer”,

Question

Suppose you have the following data framework:

import pandas as pd import numpy as np df = pd.DataFrame(np.nan,columns=['A','B','C'],index=[0,1,2]) 

Suppose I need an extra “layer” on top of this pandas frame, so column A, row 0 will have its value, column B, row 0 will have a different value, column C of rows 0 will have something, column Row 1 and etc. So, like a dataframe on top of this existing one.

Can I add other layers? How to access these layers? Is this efficient, that is, should I just use a separate data frame? And could it be possible to save these several layers as csv, referring to separate layers ?, or is there a function that would split them into different worksheets in one book?

Answer 1

pandas.DataFrame cannot have 3 dimensions :

DataFrame is a 2-dimensional labeled data structure with potentially different types of columns.

However, there is a way to fake 3-dimensional dimensions using MultiIndex / Advanced Indexing :

Hierarchical Indexing (MultiIndex)

Hierarchical / multi-level indexing is very interesting because it opens the door to some rather complex data analysis and manipulation, especially for working with higher dimensional data. Essentially, it allows you to store and manipulate data with an arbitrary number of dimensions in data of a lower structure size, such as Series (1d) and DataFrame (2d).

If you really need this extra size, go to pandas.Panel :

Panel is a slightly less used, but still important container for 3D data.

but do not miss this important disclaimer:

Note: Unfortunately, Panel , which is less commonly used than Series and DataFrame , was slightly ignored in functionality. The number of methods and parameters available in the DataFrame are not available in the Panel .

There is also pandas.Panel4D (experimental) in the unlikely pandas.Panel4D that you need it.