How can I insert at a specific place in a DataFrame MultiIndex?

Question

Suppose I have a pandas DataFrame that looks like the following in the structure. However, in practice it can be much larger, and the number of level 1 indexes, as well as the number of level 2 indexes (for a level 1 index) will differ, so the decision should not make any assumptions about this:

index = pandas.MultiIndex.from_tuples([
    ("a", "s"),
    ("a", "u"),
    ("a", "v"),
    ("b", "s"),
    ("b", "u")])

result = pandas.DataFrame([
    [1, 2],
    [3, 4],
    [5, 6],
    [7, 8],
    [9, 10]], index=index, columns=["x", "y"])

What looks like this:

      x   y
a s   1   2
  u   3   4
  v   5   6
b s   7   8
  u   9  10

Now let's say that I want to create a “common” line for each of the levels “a” and “b”. Therefore, given the above as input, I would like my code to create something like this:

      x   y
a s   1   2
  u   3   4
  v   5   6
  t   9  12
b s   7   8
  u   9  10
b t  16  18

Here is the code that I still have:

# Calculate totals
for level, _ in result.groupby(level=0):

    # work out the global total for that desk:
    x_sum = result.loc[level]["x"].sum()
    y_sum = result.loc[level]["y"].sum()

    result = result.append(pandas.DataFrame([[x_sum, y_sum]], columns=result.columns, index=pandas.MultiIndex.from_tuples([(level, "t")])))

But this leads to the addition of “summary” columns to the end:

      x   y
a s   1   2
  u   3   4
  v   5   6
b s   7   8
  u   9  10
a t   9  12
b t  16  18

Sorting using result.sort_index()does not do what I want:

      x   y
a s   1   2
  t   9  12
  u   3   4
  v   5   6
b s   7   8
  t  16  18
  u   9  10

What am I doing wrong?

Answer 1

, sorted Multiindex - . , MultiIndex UnsortedIndexError, MultiIndex.

, reindex.

df = result.groupby(level=0).sum()
df.index = [df.index, ['t'] * len(df.index)]
df1 = pd.concat([result, df]).sort_index().reindex(['s','u','t'], level=1)

df1 = pd.concat([result, df]).sort_index()
print (df1)
      x   y
a s   1   2
  t   4   6
  u   3   4
b s   5   6
  t  12  14
  u   7   8

df1 = df1.reindex(['s','u','t'], level=1)
print (df1)
      x   y
a s   1   2
  u   3   4
  t   4   6
b s   5   6
  u   7   8
  t  12  14

:

print (result.index.get_level_values(1).unique().tolist())
['s', 'u']

df1 = df1.reindex(result.index.get_level_values(1).unique().tolist() + ['t'], level=1)
print (df1)
      x   y
a s   1   2
  u   3   4
  t   4   6
b s   5   6
  u   7   8
  t  12  14

---

GroupBy.apply:

def f(x):
    x.loc[(x.name, 't'),:] = x.sum()
    return x   

df = result.groupby(level=0, group_keys=False).apply(f)
print (df)
        x     y
a s   1.0   2.0
  u   3.0   4.0
  t   4.0   6.0
b s   5.0   6.0
  u   7.0   8.0
  t  12.0  14.0

Answer 2

result.reindex(pandas.MultiIndex.from_tuples([
("a", "s"),
("a", "u"),
("a", "t"),
("b", "s"),
("b", "u"),
("b", "t")
]))

        x   y
a   s   1   2
    u   3   4
    t   4   6
b   s   5   6
    u   7   8
    t   12  14