ENH: Feature Request for Ungroup Method for Grouped Data Frames

[mdancho84]

Hi, thanks for your work developing pandas. I'd like to request a feature to add an ungroup() method for grouped data frames. It's related to this StackOverflow where I've developed a hack to extract using the .obj to pull out the original data frame from the grouped data frame.

However, it would be helpful to have a method developed, which would do the extraction to prevent users from depending on my hack.

>>> dat_1 = df.groupby("category_2")
>>> dat_1
<pandas.core.groupby.generic.DataFrameGroupBy object at 0x7fce78b3dd00>
>>> dat_1.obj
    order_date          category_2     value
1   2011-02-01  Cross Country Race  324400.0
2   2011-03-01  Cross Country Race  142000.0
3   2011-04-01  Cross Country Race  498580.0
4   2011-05-01  Cross Country Race  220310.0
5   2011-06-01  Cross Country Race  364420.0
..         ...                 ...       ...
535 2015-08-01          Triathalon   39200.0
536 2015-09-01          Triathalon   75600.0
537 2015-10-01          Triathalon   58600.0
538 2015-11-01          Triathalon   70050.0
539 2015-12-01          Triathalon   38600.0

[531 rows x 3 columns]

[jreback]

-1 as pd.concat([grp for g, grp in df.groupby...()]) is idiomatic. this is not worth a method.

[hnagaty]

That would be a nice feature and may come handy at some times.

[mdancho84]

It's definitely a handy tool to implement that would help beginners seeking to extract groups. It also parallels R's tidyverse which has ungroup() in dplyr, so it might make it easier for R users to transition to pandas.

[s-pike]

For me, this feature could be useful.

For reference, pd.concat([grp for g, grp in df.groupby...()]) doesn't seem to have quite the same output as df.groupby().obj. The former sorts the dataframe into groups, whereas the latter maintains the original order. The .obj hack is also an order of magnitude faster in my tests (even if you df.sort_values(group), although again, the results aren't identical).

It would really come into its own if the DataFrameGroupBy object had its own assign method, equivalent to dplyr's group_by %>% mutate functionality (see this stackoverflow question). If you like method chaining, the best current approach is using x.groupby('group').transform('fn')['value'], but that's potentially awkward if you want to use the group for multiple assignments, e.g.:

(df.assign(normalised_value = lambda x: x['value'] / x.groupby('group').transform('sum')['value'],
           normalising_value = lambda x: x.groupby('group').transform('sum')['value'])
  .more_methods...()
)

It'd be nice to have something like:

(df.groupby('group')
  .assign(normalised_value = lambda x: x['value']/x['value'].sum(),
          normalising_value = lambda x: x['value'].sum())
  .ungroup()
  .more_methods...()
)

The R dplyr equivalent being:

df %>%
  group_by(group) %>%
  mutate(normalised_value = value / sum(value)
         normalising_value = sum(value)) %>%
  ungroup() %>%
  more_methods...()

[pwwang]

Looks like some of you are leaning toward R/dplyr styles.

Check out datar, which reimages pandas APIs to align with R/dplyr's.

An example with @s-pike 's R code:

>>> from datar.all import f, tibble, group_by, mutate, ungroup, row_number, sum
[2022-03-17 11:25:33][datar][WARNING] Builtin name "sum" has been overriden by datar.
>>> df = tibble(group=[1,1,2,2], value=[1,2,3,4])
>>> (
...     df
...     >> group_by(f.group)
...     >> mutate(normalised_value=f.value/sum(f.value), normalising_value=sum(f.value))
...     >> ungroup()
...     >> mutate(n=row_number())
... )
    group   value  normalised_value  normalising_value         n
  <int64> <int64>         <float64>            <int64> <float64>
0       1       1          0.333333                  3       1.0
1       1       2          0.666667                  3       2.0
2       2       3          0.428571                  7       3.0
3       2       4          0.571429                  7       4.0

[M-Harrington]

ungroup as a simple wrapper seems like a no brainer. Especially for people new to python that came from R. But in general why would you write pd.concat([grp for g, grp in df.groupby...()]) if there could exist a method as simple as df.ungroup() even if ungroup just calls the prior function. Seems like a simple change to clear up the multiple ways that "ungrouping" can be done and decrease the fatigue of choice.

[jreback]

if u think this is a useful then show a complete example

the above is not very compelling

[M-Harrington]

I'm not sure I understand what you're looking for @jreback. Especially if you're referring to @s-pike 's example. An example of why it might be useful to have a wrapper for ungrouping a dataframe? If you need to recover the original order as is common with unlabeled numpy data for machine learning, having an ordered df by group makes matching the two datasets difficult.

This is a task that happens to me frequently.

[jreback]

a compelling example in code not words

[M-Harrington]

Can you answer my question by any chance? That'll make it easier for me to know what you're looking for.