arrange() not working

[deepanshu88]

I am trying to calculate the summary statistics by grouping variable and then sorting the result in descending order.

#Import Data
import pandas as pd
mydata=pd.read_csv("http://winterolympicsmedals.com/medals.csv")

#2006 Gold Medal Count 
mydata >> mask(X.Year==2006 , X.Medal =='Gold') >> group_by(X.NOC) >> summarize(N=n(X.NOC)) >> arrange(X.N, ascending=False)

Gold Medal Count (i.e. variable N) is not sorted in descending order

[sharpe5]

Try adding >> ungroup() before the arrange.

I regularly use group_by(), ungroup() followed by arrange(), and it works perfectly every time.

In addition, please ensure that the dataframe is sorted by the same key as in the "group_by" before using "group_by". This is a key difference between SQL and Pandas - grouping requires pre-sorting first or it may not work.

[ghost]

@sharpe5's answer is correct, but this behavior unexpectedly diverges from dplyr's.

As far as I can tell, after the summarize, in dplyr the groups have been eliminated, while in dfply they remain attached to the dataframe.

Is this the intended behavior? If so, should the difference be documented?

Thanks!

In R with dplyr:

r$> library(tidyverse)   

r$> diamonds %>%  
        group_by(cut) %>%  
        summarize(count_color=n()) %>% 
        arrange(-count_color)                                                                                                                                                                                        
# A tibble: 5 x 2
  cut       count_color
  <ord>           <int>
1 Ideal           21551
2 Premium         13791
3 Very Good       12082
4 Good             4906
5 Fair             1610


r$> diamonds %>%  
        group_by(cut) %>%  
        summarize(count_color=n()) %>% 
        ungroup() %>% 
        arrange(-count_color)                                                                                                                                                                                        
# A tibble: 5 x 2
  cut       count_color
  <ord>           <int>
1 Ideal           21551
2 Premium         13791
3 Very Good       12082
4 Good             4906
5 Fair             1610

In Python with dfply:

In [8]: (dfply.diamonds >>  
   ...:     group_by('cut') >>  
   ...:     summarize(count_color=dfply.n(X.color)) >>  
   ...:     arrange(X.count_color, ascending=False))                                                                                                                                                                 
Out[8]: 
         cut  count_color
0       Fair         1610
1       Good         4906
2      Ideal        21551
3    Premium        13791
4  Very Good        12082


In [7]: (dfply.diamonds >>  
   ...:     group_by('cut') >>  
   ...:     summarize(count_color=dfply.n(X.color)) >>  
   ...:     ungroup() >>  
   ...:     arrange(X.count_color, ascending=False))                                                                                                                                                                 
Out[7]: 
         cut  count_color
2      Ideal        21551
3    Premium        13791
4  Very Good        12082
1       Good         4906
0       Fair         1610

[kieferk]

Yes this is the intended behavior. I am aware that it diverges from dplyr. I understand the rationale in dplyr that summarize would eliminate the groupings, since it is "collapsing" the groups to single rows and therefore the groupings become "meaningless".

I am not opposed to changing it to match the dplyr behavior if that is what the people want. My rationale for not doing this despite the collapsing into rows is that the ungrouping becomes implicit rather than explicit. My reasoning was that the grouping should be preserved until you explicitly state that groupings should be collapsed into a single dataframe.

Not sure which direction to go. If you think that the dplyr way is superior I am happy to change it to that.

[sharpe5]

I can confirm that I got stuck, and it was only after a lot of false starts and experimentation that I worked out that ungroup() was required.

My vote is to have dplyr behave similarly to dfply, so code can be directly ported from R to python and back again without the need to add extra clauses.

However - is there any technical reason why this change might be a disadvantage? Would this prevent some problems from being solved?

On Fri, 18 Jan 2019 at 18:49, Kiefer Katovich [email protected] wrote:

Yes this is the intended behavior. I am aware that it diverges from dplyr. I understand the rationale in dplyr that summarize would eliminate the groupings, since it is "collapsing" the groups to single rows and therefore the groupings become "meaningless".

I am not opposed to changing it to match the dplyr behavior if that is what the people want. My rationale for not doing this despite the collapsing into rows is that the ungrouping becomes implicit rather than explicit. My reasoning was that the grouping should be preserved until you explicitly state that groupings should be collapsed into a single dataframe.

Not sure which direction to go. If you think that the dplyr way is superior I am happy to change it to that.

— You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub https://github.com/kieferk/dfply/issues/56#issuecomment-455648676, or mute the thread https://github.com/notifications/unsubscribe-auth/ABOypE2T_Yy-RYUH6T-sH1euuZMToe-fks5vEhcQgaJpZM4UO4Jw .

[ghost]

I'd prefer that summarize() retain the ungrouping behavior of dplyr, and that if a group-retaining version is desired, that it have a different name. Since I regularly use both dplyr and dfply, it makes me nervous to have the same function name with substantially different behavior. I am worried about confusing them and getting wrong results without noticing.

[jtrakk]

I'm not wedded to any particular names, but I do think it's important that different functions have different names.

How would you feel about:

• adding a function with a new name, digest(), that does ungrouping aggregation,
• adding a function with a new name, review(), that does aggregation retaining groups, and
• removing the summarize() function to eliminate that source of confusion with dplyr.

This way there's no confusion and I can just pick the one I want without fear, and I don't have to worry about which version of dfply I'm on.