Predictable order for columns in assign through sorting

[mrocklin]

The order of columns coming out of .assign is hard to predict

In [1]: import pandas as pd

In [2]: df = pd.DataFrame({'a': [1], 'b': [2]})

In [3]: df
Out[3]: 
   a  b
0  1  2

In [4]: df.assign(d=df.a*10, a=df.b*10, e=df.a*20)
Out[4]: 
    a  b   e   d
0  20  2  20  10

Would it be reasonable to keep all existing keys in their current location and then add the remaining keys in sorted order at the end?

[jreback]

this does seem reasonable

ther args are passed as kwargs where ordering is not defined
but existing columns are known and new keys could be sorted

[shoyer]

Agreed, this seems like a sane change.

[jreback]

@TomAugspurger want this issue?

[TomAugspurger]

I thought about doing this initially but then backed off when I saw PEP 468: https://www.python.org/dev/peps/pep-0468/ which is for preserving the order of **kwargs. Plus the eventual future where pandas runs on pypy, whose dicts are ordered.
That said, I'm not sure we should make design decisions around those, which could be very far off in the future or not happen at all. Having a predictable order now is probably a good thing.

It'll be a few days before I can get to it probably.

[jreback]

its a nice idea to have the ordering for kwargs, but practically that is long off (and pypy is even longer IMHO / never be adopted).

[TomAugspurger]

FYI https://www.python.org/dev/peps/pep-0468/ was actually accepted for 3.6 at the last minute (thanks to dicts maintaining sort order now).

Is this worth revisiting this now? It'd only be for 3.6+, so users writing code would need to be aware of that. Seems like a cause for subtle bugs if you're writing code that should support earlier versions.

[shoyer]

Yes, we should revisit this, though I don't see it being a major pain point so there's no rush. Maybe for pandas 2.0? By the time pandas 2.0 is out we'll probably all be using Python 3.6 day to day anyways.