DOC: more documentation that pandas by-default aligns when setting

[IamGianluca]

Hi,

I'm not sure this is a bug but the behaviour is far from intuitive. I'm trying to assign the values of a slice of a DataFrame into another slice (of same size) of the same DataFrame.

In [1]: import pandas as pd
In [2]: import numpy as np
In [3]: a = pd.DataFrame({"a": [1, 2, 3, 4, 5], "b": [6, 7, 8, 9, 10]})
In [4]: a
Out[4]: 
   a   b
0  1   6
1  2   7
2  3   8
3  4   9
4  5  10
In [5]: a.iloc[0:2,:] = a.iloc[3:5, :]

In [6]: a
Out[6]: 
     a     b
0  NaN   NaN
1  NaN   NaN
2  3.0   8.0
3  4.0   9.0
4  5.0  10.0

Specifically, in the current implementation the data type is not preserved (int to float) and the value copied are np.NaN instead of the correct values.

The expected output can be obtained using this counter-intuitive command:

In [1]: import pandas as pd
In [2]: import numpy as np
In [3]: a = pd.DataFrame({"a": [1, 2, 3, 4, 5], "b": [6, 7, 8, 9, 10]})
In [4]: a.iloc[0:2,:] = (np.array(a.iloc[3:5, :])).astype(int)

In [5]: a
Out[5]: 
   a   b
0  4   9
1  5  10
2  3   8
3  4   9
4  5  10

This is the environment I'm working on:

INSTALLED VERSIONS
------------------
commit: None
python: 3.5.1.final.0
python-bits: 64
OS: Linux
OS-release: 4.4.0-31-generic
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8

pandas: 0.18.1
nose: 1.3.7
pip: 8.1.2
setuptools: 24.0.2
Cython: None
numpy: 1.11.1
scipy: 0.17.1
statsmodels: None
xarray: None
IPython: 5.0.0
sphinx: 1.4.5
patsy: 0.4.1
dateutil: 2.5.3
pytz: 2016.6.1
blosc: None
bottleneck: None
tables: None
numexpr: None
matplotlib: 1.5.1
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: None
html5lib: None
httplib2: 0.9.2
apiclient: None
sqlalchemy: 1.1.0b2
pymysql: None
psycopg2: None
jinja2: 2.8
boto: None
pandas_datareader: None

[TomAugspurger]

I think this comes down to pandas aligning by row labels before updating.

In [7]: values = pd.DataFrame({'a': [4, 5], 'b': [9, 10]}, index=[3, 4])

In [8]: values
Out[8]:
   a   b
3  4   9
4  5  10

In [9]: a.iloc[0:2] = values

In [10]: a
Out[10]:
     a     b
0  NaN   NaN
1  NaN   NaN
2  3.0   8.0
3  4.0   9.0
4  5.0  10.0

Here's the alignment

In [28]: l, r = a.iloc[0:2].align(values)

In [29]: l
Out[29]:
     a    b
0  1.0  6.0
1  2.0  7.0
3  NaN  NaN
4  NaN  NaN

In [30]: r
Out[30]:
      a      b
0   NaN    NaN
1   NaN    NaN
3  40.0   90.0

So you can see where the NaNs come in.

What you probably want is

In [33]: a = pd.DataFrame({"a": [1, 2, 3, 4, 5], "b": [6, 7, 8, 9, 10]})

In [34]: values = pd.DataFrame({'a': [4, 5], 'b': [9, 10]}, index=[3, 4])

In [35]: a.iloc[0:2] = np.asarray(values)  # drops the row & column labels

In [36]: a
Out[36]:
   a   b
0  4   9
1  5  10
2  3   8
3  4   9
4  5  10

I belive aligning is indented for .iloc, but I'll let Jeff or someone more knowledgeable chime in.

[jreback]

The is as-expected. pandas by definition will align on assignment. Doesn't matter how you select the values. If you don't want to align, then you can pass a same shaped non-pandas object.

In [9]: a.iloc[0:2,:] = a.iloc[3:5, :].values
Out[9]: 
   a   b
0  4   9
1  5  10
2  3   8
3  4   9
4  5  10

The docs were updated w.r.t. to a very similar issue here, and appears here

I suppose an additional doc note could be added, maybe as a separate sub-section, maybe after here. Or as a note.

So will repurpose this issue.

[raj-thapa]

take

[raj-thapa]

In the latest official documentation (see the warning note) here, it is mentioned that:

pandas aligns all AXES when setting Series and DataFrame from .loc, and .iloc.

As seen in example below, .iloc no longer aligns before assignment, while .loc does. I would like to update the documentation to accurately reflect this behavior of .loc and .iloc.

>>> a = pd.DataFrame({"a": [1, 2, 3, 4, 5], "b": [6, 7, 8, 9, 10]})
>>> a
   a   b
0  1   6
1  2   7
2  3   8
3  4   9
4  5  10
>>> a.iloc[0:2,:] = a.iloc[3:5,:]
>>> a
   a   b
0  4   9
1  5  10
2  3   8
3  4   9
4  5  10
>>> a.loc[0:1,:] = a.loc[3:5,:]
>>> a
     a     b
0  NaN   NaN
1  NaN   NaN
2  3.0   8.0
3  4.0   9.0
4  5.0  10.0