BUG: broadcasting subtraction

[slowkow]

This bug applies to version 0.11.0. I observe correct behavior in version 0.12.0.dev-84fb0e0, so the issue seems to have been fixed. Still, I thought someone might find it useful to see this report.

---

EDIT: Replaced a larger and more complicated example with a minimal example.

In [84]: df = pd.DataFrame({'A':['a1','a2','a3'],'B':[4,5,6]})

In [85]: df
Out[85]: 
    A  B
0  a1  4
1  a2  5
2  a3  6

In [86]: df.ix[[0,1], 'B'] -= 10

In [87]: df
Out[87]: 
    A  B
0  a1 -6
1  a2 -6
2  a3  6

[jreback]

thanks for the report, this appears as a dup of #3697, fixed in #3533 (and as you observed, fixed in master)

[slowkow]

Can you recommend how to get around this bug without upgrading to pandas 0.11.1 or 0.12?

[jreback]

try df.to_csv(.....engine='python') this invokes the same writer as existing < 0.11, which is slower, but correct

[slowkow]

Sorry, I want to subtract a value from each cell in the dataframe. I don't want to write anything to a file.

[jreback]

I don't think there is a problem in 0.11.0..

is this what you are trying to do?

In [26]: df = DataFrame(np.random.rand(5,2),columns=list('AB'),index=list('abcde'))

In [27]: df
Out[27]: 
          A         B
a  0.987604  0.267752
b  0.284552  0.853560
c  0.120987  0.600705
d  0.857089  0.905863
e  0.708717  0.489044

In [28]: df.loc[['b','c','d'],'A'] -= 5

In [29]: df
Out[29]: 
          A         B
a  0.987604  0.267752
b -4.715448  0.853560
c -4.879013  0.600705
d -4.142911  0.905863
e  0.708717  0.489044

In [30]: df = DataFrame(np.random.rand(5,2),columns=list('AB'),index=list('abcde'))

In [31]: df
Out[31]: 
          A         B
a  0.184312  0.759196
b  0.474819  0.428043
c  0.682918  0.848248
d  0.086138  0.508250
e  0.331544  0.568149

In [32]: df.iloc[[1,2,3],0] -= 5

In [33]: df
Out[33]: 
          A         B
a  0.184312  0.759196
b -4.525181  0.428043
c -4.317082  0.848248
d -4.913862  0.508250
e  0.331544  0.568149

In [34]: pd.__version__
Out[34]: '0.11.0'

In [35]: df.ix[[1,2,3],'A'] -= 5

In [36]: df
Out[36]: 
          A         B
a  0.184312  0.759196
b -9.525181  0.428043
c -9.317082  0.848248
d -9.913862  0.508250
e  0.331544  0.568149

[slowkow]

There is a problem, but it's more subtle than you imagine. That's what makes this bug so frustrating. My first post shows exactly how to reproduce the problem.

[slowkow]

Here's a much simpler example:

In [84]: df = pd.DataFrame({'A':['a1','a2','a3'],'B':[4,5,6]})

In [85]: df
Out[85]: 
    A  B
0  a1  4
1  a2  5
2  a3  6

In [86]: df.ix[[0,1], 'B'] -= 10

In [87]: df
Out[87]: 
    A  B
0  a1 -6
1  a2 -6
2  a3  6

In [88]: pd.__version__
Out[88]: '0.11.0'

[slowkow]

I figured out a way to get around this bug:

In [94]: df = pd.DataFrame({'A':['a1','a2','a3'],'B':[4,5,6]})

In [95]: df2 = df.ix[[0,1]]

In [96]: df2['B'] -= 10

In [97]: df.ix[[0,1]] = df2

In [98]: df
Out[98]: 
    A  B
0  a1 -6
1  a2 -5
2  a3  6

[jreback]

yeh, the bug in 0.11.0 is because its a mixed_frame (e.g. more than one dtype), if its not mixed it will work fine; this is fixed in 0.11.1