ValueError: buffer source array is read-only

[kyleabeauchamp]

After upgrading to 0.17 (anaconda 2.7 + OSX), I'm finding that some previously working code is failing with ValueError: buffer source array is read-only. Driver code is below. The issue is basically the series version of #10043. I think the problem is that the patch for #10043 (#10070) only fixed the take_2d templates, leaving the take_1d (Series) versions with the old behavior. Based on my naive understanding, it might be straightforward to extend #10070 for the 1D cases.

import pandas as pd
import numpy as np

array = np.arange(10)
array.setflags(write=False)

X = pd.Series(array)

key = 0
keys = [key]



X.iloc[keys]
# OK

X.loc[keys]
# Fail
X.ix[keys]
# Fail

X.iloc[key]
# OK
X.loc[key]
# OK
X.ix[key]
# OK

Stacktrace is below.

ValueError                                Traceback (most recent call last)
<ipython-input-53-928f0ea51c58> in <module>()
----> 1 X.loc[keys]

~/opt/lib/python2.7/site-packages/pandas/core/indexing.py in __getitem__(self, key)
   1196             return self._getitem_tuple(key)
   1197         else:
-> 1198             return self._getitem_axis(key, axis=0)
   1199 
   1200     def _getitem_axis(self, key, axis=0):

~/opt/lib/python2.7/site-packages/pandas/core/indexing.py in _getitem_axis(self, key, axis)
   1330                     raise ValueError('Cannot index with multidimensional key')
   1331 
-> 1332                 return self._getitem_iterable(key, axis=axis)
   1333 
   1334             # nested tuple slicing

~/opt/lib/python2.7/site-packages/pandas/core/indexing.py in _getitem_iterable(self, key, axis)
    973 
    974                 try:
--> 975                     result = self.obj.reindex_axis(keyarr, axis=axis, level=level)
    976 
    977                     # this is an error as we are trying to find

~/opt/lib/python2.7/site-packages/pandas/core/series.py in reindex_axis(self, labels, axis, **kwargs)
   2276         if axis != 0:
   2277             raise ValueError("cannot reindex series on non-zero axis!")
-> 2278         return self.reindex(index=labels, **kwargs)
   2279 
   2280     def take(self, indices, axis=0, convert=True, is_copy=False):

~/opt/lib/python2.7/site-packages/pandas/core/series.py in reindex(self, index, **kwargs)
   2257     @Appender(generic._shared_docs['reindex'] % _shared_doc_kwargs)
   2258     def reindex(self, index=None, **kwargs):
-> 2259         return super(Series, self).reindex(index=index, **kwargs)
   2260 
   2261     @Appender(generic._shared_docs['fillna'] % _shared_doc_kwargs)

~/opt/lib/python2.7/site-packages/pandas/core/generic.py in reindex(self, *args, **kwargs)
   1846         # perform the reindex on the axes
   1847         return self._reindex_axes(axes, level, limit, tolerance,
-> 1848                                   method, fill_value, copy).__finalize__(self)
   1849 
   1850     def _reindex_axes(self, axes, level, limit, tolerance, method,

~/opt/lib/python2.7/site-packages/pandas/core/generic.py in _reindex_axes(self, axes, level, limit, tolerance, method, fill_value, copy)
   1865             obj = obj._reindex_with_indexers(
   1866                 {axis: [new_index, indexer]},
-> 1867                 fill_value=fill_value, copy=copy, allow_dups=False)
   1868 
   1869         return obj

~/opt/lib/python2.7/site-packages/pandas/core/generic.py in _reindex_with_indexers(self, reindexers, fill_value, copy, allow_dups)
   1957                                                 fill_value=fill_value,
   1958                                                 allow_dups=allow_dups,
-> 1959                                                 copy=copy)
   1960 
   1961         if copy and new_data is self._data:

~/opt/lib/python2.7/site-packages/pandas/core/internals.py in reindex_indexer(self, new_axis, indexer, axis, fill_value, allow_dups, copy)
   3397         if axis == 0:
   3398             new_blocks = self._slice_take_blocks_ax0(
-> 3399                 indexer, fill_tuple=(fill_value,))
   3400         else:
   3401             new_blocks = [blk.take_nd(indexer, axis=axis,

~/opt/lib/python2.7/site-packages/pandas/core/internals.py in _slice_take_blocks_ax0(self, slice_or_indexer, fill_tuple)
   3438                 return [blk.take_nd(slobj, axis=0,
   3439                                     new_mgr_locs=slice(0, sllen),
-> 3440                                     fill_tuple=fill_tuple)]
   3441 
   3442         if sl_type in ('slice', 'mask'):

~/opt/lib/python2.7/site-packages/pandas/core/internals.py in take_nd(self, indexer, axis, new_mgr_locs, fill_tuple)
    933             fill_value = fill_tuple[0]
    934             new_values = com.take_nd(values, indexer, axis=axis,
--> 935                                      allow_fill=True, fill_value=fill_value)
    936 
    937         if new_mgr_locs is None:

~/opt/lib/python2.7/site-packages/pandas/core/common.py in take_nd(arr, indexer, axis, out, fill_value, mask_info, allow_fill)
    782                                  axis=axis, mask_info=mask_info)
    783     indexer = _ensure_int64(indexer)
--> 784     func(arr, indexer, out, fill_value)
    785 
    786     if flip_order:

pandas/src/generated.pyx in pandas.algos.take_1d_int64_int64 (pandas/algos.c:87226)()

~/opt/lib/python2.7/site-packages/pandas/algos.so in View.MemoryView.memoryview_cwrapper (pandas/algos.c:176760)()

~/opt/lib/python2.7/site-packages/pandas/algos.so in View.MemoryView.memoryview.__cinit__ (pandas/algos.c:173129)()

ValueError: buffer source array is read-only

[jreback]

yep looks like the same issue

why do you have the read only flag set in the first place?

[kyleabeauchamp]

I have no idea. I have a pipeline of like 20 munging steps, but I haven't actually debugged why things end up read-only. Presumably, the issue is related to the usual "copy versus view" slicing stuff that always trips people up.

[jreback]

@kyleabeauchamp ok. would love to see a pull-request to fix (along the same lines as the issue you mentioned above). should be straightforward to do.

[invisibleroads]

This error appears in 0.17.1 and 0.17.0 when trying to aggregate large tables.

This error does not appear in 0.16.2.

Something broke between 0.16.2 and 0.17.0.

[EvanZ]

Glad I stumbled upon this issue. I am running into issue training a pipeline in sklearn, and found this error "ValueError: buffer source array is read-only". It seems to only occur under sklearn 0.17.0. When I downgrade to 0.15.2 (the version I was using when I built the pipeline originally), the issue doesn't occur. I assume it's related to whatever this bug is.

[max-sixty]

+1

[jreback]

closed by #12033