ValueError: cannot set WRITEABLE flag to True of this array

[macd2]

will need to revert the xfail decorator in: #25517 when this is fixed

Code Sample, a copy-pastable example if possible

Im getting all of a sudden this Error, any idea?

# Your code here
 input_df = pd.read_hdf(path_or_buf='x.hdf5',key='/x',mode='r')

Problem description

Traceback :

Traceback (most recent call last):
  File "...", line 115, in <module>
    input_df = pd.read_hdf(path_or_buf='x.hdf5',key='/x',mode='r')
  File "/usr/local/lib/python3.6/dist-packages/pandas/io/pytables.py", line 394, in read_hdf
    return store.select(key, auto_close=auto_close, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/pandas/io/pytables.py", line 741, in select
    return it.get_result()
  File "/usr/local/lib/python3.6/dist-packages/pandas/io/pytables.py", line 1483, in get_result
    results = self.func(self.start, self.stop, where)
  File "/usr/local/lib/python3.6/dist-packages/pandas/io/pytables.py", line 734, in func
    columns=columns)
  File "/usr/local/lib/python3.6/dist-packages/pandas/io/pytables.py", line 2937, in read
    start=_start, stop=_stop)
  File "/usr/local/lib/python3.6/dist-packages/pandas/io/pytables.py", line 2489, in read_array
    ret = node[0][start:stop]
  File "/usr/local/lib/python3.6/dist-packages/tables/vlarray.py", line 681, in __getitem__
    return self.read(start, stop, step)[0]
  File "/usr/local/lib/python3.6/dist-packages/tables/vlarray.py", line 821, in read
    listarr = self._read_array(start, stop, step)
  File "tables/hdf5extension.pyx", line 2155, in tables.hdf5extension.VLArray._read_array
ValueError: cannot set WRITEABLE flag to True of this array
```

[gfyoung]

@macd2 : Thanks for reporting this! A couple of things:

• In the issue, could you provide your environment information from pandas.show_versions ?
• Assuming you can, do you mind sharing the file that triggers this error?

Im getting all of a sudden this Error, any idea?

• It sounds like this was working for you on a previous versions of pandas . When did this code last work for you? What version are you using now (related to the first question)?

cc @jreback

[vvvlc]

When using numpy=1.16.0 I get this error, when I downgrade numpy=1.15.4 problem is gone

[macd2]

@gfyoung Sure here is the Versions:

'dependencies': 
{'pandas': '0.23.4', 'pytest': '3.4.0', 'pip': '18.1', 'setuptools': '40.6.3', 'Cython': '0.29.3', 'numpy': '1.16.0', 'scipy': '1.2.0', 'pyarrow': None, 'xarray': None, 'IPython': '6.5.0', 'sphinx': None, 'patsy': '0.5.0', 'dateutil': '2.7.5', 'pytz': '2018.7', 'blosc': None, 'bottleneck': '1.2.1', 'tables': '3.4.4', 'numexpr': '2.6.8', 'feather': None, 'matplotlib': '3.0.2', 'openpyxl': '2.5.12', 'xlrd': '1.1.0', 'xlwt': '1.3.0', 'xlsxwriter': '0.7.3', 'lxml': '4.1.1', 'bs4': '4.4.1', 'html5lib': '1.0b8', 'sqlalchemy': '1.2.15', 'pymysql': '0.9.2', 'psycopg2': '2.7.6.1 (dt dec pq3 ext lo64)', 'jinja2': '2.10', 's3fs': None, 'fastparquet': None, 'pandas_gbq': None, 'pandas_datareader': '0.7.0'}

unfortunately i can not share the file but i think the issues indeed comes from numpy as @vvvlc said here is an other issue on their git:

nipy/nibabel#697

PS: just downgraded to numpy=1.15.4 and indeed it resolves the issue

[TomAugspurger]

Is there anything pandas can do in the meantime? Or just wait for the next pytables release?

…

On Mon, Jan 21, 2019 at 5:34 AM macd2 ***@***.***> wrote: @gfyoung <https://github.com/gfyoung> Sure here is the Versions: 'dependencies': {'pandas': '0.23.4', 'pytest': '3.4.0', 'pip': '18.1', 'setuptools': '40.6.3', 'Cython': '0.29.3', 'numpy': '1.16.0', 'scipy': '1.2.0', 'pyarrow': None, 'xarray': None, 'IPython': '6.5.0', 'sphinx': None, 'patsy': '0.5.0', 'dateutil': '2.7.5', 'pytz': '2018.7', 'blosc': None, 'bottleneck': '1.2.1', 'tables': '3.4.4', 'numexpr': '2.6.8', 'feather': None, 'matplotlib': '3.0.2', 'openpyxl': '2.5.12', 'xlrd': '1.1.0', 'xlwt': '1.3.0', 'xlsxwriter': '0.7.3', 'lxml': '4.1.1', 'bs4': '4.4.1', 'html5lib': '1.0b8', 'sqlalchemy': '1.2.15', 'pymysql': '0.9.2', 'psycopg2': '2.7.6.1 (dt dec pq3 ext lo64)', 'jinja2': '2.10', 's3fs': None, 'fastparquet': None, 'pandas_gbq': None, 'pandas_datareader': '0.7.0'} unfortunately i can not share the file but i think the issues indeed comes from numpy as @vvvlc <https://github.com/vvvlc> said here is an other issue on their git: nipy/nibabel#697 <nipy/nibabel#697> — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#24839 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABQHIqiIyuE-lmMlN2Ep-4htyMrTBdefks5vFaWqgaJpZM4aJQqs> .

[mikkokotila]

For me pip install numpy==1.15.4 also resolves this issue.

@TomAugspurger this issue comes up on Google on top (at least for me it did) and it seems that the remedy is simple, so maybe it's enough to just wait for an upstream fix. Really great that you would ask :)

[swinman]

pip3 install numpy==1.15.4 also solved for me..

However, when I downgraded numpy i kept getting this error ImportError: No module named 'numpy.core._multiarray_umath'. Finally figured out it was happening because I stored the .h5 file with numpy 1.16 installed and it wouldn't reopen with the downgraded numpy...

[dev72]

you can avoid the error:
ValueError: cannot set WRITEABLE flag to True of this array
passing format='table' to HDFStore.append or HDFStore.put when you save data with pandas.

This will likely solve your problem, tested with pandas 0.24 and numpy 1.16+

[macd2]

@dev72
ok but how about old HDF files that already exist?

[dev72]

I think the best way to read old hdf files is to downgrade your pandas+numpy version, read all data and write it in a new hdf store with format='table'.
Then it should work with newer numpy and pandas versions.

[macd2]

@dev72 yes right im at the same point, but than i rather stick with the downgrade until this is properly fixed

[swyxio]

yes same came from google, thanks for filing this issue @macd2

[esvhd]

So after chasing around a few issues submitted at numpy and PyTables this post by @avalentino would suggest this issue is fixed in PyTables master but not in a release yet.

PyTables/PyTables#719 (comment)

Has anyone tried using PyTables master w/ numpy >= 0.16?

[ghost]

I got it to work using this

HDF5_DIR={HDF5_PATH} pip install -e git+https://github.com/PyTables/PyTables@492ee2f#egg=tables
pip install numpy==1.16.0

Make sure cython and hdf5 are installed.

[TomAugspurger]

What versions of pytables and numpy reproduce this? Is it specific to the data?

with

pandas: 0.24.1
numpy: 1.16.2
tables: 3.4.4

this doesn't raise,

In [8]: df = pd.DataFrame({"A": [1, 2]})

In [9]: df.to_hdf('x.hdf5', key='x')

In [10]: pd.read_hdf('x.hdf5', 'x', mode='r')
Out[10]:
   A
0  1
1  2

FYI, pytables 3.5.0 and 3.5.1 are on PyPI with the fix from the pytables side.

[goriccardo]

Upgrading to pytables 3.5.1 fixes the problem for me also with numpy 1.16.2

[mirameshs]

dealing with tf 2.0.0 need numpy (at least) 1.16.0. downgrading numpy to the previous version won't work on tf 2.0.0