Skip to content

packers tests failing on python 2 with blosc 1.14.4 #22430

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
TomAugspurger opened this issue Aug 20, 2018 · 2 comments
Closed

packers tests failing on python 2 with blosc 1.14.4 #22430

TomAugspurger opened this issue Aug 20, 2018 · 2 comments

Comments

@TomAugspurger
Copy link
Contributor

Using the ci/travis-27.yml:

pytest pandas/tests/io/test_packers.py::TestCompression::test_small_strings_no_warn_blosc -x --pdb
==================================================================== test session starts ====================================================================
platform darwin -- Python 2.7.15, pytest-3.7.1, py-1.5.4, pluggy-0.7.1
rootdir: /Users/taugspurger/sandbox/pandas-27, inifile: setup.cfg
plugins: xdist-1.22.5, forked-0.2
collected 1 item

pandas/tests/io/test_packers.py F
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> captured stderr >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
Output buffer size should be larger than 16 bytes
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> traceback >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

self = <pandas.tests.io.test_packers.TestCompression object at 0x1a180a1a10>

    def test_small_strings_no_warn_blosc(self):
        if not _BLOSC_INSTALLED:
            pytest.skip('no blosc')
>       self._test_small_strings_no_warn('blosc')

pandas/tests/io/test_packers.py:758:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
pandas/tests/io/test_packers.py:726: in _test_small_strings_no_warn
    empty_unpacked = self.encode_decode(empty, compress=compress)
pandas/tests/io/test_packers.py:104: in encode_decode
    to_msgpack(p, x, compress=compress, **kwargs)
pandas/io/packers.py:156: in to_msgpack
    writer(fh)
pandas/io/packers.py:151: in writer
    fh.write(pack(a, **kwargs))
pandas/io/packers.py:727: in pack
    use_bin_type=use_bin_type).pack(o)
pandas/io/msgpack/_packer.pyx:237: in pandas.io.msgpack._packer.Packer.pack
    cpdef pack(self, object obj):
pandas/io/msgpack/_packer.pyx:239: in pandas.io.msgpack._packer.Packer.pack
    ret = self._pack(obj, DEFAULT_RECURSE_LIMIT)
pandas/io/msgpack/_packer.pyx:230: in pandas.io.msgpack._packer.Packer._pack
    o = self._default(o)
pandas/io/packers.py:551: in encode
    u'data': convert(obj),
pandas/io/packers.py:306: in convert
    return ExtType(0, blosc.compress(v, typesize=dtype.itemsize))
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

bytesobj = '', typesize = 1, clevel = 9, shuffle = 1, cname = 'blosclz'

    def compress(bytesobj, typesize=8, clevel=9, shuffle=blosc.SHUFFLE,
                 cname='blosclz'):
        """compress(bytesobj[, typesize=8, clevel=9, shuffle=blosc.SHUFFLE, cname='blosclz']])

        Compress bytesobj, with a given type size.

        Parameters
        ----------
        bytesobj : bytes-like object (supporting the buffer interface)
            The data to be compressed.
        typesize : int
            The data type size.
        clevel : int (optional)
            The compression level from 0 (no compression) to 9
            (maximum compression).  The default is 9.
        shuffle : int (optional)
            The shuffle filter to be activated.  Allowed values are
            blosc.NOSHUFFLE, blosc.SHUFFLE and blosc.BITSHUFFLE.  The
            default is blosc.SHUFFLE.
        cname : string (optional)
            The name of the compressor used internally in Blosc. It can be
            any of the supported by Blosc ('blosclz', 'lz4', 'lz4hc',
            'snappy', 'zlib', 'zstd' and maybe others too). The default is
            'blosclz'.

        Returns
        -------
        out : str / bytes
            The compressed data in form of a Python str / bytes object.

        Raises
        ------
        TypeError
            If bytesobj doesn't support the buffer interface.
        ValueError
            If bytesobj is too long.
            If typesize is not within the allowed range.
            If clevel is not within the allowed range.
            If cname is not a valid codec.

        Examples
        --------

        >>> import array
        >>> a = array.array('i', range(1000*1000))
        >>> a_bytesobj = a.tostring()
        >>> c_bytesobj = blosc.compress(a_bytesobj, typesize=4)
        >>> len(c_bytesobj) < len(a_bytesobj)
        True

        """

        _check_input_length('bytesobj', len(bytesobj))
        _check_typesize(typesize)
        _check_shuffle(shuffle)
        _check_clevel(clevel)
        _check_cname(cname)

>       return _ext.compress(bytesobj, typesize, clevel, shuffle, cname)
E       error: Error -1 while compressing data

../../miniconda3/envs/travis-27/lib/python2.7/site-packages/blosc/toplevel.py:367: error
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> entering PDB >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> /Users/taugspurger/miniconda3/envs/travis-27/lib/python2.7/site-packages/blosc/toplevel.py(367)compress()
    365     _check_cname(cname)
    366
--> 367     return _ext.compress(bytesobj, typesize, clevel, shuffle, cname)
    368
    369


# packages in environment at /Users/taugspurger/miniconda3/envs/travis-27:
#
# Name                    Version                   Build  Channel
apipkg                    1.5                      py27_0  
appnope                   0.1.0            py27hb466136_0  
arrow-cpp                 0.4.1               np113py27_2    conda-forge
asn1crypto                0.24.0                   py27_0  
atomicwrites              1.1.5                    py27_0  
attrs                     18.1.0                   py27_0  
backports                 1.0                      py27_1  
backports.functools_lru_cache 1.5                      py27_1  
backports.lzma            0.0.13                    <pip>
backports.shutil_get_terminal_size 1.0.0                    py27_2  
backports_abc             0.5              py27h6972548_0  
beautifulsoup4            4.6.3                    py27_0  
blas                      1.0                    openblas  
blinker                   1.4                      py27_0  
blosc                     1.14.4               hd9629dc_0  
boto                      2.49.0                   py27_0  
boto3                     1.7.62                   py27_1  
botocore                  1.10.62                  py27_0  
bottleneck                1.2.1            py27h1d22016_1  
bzip2                     1.0.6                h1de35cc_5  
ca-certificates           2018.03.07                    0  
cachetools                2.1.0                      py_0    conda-forge
certifi                   2018.8.13                py27_0  
cffi                      1.11.5           py27h342bebf_0  
chardet                   3.0.4                    py27_1  
click                     6.7              py27h2b86a94_0  
configparser              3.5.0            py27hc7edf1b_0  
cookies                   2.2.1                    py27_0  
cpplint                   1.3.0                     <pip>
cryptography              2.3              py27hdbc3d79_0  
cryptography-vectors      2.3                      py27_0  
cycler                    0.10.0           py27hfc73c78_0  
cython                    0.28.2           py27h1de35cc_0  
decorator                 4.3.0                    py27_0  
dicttoxml                 1.7.4                    py27_0    conda-forge
docutils                  0.14             py27h0befae3_0  
enum34                    1.1.6                    py27_1  
et_xmlfile                1.0.1            py27hc42f929_0  
execnet                   1.5.0                    py27_0  
fastparquet               0.1.5            py27h1de35cc_0  
feather-format            0.4.0                    py27_2    conda-forge
flake8                    3.4.1            py27haba0547_0  
flake8-comprehensions     1.4.1                      py_1    conda-forge
flask                     0.12.2           py27h3ac5568_0  
freetype                  2.9.1                hb4e5f40_0  
funcsigs                  1.0.2            py27hb9f6266_0  
functools32               3.2.3.2                  py27_1  
futures                   3.2.0                    py27_0  
gcsfs                     0.1.2                      py_0    conda-forge
google-api-core           1.3.0                     <pip>
google-auth               1.5.1                      py_0    conda-forge
google-auth-oauthlib      0.2.0                    py27_0    conda-forge
google-cloud-bigquery     1.5.0                     <pip>
google-cloud-core         0.28.1                    <pip>
google-resumable-media    0.3.1                     <pip>
googleapis-common-protos  1.5.3                     <pip>
hdf5                      1.10.2               hfa1e0ec_1  
html5lib                  1.0.1                    py27_0  
icu                       58.2                 h4b95b61_1  
idna                      2.7                      py27_0  
ipaddress                 1.0.22                   py27_0  
ipython                   5.8.0                    py27_0  
ipython_genutils          0.2.0            py27h8b9a179_0  
itsdangerous              0.24                     py27_1  
jdcal                     1.4                      py27_0  
jemalloc                  4.5.0.post                    0    conda-forge
jinja2                    2.8.1                    py27_0  
jmespath                  0.9.3                    py27_0  
kiwisolver                1.0.1            py27h0a44026_0  
krb5                      1.16.1               h24a3359_6  
libcxx                    4.0.1                h579ed51_0  
libcxxabi                 4.0.1                hebd6815_0  
libedit                   3.1.20170329         hb402a30_2  
libffi                    3.2.1                h475c297_4  
libgfortran               3.0.1                h93005f0_2  
libiconv                  1.15                 hdd342a3_7  
libopenblas               0.2.20               hdc02c5d_7  
libpng                    1.6.34               he12f830_0  
libpq                     10.5                 hf30b1f0_0  
libxml2                   2.9.8                hab757c2_1  
libxslt                   1.1.32               hb819dd2_0  
llvmlite                  0.24.0           py27hc454e04_0  
lxml                      4.2.4            py27hef8c89e_0  
lzo                       2.10                 h362108e_2  
markupsafe                1.0              py27h1de35cc_1  
matplotlib                2.2.3            py27h54f8f79_0  
mccabe                    0.6.1                    py27_1  
mock                      2.0.0            py27h300204c_0  
more-itertools            4.3.0                    py27_0  
moto                      1.1.1                    py27_0    conda-forge
ncurses                   6.1                  h0a44026_0  
nomkl                     3.0                           0  
numba                     0.39.0           py27h6440ff4_0  
numexpr                   2.6.5           py27_nomklhd0a1019_0  [nomkl]
numpy                     1.13.3          py27_nomklh9797aa9_4  [nomkl]
oauthlib                  2.1.0                    py27_0  
openpyxl                  2.4.0                    py27_0  
openssl                   1.0.2p               h1de35cc_0  
pandas                    0.24.0.dev0+492.g513c02cb7           <pip>
pandas                    0.23.4           py27h6440ff4_0  
pandas-gbq                0.6.0                     <pip>
parquet-cpp               1.1.0.post                    0    conda-forge
pathlib                   1.0.1                     <pip>
pathlib2                  2.3.2                    py27_0  
patsy                     0.5.0                    py27_0  
pbr                       4.2.0                    py27_0  
pexpect                   4.6.0                    py27_0  
pickleshare               0.7.4            py27h37e3d41_0  
pip                       10.0.1                   py27_0  
pluggy                    0.7.1                    py27_0  
prompt_toolkit            1.0.15           py27h4a7b9c2_0  
protobuf                  3.6.1                     <pip>
psycopg2                  2.7.5            py27hdbc3d79_0  
ptyprocess                0.6.0                    py27_0  
py                        1.5.4                    py27_0  
pyaml                     17.12.1                    py_0    conda-forge
pyarrow                   0.4.1               np113py27_1    conda-forge
pyasn1                    0.4.4                    py27_0  
pyasn1-modules            0.2.2                    py27_0  
pycodestyle               2.3.1            py27h5b634e0_0  
pycparser                 2.18                     py27_1  
pycrypto                  2.6.1            py27h1de35cc_9  
pyflakes                  1.5.0            py27h7dfcaa9_1  
pygments                  2.2.0            py27h1a556bb_0  
pyjwt                     1.6.4                    py27_0  
pymysql                   0.6.3                    py27_0  
pyopenssl                 18.0.0                   py27_0  
pyparsing                 2.2.0                    py27_1  
pysocks                   1.6.8                    py27_0  
pytables                  3.4.4            py27h13cba08_0  
pytest                    3.7.1                    py27_0  
pytest-forked             0.2                      py27_0  
pytest-xdist              1.22.5                   py27_0  
python                    2.7.15               h138c1fe_0  
python-blosc              1.5.1            py27h1de35cc_2  
python-dateutil           2.5.0                    py27_0  
pytz                      2013b                    py27_0  
pyyaml                    3.13             py27h1de35cc_0  
readline                  7.0                  hc1231fa_4  
requests                  2.19.1                   py27_0  
requests-oauthlib         1.0.0                    py27_1  
rsa                       3.4.2                      py_1    conda-forge
s3fs                      0.1.5                    py27_0  
s3transfer                0.1.13                   py27_0  
scandir                   1.9.0            py27h1de35cc_0  
scipy                     1.1.0           py27_nomklh7cd7d8e_0  [nomkl]
setuptools                40.0.0                   py27_0  
simplegeneric             0.8.1                    py27_2  
singledispatch            3.4.0.3          py27he22c18d_0  
six                       1.11.0                   py27_1  
snappy                    1.1.7                he62c110_3  
sqlalchemy                0.9.6                    py27_0  
sqlite                    3.24.0               ha441bb4_0  
subprocess32              3.5.2            py27h1de35cc_0  
thrift                    0.11.0           py27h0a44026_0  
tk                        8.6.7                h35a86e2_3  
tornado                   5.1              py27h1de35cc_0  
traitlets                 4.3.2            py27hcf08151_0  
urllib3                   1.23                     py27_0  
wcwidth                   0.1.7            py27h817c265_0  
webencodings              0.5.1                    py27_1  
werkzeug                  0.14.1                   py27_0  
wheel                     0.31.1                   py27_0  
xarray                    0.8.0                    py27_0  
xlrd                      0.9.2                    py27_0  
xlsxwriter                0.5.2                    py27_0  
xlwt                      0.7.5                    py27_0  
xmltodict                 0.11.0                     py_1    conda-forge
xz                        5.2.4                h1de35cc_4  
yaml                      0.1.7                hc338f04_2  
zlib                      1.2.11               hf3cbc9b_2

Running

pytest pandas/tests/io/test_packers.py::TestCompression::test_small_strings_no_warn_blosc pandas/tests/io/test_packers.py::TestCompression::test_readonly_axis_blosc

fails the first test and hangs on the second.
@TomAugspurger TomAugspurger added CI Continuous Integration IO Msgpack and removed CI Continuous Integration labels Aug 20, 2018
@TomAugspurger
Copy link
Contributor Author

Should revert the pin in #22429 as well.

@TomAugspurger TomAugspurger mentioned this issue Aug 20, 2018
Closed
@jreback
Copy link
Contributor

jreback commented Jun 27, 2019

we no longer support python 2

@jreback jreback closed this as completed Jun 27, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@jreback @TomAugspurger