Skip to content

BUG: syntax error in hdf query with ts #15544

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 5 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 28 additions & 1 deletion doc/source/whatsnew/v0.20.0.txt
Original file line number Diff line number Diff line change
Expand Up @@ -450,6 +450,33 @@ New Behavior:

df.groupby('A').agg([np.mean, np.std, np.min, np.max])

HDFStore where string comparison
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

In previous versions most types could be compared to string column in a ``HDFStore``
usually resulting in an invalid comparsion. These comparisions will now raise a
``TypeError`` (:issue:`15492`)

New Behavior:

.. code-block:: ipython

In [15]: df = pd.DataFrame({'unparsed_date': ['2014-01-01', '2014-01-01']})

In [16]: df.dtypes
Out[16]:
unparsed_date object
dtype: object

In [17]: df.to_hdf('store.h5', 'key', format='table', data_columns=True)

In [18]: ts = pd.Timestamp('2014-01-01')

In [19]: pd.read_hdf('store.h5', 'key', where='unparsed_date > ts')
TypeError: Cannot compare 2014-01-01 00:00:00 of
type <class 'pandas.tslib.Timestamp'> to string column


.. _whatsnew_0200.api:

Other API Changes
Expand Down Expand Up @@ -602,7 +629,7 @@ Bug Fixes
- Bug in ``pd.merge_asof()`` where ``left_index``/``right_index`` together caused a failure when ``tolerance`` was specified (:issue:`15135`)



- Bug in ``pd.read_hdf()`` passing a ``Timestamp`` to the ``where`` parameter with a non date column (:issue:`15492`)


- Bug in ``Series`` constructor when both ``copy=True`` and ``dtype`` arguments are provided (:issue:`15125`)
Expand Down
21 changes: 8 additions & 13 deletions pandas/computation/pytables.py
Original file line number Diff line number Diff line change
@@ -1,7 +1,6 @@
""" manage PyTables query interface via Expressions """

import ast
import time
import warnings
from functools import partial
from datetime import datetime, timedelta
Expand Down Expand Up @@ -188,10 +187,6 @@ def stringify(value):
if v.tz is not None:
v = v.tz_convert('UTC')
return TermValue(v, v.value, kind)
elif (isinstance(v, datetime) or hasattr(v, 'timetuple') or
kind == u('date')):
v = time.mktime(v.timetuple())
return TermValue(v, pd.Timestamp(v), kind)
elif kind == u('timedelta64') or kind == u('timedelta'):
v = _coerce_scalar_to_timedelta_type(v, unit='s').value
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jreback - any idea what this was supposed to be doing? This doesn't impact any tested behavior and not sure it ever would have worked.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this was some really old compat IIRC. (or maybe just copied it from somewhere :>)

improving test coverage by deletions!

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah that isn't hit at all in the tests

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

there is another reference to timetuple in that same file, you can prob take that out too.

return TermValue(int(v), v, kind)
Expand All @@ -218,12 +213,13 @@ def stringify(value):
else:
v = bool(v)
return TermValue(v, v, kind)
elif not isinstance(v, string_types):
v = stringify(v)
elif isinstance(v, string_types):
# string quoting
return TermValue(v, stringify(v), u('string'))

# string quoting
return TermValue(v, stringify(v), u('string'))
else:
raise TypeError(("Cannot compare {v} of type {typ}"
" to {kind} column").format(v=v, typ=type(v),
kind=kind))

def convert_values(self):
pass
Expand Down Expand Up @@ -558,9 +554,8 @@ def parse_back_compat(self, w, op=None, value=None):

# stringify with quotes these values
def convert(v):
if (isinstance(v, (datetime, np.datetime64,
timedelta, np.timedelta64)) or
hasattr(v, 'timetuple')):
if isinstance(v, (datetime, np.datetime64,
timedelta, np.timedelta64)):
return "'{0}'".format(v)
return v

Expand Down
44 changes: 44 additions & 0 deletions pandas/tests/io/test_pytables.py
Original file line number Diff line number Diff line change
Expand Up @@ -5080,6 +5080,50 @@ def test_query_long_float_literal(self):
expected = df.loc[[1], :]
tm.assert_frame_equal(expected, result)

def test_query_compare_column_type(self):
# GH 15492
df = pd.DataFrame({'date': ['2014-01-01', '2014-01-02'],
'real_date': date_range('2014-01-01', periods=2),
'float': [1.1, 1.2],
'int': [1, 2]},
columns=['date', 'real_date', 'float', 'int'])

with ensure_clean_store(self.path) as store:
store.append('test', df, format='table', data_columns=True)

ts = pd.Timestamp('2014-01-01') # noqa
result = store.select('test', where='real_date > ts')
expected = df.loc[[1], :]
tm.assert_frame_equal(expected, result)

for op in ['<', '>', '==']:
# non strings to string column always fail
for v in [2.1, True, pd.Timestamp('2014-01-01'),
pd.Timedelta(1, 's')]:
query = 'date {op} v'.format(op=op)
with tm.assertRaises(TypeError):
result = store.select('test', where=query)

# strings to other columns must be convertible to type
v = 'a'
for col in ['int', 'float', 'real_date']:
query = '{col} {op} v'.format(op=op, col=col)
with tm.assertRaises(ValueError):
result = store.select('test', where=query)

for v, col in zip(['1', '1.1', '2014-01-01'],
['int', 'float', 'real_date']):
query = '{col} {op} v'.format(op=op, col=col)
result = store.select('test', where=query)

if op == '==':
expected = df.loc[[0], :]
elif op == '>':
expected = df.loc[[1], :]
else:
expected = df.loc[[], :]
tm.assert_frame_equal(expected, result)


class TestHDFComplexValues(Base):
# GH10447
Expand Down