Skip to content

Accessing with .ix removes time zone info #13697

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
ekipmanager opened this issue Jul 18, 2016 · 4 comments
Closed

Accessing with .ix removes time zone info #13697

ekipmanager opened this issue Jul 18, 2016 · 4 comments
Labels
Bug Duplicate Report Duplicate issue or pull request Indexing Related to indexing on series/frames, not to indexes themselves Timezones Timezone data dtype

Comments

@ekipmanager
Copy link

Code Sample, a copy-pastable example if possible

I have a dataframe with two columns start and end both have the dtype of datetime64[ns, tzoffset(None, -21600)] but depending on how I access it the output removes the timezone info for for example

df.ix[0, 'end']
Out[30]: Timestamp('2016-07-17 06:26:00')

but

df['end'][0]
Out[32]: Timestamp('2016-07-17 00:26:00-0600', tz='tzoffset(None, -21600)')

Issue rises when I am doing something like this (length is pandas.tslib.Timedelta):

df.ix[0, 'end'] = df.ix[0, 'start'] + df.ix[0, 'length']

ValueError                                Traceback (most recent call last)
<ipython-input-34-f63ebfd12a2e> in <module>()
----> 1 df.ix[0, 'end'] = df.ix[0, 'start'] + df.ix[0, 'length']
/usr/lib64/python2.7/site-packages/pandas/core/indexing.pyc in __setitem__(self, key, value)
    130             key = com._apply_if_callable(key, self.obj)
    131         indexer = self._get_setitem_indexer(key)
--> 132         self._setitem_with_indexer(indexer, value)
    133 
    134     def _has_valid_type(self, k, axis):

/usr/lib64/python2.7/site-packages/pandas/core/indexing.pyc in _setitem_with_indexer(self, indexer, value)
    534                 # scalar
    535                 for item in labels:
--> 536                     setter(item, value)
    537 
    538         else:

/usr/lib64/python2.7/site-packages/pandas/core/indexing.pyc in setter(item, v)
    459                     s._consolidate_inplace()
    460                     s = s.copy()
--> 461                     s._data = s._data.setitem(indexer=pi, value=v)
    462                     s._maybe_update_cacher(clear=True)
    463 

/usr/lib64/python2.7/site-packages/pandas/core/internals.pyc in setitem(self, **kwargs)
   2915 
   2916     def setitem(self, **kwargs):
-> 2917         return self.apply('setitem', **kwargs)
   2918 
   2919     def putmask(self, **kwargs):

/usr/lib64/python2.7/site-packages/pandas/core/internals.pyc in apply(self, f, axes, filter, do_integrity_check, consolidate, raw, **kwargs)
   2888 
   2889             kwargs['mgr'] = self
-> 2890             applied = getattr(b, f)(**kwargs)
   2891             result_blocks = _extend_blocks(applied, result_blocks)
   2892 

/usr/lib64/python2.7/site-packages/pandas/core/internals.pyc in setitem(self, indexer, value, mgr)
    640 
    641         # coerce args
--> 642         values, _, value, _ = self._try_coerce_args(self.values, value)
    643         arr_value = np.array(value)
    644 

/usr/lib64/python2.7/site-packages/pandas/core/internals.pyc in _try_coerce_args(self, values, other)
   2270             # test we can have an equal time zone
   2271             if tz is None or str(tz) != str(self.values.tz):
-> 2272                 raise ValueError("incompatible or non tz-aware value")
   2273             other_mask = isnull(other)
   2274             other = other.tz_localize(None).value

ValueError: incompatible or non tz-aware value

However if I do it like this:
df.ix[0, 'end'] = df['start'][0] + df.ix[0, 'length']
It works fine since that type of selection does not lose the timezone info.

output of pd.show_versions()

INSTALLED VERSIONS

commit: None
python: 2.7.11.final.0
python-bits: 64
OS: Linux
OS-release: 4.5.7-200.fc23.x86_64
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_US.utf8

pandas: 0.18.1
nose: 1.3.7
pip: 8.1.2
setuptools: 20.1.1
Cython: 0.23.4
numpy: 1.11.1
scipy: 0.17.1
statsmodels: 0.6.1
xarray: None
IPython: 4.1.1
sphinx: 1.2.3
patsy: 0.4.1
dateutil: 2.5.3
pytz: 2016.6.1
blosc: None
bottleneck: 0.6.0
tables: None
numexpr: 2.4.6
matplotlib: 1.5.1
openpyxl: 1.8.6
xlrd: None
xlwt: None
xlsxwriter: None
lxml: 3.4.4
bs4: None
html5lib: None
httplib2: None
apiclient: None
sqlalchemy: 1.0.12
pymysql: None
psycopg2: 2.6.1 (dt dec pq3 ext lo64)
jinja2: 2.8
boto: 2.40.0
pandas_datareader: None

@jreback
Copy link
Contributor

jreback commented Jul 18, 2016

pls show a complete copy-pastable example

@ekipmanager
Copy link
Author

This fails

from dateutil.tz import tzoffset
df = pd.DataFrame({'A': pd.Timestamp('20130102', tz=tzoffset(None, -21600)),
                    'B': pd.Timestamp('20130102', tz=tzoffset(None, -21600)),
                    'C': np.array([3] * 4, dtype='int32'), })
df.ix[0, 'A'] = df.ix[0, 'B'] + pd.tslib.Timedelta(1e9)

But not this:
df.ix[0, 'A'] = df['B'][0] + pd.tslib.Timedelta(1e9)

@jreback
Copy link
Contributor

jreback commented Jul 18, 2016

dupe of #12938
thanks for the report

@jreback jreback closed this as completed Jul 18, 2016
@jreback jreback added Bug Indexing Related to indexing on series/frames, not to indexes themselves Duplicate Report Duplicate issue or pull request Timezones Timezone data dtype labels Jul 18, 2016
@jreback
Copy link
Contributor

jreback commented Jul 18, 2016

in general we don't recommend using .ix as is not semantically as correct as .loc; further we NEVER recommend using chained indexing df['B'][0], but it is marked as a bug in any event. pull requests to fix are welcome!

@jreback jreback added this to the No action milestone Jul 18, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Bug Duplicate Report Duplicate issue or pull request Indexing Related to indexing on series/frames, not to indexes themselves Timezones Timezone data dtype
Projects
None yet
Development

No branches or pull requests

2 participants