Skip to content

BUG: Unexpected behaviour when inserting timestamps into Series #57596

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
2 of 3 tasks
KowalMar opened this issue Feb 24, 2024 · 6 comments
Open
2 of 3 tasks

BUG: Unexpected behaviour when inserting timestamps into Series #57596

KowalMar opened this issue Feb 24, 2024 · 6 comments
Labels
Bug Needs Triage Issue that has not been reviewed by a pandas team member

Comments

@KowalMar
Copy link

Pandas version checks

  • I have checked that this issue has not already been reported.

  • I have confirmed this bug exists on the latest version of pandas.

  • I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import pandas as pd

dt = pd.to_datetime(['2024-02-24 10:00'])
s = pd.Series([1],index=dt)

s.loc['2024-02-24 9:57'] = None     # is inserted into s
print(s)
s.loc['2024-02-24 10:2:30'] = None  # is inserted into s
print(s)
s.loc['2024-02-24 10:08'] = None    # is _not_ inserted into s
print(s)

s.loc['2024-02-24 10:08:00'] = None # is inserted into s
print(s)

Issue Description

When inserting timestamps into a pandas Series, beginning with the format 'yyyy-mm-dd hh:mm', they are inserted correctly, until first time switching to format 'yyyy-mm-dd hh:mm:ss'. From then on format 'yyyy-mm-dd hh:mm' is silently not inserted, ending with seconds required.

Not sure if this is intended, unexpected behaviour in my opinion.

Expected Behavior

Insert timestamps as long as their format is parsed unambiguously.

Installed Versions

pandas 2.1.4 py311hf62ec03_0
@phofl
Copy link
Member

phofl commented Mar 18, 2024

cc @jbrockmendel

@wleong1
Copy link

wleong1 commented Mar 22, 2024

take

@jbrockmendel
Copy link
Member

This is a tough one. The '2024-02-24 10:08' case goes through the if self._can_partial_date_slice(reso): path in DatetimeIndex.get_loc, which in this case returns an empty array. One possible place to catch this would be to check for it in Loc._get_setitem_indexer. Not sure if there is a less-ugly/special-case-y place to do it.

@wleong1
Copy link

wleong1 commented Mar 24, 2024

Correct me if I'm wrong, from what I understand, the if self._can_partial_date_slice(reso): (in DatetimeIndex.get_loc) checks if reso > self._resolution_obj (in DatetimeIndexOpsMixin._can_partial_date_slice), with reso being the resolution of the current key being checked and self._resolution_obj being the lowest resolution thus far. If the if self._can_partial_date_slice(reso): returns True, it will then do return self._partial_date_slice(reso, parsed) (in DatetimeIndexOpsMixin._partial_date_slice), which returns the empty array. Would it be possible to just convert all entries to be the same as the lowest resolution thus far, e.g. if the lowest resolution thus far is Resolution.RESO_SEC, then any entries before and after would be converted into Resolution.RESO_SEC?

@jbrockmendel
Copy link
Member

Changing this within DTI.get_loc doesn't make sense; you'd need to catch it elsewhere

@wleong1
Copy link

wleong1 commented Mar 26, 2024

_LocationIndexer._get_setitem_indexer calls _LocIndexer._convert_to_indexer which goes through the if is_scalar(key) or (isinstance(labels, MultiIndex) and is_hashable(key) and not contains_slice): check. As is_scalar(key) is True, it returns return labels.get_loc(key) which is an empty array in this case, would it be possible to catch it here by checking if the array is empty, if it is then return {"key": key}?

Just wondering if catching it in _LocationIndexer._get_setitem_indexer could be repeating the checks done at _LocIndexer._convert_to_indexer, unless there is another way of approaching it.

@wleong1 wleong1 removed their assignment Dec 16, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Bug Needs Triage Issue that has not been reviewed by a pandas team member
Projects
None yet
4 participants