BUG: Unexpected behaviour when inserting timestamps into Series #57596 #57628
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…d on values for Series with string dtype (#57116)
* TST: Interchange implementation for timestamp[ns][pyarrow] * Check timestamp explicitly * Use pytest.importorskip --------- Co-authored-by: Marco Edward Gorelli <[email protected]>
…s.read_hdf, pandas.HDFStore.append (#57114) * Resolve PR02 errors in docstrings: pandas.Series.interpolate, pandas.read_hdf, pandas.HDFStore.append * Update code_checks.sh
Update missing_data.rst Fix typo
Bumps [pypa/cibuildwheel](https://github.com/pypa/cibuildwheel) from 2.16.2 to 2.16.4. - [Release notes](https://github.com/pypa/cibuildwheel/releases) - [Changelog](https://github.com/pypa/cibuildwheel/blob/main/docs/changelog.md) - [Commits](pypa/cibuildwheel@v2.16.2...v2.16.4) --- updated-dependencies: - dependency-name: pypa/cibuildwheel dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…et_option, pandas.reset_option (#57117) * Copy the signature from the implementation * updated code_checs.sh
…ixed type index (#57101) * fix regression in join with empty * mypy * move empty check
* CI: autouse add_doctest_imports * add ref
The smallest typo correction, "engine" lacked an "n".
* ambiguous takes bool as an argument, not bool-ndarray ambiguous takes bool as an argument, not bool-ndarray * Update pandas/core/generic.py Co-authored-by: Matthew Roeschke <[email protected]> * Update pandas/core/generic.py Co-authored-by: Matthew Roeschke <[email protected]> --------- Co-authored-by: Marc Garcia <[email protected]> Co-authored-by: Matthew Roeschke <[email protected]>
* ENH: Add skipna to groupby.first and groupby.last * resample & tests * Improve test * Fixups * fixup test * Rework na_value determination
* fix masked indexing regression * fix test * fix test * dedup resizing logic * add types
* fix PR02 error in clip function * fix PR02 error in argsort function
…orrectly (#57175) fix from_dataframe for empty dataframes
DataFrame.sort_index not producing stable sort
…moval is … (#57162) 'freq' is removed in GH 14146), not moved to kw-only arg. Removal is already captured under 'Removal of prior version deprecations/changes'
Remove redundant code style check
* DEPR: Enforce deprecation of removing axis from all groupby ops * Add note on fillna and cleanup * Doc fixups * Remove corrwith axis=1 test * Remove corrwith axis=1 test * Skip corrwith docstring validation
* Document some more categorical methods * Render pages for Categorical methods
fix ES01 for pandas.Flags
* Revert "DEPS: Add warning if pyarrow is not installed (#56896)" This reverts commit 5c2a407 * Add whatsnew * Update * Update doc/source/whatsnew/v2.2.1.rst Co-authored-by: Matthew Roeschke <[email protected]> --------- Co-authored-by: Matthew Roeschke <[email protected]>
* Centeralize methods to class * Cleanups * CLN: Remove pickle support pre-pandas 1.0 * Typing * clean:
* attempt failing test * expand test for demonstration purposes * fix near-minimum timestamp overflow when scaling from microseconds to nanoseconds * minor refactor * add comments around specifically handling near-minimum microsecond and nanosecond timestamps * consolidate comments --------- Co-authored-by: Robert Schmidtke <[email protected]>
* CLN: Assort khash-python cleanups * add static inline to memory tracers * revert mistake with hashdouble * try less
…ame incorrect (#57570) string
….__array__ (#57561) * fixing PR01 errors for pandas.Categorical and pandas.Categorical.__array__ * updated dtype to np.dtype Co-authored-by: Matthew Roeschke <[email protected]> --------- Co-authored-by: Matthew Roeschke <[email protected]>
* ENH: Allow performance warnings to be disabled * ENH: Allow performance warnings to be disabled * Add tests * fixup
* PERF: groupby(...).__len__ * GH#
* CoW: Remove a few copy=False statements * Cow: Deprecate copy keyword from first set of methods * Fixup * Update * Update * Update
…egories (#57603) * Add description to Index._to_numpy method. * Fix description of default value for parameter "ordered" in set_categories * Add description to return value of Categorical.set_categories and fix typo in description. * Remove fixed docstrings from code_checks.sh
* Remove use_numexpr * remove return_filelike in ensure_clean * Start using temp_file * Use more unique name?' * Fix failure * remove from all
wleong1
requested review from
attack68,
rhshadrach,
MarcoGorelli,
Dr-Irv,
WillAyd,
noatamir,
datapythonista and
mroeschke
as code owners
|
Thanks for the contribution @wleong1. Seems like something is wrong with this PR. I think you may want to clone pandas again, and before implementing your changes create a branch in git. If you work in the main branch, things get mixed with other changes, as I think it's the case here. I'll close this, since you'll have to create a new PR if you work work in a different branch. Please let us know if you need help. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Resolution info:
Resolution.RESO_SEC - 3 for seconds
Resolution.RESO_MIN - 4 for minute
Observation:
With the previous code, when the current entry (resolution: 4, entry: '2024-02-24 10:08') has a greater resolution value than the smallest amongst previous entries (in this case resolution: 3, entry: '2024-02-24 10:2:30'), it is shown that the key is said to exist within the list of keys within the Series even though it shouldn't. This does not happen when a smaller or equal resolution value entry is inserted into the Series.
Thought process:
With the previous code, if the current entry's resolution value is greater than the smallest resolution value amongst the previous entries, the _LocIndexer._convert_to_indexer method will only return an empty list (output of the method labels.get_loc(key)), which might be causing entries with larger resolution values not being inserted into the Series correctly.
Potential solution:
I have changed the method _LocIndexer._convert_to_indexer to return the key ('2024-02-24 10:08') instead of an empty list when the resolution value is greater than the smallest resolution value.
doc/source/whatsnew/v2.2.1.rstfile if fixing a bug or adding a new feature.