ENH/PERF: add ExtensionArray.duplicated #55255

lukemanley · 2023-09-23T02:31:29Z

closes ENH: support 'duplicated' functionality for ExtensionArrays #27264
closes ENH: Add duplicated to MaskedArray/ExtensionArray interface #48424
closes BUG: pd.Series.duplicated/pd.core.algorithms.duplicated fails with float-based SparseArray #48788
Tests added and passed if fixing a bug or adding a new feature
All code checks passed.
Added type annotations to new arguments/methods/functions.
Added an entry in the latest doc/source/whatsnew/v2.2.0.rst file if fixing a bug or adding a new feature.

[updated]

asv continuous -f 1.1 upstream/main arrow-temporal-duplicated -b algorithms.Duplicated

       before           after         ratio
-         101±2ms         51.6±3ms     0.51  algorithms.Duplicated.time_duplicated(False, False, 'string[pyarrow]')
-        98.5±3ms         47.3±4ms     0.48  algorithms.Duplicated.time_duplicated(False, 'first', 'string[pyarrow]')
-        96.0±4ms         45.5±4ms     0.47  algorithms.Duplicated.time_duplicated(False, 'last', 'string[pyarrow]')
-      1.57±0.03s       13.6±0.3ms     0.01  algorithms.Duplicated.time_duplicated(False, False, 'timestamp[ms][pyarrow]')
-      1.54±0.02s       13.1±0.7ms     0.01  algorithms.Duplicated.time_duplicated(False, 'first', 'timestamp[ms][pyarrow]')
-         1.55±0s       12.8±0.7ms     0.01  algorithms.Duplicated.time_duplicated(False, 'last', 'timestamp[ms][pyarrow]')

jbrockmendel · 2023-09-24T14:46:48Z

pandas/core/algorithms.py

+                values = values._to_masked()  # type: ignore[union-attr]
+            else:
+                values = (
+                    values._maybe_convert_datelike_array()  # type: ignore[union-attr]


let's just make a EA.duplicated for this? xref #27264, #48424

updated to implement ExtensionArray.duplicated. This is now a bit more general in that it is faster for more than just the pyarrow timestamp and duration types. See updated timings in OP.

…icated

pandas/core/arrays/base.py

…icated

mroeschke · 2023-09-28T16:30:25Z

pandas/core/arrays/sparse/array.py

+    def duplicated(
+        self, keep: Literal["first", "last", False] = "first"
+    ) -> npt.NDArray[np.bool_]:
+        return algos.duplicated(np.asarray(self), keep=keep)


Does sparse need to pass mask for it's NA values?

updated to pass the mask

mroeschke

One comment otherwise looks good

…icated

jbrockmendel · 2023-09-29T15:18:32Z

asv_bench/benchmarks/algorithms.py

@@ -80,7 +81,7 @@ class Duplicated:
            "datetime64[ns]",
            "datetime64[ns, tz]",
            "timestamp[ms][pyarrow]",
-            "duration[s][pyarrow]",
+            "string[pyarrow]",


why getting rid of duration here?

i think at some pint "string" on L80 will become redundant with "string[pyarrow]" here, so maybe a TODO(3.0) to get rid of one when that happens?

The original version of this PR only targeted pyarrow timestamp and duration types. The current version improves perf for a larger number of pyarrow types so thought I'd add a non-temporal type as well. I can add it back if you want.

im fine either way, just curious. usually for asvs we go throw the kitchen sink at it but there are downsides to that

pandas/core/algorithms.py

pandas/core/arrays/base.py

…icated

jbrockmendel

LGTM

mroeschke · 2023-10-03T15:48:08Z

Thanks @lukemanley

rhshadrach · 2023-10-20T20:33:05Z

This patch may have induced a performance regression. If it was a necessary behavior change, this may have been expected and everything is okay.

Please check the links below. If any ASVs are parameterized, the combinations of parameters that a regression has been detected for appear as subbullets.

period.Algorithms.time_drop_duplicates
- typ='index'

Subsequent benchmarks may have skipped some commits. The link below lists the commits that are between the two benchmark runs where the regression was identified.

6e6a683...3bf0f64

PERF: Series.duplicated for pyarrow timestamp and duration types

c614bce

lukemanley added Performance Memory or execution speed performance Arrow pyarrow functionality labels Sep 23, 2023

lukemanley added this to the 2.2 milestone Sep 23, 2023

lukemanley added 2 commits September 22, 2023 22:32

whatsnew

b508d13

fix setup

bbf5c6e

jbrockmendel reviewed Sep 24, 2023

View reviewed changes

lukemanley added 5 commits September 26, 2023 07:05

add ExtensionArray.duplicated

ccdec9e

Merge remote-tracking branch 'upstream/main' into arrow-temporal-dupl…

8ea5d97

…icated

fix

f1ebefa

simplify

d010745

Merge remote-tracking branch 'upstream/main' into arrow-temporal-dupl…

db1ef4f

…icated

lukemanley changed the title ~~PERF: Series.duplicated for pyarrow timestamp and duration types~~ PERF: Series.duplicated for pyarrow dtypes Sep 26, 2023

add SparseArray.duplicated

4c9e4dd

lukemanley changed the title ~~PERF: Series.duplicated for pyarrow dtypes~~ ENH/PERF: add ExtensionArray.duplicated Sep 27, 2023

lukemanley added 2 commits September 27, 2023 05:46

simplify

0af7e27

docs

6aca6e5

mroeschke reviewed Sep 27, 2023

View reviewed changes

pandas/core/arrays/base.py Outdated Show resolved Hide resolved

lukemanley added 3 commits September 27, 2023 17:58

pass mask

8690245

Merge remote-tracking branch 'upstream/main' into arrow-temporal-dupl…

5a8508b

…icated

mypy

c5acd57

mroeschke reviewed Sep 28, 2023

View reviewed changes

lukemanley added 2 commits September 28, 2023 18:20

Merge remote-tracking branch 'upstream/main' into arrow-temporal-dupl…

0d0ac49

…icated

use mask

cc7cf14

jbrockmendel reviewed Sep 29, 2023

View reviewed changes

pandas/core/algorithms.py Outdated Show resolved Hide resolved

jbrockmendel reviewed Sep 29, 2023

View reviewed changes

pandas/core/algorithms.py Show resolved Hide resolved

jbrockmendel reviewed Sep 29, 2023

View reviewed changes

pandas/core/arrays/base.py Show resolved Hide resolved

Merge remote-tracking branch 'upstream/main' into arrow-temporal-dupl…

ad39a6f

…icated

lukemanley added 3 commits September 29, 2023 21:36

add optional to docstring

1bdbe6e

Merge remote-tracking branch 'upstream/main' into arrow-temporal-dupl…

f1984ab

…icated

revert asv change

12cec9b

jbrockmendel approved these changes Oct 3, 2023

View reviewed changes

mroeschke approved these changes Oct 3, 2023

View reviewed changes

mroeschke merged commit 3bf0f64 into pandas-dev:main Oct 3, 2023

MichaelTiemannOSC mentioned this pull request Oct 12, 2023

Enable many complex number tests #54761

Closed

5 tasks

lukemanley deleted the arrow-temporal-duplicated branch November 16, 2023 12:56

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ENH/PERF: add ExtensionArray.duplicated #55255

ENH/PERF: add ExtensionArray.duplicated #55255

lukemanley commented Sep 23, 2023 •

edited

Loading

jbrockmendel Sep 24, 2023

lukemanley Sep 26, 2023

mroeschke Sep 28, 2023

lukemanley Sep 30, 2023

mroeschke left a comment

jbrockmendel Sep 29, 2023

lukemanley Sep 30, 2023

jbrockmendel Sep 30, 2023

jbrockmendel left a comment

mroeschke commented Oct 3, 2023

rhshadrach commented Oct 20, 2023

ENH/PERF: add ExtensionArray.duplicated #55255

ENH/PERF: add ExtensionArray.duplicated #55255

Conversation

lukemanley commented Sep 23, 2023 • edited Loading

jbrockmendel Sep 24, 2023

Choose a reason for hiding this comment

lukemanley Sep 26, 2023

Choose a reason for hiding this comment

mroeschke Sep 28, 2023

Choose a reason for hiding this comment

lukemanley Sep 30, 2023

Choose a reason for hiding this comment

mroeschke left a comment

Choose a reason for hiding this comment

jbrockmendel Sep 29, 2023

Choose a reason for hiding this comment

lukemanley Sep 30, 2023

Choose a reason for hiding this comment

jbrockmendel Sep 30, 2023

Choose a reason for hiding this comment

jbrockmendel left a comment

Choose a reason for hiding this comment

mroeschke commented Oct 3, 2023

rhshadrach commented Oct 20, 2023

lukemanley commented Sep 23, 2023 •

edited

Loading