PERF: faster groupby diff #45575

lukemanley · 2022-01-23T20:01:51Z

closes PERF: groupby-diff #16706
tests added / passed
Ensure all linting tests pass, see here for how to run them
whatsnew entry

As @jreback suggests in the original issue, there is a large perf improvement to groupby.diff if implemented directly as a groupby method.

The int8/int16 type coercion feels a bit odd, but was needed to retain existing behavior as well as pass existing unit tests that explicitly test those dtypes. Separate issue to discuss that: #45562

       before           after         ratio
     [2db3b0a0]       [6edfb839]
                      <faster-groupby-diff>
-     1.98±0.02ms      1.35±0.02ms     0.68  groupby.GroupByMethods.time_dtype_as_group('object', 'diff', 'transformation', 5)
-         799±9μs          472±6μs     0.59  groupby.GroupByMethods.time_dtype_as_group('object', 'diff', 'transformation', 1)
-         461±5μs        166±0.8μs     0.36  groupby.GroupByMethods.time_dtype_as_group('object', 'diff', 'direct', 1)
-        94.0±4ms       2.22±0.9ms     0.02  groupby.GroupByMethods.time_dtype_as_group('uint', 'diff', 'transformation', 5)
-        69.1±1ms      1.51±0.02ms     0.02  groupby.GroupByMethods.time_dtype_as_field('uint', 'diff', 'transformation', 5)
-        69.3±2ms      1.48±0.01ms     0.02  groupby.GroupByMethods.time_dtype_as_field('float', 'diff', 'transformation', 5)
-        78.5±4ms      1.45±0.01ms     0.02  groupby.GroupByMethods.time_dtype_as_field('int', 'diff', 'transformation', 5)
-        94.0±6ms      1.46±0.01ms     0.02  groupby.GroupByMethods.time_dtype_as_group('int', 'diff', 'transformation', 5)
-         122±3ms      1.55±0.01ms     0.01  groupby.GroupByMethods.time_dtype_as_field('datetime', 'diff', 'transformation', 5)
-         140±2ms         1.45±0ms     0.01  groupby.GroupByMethods.time_dtype_as_group('float', 'diff', 'transformation', 5)
-         140±2ms      1.41±0.02ms     0.01  groupby.GroupByMethods.time_dtype_as_group('datetime', 'diff', 'transformation', 5)
-      66.0±0.7ms          502±5μs     0.01  groupby.GroupByMethods.time_dtype_as_field('uint', 'diff', 'transformation', 1)
-      64.3±0.5ms         488±10μs     0.01  groupby.GroupByMethods.time_dtype_as_field('float', 'diff', 'transformation', 1)
-         106±8ms        767±200μs     0.01  groupby.GroupByMethods.time_dtype_as_group('uint', 'diff', 'transformation', 1)
-        75.9±2ms         497±20μs     0.01  groupby.GroupByMethods.time_dtype_as_field('int', 'diff', 'transformation', 1)
-        89.8±3ms        587±200μs     0.01  groupby.GroupByMethods.time_dtype_as_group('uint', 'diff', 'direct', 5)
-        88.5±1ms          526±5μs     0.01  groupby.GroupByMethods.time_dtype_as_group('int', 'diff', 'direct', 5)
-         102±2ms         552±10μs     0.01  groupby.GroupByMethods.time_dtype_as_group('int', 'diff', 'transformation', 1)
-         120±2ms          614±4μs     0.01  groupby.GroupByMethods.time_dtype_as_field('datetime', 'diff', 'transformation', 1)
-       161±0.6ms          604±7μs     0.00  groupby.GroupByMethods.time_dtype_as_group('float', 'diff', 'transformation', 1)
-         138±3ms          500±3μs     0.00  groupby.GroupByMethods.time_dtype_as_group('float', 'diff', 'direct', 5)
-         139±3ms          498±5μs     0.00  groupby.GroupByMethods.time_dtype_as_group('datetime', 'diff', 'direct', 5)
-         165±1ms          502±3μs     0.00  groupby.GroupByMethods.time_dtype_as_group('datetime', 'diff', 'transformation', 1)
-        66.5±1ms          192±3μs     0.00  groupby.GroupByMethods.time_dtype_as_field('uint', 'diff', 'direct', 1)
-        71.2±1ms          201±3μs     0.00  groupby.GroupByMethods.time_dtype_as_field('uint', 'diff', 'direct', 5)
-         124±3ms          326±2μs     0.00  groupby.GroupByMethods.time_dtype_as_field('datetime', 'diff', 'direct', 5)
-         119±2ms          306±3μs     0.00  groupby.GroupByMethods.time_dtype_as_field('datetime', 'diff', 'direct', 1)
-        78.6±1ms          196±5μs     0.00  groupby.GroupByMethods.time_dtype_as_field('int', 'diff', 'direct', 5)
-      70.1±0.8ms          170±5μs     0.00  groupby.GroupByMethods.time_dtype_as_field('float', 'diff', 'direct', 5)
-        67.4±4ms          159±2μs     0.00  groupby.GroupByMethods.time_dtype_as_field('float', 'diff', 'direct', 1)
-        78.4±1ms          183±2μs     0.00  groupby.GroupByMethods.time_dtype_as_field('int', 'diff', 'direct', 1)
-         108±8ms         249±80μs     0.00  groupby.GroupByMethods.time_dtype_as_group('uint', 'diff', 'direct', 1)
-         102±3ms          182±2μs     0.00  groupby.GroupByMethods.time_dtype_as_group('int', 'diff', 'direct', 1)
-         160±2ms          170±3μs     0.00  groupby.GroupByMethods.time_dtype_as_group('float', 'diff', 'direct', 1)
-         165±3ms          171±2μs     0.00  groupby.GroupByMethods.time_dtype_as_group('datetime', 'diff', 'direct', 1)

jreback · 2022-01-23T20:07:43Z

pandas/core/groupby/groupby.py

+        # int8 and int16 are coerced to float32 rather than float64.
+        dtypes_to_f32 = ["int8", "int16"]
+        if obj.ndim == 1:
+            if obj.dtype in dtypes_to_f32:


this will only work on the series case. why is this necessary? i think upcasting is fine here (likey what we do now)

jreback · 2022-01-23T20:08:07Z

pandas/core/groupby/groupby.py

+            if obj.dtype in dtypes_to_f32:
+                shifted = shifted.astype("float32")
+        else:
+            mask = obj.dtypes.astype(str).isin(dtypes_to_f32).values


umm what is this for?

the existing groupby diff behavior upcast int8 and int16 to float32, not float64. However, groupby shift upcasts to float64. See #45562 for more discussion.

its fine to break it here (and change the tests) upcatsing to float32 is weird

Ok, will do. That means groupby.diff will upcast differently vs Series.diff, but that can be discussed in #45562.

no you would fix that too

oh i see we did that because of some cython issue. ok then let's preserve it (you will need an explict test for this ) and i don't think what you are doing is actually correct for frames

thanks, fixed the type coercion and added an explicit test.

lukemanley · 2022-01-25T02:32:13Z

Might be worth deciding on #45609 before proceeding here.

jreback · 2022-02-26T19:51:22Z

ok let's use this PR. can you rebase. we have sufficient asv's here (for the small dtypes as well)?

jreback · 2022-02-26T19:51:44Z

pandas/core/groupby/groupby.py

+    @final
+    @Substitution(name="groupby")
+    @Appender(_common_see_also)
+    def diff(self, periods=1, axis=0):


can you type these args & return

Typing added

lukemanley · 2022-02-26T20:56:21Z

ok let's use this PR. can you rebase. we have sufficient asv's here (for the small dtypes as well)?

Added int16 to the GroupByMethods asv. Shows a similar improvement as other dtypes:

       before           after         ratio
     [19aa5ab5]       [c4459b4e]
     <main>           <faster-groupby-diff>
-      77.7±0.5ms      1.60±0.02ms     0.02  groupby.GroupByMethods.time_dtype_as_field('int16', 'diff', 'transformation', 5)
-        88.3±1ms      1.33±0.07ms     0.02  groupby.GroupByMethods.time_dtype_as_group('int16', 'diff', 'transformation', 5)
-        74.8±1ms          601±3μs     0.01  groupby.GroupByMethods.time_dtype_as_field('int16', 'diff', 'transformation', 1)
-         109±1ms         577±10μs     0.01  groupby.GroupByMethods.time_dtype_as_group('int16', 'diff', 'transformation', 1)
-      79.2±0.9ms          308±3μs     0.00  groupby.GroupByMethods.time_dtype_as_field('int16', 'diff', 'direct', 5)
-      75.6±0.9ms          288±6μs     0.00  groupby.GroupByMethods.time_dtype_as_field('int16', 'diff', 'direct', 1)
-        89.6±2ms          322±3μs     0.00  groupby.GroupByMethods.time_dtype_as_group('int16', 'diff', 'direct', 5)
-         109±2ms          192±7μs     0.00  groupby.GroupByMethods.time_dtype_as_group('int16', 'diff', 'direct', 1)

jreback · 2022-02-27T15:23:35Z

https://github.com/pandas-dev/pandas/runs/5346468863?check_suite_focus=true the asv benches are failing, can you take a look

lukemanley · 2022-02-28T10:17:56Z

@jreback - green, asv issues fixed

jreback · 2022-02-28T13:41:13Z

thanks @lukemanley very nice!

jbrockmendel · 2022-05-06T23:01:53Z

@lukemanley running asvs comparing main (as of yesterday) to v.1.4.0 i'm seeing some pretty big slowdowns in groupby.diff. For whatever reason I don't see them here (https://pandas.pydata.org/speed/pandas/index.html#regressions?sort=3&dir=desc). Any idea what might have happened?

lukemanley · 2022-05-07T00:54:58Z

hmmm... I still see a significant speedup when comparing main to the latest 1.4.x commit. I just ran the following:

asv continuous -f 1.1 upstream/1.4.x upstream/main -b groupby.GroupByMethods

       before           after         ratio
     [21de060f]       [d86e2006]
                      <main>    
-      75.4±0.8ms      1.52±0.01ms     0.02  groupby.GroupByMethods.time_dtype_as_field('uint', 'diff', 'transformation', 5)
-        78.6±4ms      1.51±0.02ms     0.02  groupby.GroupByMethods.time_dtype_as_field('float', 'diff', 'transformation', 5)
-       89.1±10ms      1.65±0.01ms     0.02  groupby.GroupByMethods.time_dtype_as_field('int16', 'diff', 'transformation', 5)
-        93.4±1ms      1.35±0.01ms     0.01  groupby.GroupByMethods.time_dtype_as_group('uint', 'diff', 'transformation', 5)
-        97.0±1ms      1.38±0.05ms     0.01  groupby.GroupByMethods.time_dtype_as_group('int', 'diff', 'transformation', 5)
-        115±40ms      1.55±0.05ms     0.01  groupby.GroupByMethods.time_dtype_as_field('int', 'diff', 'transformation', 5)
-        100±10ms      1.31±0.01ms     0.01  groupby.GroupByMethods.time_dtype_as_group('int16', 'diff', 'transformation', 5)
-         133±2ms      1.67±0.03ms     0.01  groupby.GroupByMethods.time_dtype_as_field('datetime', 'diff', 'transformation', 5)
-         148±2ms      1.37±0.01ms     0.01  groupby.GroupByMethods.time_dtype_as_group('float', 'diff', 'transformation', 5)
-         144±1ms      1.27±0.02ms     0.01  groupby.GroupByMethods.time_dtype_as_group('datetime', 'diff', 'transformation', 5)
-        70.7±1ms          550±6μs     0.01  groupby.GroupByMethods.time_dtype_as_field('uint', 'diff', 'transformation', 1)
-       88.1±20ms         623±10μs     0.01  groupby.GroupByMethods.time_dtype_as_field('int16', 'diff', 'transformation', 1)
-        76.5±5ms         510±10μs     0.01  groupby.GroupByMethods.time_dtype_as_field('float', 'diff', 'transformation', 1)
-         109±2ms         582±80μs     0.01  groupby.GroupByMethods.time_dtype_as_group('uint', 'diff', 'transformation', 1)
-         113±3ms          589±9μs     0.01  groupby.GroupByMethods.time_dtype_as_group('int16', 'diff', 'transformation', 1)
-         131±1ms         675±20μs     0.01  groupby.GroupByMethods.time_dtype_as_field('datetime', 'diff', 'transformation', 1)
-         117±7ms          586±6μs     0.00  groupby.GroupByMethods.time_dtype_as_group('int', 'diff', 'transformation', 1)
-        112±40ms          522±5μs     0.00  groupby.GroupByMethods.time_dtype_as_field('int', 'diff', 'transformation', 1)
-         172±2ms          639±7μs     0.00  groupby.GroupByMethods.time_dtype_as_group('float', 'diff', 'transformation', 1)
-        95.2±1ms         349±10μs     0.00  groupby.GroupByMethods.time_dtype_as_group('uint', 'diff', 'direct', 5)
-      94.4±0.8ms          343±1μs     0.00  groupby.GroupByMethods.time_dtype_as_group('int', 'diff', 'direct', 5)
-       90.0±20ms          317±3μs     0.00  groupby.GroupByMethods.time_dtype_as_field('int16', 'diff', 'direct', 5)
-        99.4±7ms          328±2μs     0.00  groupby.GroupByMethods.time_dtype_as_group('int16', 'diff', 'direct', 5)
-         172±2ms          533±9μs     0.00  groupby.GroupByMethods.time_dtype_as_group('datetime', 'diff', 'transformation', 1)
-       94.4±20ms          285±7μs     0.00  groupby.GroupByMethods.time_dtype_as_field('int16', 'diff', 'direct', 1)
-      72.9±0.7ms          207±1μs     0.00  groupby.GroupByMethods.time_dtype_as_field('uint', 'diff', 'direct', 5)
-      68.9±0.5ms          191±4μs     0.00  groupby.GroupByMethods.time_dtype_as_field('uint', 'diff', 'direct', 1)
-         134±1ms          367±7μs     0.00  groupby.GroupByMethods.time_dtype_as_field('datetime', 'diff', 'direct', 5)
-         133±3ms          329±4μs     0.00  groupby.GroupByMethods.time_dtype_as_field('datetime', 'diff', 'direct', 1)
-         145±1ms          325±3μs     0.00  groupby.GroupByMethods.time_dtype_as_group('float', 'diff', 'direct', 5)
-         145±5ms          322±3μs     0.00  groupby.GroupByMethods.time_dtype_as_group('datetime', 'diff', 'direct', 5)
-        82.2±3ms          177±5μs     0.00  groupby.GroupByMethods.time_dtype_as_field('float', 'diff', 'direct', 5)
-        76.4±5ms        163±0.9μs     0.00  groupby.GroupByMethods.time_dtype_as_field('float', 'diff', 'direct', 1)
-        102±40ms          202±1μs     0.00  groupby.GroupByMethods.time_dtype_as_field('int', 'diff', 'direct', 5)
-         107±2ms         203±20μs     0.00  groupby.GroupByMethods.time_dtype_as_group('uint', 'diff', 'direct', 1)
-       107±0.7ms          188±1μs     0.00  groupby.GroupByMethods.time_dtype_as_group('int', 'diff', 'direct', 1)
-         113±5ms          186±2μs     0.00  groupby.GroupByMethods.time_dtype_as_group('int16', 'diff', 'direct', 1)
-        127±60ms         190±20μs     0.00  groupby.GroupByMethods.time_dtype_as_field('int', 'diff', 'direct', 1)
-         176±9ms          183±3μs     0.00  groupby.GroupByMethods.time_dtype_as_group('float', 'diff', 'direct', 1)
-         171±2ms          176±2μs     0.00  groupby.GroupByMethods.time_dtype_as_group('datetime', 'diff', 'direct', 1)

jbrockmendel · 2022-05-07T00:57:58Z

Can you compare against v1.4.0 instead of 1.4.x? If the slowdown is between 1.4.0 and 1.4.x then it must be for a bugfix, in which case we can ignore it.

lukemanley · 2022-05-07T01:32:34Z

@jbrockmendel - seeing a similar speedup vs. 1.4.0 as well:

       before           after         ratio
     [bb1f6515]       [d86e2006]
     <v1.4.0^0>       <main>    
-        75.3±2ms       2.18±0.7ms     0.03  groupby.GroupByMethods.time_dtype_as_field('uint', 'diff', 'transformation', 5)
-        80.4±1ms       1.98±0.5ms     0.02  groupby.GroupByMethods.time_dtype_as_field('int16', 'diff', 'transformation', 5)
-        79.6±2ms       1.73±0.2ms     0.02  groupby.GroupByMethods.time_dtype_as_field('float', 'diff', 'transformation', 5)
-        81.5±3ms       1.75±0.2ms     0.02  groupby.GroupByMethods.time_dtype_as_field('int', 'diff', 'transformation', 5)
-        94.7±3ms       1.43±0.1ms     0.02  groupby.GroupByMethods.time_dtype_as_group('int', 'diff', 'transformation', 5)
-        94.2±2ms      1.31±0.02ms     0.01  groupby.GroupByMethods.time_dtype_as_group('int16', 'diff', 'transformation', 5)
-         133±2ms       1.79±0.1ms     0.01  groupby.GroupByMethods.time_dtype_as_field('datetime', 'diff', 'transformation', 5)
-        95.2±5ms      1.28±0.02ms     0.01  groupby.GroupByMethods.time_dtype_as_group('uint', 'diff', 'transformation', 5)
-         149±8ms      1.41±0.03ms     0.01  groupby.GroupByMethods.time_dtype_as_group('float', 'diff', 'transformation', 5)
-      76.7±0.8ms        705±100μs     0.01  groupby.GroupByMethods.time_dtype_as_field('int16', 'diff', 'transformation', 1)
-         148±8ms      1.24±0.01ms     0.01  groupby.GroupByMethods.time_dtype_as_group('datetime', 'diff', 'transformation', 5)
-        74.6±2ms        627±200μs     0.01  groupby.GroupByMethods.time_dtype_as_field('int', 'diff', 'transformation', 1)
-        73.7±1ms         548±40μs     0.01  groupby.GroupByMethods.time_dtype_as_field('float', 'diff', 'transformation', 1)
-        73.5±2ms          543±8μs     0.01  groupby.GroupByMethods.time_dtype_as_field('uint', 'diff', 'transformation', 1)
-       111±0.7ms         641±70μs     0.01  groupby.GroupByMethods.time_dtype_as_group('int', 'diff', 'transformation', 1)
-         129±3ms         698±40μs     0.01  groupby.GroupByMethods.time_dtype_as_field('datetime', 'diff', 'transformation', 1)
-         110±4ms         588±20μs     0.01  groupby.GroupByMethods.time_dtype_as_group('int16', 'diff', 'transformation', 1)
-         112±2ms          571±5μs     0.01  groupby.GroupByMethods.time_dtype_as_group('uint', 'diff', 'transformation', 1)
-        76.1±3ms         334±70μs     0.00  groupby.GroupByMethods.time_dtype_as_field('int16', 'diff', 'direct', 1)
-        79.3±3ms         343±30μs     0.00  groupby.GroupByMethods.time_dtype_as_field('int16', 'diff', 'direct', 5)
-        92.7±2ms         379±50μs     0.00  groupby.GroupByMethods.time_dtype_as_group('int', 'diff', 'direct', 5)
-         173±2ms         641±20μs     0.00  groupby.GroupByMethods.time_dtype_as_group('float', 'diff', 'transformation', 1)
-        92.2±3ms          341±5μs     0.00  groupby.GroupByMethods.time_dtype_as_group('int16', 'diff', 'direct', 5)
-      92.6±0.7ms         342±20μs     0.00  groupby.GroupByMethods.time_dtype_as_group('uint', 'diff', 'direct', 5)
-         135±4ms         405±40μs     0.00  groupby.GroupByMethods.time_dtype_as_field('datetime', 'diff', 'direct', 5)
-         181±2ms          537±8μs     0.00  groupby.GroupByMethods.time_dtype_as_group('datetime', 'diff', 'transformation', 1)
-        75.1±1ms         219±30μs     0.00  groupby.GroupByMethods.time_dtype_as_field('int', 'diff', 'direct', 1)
-        80.5±3ms         226±30μs     0.00  groupby.GroupByMethods.time_dtype_as_field('int', 'diff', 'direct', 5)
-        77.2±5ms          216±3μs     0.00  groupby.GroupByMethods.time_dtype_as_field('uint', 'diff', 'direct', 5)
-         130±4ms         346±40μs     0.00  groupby.GroupByMethods.time_dtype_as_field('datetime', 'diff', 'direct', 1)
-        74.6±3ms          192±2μs     0.00  groupby.GroupByMethods.time_dtype_as_field('uint', 'diff', 'direct', 1)
-        80.4±5ms         206±20μs     0.00  groupby.GroupByMethods.time_dtype_as_field('float', 'diff', 'direct', 5)
-        73.3±1ms         185±20μs     0.00  groupby.GroupByMethods.time_dtype_as_field('float', 'diff', 'direct', 1)
-         148±5ms          323±3μs     0.00  groupby.GroupByMethods.time_dtype_as_group('float', 'diff', 'direct', 5)
-         149±1ms          315±4μs     0.00  groupby.GroupByMethods.time_dtype_as_group('datetime', 'diff', 'direct', 5)
-         112±6ms         200±20μs     0.00  groupby.GroupByMethods.time_dtype_as_group('int', 'diff', 'direct', 1)
-         107±3ms          190±2μs     0.00  groupby.GroupByMethods.time_dtype_as_group('int16', 'diff', 'direct', 1)
-         112±2ms          185±1μs     0.00  groupby.GroupByMethods.time_dtype_as_group('uint', 'diff', 'direct', 1)
-         174±8ms          179±3μs     0.00  groupby.GroupByMethods.time_dtype_as_group('float', 'diff', 'direct', 1)
-         179±9ms          175±2μs     0.00  groupby.GroupByMethods.time_dtype_as_group('datetime', 'diff', 'direct', 1)

jbrockmendel · 2022-05-07T03:12:25Z

thanks. i found a slowdown on my linux machine, will try again on mac and see if i can figure out whats going on

lukemanley added 3 commits January 23, 2022 13:49

faster groupby.diff

6edfb83

Merge branch 'main' into faster-groupby-diff

97d65ea

whatsnew

3badfad

jreback requested changes Jan 23, 2022

View reviewed changes

jreback added Groupby Performance Memory or execution speed performance labels Jan 23, 2022

fix type coercion, add test

1c730a0

jreback added this to the 1.5 milestone Jan 27, 2022

Merge branch 'main' into faster-groupby-diff

760e13e

jreback reviewed Feb 26, 2022

View reviewed changes

lukemanley added 3 commits February 26, 2022 14:59

Merge branch 'main' into faster-groupby-diff

84a4f8b

typing

3d5ca7a

add int16 to GroupByMethods asv

c4459b4

jreback approved these changes Feb 27, 2022

View reviewed changes

Merge remote-tracking branch 'upstream/main' into faster-groupby-diff

c6668fc

lukemanley mentioned this pull request Feb 27, 2022

REF/TST: misplaced hashtable tests, rename HashTable.map map_keys_to_values #46106

Merged

4 tasks

lukemanley added 2 commits February 27, 2022 19:05

Merge remote-tracking branch 'upstream/main' into faster-groupby-diff

90a7b20

fix asv

8e990c0

jreback merged commit e70d310 into pandas-dev:main Feb 28, 2022

lukemanley deleted the faster-groupby-diff branch March 2, 2022 01:12

yehoshuadimarsky pushed a commit to yehoshuadimarsky/pandas that referenced this pull request Jul 13, 2022

PERF: faster groupby diff (pandas-dev#45575)

fbbe02a

lukemanley mentioned this pull request Feb 12, 2023

ENH: optimized Groupby.diff() #33658

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

PERF: faster groupby diff #45575

PERF: faster groupby diff #45575

lukemanley commented Jan 23, 2022

jreback Jan 23, 2022

jreback Jan 23, 2022

lukemanley Jan 23, 2022

jreback Jan 23, 2022

lukemanley Jan 23, 2022

jreback Jan 23, 2022

jreback Jan 23, 2022

lukemanley Jan 23, 2022

lukemanley commented Jan 25, 2022

jreback commented Feb 26, 2022

jreback Feb 26, 2022

lukemanley Feb 26, 2022

lukemanley commented Feb 26, 2022

jreback commented Feb 27, 2022

lukemanley commented Feb 28, 2022

jreback commented Feb 28, 2022

jbrockmendel commented May 6, 2022

lukemanley commented May 7, 2022

jbrockmendel commented May 7, 2022

lukemanley commented May 7, 2022

jbrockmendel commented May 7, 2022

PERF: faster groupby diff #45575

PERF: faster groupby diff #45575

Conversation

lukemanley commented Jan 23, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lukemanley commented Jan 25, 2022

jreback commented Feb 26, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lukemanley commented Feb 26, 2022

jreback commented Feb 27, 2022

lukemanley commented Feb 28, 2022

jreback commented Feb 28, 2022

jbrockmendel commented May 6, 2022

lukemanley commented May 7, 2022

jbrockmendel commented May 7, 2022

lukemanley commented May 7, 2022

jbrockmendel commented May 7, 2022