CLN: Make Series._values match Index._values

[jbrockmendel]

Discussed in #31037

• Note this is not an alternative to that, as this should not be considered for 1.0.
• This does not implement any of the simplifications that it makes available.
• There is a decent chance that the check in core.apply can be changed so that we can still use the fastpath for these dtypes.
• The PandasDType edit is not central to the change, just included for perf comparison against PERF: improve access of .array #31037.

[jorisvandenbossche]

Nice

[jorisvandenbossche]

Yes, I was also just thinking while looking above at the .array implementation, that we could do the same here instead of going through the ".array -> wrap in PandasArray -> extract the numpy array again" route, that will further reduce some overhead of extract_array(..., extract_numpy=True).

Could also do a arr = PandasArray(arr) here for being explicit (it's not that it duplicates a lot from .array)

[jbrockmendel]

rebased, this now removes array_values

[jreback]

lgtm.

[jreback]

@jorisvandenbossche

[jorisvandenbossche]

You will need to update the Series._values docstring I wrote.

Also, can you show similar timings as you showed on the other PR but now compared to latest master?

[jorisvandenbossche]

Can you add a comment here on why this is needed

[jbrockmendel]

Updated timings vs current master are a mixed bag:

import pandas as pd
from pandas.core.construction import extract_array

s1 = pd.Series(pd.date_range("2012-01-01", periods=3, tz='UTC'))
s2 = pd.Series(pd.date_range("2012-01-01", periods=3))
s3 = pd.Series([1, 2, 3])

%timeit s1.array
442 ns ± 16.2 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)  # <-- master
913 ns ± 60.1 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)  # <-- PR

%timeit s2.array
1.26 µs ± 40.9 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)  # <-- master
972 ns ± 33.2 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)  # <-- PR

%timeit s3.array
1.89 µs ± 71.5 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)  # <-- master
3.44 µs ± 28.4 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)  # <-- PR

%timeit extract_array(s1, extract_numpy=True)
1.85 µs ± 111 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)  # <-- master
1.62 µs ± 46.7 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)  # <-- PR

%timeit extract_array(s2, extract_numpy=True)
2.61 µs ± 29.4 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)  # <-- master
1.74 µs ± 66.8 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)  # <-- PR

%timeit extract_array(s3, extract_numpy=True)
3.95 µs ± 32.1 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)  # <-- master
1.6 µs ± 34.5 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)  # <-- PR

%timeit extract_array(s3, extract_numpy=False)
3.1 µs ± 115 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)  # <-- master
5.29 µs ± 160 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)  # <-- PR

[jorisvandenbossche]

Do we understand where the slowdown is coming from? From looking at the code, from the extra if isinstance(result, np.ndarray) in the .array attribute, I wouldn't expect such a difference.

[jbrockmendel]

Do we understand where the slowdown is coming from?

It looks like restoring array_values and Series.array does the trick. With those two changes, all of the timings become slightly better than master, with the exception of extract_array(s3, extract_numpy=False) which is ~30% slower

[jbrockmendel]

Restored array_values to avoid a perf hit; im ambivalent on if its worthwhile.

import pandas as pd 
from pandas.core.construction import extract_array 

s1 = pd.Series(pd.date_range("2012-01-01", periods=3, tz='UTC')) 
s2 = pd.Series(pd.date_range("2012-01-01", periods=3)) 
s3 = pd.Series([1, 2, 3])                                                                                                        

In [2]: %timeit s1.array                                                                                                                 
391 ns ± 3.96 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)  # <-- master
408 ns ± 17.7 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)  # <-- PR

In [3]: %timeit s2.array                                                                                                                 
1.1 µs ± 6.44 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)  # <-- master
514 ns ± 43 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)  # <-- PR

In [4]: %timeit s3.array                                                                                                                 
1.68 µs ± 16.8 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)  # <-- master
509 ns ± 4.19 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)  # <-- PR

In [5]: %timeit extract_array(s1, extract_numpy=True)                                                                                    
1.77 µs ± 35 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)  # <-- master
1.72 µs ± 37.3 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)  # <-- PR

In [6]: %timeit extract_array(s2, extract_numpy=True)                                                                                    
2.59 µs ± 38.2 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)  # <-- master
1.78 µs ± 14.6 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)  # <-- PR

In [7]: %timeit extract_array(s3, extract_numpy=True)                                                                                    
3.89 µs ± 123 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)  # <-- master
2.58 µs ± 27.6 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)  # <-- PR

In [8]: %timeit extract_array(s3, extract_numpy=False)                                                                                   
2.71 µs ± 72.4 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)  # <-- master
1.46 µs ± 16.1 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)  # <-- PR

[jorisvandenbossche]

Ah, I suppose the use of _simple_new was certainly part of the reason (I remember now that I changed to use that in the DatetimeBlock.array_values() at some point in the other PR)

[jorisvandenbossche]

It might be that now you added to use _simple_new for internal_values as well, you can again combine internal_values and array_values as you did before with less performance loss ?

[jbrockmendel]

OK, updated one more time, this time just changing DatetimeBlock/TimedeltaBlock.internal_values to return self.array_values(). This should have zero effect on array or extract_array, and local timeit results bear that out (not posting for brevity)

We can worry about optimizing array/extract_array further separately, as that should really be orthogonal to this.

[jorisvandenbossche]

A bit above (beginning of the docstring), the sentence "This are the values as stored in the Block" is no longer adequate I think?

[jorisvandenbossche]

The sentence I am quoting (the second line of the docstring) still needs to be updated

[jbrockmendel]

(btw, the comment "Disallow dtypes that have blocks backed by EAs" is not fully correct, as DatetimeBlock is still backed by ndarray no?)

[jreback]

lgtm needs a rebase

[jorisvandenbossche]

Thanks!