Skip to content

PERF: quantile now operates per block boosting perf / fix quantile with nan #13122

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 1 commit into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 12 additions & 1 deletion asv_bench/benchmarks/frame_methods.py
Original file line number Diff line number Diff line change
Expand Up @@ -423,7 +423,7 @@ class frame_get_dtype_counts(object):
goal_time = 0.2

def setup(self):
self.df = pandas.DataFrame(np.random.randn(10, 10000))
self.df = DataFrame(np.random.randn(10, 10000))

def time_frame_get_dtype_counts(self):
self.df.get_dtype_counts()
Expand Down Expand Up @@ -985,3 +985,14 @@ def setup(self):

def time_series_string_vector_slice(self):
self.s.str[:5]


class frame_quantile_axis1(object):
goal_time = 0.2

def setup(self):
self.df = DataFrame(np.random.randn(1000, 3),
columns=list('ABC'))

def time_frame_quantile_axis1(self):
self.df.quantile([0.1, 0.5], axis=1)
3 changes: 2 additions & 1 deletion codecov.yml
Original file line number Diff line number Diff line change
Expand Up @@ -9,4 +9,5 @@ coverage:
branches: null
changes:
default:
branches: null
branches:
- master
1 change: 0 additions & 1 deletion doc/source/whatsnew/v0.18.1.txt
Original file line number Diff line number Diff line change
Expand Up @@ -563,7 +563,6 @@ Performance Improvements
- Improved speed of SAS reader (:issue:`12656`, :issue:`12961`)
- Performance improvements in ``.groupby(..).cumcount()`` (:issue:`11039`)
- Improved memory usage in ``pd.read_csv()`` when using ``skiprows=an_integer`` (:issue:`13005`)

- Improved performance of ``DataFrame.to_sql`` when checking case sensitivity for tables. Now only checks if table has been created correctly when table name is not lower case. (:issue:`12876`)
- Improved performance of ``Period`` construction and time series plotting (:issue:`12903`, :issue:`11831`).
- Improved performance of ``.str.encode()`` and ``.str.decode()`` methods (:issue:`13008`)
Expand Down
4 changes: 4 additions & 0 deletions doc/source/whatsnew/v0.18.2.txt
Original file line number Diff line number Diff line change
Expand Up @@ -97,6 +97,9 @@ Performance Improvements

- Improved performance of sparse ``IntIndex.intersect`` (:issue:`13082`)
- Improved performance of sparse arithmetic with ``BlockIndex`` when the number of blocks are large, though recommended to use ``IntIndex`` in such cases (:issue:`13082`)
- increased performance of ``DataFrame.quantile()`` as it now operates per-block (:issue:`11623`)





Expand All @@ -110,6 +113,7 @@ Bug Fixes



- Regression in ``Series.quantile`` with nans (:issue:`13098`)



Expand Down
34 changes: 15 additions & 19 deletions pandas/core/frame.py
Original file line number Diff line number Diff line change
Expand Up @@ -4989,31 +4989,27 @@ def quantile(self, q=0.5, axis=0, numeric_only=True,
0.5 2.5 55.0
"""
self._check_percentile(q)
if not com.is_list_like(q):
q = [q]
squeeze = True
else:
squeeze = False

data = self._get_numeric_data() if numeric_only else self
axis = self._get_axis_number(axis)
is_transposed = axis == 1

def _quantile(series):
res = series.quantile(q, interpolation=interpolation)
return series.name, res

if axis == 1:
if is_transposed:
data = data.T

# unable to use DataFrame.apply, becasuse data may be empty
result = dict(_quantile(s) for (_, s) in data.iteritems())
result = self._constructor(result, columns=data.columns)
if squeeze:
if result.shape == (1, 1):
result = result.T.iloc[:, 0] # don't want scalar
else:
result = result.T.squeeze()
result.name = None # For groupby, so it can set an index name
result = data._data.quantile(qs=q,
axis=1,
interpolation=interpolation,
transposed=is_transposed)

if result.ndim == 2:
result = self._constructor(result)
else:
result = self._constructor_sliced(result, name=q)

if is_transposed:
result = result.T

return result

def to_timestamp(self, freq=None, how='start', axis=0, copy=True):
Expand Down
Loading