Skip to content

DOC: update the pandas.DataFrame.clip docstring #20212

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 25 commits into from
Closed
Changes from 17 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
102 changes: 66 additions & 36 deletions pandas/core/generic.py
Original file line number Diff line number Diff line change
Expand Up @@ -5601,53 +5601,83 @@ def clip(self, lower=None, upper=None, axis=None, inplace=False,
"""
Trim values at input threshold(s).

Elements above/below the upper/lower thresholds will be changed to
upper/lower thresholds.

Parameters
----------
lower : float or array_like, default None
upper : float or array_like, default None
axis : int or string axis name, optional
Align object with lower and upper along the given axis.
lower : float, array-like or None, default None
Lower threshold for clipping. Values smaller than `lower` will be
converted to `lower`.
upper : float, array-like or None, default None
Upper threshold for clipping. Values larger than `upper` will be
converted to `upper`.
axis : {0 or 'index', 1 or 'columns', None}, default None
Apply clip by index (i.e. by rows) or columns.
inplace : boolean, default False
Whether to perform the operation in place on the data
.. versionadded:: 0.21.0
.. versionadded:: 0.21.0.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I find the description of axis a bit complex.

*args : Additional keywords have no effect but might be accepted
for compatibility with numpy.
**kwargs : Additional keywords have no effect but might be accepted
for compatibility with numpy.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It was some change to the convention during the sprint, and args and kwargs should be in the same line *args, **kwargs and a common description in the next line. We didn't find any case where args and kwargs have different description, and it's repetitive the way it was originally defined.

Also, note that *args are positional arguments, I don't think it's correct to name they keywords.


Returns
-------
clipped : Series
`Series` or `DataFrame`.
DataFrame is returned with those values above/below the
`upper`/`'lower` thresholds set to the threshold values.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If return can be Series, the description is not accurate, saying the DataFrame is returned. May be something like "Original input with values above/below..."?


See Also
--------
pandas.Series.clip : Trim values at input threshold(s).

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the prefix pandas is not needed, just Series.clip. Also, not sure if that was already discussed, but don't we want clip_lower and clip_upper here?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you previously mentioned that those were generic. Did we want them included still?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry, I think I wasn't clear enough. You should check it, but as this docstring is in generic.py I assume it'll be used by both Series and DataFrame. That's why I said it was (the docstring) was generic.

So, as your assigned docstring was DataFrame.clip (and it's what it's in the title of the PR) but you're actually working also in Series.clip, I don't think it makes sense to only add Series.clip as @jreback suggested. I'd say that we probably want also DataFrame.clip. Even if it'll be a bit weird to have in the See Also the same method which is being documented, but I think it's all right. In a separate PR we could use a template and just have the right one.

So, summarizing, the see also should contain Series.clip, DataFrame.clip, and the clip_lower and clip_upper for each of them (I assume both classes have it). And assuming that clip is commonly used with quantile as in your example (I don't know), I would also add it.

@jreback do you agree?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, yes I see, I was mistaken. Sure, I can add all those.

Examples
--------
>>> some_data = {'a': [1, 2, 3], 'b': [4, 5, 6], 'c': [7, 8, 9001]}
>>> df = pd.DataFrame(some_data, index = ['foo', 'bar', 'foobar'])
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The spaces around = are not PEP-8 compliant. Sorry the validation script doesn't check it yet.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PEP-8 issue with spaces around =.

Also, as you need to change this, I'd directly have the dictionary in the constructor, and avoid creating some_data. Just for the sake of consistency, as all examples I've seen use this way. With one key of the dictionary in one line, and the index in a third one, I think line width should be a problem.

>>> df
0 1
0 0.335232 -1.256177
1 -1.367855 0.746646
2 0.027753 -1.176076
3 0.230930 -0.679613
4 1.261967 0.570967

>>> df.clip(-1.0, 0.5)
0 1
0 0.335232 -1.000000
1 -1.000000 0.500000
2 0.027753 -1.000000
3 0.230930 -0.679613
4 0.500000 0.500000

>>> t
0 -0.3
1 -0.2
2 -0.1
3 0.0
4 0.1
dtype: float64

>>> df.clip(t, t + 1, axis=0)
0 1
0 0.335232 -0.300000
1 -0.200000 0.746646
2 0.027753 -0.100000
3 0.230930 0.000000
4 1.100000 0.570967
a b c
foo 1 4 7
bar 2 5 8
foobar 3 6 9001

>>> df.clip(lower=1, upper=9)
a b c
foo 1 4 7
bar 2 5 8
foobar 3 6 9
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this example could be more concise, and help people understand the concept faster.

I think 2 columns is enough, have 3 doesn't add much value, but takes some extra time to check what's going on with the example.

Also, the lower trimming by 1 not having effect, makes me a bit confused, on whether I'm understanding it or not. Also it's somethimg really minor, but I found the 9001 unnecessarily big and arbitrary (using 100 wouldn't make me thing there is any special on it..

Feel free to use the example you want, but using something like -1, -2, -100, 1, 2, 100 and trimming at -10 and 10 would make it more clear and straight to the point.


You can clip each column or row with different thresholds by passing
a ``Series`` to the lower/upper argument. Use the axis argument to clip
by column or rows.

>>> col_thresh = pd.Series({'a':4, 'b':5, 'c':6})
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

spaces after colons for PEP-8

>>> df.clip(lower=col_thresh, axis='columns')
a b c
foo 4 5 7
bar 4 5 8
foobar 4 6 9001

Clip the foo, bar, and foobar rows with lower thresholds 5, 7, and 10.

>>> row_thresh = pd.Series({'foo': 5, 'bar': 7, 'foobar': 10})
>>> df.clip(lower=row_thresh, axis='index')
a b c
foo 5 5 7
bar 7 7 8
foobar 10 10 9001

Clipping data is a method for dealing with out-of-range elements.
If some elements are too large or too small, clipping is one way to
transform the data into a reasonable range.
`Winsorizing <https://en.wikipedia.org/wiki/Winsorizing>`__ is a
related method, whereby the data are clipped at
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

put this reference in the Notes section

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you want me to move the entire part about winsorization or just the link?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just the link, you can see the format in the 13. References section of this document: http://numpydoc.readthedocs.io/en/latest/format.html

the 5th and 95th percentiles.

>>> lwr_thresh = df.quantile(0.05)
>>> upr_thresh = df.quantile(0.95)
>>> dfw = df.clip(lower=lwr_thresh, upper=upr_thresh, axis='columns')
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

as you're using df now, I think you should show the output, instead of saving it to a variable.

Also, it's more a personal taste, feel free to ignore the commend, but these abbreviations feel a bit cryptic. lower_threshold would be better, but I think just lower it's probably even better.

And just an idea, in case you like it (it's how I'd write it and find more readable, but feel free to leave it as it):

>>> lower, upper = df.quantile(0.05), df.quantile(0.95)

"""
if isinstance(self, ABCPanel):
raise NotImplementedError("clip is not supported yet for panels")
Expand Down