Skip to content

DOC: update the pandas.DataFrame.clip docstring #20212

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 25 commits into from
Closed
Changes from 9 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
110 changes: 78 additions & 32 deletions pandas/core/generic.py
Original file line number Diff line number Diff line change
Expand Up @@ -5601,53 +5601,99 @@ def clip(self, lower=None, upper=None, axis=None, inplace=False,
"""
Trim values at input threshold(s).

Elements above the upper threshold will be changed to upper threshold.
Elements below the lower threshold will be changed to lower threshold.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems duplicated.


Parameters
----------
lower : float or array_like, default None
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not sure if it's better or worse, but I think the standard we defined is to use float, array-like or None, default None (so the None is duplicated). Also note the hyphen and not underscore in array-like

Lower threshold for clipping. Values smaller than upper will be
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For naming parameters it's better to have them in backticks. In this case I think it adds value, making it clearer that upper is a parameter (and lower).

converted to lower.
upper : float or array_like, default None
Upper threshold for clipping. Values larger than upper will be
converted to upper.
axis : int or string axis name, optional
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We added a standard for axis to the documentation, at the end of the Paramters section. Basically it's axis : {0 or 'index', 1 or 'columns', None}, default None

Align object with lower and upper along the given axis.
inplace : boolean, default False
Whether to perform the operation in place on the data
.. versionadded:: 0.21.0
.. versionadded:: 0.21.0.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I find the description of axis a bit complex.

args : dictionary of arguments arguments passed to pandas.compat.numpy
kwargs : dictionary of keyword arguments passed to pandas.compat.numpy
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

*args and **kwargs with the stars (even if the validate script complaints), and without the type (the description part goes in the nest line)


Returns
-------
clipped : Series
clipped : DataFrame/Series
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Series or DataFrame

Elements above or below the upper and lower thresholds converted to
threshold values.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can just have the type in the first row of Returns, providing a name doesn't add much value.

The description sounds a bit like if we could be returning only part of the original values.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure thing.


Notes
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it should be in References and not in Notes. It wasn't in the documentation for the sprint, for simplicity, but you can check this document: http://numpydoc.readthedocs.io/en/latest/format.html

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yea, I agree. Done.

-----
Clipping data is a method for dealing with dubious elements.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

prefer, out-of-range to dubious

If some elements are too large or too small, clipping is one way to
transform the data into a reasonable range.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This sounds more like part of the extended summary to me, than Notes, which is usually left for details on the implementaiton (e.g. calling this function makes a copy of the data)

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should I just remove it then?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's an interesting comment. May be you can also mention about the outlier thing you show in the example. But I'd have it in the extended summary. After moving it, make sure the whole summary makes sense and doesn't sound repetitive. Usually happens when you move blocks.


See Also
--------
pandas.DataFrame.clip_upper : Return copy of input with values
above given value(s) truncated.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add Series.clip

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is in generic, isn't it being reused by Series.clip?

pandas.DataFrame.clip_lower : Return copy of input with values
below given value(s) truncated.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the prefix pandas is not needed, just Series.clip. Also, not sure if that was already discussed, but don't we want clip_lower and clip_upper here?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you previously mentioned that those were generic. Did we want them included still?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry, I think I wasn't clear enough. You should check it, but as this docstring is in generic.py I assume it'll be used by both Series and DataFrame. That's why I said it was (the docstring) was generic.

So, as your assigned docstring was DataFrame.clip (and it's what it's in the title of the PR) but you're actually working also in Series.clip, I don't think it makes sense to only add Series.clip as @jreback suggested. I'd say that we probably want also DataFrame.clip. Even if it'll be a bit weird to have in the See Also the same method which is being documented, but I think it's all right. In a separate PR we could use a template and just have the right one.

So, summarizing, the see also should contain Series.clip, DataFrame.clip, and the clip_lower and clip_upper for each of them (I assume both classes have it). And assuming that clip is commonly used with quantile as in your example (I don't know), I would also add it.

@jreback do you agree?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, yes I see, I was mistaken. Sure, I can add all those.

Examples
--------
>>> df=pd.DataFrame({'a':[1, 2, 3], 'b':[4, 5, 6], 'c':[7, 8, 9001]})
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This have several PEP-8 issues because of missing spaces

>>> df
0 1
0 0.335232 -1.256177
1 -1.367855 0.746646
2 0.027753 -1.176076
3 0.230930 -0.679613
4 1.261967 0.570967

>>> df.clip(-1.0, 0.5)
0 1
0 0.335232 -1.000000
1 -1.000000 0.500000
2 0.027753 -1.000000
3 0.230930 -0.679613
4 0.500000 0.500000

>>> t
0 -0.3
1 -0.2
2 -0.1
3 0.0
4 0.1
dtype: float64

>>> df.clip(t, t + 1, axis=0)
0 1
0 0.335232 -0.300000
1 -0.200000 0.746646
2 0.027753 -0.100000
3 0.230930 0.000000
4 1.100000 0.570967
a b c
0 1 4 7
1 2 5 8
2 3 6 9001

>>> df.clip(lower=1, upper=9)
a b c
0 1 4 7
1 2 5 8
2 3 6 9

You can clip each column or row with different thresholds by passing
a ``Series`` to the lower/upper argument.

>>> some_data={'A':[-19, 12, -5],'B':[1, 100, -5]}
>>> df=pd.DataFrame(data=some_data, index=['foo', 'bar', 'bizz'])
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

more PEP-8

>>> df
A B
foo -19 1
bar 12 100
bizz -5 -5

Use the axis argument to clip by column or rows. Clip column A with
lower threshold of -10 and column B has lower threshold of 10.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

two spaces after the dot?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep, bad habit.


>>> df.clip(lower=pd.Series({'A':-10, 'B':10}), axis=1)
A B
foo -10 10
bar 12 100
bizz -5 10

Clip the foo, bar, and bizz rows with lower thresholds -10, 0, and 10.

>>> row_thresh=pd.Series({'foo':-10, 'bar':0, 'bizz':10})
>>> df.clip(lower=row_thresh, axis=0)
A B
foo -10 1
bar 12 100
bizz 10 10

`Winsorizing <https://en.wikipedia.org/wiki/Winsorizing>`__ is a way
of removing outliers from data. Columns of a DataFrame can be
winsorized by using clip.

>>> import numpy as np
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

don't need the numpy import

>>> x=np.random.normal(size=(1000,3))
>>> df=pd.DataFrame(x, columns=['a','b','c'])
>>> #Winsorize columns at 5% and 95%
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you can add this as part of the text above

>>> U=df.quantile(0.95)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

spaces around equals

>>> L=df.quantile(0.5)
>>> winsorized_df=df.clip(lower=L, upper=U, axis = 1)
"""
if isinstance(self, ABCPanel):
raise NotImplementedError("clip is not supported yet for panels")
Expand Down