DOC: move info docs to DataFrameInfo #38062

ivanovmg · 2020-11-25T12:12:54Z

Precursor for #37320
Move info docstring from pandas/core/frame.py to class DataFrameInfo using doc decorator.

@simonjayhawkins need your help here as discussed previously.

jreback · 2021-01-01T21:48:27Z

@ivanovmg can you merge master

github-actions · 2021-02-10T00:12:41Z

This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.

jreback · 2021-02-11T01:29:32Z

@ivanovmg if you can merge master

ivanovmg · 2021-02-11T11:57:44Z

Merged.
@simonjayhawkins I need help on creating a shared docstring for DataFrame.info() and Series.info().
Currently this implementation fails as apparently I do not use doc properly.
Can you please help?

datapythonista

The logs are expired and I can't see them. If the comments don't fix the issue you mention, I'll have a look after the CI reruns.

datapythonista · 2021-06-02T20:58:23Z

pandas/io/formats/info.py

+)
+
+
+frame_subs = {


Why do you do this? This could be simply specified in the decorator, doesn't seem to add value.

So, instead of:

@doc(BaseInfo.render, **frame_subs)

You can use:

@doc(BaseInfo.render, klass='DataFrame', type_sub='and colmns', ...)

datapythonista · 2021-06-02T20:59:18Z

pandas/io/formats/info.py

@@ -174,26 +302,26 @@ def render(
        show_counts: Optional[bool],
    ) -> None:
        """
-        Print a concise summary of a %(klass)s.
+        Print a concise summary of a {klass}.


If you have variables in this docstring, this render should also have the @doc decorator, with the values of the variables here, no?

Thank you for the comment!
I tried that, but the problem is that in future I would like to share the docstring with Series.info().
If we put @doc with params pertaining to dataframes onto BaseInfo.render, then how would we handle the same for series?

The @doc decorator does some magic. Basically it renders the docstring (template) with the provided variables, and it stores it as the __doc__, so that's the docstring of the function/class. But also, it saves the original docstring before rendering the variables in something like __doc_original__ (forgot the actual name), so if you reuse that docstring somewhere else with different variables, it'll work as "expected".

ivanovmg · 2021-06-03T11:26:50Z

@datapythonista can you please review the workaround I have just commited?

datapythonista · 2021-06-03T16:53:08Z

My understanding here is that we have a docstring in an object, and we want to reuse in another object too. With some small changes. This is done by:

@doc(name='render1')
def render1():
    """
    Docstring or {name}.
    """

@doc(render1, name='render2')
def render2():
    pass

With Appender and Supstitution, it made more sense to have docstrings in variables. here you write it normally in one object, and then pass that object name for doc for the other object to know where to get the base docstring for.

Then, if it makes things simpler, change what's being reused. In many cases it's preferrable to duplicate a bit of the docstring, or make a part of it not so generic, that have huge blocks of codes in the variables.

The important part is that once you finished the changes, you run validate_docstrings.py for your objects, check that things look good, that the variables are rendering well and with the values corresponding to the right object. And ideally, you can also build the html for the docs locally, and check that things look good.

ivanovmg · 2021-06-04T06:50:59Z

My original intention was to do the following.

from abc import ABC, abstractmethod

from pandas.util._decorators import doc


class BaseInfo(ABC):
    @doc(klass='Base')
    @abstractmethod
    def render(self):
        """Info for {klass}"""


class DataFrameInfo(BaseInfo):
    @doc(BaseInfo.render, klass='DataFrame')
    def render(self):
        pass


class SeriesInfo(BaseInfo):
    @doc(BaseInfo.render, klass='Series')
    def render(self):
        pass


class DataFrame:
    @doc(DataFrameInfo.render)
    def info(self):
        return DataFrameInfo().render()


if __name__ == '__main__':
    print(f'BaseInfo.render(): {BaseInfo.render.__doc__}')
    print(f'DataFrameInfo.render(): {DataFrameInfo.render.__doc__}')
    print(f'SeriesInfo.render(): {SeriesInfo.render.__doc__}')
    print(f'DataFrame.info(): {DataFrame.info.__doc__}')

However, the problem is that when executing this script we get the following error:

Traceback (most recent call last):
  File "pandas\temp_doc.py", line 26, in <module>
    class DataFrame:
  File "pandas\temp_doc.py", line 28, in DataFrame
    def info(self):
  File "c:\users\mivanov4\code\pandas\pandas\util\_decorators.py", line 387, in decorator
    decorated.__doc__ = "".join(
  File "c:\users\mivanov4\code\pandas\pandas\util\_decorators.py", line 388, in <genexpr>
    component.format(**params)
KeyError: 'klass'

I guess the problem is that doc decorator does not support such multiple redirections.

Therefore, I extracted a constant INFO_DOCSTRING and formatted it.

======================
In comparison:

from abc import ABC, abstractmethod

from pandas.util._decorators import doc

INFO_DOCSTRING = """Info for {klass} in the constant"""

class BaseInfo(ABC):
    @doc(klass='Base')
    @abstractmethod
    def render(self):
        """Info for {klass} in the base class"""


class DataFrameInfo(BaseInfo):
    @doc(BaseInfo.render, klass='DataFrame')
    def render(self):
        pass


class SeriesInfo(BaseInfo):
    @doc(BaseInfo.render, klass='Series')
    def render(self):
        pass


class DataFrame:
    @doc(INFO_DOCSTRING, klass='DataFrame')
    def info(self):
        return DataFrameInfo().render()


if __name__ == '__main__':
    print(f'BaseInfo.render(): {BaseInfo.render.__doc__}')
    print(f'DataFrameInfo.render(): {DataFrameInfo.render.__doc__}')
    print(f'SeriesInfo.render(): {SeriesInfo.render.__doc__}')
    print(f'DataFrame.info(): {DataFrame.info.__doc__}')

Gives output:

BaseInfo.render(): Info for Base in the base class
DataFrameInfo.render(): Info for DataFrame in the base class
SeriesInfo.render(): Info for Series in the base class
DataFrame.info(): Info for DataFrame in the constant

So, docstrings for DataFrameInfo and SeriesInfo work fine, as expected.
But additional redirection from DataFrame.info() was not possible due to originally empty __doc__ for DataFrameInfo and SeriesInfo, I suppose.

Linking previous comment on the docstrings: #37320 (review) (the reason the docstrings are to be generic for both DataFrame and Series and done via doc decorator).

datapythonista · 2021-06-04T09:29:09Z

This is cumbersome, better to avoid even if it was supported. If it's the same docstring make all them be based in the same base one. Also, are all these public? If they are not rendered in the docs, we probably don't care much on having tge docstring.

ivanovmg · 2021-06-04T11:02:50Z

This is cumbersome, better to avoid even if it was supported. If it's the same docstring make all them be based in the same base one. Also, are all these public? If they are not rendered in the docs, we probably don't care much on having tge docstring.

Only DataFrame.info() is public.
Just to clarify, do you consider the current approach (committed) as cumbersome?

Another option, as you already mentioned above, is to simply have different docstrings for DataFrame.info and Series.info, but looks like @jreback preferred a more generic approach.

datapythonista · 2021-06-04T12:04:04Z

Sorry, what I meant that is cumbersome is your example, where doc is being used to copy one docstring, and then used again to copy the copy.

Not so familiar with those DataFrameInfo class, but if the mehods you are documenting are not public, having those docstrings it's not very useful. Docstrings are useful for two things, for developers reading the code, and for generating the docs as html. With @doc, we ruin the former, so we don't need to repeat long docstrings rendered in the documentation. So, if a method is not public, in my opinion is as useful to have a doctring """Look this other object, does the same.""" as using @doc.

ivanovmg · 2021-06-04T12:27:44Z

Right, there is no need to have docstrings inside the internal classes.
But the idea was to generate docstrings for DataFrame.info() and Series.info() (to be added) from within pandas/io/formats/info.py.

jbrockmendel · 2021-06-22T16:43:22Z

pandas/io/formats/info.py

@@ -284,6 +425,17 @@ def memory_usage_bytes(self) -> int:
            deep = False
        return self.data.memory_usage(index=True, deep=deep).sum()

+    @doc(
+        INFO_DOCSTRING,


can you keep INFO_DOCSTRING where it is and reference it here as BaseInfo.render.__doc__?

I have just tried that, but it messes with the indentation for some reason

This reverts commit d326198.

jreback · 2021-10-04T00:21:19Z

@ivanovmg in theory this is a good change, if you want to merge master. ideally small changes in a series of PRs are better.

jreback · 2021-11-28T21:11:03Z

@ivanovmg can you merge master again pls. and ping when green.

ivanovmg · 2021-11-29T08:03:10Z

@jreback, I merged master.
It is green.

jreback · 2021-11-29T14:47:03Z

thanks @ivanovmg

DOC: move info docs to DataFrameInfo

5151e33

jreback added the Docs label Nov 26, 2020

ivanovmg requested a review from simonjayhawkins December 2, 2020 18:34

Merge branch 'master' into doc/info

53a47cd

Merge branch 'master' into doc/info

ffce3d5

github-actions bot added the Stale label Feb 10, 2021

Merge branch 'master' into doc/info

b632c6c

mroeschke mentioned this pull request Mar 10, 2021

ENH: enable Series.info() #37320

Merged

5 tasks

datapythonista reviewed Jun 2, 2021

View reviewed changes

ivanovmg added 3 commits June 3, 2021 11:00

Merge branch 'master' into doc/info

642a308

Drop variable frame_subs

751143c

Extract INFO_DOCSTRING to fix doc decorator

7e21334

ivanovmg force-pushed the doc/info branch from 8ceb0e1 to 7e21334 Compare June 3, 2021 10:33

ivanovmg marked this pull request as ready for review June 3, 2021 11:25

ivanovmg requested a review from datapythonista June 3, 2021 11:27

simonjayhawkins removed the Stale label Jun 8, 2021

jbrockmendel reviewed Jun 22, 2021

View reviewed changes

ivanovmg added 2 commits June 23, 2021 00:35

Move INFO_DOCSTRING back to BaseInfo and fix reference

d326198

Revert "Move INFO_DOCSTRING back to BaseInfo and fix reference"

05efde0

This reverts commit d326198.

ivanovmg added 2 commits June 23, 2021 00:56

Handle merge conflict missed: null_counts and show_counts

a686a22

Merge branch 'master' into doc/info

f1b3371

Merge branch 'master' into doc/info

6041cd9

ivanovmg added 2 commits November 29, 2021 11:37

Merge branch 'master' into doc/info

013ee3b

FIX: newline in docstring

5814bc9

jreback added this to the 1.4 milestone Nov 29, 2021

jreback merged commit c448ec7 into pandas-dev:master Nov 29, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

DOC: move info docs to DataFrameInfo #38062

DOC: move info docs to DataFrameInfo #38062

ivanovmg commented Nov 25, 2020 •

edited

Loading

jreback commented Jan 1, 2021

github-actions bot commented Feb 10, 2021

jreback commented Feb 11, 2021

ivanovmg commented Feb 11, 2021

datapythonista left a comment

datapythonista Jun 2, 2021

datapythonista Jun 2, 2021

ivanovmg Jun 3, 2021

datapythonista Jun 3, 2021

ivanovmg commented Jun 3, 2021

datapythonista commented Jun 3, 2021

ivanovmg commented Jun 4, 2021 •

edited

Loading

datapythonista commented Jun 4, 2021

ivanovmg commented Jun 4, 2021

datapythonista commented Jun 4, 2021

ivanovmg commented Jun 4, 2021

jbrockmendel Jun 22, 2021

ivanovmg Jun 22, 2021

jreback commented Oct 4, 2021

jreback commented Nov 28, 2021

ivanovmg commented Nov 29, 2021

jreback commented Nov 29, 2021

		)


		frame_subs = {

DOC: move info docs to DataFrameInfo #38062

DOC: move info docs to DataFrameInfo #38062

Conversation

ivanovmg commented Nov 25, 2020 • edited Loading

jreback commented Jan 1, 2021

github-actions bot commented Feb 10, 2021

jreback commented Feb 11, 2021

ivanovmg commented Feb 11, 2021

datapythonista left a comment

Choose a reason for hiding this comment

datapythonista Jun 2, 2021

Choose a reason for hiding this comment

datapythonista Jun 2, 2021

Choose a reason for hiding this comment

ivanovmg Jun 3, 2021

Choose a reason for hiding this comment

datapythonista Jun 3, 2021

Choose a reason for hiding this comment

ivanovmg commented Jun 3, 2021

datapythonista commented Jun 3, 2021

ivanovmg commented Jun 4, 2021 • edited Loading

datapythonista commented Jun 4, 2021

ivanovmg commented Jun 4, 2021

datapythonista commented Jun 4, 2021

ivanovmg commented Jun 4, 2021

jbrockmendel Jun 22, 2021

Choose a reason for hiding this comment

ivanovmg Jun 22, 2021

Choose a reason for hiding this comment

jreback commented Oct 4, 2021

jreback commented Nov 28, 2021

ivanovmg commented Nov 29, 2021

jreback commented Nov 29, 2021

ivanovmg commented Nov 25, 2020 •

edited

Loading

ivanovmg commented Jun 4, 2021 •

edited

Loading