Skip to content

DOC: Add TextFileReader to docs #46308

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 8 commits into from
Closed

DOC: Add TextFileReader to docs #46308

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
1e91695
DOC: Add TextFileReader to docs
phofl Mar 10, 2022
d10d829
Fix docstring
phofl Mar 10, 2022
59ecfdf
Merge remote-tracking branch 'upstream/main' into 46187
phofl Jun 24, 2022
1e487ad
Add docstrings
phofl Jun 24, 2022
0192267
Update pandas/io/parsers/readers.py
datapythonista Jan 5, 2023
6db27ea
Merge branch 'main' into 46187
datapythonista Jan 5, 2023
c367d3d
Merge branch 'main' into 46187
phofl Feb 23, 2023
db74d99
Merge branch 'main' into 46187
datapythonista Apr 27, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 13 additions & 0 deletions doc/source/reference/io.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,19 @@ Flat file
DataFrame.to_csv
read_fwf

.. currentmodule:: pandas.io.parsers

.. autosummary::
:toctree: api/

TextFileReader

TextFileReader.get_chunk
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We obviously don't want to add these. But when I remove them, sphinx complains that it can not find them in any toctree. Anyone any ideas how to solve this? For ExcelWriter below this works, so I am probably missing something.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think in the past we used a section with :hidden:, like in https://github.com/pandas-dev/pandas/blame/main/doc/source/getting_started/index.rst#L641

But I don't see it being used for the API anymore. I guess we should just make them private by using _get_chunk... if we don't want them public and in the documentation.

Or maybe just _TextFileReader and make the whole class private if we don't want it being part of our public API.

I'm personally fine with any of them.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we could deprecate it and remove it later, that sounds good to me.

We can't makeTextFileReader private, since it is returned by read_csv if you are reading the file in chunks

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, true. Sounds good to me then. Also find to simply document the methods and publish them in the docs. Maybe we can just add a not for now that we recommend using the magic methods way.

TextFileReader.close
TextFileReader.read
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

close and read can probably be public but more important are the magic methods __enter__, __exit__, and __next__. Ideally, people interact with TextFileReader in this manner:

with pd.read_csv("test.csv", iterator=True) as reader:
    for chunk in reader:
        print(chunk)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would need to add a doc-string to read (and maybe also to close)


.. currentmodule:: pandas

Clipboard
~~~~~~~~~
.. autosummary::
Expand Down
3 changes: 2 additions & 1 deletion pandas/io/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,9 @@
from pandas.io import (
formats,
json,
parsers,
stata,
)

# and mark only those modules as public
__all__ = ["formats", "json", "stata"]
__all__ = ["formats", "json", "parsers", "stata"]
27 changes: 26 additions & 1 deletion pandas/io/parsers/readers.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1367,9 +1367,21 @@ def read_fwf(

class TextFileReader(abc.Iterator):
"""
Passed dialect overrides any of the related parser options.

Passed dialect overrides any of the related parser options
Iterator class used to process to text files read via read_csv in
chunks.

An instance of this class is returned by `read_csv` when it is processed in
chunks, instead of returning a single `DataFrame`.

When iterating over `TextFileReader`, every item returned will be a DataFrame.

Examples
---------
>>> with pandas.read_csv(..., iterator=True) as text_file_reader:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You need to import pandas or use pd which is imported by default to make the CI happy: https://github.com/pandas-dev/pandas/runs/7043600368?check_suite_focus=true#step:8:132

... for df in text_file_reader:
... ...
"""

def __init__(
Expand Down Expand Up @@ -1422,6 +1434,7 @@ def __init__(
self._engine = self._make_engine(f, self.engine)

def close(self) -> None:
"""Closes the file handle."""
if self.handles is not None:
self.handles.close()
self._engine.close()
Expand Down Expand Up @@ -1730,6 +1743,18 @@ def _failover_to_python(self) -> None:
raise AbstractMethodError(self)

def read(self, nrows: int | None = None) -> DataFrame:
"""
Reads the text file and stores the result in a DataFrame.

Parameters
----------
nrows: int, optional, default None
The number of rows to read in one go.

Returns
-------
DataFrame
"""
if self.engine == "pyarrow":
try:
# error: "ParserBase" has no attribute "read"
Expand Down