BUG: Formatting issues with stderr output of read_csv() with warn_bad_lines=True

[dericke]

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• (optional) I have confirmed this bug exists on the master branch of pandas.

---

Code Sample, a copy-pastable example

badly-formatted.csv

lname	fname	birthday	hometown
Deere	Jane	2003-01-02	Philadelphia, PA
Doe	John	2001-04-01	New York,	NY
Jefferson		Thomas	1743-04-13	Shadwell, VA
Lincoln	Abraham	1809-02-12	Sinking Spring, KY
Washington	George		1732-02-22	Pope's Creek, VA

(note extra tab characters)

main.py

import pandas

pandas.read_csv("badly-formatted.csv", sep="\t", error_bad_lines=False, warn_bad_lines=True)

Problem description

When using pandas.read_csv() with the error_bad_lines=False argument (and the default warn_bad_lines=True arg), the output being printed to stderr appears to be bytes type that has been cast to str. This adds noise to the console output, and is making it difficult to properly capture stderr output from this constructor.

Expected stderr Output from MRE

Skipping line 3: expected 4 fields, saw 5
Skipping line 4: expected 4 fields, saw 5
Skipping line 6: expected 4 fields, saw 5

Actual stderr Output from MRE

b'Skipping line 3: expected 4 fields, saw 5\nSkipping line 4: expected 4 fields, saw 5\nSkipping line 6: expected 4 fields, saw 5\n'

Inconsistent line breaks with larger files

I have also seen line breaks appear inconsistently when reading in larger files. I haven't been able to isolate any causes of this inconsistency yet

Actual stderr output from large file

b'Skipping line 342635: expected 25 fields, saw 26\n'
b'Skipping line 392116: expected 25 fields, saw 26\n'
b'Skipping line 544651: expected 25 fields, saw 26\nSkipping line 553773: expected 25 fields, saw 26\n'
b'Skipping line 559046: expected 25 fields, saw 26\nSkipping line 559146: expected 25 fields, saw 26\nSkipping line 559148: expected 25 fields, saw 26\nSkipping line 559155: expected 25 fields, saw 26\n'
b'Skipping line 596525: expected 25 fields, saw 26\n'
b'Skipping line 634470: expected 25 fields, saw 26\n'
b'Skipping line 777743: expected 25 fields, saw 26\nSkipping line 777744: expected 25 fields, saw 26\nSkipping line 777745: expected 25 fields, saw 26\nSkipping line 777746: expected 25 fields, saw 26\nSkipping line 777747: expected 25 fields, saw 26\nSkipping line 777748: expected 25 fields, saw 26\nSkipping line 777749: expected 25 fields, saw 26\nSkipping line 777750: expected 25 fields, saw 26\nSkipping line 777831: expected 25 fields, saw 26\nSkipping line 777832: expected 25 fields, saw 26\nSkipping line 778297: expected 25 fields, saw 26\n'

Expected stderr output from large file

Skipping line 342635: expected 25 fields, saw 26
Skipping line 392116: expected 25 fields, saw 26
Skipping line 544651: expected 25 fields, saw 26
Skipping line 553773: expected 25 fields, saw 26
Skipping line 559046: expected 25 fields, saw 26
Skipping line 559146: expected 25 fields, saw 26
Skipping line 559148: expected 25 fields, saw 26
Skipping line 559155: expected 25 fields, saw 26
Skipping line 596525: expected 25 fields, saw 26
Skipping line 634470: expected 25 fields, saw 26
Skipping line 777743: expected 25 fields, saw 26
Skipping line 777744: expected 25 fields, saw 26
Skipping line 777745: expected 25 fields, saw 26
Skipping line 777746: expected 25 fields, saw 26
Skipping line 777747: expected 25 fields, saw 26
Skipping line 777748: expected 25 fields, saw 26
Skipping line 777749: expected 25 fields, saw 26
Skipping line 777750: expected 25 fields, saw 26
Skipping line 777831: expected 25 fields, saw 26
Skipping line 777832: expected 25 fields, saw 26
Skipping line 778297: expected 25 fields, saw 26

Output of pd.show_versions()

INSTALLED VERSIONS

commit : 2cb9652
python : 3.9.5.final.0
python-bits : 64
OS : Darwin
OS-release : 18.7.0
Version : Darwin Kernel Version 18.7.0: Mon Mar 8 22:11:48 PST 2021; root:xnu-4903.278.65~1/RELEASE_X86_64
machine : x86_64
processor : i386
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8

pandas : 1.2.4
numpy : 1.19.4
pytz : 2020.5
dateutil : 2.8.1
pip : 21.1.2
setuptools : 56.0.0
Cython : None
pytest : 6.2.1
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : 4.6.2
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : None
IPython : None
pandas_datareader: None
bs4 : None
bottleneck : None
fsspec : None
fastparquet : None
gcsfs : None
matplotlib : 2.2.5
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pyxlsb : None
s3fs : None
scipy : 1.6.3
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
xlwt : None
numba : None

[lithomas1]

Thanks for the bug report @dericke. This indeed does look buggy and I can also reproduce this on master. PRs to fix are welcome.

[terryluan12]

I can take this issue.

[terryluan12]

take