Skip to content

BUG: read_fwf doesn't allow names with colspec='infer' #45337

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
2 of 3 tasks
dopplershift opened this issue Jan 12, 2022 · 5 comments · Fixed by #45522
Closed
2 of 3 tasks

BUG: read_fwf doesn't allow names with colspec='infer' #45337

dopplershift opened this issue Jan 12, 2022 · 5 comments · Fixed by #45522
Labels
Bug IO Fixed Width read_fwf Regression Functionality that used to work in a prior pandas version
Milestone

Comments

@dopplershift
Copy link

dopplershift commented Jan 12, 2022

Pandas version checks

  • I have checked that this issue has not already been reported.

  • I have confirmed this bug exists on the latest version of pandas.

  • I have confirmed this bug exists on the master branch of pandas.

Reproducible Example

from io import StringIO
import pandas as pd

test_data = StringIO('''-----------------------------------------------------------------------------
   PRES   HGHT   TEMP   DWPT   RELH   MIXR   DRCT   SKNT   THTA   THTE   THTV
    hPa     m      C      C      %    g/kg    deg   knot     K      K      K 
-----------------------------------------------------------------------------
 1000.0     -7                                                               
  959.0    345   22.2   19.0     82  14.64    160     18  298.9  341.8  301.5
  931.3    610   20.2   17.5     84  13.66    165     40  299.4  339.5  301.9
  925.0    671   19.8   17.1     84  13.44    165     38  299.6  339.0  302.0
  899.3    914   18.4   16.9     91  13.63    175     39  300.5  340.7  303.0
''')

col_names = ['pressure', 'height', 'temperature', 'dewpoint', 'direction', 'speed']
df = pd.read_fwf(test_data,
                 skiprows=5, usecols=[0, 1, 2, 3, 6, 7], names=col_names)

Issue Description

On pandas 1.4.0rc0, that code produces:

Traceback (most recent call last):
  File "/Users/rmay/repos/metpy/examples/Advanced_Sounding.py", line 30, in <module>
    df = pd.read_fwf(get_test_data('may4_sounding.txt', as_file_obj=False),
  File "/Users/rmay/miniconda3/envs/py310/lib/python3.10/site-packages/pandas/util/_decorators.py", line 311, in wrapper
    return func(*args, **kwargs)
  File "/Users/rmay/miniconda3/envs/py310/lib/python3.10/site-packages/pandas/io/parsers/readers.py", line 854, in read_fwf
    raise ValueError("Length of colspecs must match length of names")
ValueError: Length of colspecs must match length of names

It seems the problem is that it's checking names while colspecs is still set to 'infer':

if len(names) != len(colspecs):

I don't see a quick fix since the handling of "infer" appears to be buried somewhere else.

Expected Behavior

This code ran fine in Pandas <1.4rc0. Looks like the check was added in #40830.

Installed Versions

INSTALLED VERSIONS

commit : d023ba7
python : 3.10.1.final.0
python-bits : 64
OS : Darwin
OS-release : 21.2.0
Version : Darwin Kernel Version 21.2.0: Sun Nov 28 20:28:54 PST 2021; root:xnu-8019.61.5~1/RELEASE_X86_64
machine : x86_64
processor : i386
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8

pandas : 1.4.0rc0
numpy : 1.22.0
pytz : 2021.3
dateutil : 2.8.2
pip : 21.3.1
setuptools : 59.7.0
Cython : None
pytest : 6.2.5
hypothesis : None
sphinx : 4.3.2
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : 3.0.3
IPython : 7.31.0
pandas_datareader: None
bs4 : 4.10.0
bottleneck : None
fsspec : 2021.11.1
fastparquet : None
gcsfs : None
matplotlib : 3.5.1
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pyxlsb : None
s3fs : 2021.11.1
scipy : 1.7.3
sqlalchemy : None
tables : None
tabulate : None
xarray : 0.20.2
xlrd : None
xlwt : None
numba : None
zstandard : None

@dopplershift dopplershift added Bug Needs Triage Issue that has not been reviewed by a pandas team member labels Jan 12, 2022
@mroeschke
Copy link
Member

Could you share a sample input of the data or a StringIO version that could be used for a unit test if the issue is confirmed?

@mroeschke mroeschke added IO Fixed Width read_fwf Needs Info Clarification about behavior needed to assess issue and removed Needs Triage Issue that has not been reviewed by a pandas team member labels Jan 14, 2022
@dopplershift
Copy link
Author

What's needed beyond what I put under

Here's a sample of the data file:

in the original report?

@mroeschke
Copy link
Member

Ideally it helps if the example is copy-pastable such that it helps with investigating the potential bug: https://matthewrocklin.com/blog/work/2018/02/28/minimal-bug-reports

Additionally, col_names is not defined in the example.

@dopplershift
Copy link
Author

I've edited the original code to include the sample data in a StringIO and added the missing col_names (oops 🐑 ). This runs and produces the error for me in a clean jupyter notebook.

@mroeschke
Copy link
Member

mroeschke commented Jan 18, 2022

Great, thank you. I can fully replicate the issue now.

@mroeschke mroeschke removed the Needs Info Clarification about behavior needed to assess issue label Jan 18, 2022
@phofl phofl added the Regression Functionality that used to work in a prior pandas version label Jan 21, 2022
@jorisvandenbossche jorisvandenbossche added this to the 1.4 milestone Jan 21, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Bug IO Fixed Width read_fwf Regression Functionality that used to work in a prior pandas version
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants