ENH: Add a Parser to read fixed-width ASCII data using a data description file or dictionary file

[AllenDowney]

As an example, I would like to be able to read a dataset like this one:

http://www.cdc.gov/nchs/nsfg/nsfg_cycle6.htm

The data files themselves are ASCII with fixed-width fields. The variables names, types, and indices are in a separate data description file, available for SAS, SPSS and STATA. I would like to add a parser that reads at least one of these description files and then parses the data file. Since two files are used, it might require changes in the Parser API.

I am happy to write a parser that reads the dictionary file and then the data file. I could use help with either setting up the new parser ahead of time or (after the fact) integrating my code with the existing structure.

Also, is there a preference for the SAS, SPSS, or Stata format?

[jreback]

does pd.read_fwf(...) do a credible job with this (w/o the columns being specified and hence being inferred)?

[AllenDowney]

Yes, once I have parsed the dictionary, it looks like I will be able to use
this to parse the data file.

But for the data files I am working with I don't think it would be possible
to infer the breaks between columns.

Thanks!
Allen

On Mon, May 5, 2014 at 9:21 AM, jreback [email protected] wrote:

does pd.read_fwf(...) do a credible job with this (w/o the columns being
specified and hence being inferred)?

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/7030#issuecomment-42186589
.

[tyler-abbot]

@AllenDowney I am thinking of working on this project. Did you ever get anywhere with it? Or do you have any sample data that you would like to see treated by a function?

[AllenDowney]

Yes, I have a dataset and a hacky solution that might be a good example.
It's all in this repo

https://github.com/AllenDowney/MarriageNSFG

In thinkstats2.py, you'll see a function called ReadStataDct that returns a
FixedWidthVariables object that provides ReadFixedWidth

I am using it to read data from the NSFG; the data is also in the repo.
You can see an example in marriage.py, specifically the
function ReadFemResp.

Please let me know if I can help.

On Wed, Jun 24, 2015 at 9:00 AM, tyler-abbot [email protected]
wrote:

@AllenDowney https://github.com/AllenDowney I am thinking of working on
this project. Did you ever get anywhere with it? Or do you have any sample
data that you would like to see treated by a function?

—
Reply to this email directly or view it on GitHub
#7030 (comment).

[jbrockmendel]

Closing and adding to a tracker issue #30407 for IO format requests, can re-open if interest is expressed.