DOC: Adding methods for "bad" lines that preserve all data #17385
Conversation
joshjacobson
commented
Aug 30, 2017
joshjacobson
commented
- closes [Feature Request] On import, allow option for number of fields to match widest row of data (filling missing with NaN) #17319
| @@ -1175,6 +1175,80 @@ data that appear in some lines but not others: | |||
| 0 1 2 3 | |||
| 1 4 5 6 | |||
| 2 8 9 10 | |||
|
|
|||
| Handling "bad" lines - preserving the data | |||
There was a problem hiding this comment.
Not sure if you need this. I think one giant section will work.
There was a problem hiding this comment.
If we eliminate the open workaround, I agree. Otherwise it felt like the open workaround might be difficult to understand in context.
There was a problem hiding this comment.
Even still, you can add a transition sentence instead between your code-blocks, adding this open workaround is also for preserving data.
doc/source/io.rst
Outdated
| Handling "bad" lines - preserving the data | ||
| '''''''''''''''''''' | ||
|
|
||
| To preserve all data, you can specify header ``names`` that are long enough: |
There was a problem hiding this comment.
"that are long enough" might come across as that the actual string name elements are long length-wise (also names is singular, not plural, in the context). I think you can reword to make it clearer that we're talking about about the array itself being long itself.
| 1 4 5 6 7 | ||
| 2 8 9 10 NaN | ||
|
|
||
| or you can use Python's ``open`` command to detect the length of the widest row: |
There was a problem hiding this comment.
I'm not sure you need to include this workaround. @jorisvandenbossche suggestion, which you added above, should suffice.
There was a problem hiding this comment.
Note, that you can always over-specify the length and cut down later, which is a lot less suffering even than this, even though I proposed it 😄
There was a problem hiding this comment.
I think the generation of arbitrary labels is generally an approach that should be avoided:
- To others reviewing the code, the DataFrame, or a potentially exported file, it implies some level of purpose, or customization in label specification. They might seek to understand why these labels were used, and if there's a matching codebook or other DataFrames.
- It's messy. Using a
forloop to create strings for the number of variables necessary doesn't feel like something that should be required.
So I like including at least the first open example, because it's the only way to preserve all data without customizing header labels.
There was a problem hiding this comment.
Not really in fact. Just use names = ['dummy'] * n, and we'll take of the duplicates for you.
There was a problem hiding this comment.
@gfyoung Can you clarify? Where's the value for n coming from?
There was a problem hiding this comment.
@joshjacobson : Replace n with whatever number of columns you want. This is a placeholder. My point is that it isn't very hard to specify a list of dummy names.
Codecov Report
@@ Coverage Diff @@
## master #17385 +/- ##
==========================================
- Coverage 91.03% 91.02% -0.02%
==========================================
Files 163 163
Lines 49580 49580
==========================================
- Hits 45137 45128 -9
- Misses 4443 4452 +9
Continue to review full report at Codecov.
|
|
closing as stale. its a worthwhile addition i you can respond to comment (ping and we can reopen). |
