Bug: read_csv does not work with chr(254) as quotechar parameter

[ghost]

No description provided.

[jreback]

pls show a copy-pastable example

[dsm054]

I can reproduce:

>>> pd.read_csv(io.StringIO(chr(127)+"a, "+chr(127)+",b"), quotechar=chr(127), header=None)

     0  1
0  a,   b

>>> pd.read_csv(io.StringIO(chr(254)+"a, "+chr(254)+",b"), quotechar=chr(254), header=None)

    0   1  2
0  þa   þ  b

The problem sets in at chr(128), which just so happens to be the first value when len(chr(x).encode("utf-8")) > 1, and we're using only char quotechar in parser.pyx.

I suspect this would be a bit of a headache to fix, and it would only really be useful for people who decided to use a small thorn as a separator, and who were also using every other viable separator character so that a pass to replace the thorns wouldn't work, and who can't use the python engine for some reason.

>>> pd.read_csv(io.StringIO(chr(254)+"a, "+chr(254)+",b"), quotechar=chr(254), 
header=None, engine='python')

     0  1
0  a,   b

I like the idea of being able to use anything anywhere, but in terms of developer time, maybe we should just raise NotimplementedError if you're using the C engine in these circumstances..

[jreback]

thanks @dsm054 for the example.

[gfyoung]

@jreback : We have a similar issue with sep in which we cannot use multi-char sep with the C engine. I'm inclined to agree with @joelschw and raise NotImplementedError (or some kind of error) for such a quotechar for the time being. Thoughts?

[jreback]

multi-char sep for sure should be NotImplementedError (when c-engine is given). so the same here is fine.