REGR: errors='replace' when encoding/errors are not specified

[twoertwein]

• closes BUG: read_csv raising when null bytes are in skipped rows #38989
• tests added / passed
• Ensure all linting tests pass, see here for how to run them
• whatsnew entry

Should 1.3 use errors='replace' when no encoding/errors are specified or use errors=None (strict)?

[phofl]

I would say replace only when encoding is Not set and inferred to utf-8.

This should go back to 1.2.x

[jreback]

tests!

[twoertwein]

Should a follow-up PR intentionally revert this PR for 1.3? Personally, I would want to know when read_csv fails to read certain characters (errors = None by default).

[phofl]

I don't think that content from skipped rows should matter and it should raise no error if the rest of the file is ok. Maybe you could add a test withouth skiprows, this raises on 1.1.5 which is good and should keep raising

[jreback]

instead why don't you

encoding_passed, encoding = encoding, encoding or 'utf-i'

and then you can test encoding_passed is not None its the same as you have but i think a bit more natural

[twoertwein]

I thought errors is exposed in read_csv. It isn't. In that case it makes sense to keep 'replace' as the default.

I don't think it is feasible to ignore encoding errors only for skipped rows. (We could read everything in binary mode but we would need to decode it to determine line endings.)

[phofl]

This errors parameter is Not exposed

[twoertwein]

yes, it isn't exposed. Sorry, that is what I meant to say :)

[jreback]

thanks @twoertwein

[jreback]

@meeseeksdev backport 1.2.x

[lumberbot-app]

Something went wrong ... Please have a look at my logs.

[phofl]

Thanks