BUG: Add errors argument to to_csv() call to enable error handling for encoders #32702

roberthdevries · 2020-03-14T15:59:58Z

closes to_csv() surrogates not allowed #22610
tests added / passed
passes black pandas
passes git diff upstream/master -u -- "*.py" | flake8 --diff
whatsnew entry

WillAyd · 2020-03-14T16:53:27Z

Cool makes sense. Can you add tests to make sure this works?

roberthdevries · 2020-03-15T15:00:32Z

@WillAyd Test code is now added, pls. review

jreback · 2020-03-16T01:28:33Z

doc/source/whatsnew/v1.1.0.rst

@@ -69,7 +69,7 @@ Other enhancements
 - `OptionError` is now exposed in `pandas.errors` (:issue:`27553`)
 - :func:`timedelta_range` will now infer a frequency when passed ``start``, ``stop``, and ``periods`` (:issue:`32377`)
 - Positional slicing on a :class:`IntervalIndex` now supports slices with ``step > 1`` (:issue:`31658`)
-
+- :meth:`to_csv` now accepts an ``error`` argument (:issue:`22610`)


use :meth:`DataFrame.to_csv` amd :meth:`Series.to_csv`

jreback · 2020-03-16T01:29:55Z

pandas/core/generic.py

        encoding: str = "UTF-8",
+        errors: str = "strict",


is there a reason you changed this?

This order is in line with the order of arguments in open() of the Python standard library. I thought it would be nice to have a consistent order of arguments.

Let’s revert, to not break users passing them by position.

You are kidding right, this is the 13'th argument of this method. Nobody in their right minds would ever use this one as a positional argument. (masochists excluded)

No, I'm not kidding.

Whatever, I'll revert

jreback · 2020-03-16T01:34:23Z

pandas/io/common.py

@@ -345,6 +346,10 @@ def get_handle(
        Mode to open path_or_buf with.
    encoding : str or None
        Encoding to use.
+    errors : str, default 'strict'


I think we called this encoding_errors elsewhere (as errors is too generic here). cc @WillAyd @TomAugspurger

The discussion is at #27899. I don't recall the resolution.

I pushed for passing that to a handle prior to the call in that conversation and just documenting that. But I think this is OK as well

This is identical to the keyword argument names of the standard Python open() function. In my opinion this allows us to point users to the standard python library documentation for more information on what these arguments mean.

pandas/io/common.py

jreback

minor comment. errors is appealing because it mirrors open, but we use errors in different context more generally, so kind of +0 on errors (and +1 on encoding_errors), but could be persuaded.

jreback · 2020-03-19T00:11:52Z

doc/source/whatsnew/v1.1.0.rst

@@ -70,7 +70,8 @@ Other enhancements
 - :func:`timedelta_range` will now infer a frequency when passed ``start``, ``stop``, and ``periods`` (:issue:`32377`)
 - Positional slicing on a :class:`IntervalIndex` now supports slices with ``step > 1`` (:issue:`31658`)
 - :meth:`DataFrame.sample` will now also allow array-like and BitGenerator objects to be passed to ``random_state`` as seeds (:issue:`32503`)
-
+- :meth:`DataFrame.to_csv` and :meth:`Series.to_csv` now accept an ``error`` argument (:issue:`22610`)


error -> errors

jreback · 2020-03-19T00:12:13Z

pandas/core/generic.py

@@ -3086,6 +3087,12 @@ def to_csv(
        encoding : str, optional
            A string representing the encoding to use in the output file,
            defaults to 'utf-8'.
+        errors : str, default 'strict'
+            Specifies how encoding and decoding errors are to be handled.
+            See the errors argument for :func:`open` for a full list


does this render (open)?

It renders as: open()
See https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.to_hdf.html
That is where I stole it from

roberthdevries · 2020-03-19T19:24:58Z

There is prior art for the errors argument for this purpose in DataFrame.to_hdf and DataFrame:read_hdf

WillAyd

comment on arg position otherwise lgtm

WillAyd · 2020-03-25T00:10:24Z

pandas/core/generic.py

@@ -3030,6 +3030,7 @@ def to_csv(
        index_label: Optional[Union[bool_t, str, Sequence[Label]]] = None,
        mode: str = "w",
        encoding: Optional[str] = None,
+        errors: str = "strict",


I think @TomAugspurger mentioned this before but should add this to the end of the function; while seemingly pedantic we have a lot of users doing different things, so no real upside to change the order of arguments here

If an interest point for you there are other issues we have to make larger functions keyword only via a deprecation that could use some help

Moved to the end, however the logic behind the argument order is now lost (just as it was for the other method/function

If you can point me to the other issues, I can see if I can help

WillAyd

lgtm

jreback · 2020-05-25T22:41:38Z

@roberthdevries this looked fine, can you merge master and ping on green.

@gfyoung if you have comments.

gfyoung · 2020-05-25T23:26:40Z

@jreback: Agreed that this looks good. Just need to merge with master.

…oders

… to open())

WillAyd · 2020-06-08T23:36:23Z

Thanks @roberthdevries

WillAyd added Error Reporting Incorrect or improved errors from pandas IO CSV read_csv, to_csv labels Mar 14, 2020

roberthdevries force-pushed the fix-22610-add-errors-argument-to-to_csv branch from c576828 to 5b51c51 Compare March 15, 2020 07:23

roberthdevries changed the title ~~WIP: Add errors argument to to_csv() call to enable error handling for encoders~~ Add errors argument to to_csv() call to enable error handling for encoders Mar 15, 2020

jreback requested changes Mar 16, 2020

View reviewed changes

roberthdevries force-pushed the fix-22610-add-errors-argument-to-to_csv branch from 3ea8ebb to 5424787 Compare March 16, 2020 20:14

roberthdevries requested a review from jreback March 17, 2020 20:55

roberthdevries force-pushed the fix-22610-add-errors-argument-to-to_csv branch 2 times, most recently from f26a492 to efd6b9b Compare March 18, 2020 19:48

roberthdevries changed the title ~~Add errors argument to to_csv() call to enable error handling for encoders~~ BUG: Add errors argument to to_csv() call to enable error handling for encoders Mar 18, 2020

jreback requested changes Mar 19, 2020

View reviewed changes

jreback added this to the 1.1 milestone Mar 19, 2020

roberthdevries requested a review from jreback March 19, 2020 19:25

roberthdevries force-pushed the fix-22610-add-errors-argument-to-to_csv branch 2 times, most recently from 8e95f27 to 23668cd Compare March 20, 2020 18:56

roberthdevries requested review from WillAyd and TomAugspurger March 24, 2020 21:23

WillAyd requested changes Mar 25, 2020

View reviewed changes

roberthdevries force-pushed the fix-22610-add-errors-argument-to-to_csv branch from 23668cd to d4ed3b0 Compare March 25, 2020 20:53

roberthdevries requested a review from WillAyd March 25, 2020 20:56

WillAyd approved these changes Mar 27, 2020

View reviewed changes

roberthdevries force-pushed the fix-22610-add-errors-argument-to-to_csv branch 2 times, most recently from ed48735 to ecc604b Compare April 2, 2020 12:22

roberthdevries force-pushed the fix-22610-add-errors-argument-to-to_csv branch from ecc604b to 1218163 Compare April 19, 2020 12:29

Add errors argument to to_csv() call to enable error handling for enc…

f6bdff6

…oders

roberthdevries added 13 commits May 28, 2020 23:32

Fix doc string validation

33d7e1a

Add docstring for errors argument

e7b7606

Move errors keyword argument after encoding keyword argument (similar…

7c03df0

… to open())

Add a test for the errors argument in to_csv()

03105b5

Add whatsnew entry

0c71fa6

Reformatting by black

8d943e4

Fix whatsnew entry

a9eb4e0

Add versionadded tag to error argument

0ba9d14

Revert change

6fba37d

Fix versionadded tag 3rd try

149ed35

Fix whatsnew entry: error -> errors

1030f6c

Move new errors argument to the end

55e6dbf

Fix rebase issues

6fa2290

roberthdevries force-pushed the fix-22610-add-errors-argument-to-to_csv branch from 1218163 to 6fa2290 Compare May 28, 2020 21:39

Merge branch 'master' into fix-22610-add-errors-argument-to-to_csv

5ef80a9

WillAyd merged commit 43a463d into pandas-dev:master Jun 8, 2020

roberthdevries deleted the fix-22610-add-errors-argument-to-to_csv branch June 9, 2020 19:13

tgmof mentioned this pull request Jun 10, 2020

Suppress UnicodeEncodeError when executing to_csv method #27750

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

BUG: Add errors argument to to_csv() call to enable error handling for encoders #32702

BUG: Add errors argument to to_csv() call to enable error handling for encoders #32702

roberthdevries commented Mar 14, 2020 •

edited

Loading

WillAyd commented Mar 14, 2020

roberthdevries commented Mar 15, 2020

jreback Mar 16, 2020

roberthdevries Mar 16, 2020

jreback Mar 16, 2020

roberthdevries Mar 16, 2020

TomAugspurger Mar 16, 2020

roberthdevries Mar 17, 2020

TomAugspurger Mar 17, 2020

roberthdevries Mar 17, 2020

jreback Mar 16, 2020

TomAugspurger Mar 16, 2020

WillAyd Mar 16, 2020

roberthdevries Mar 16, 2020

jreback left a comment

jreback Mar 19, 2020

roberthdevries Mar 19, 2020

jreback Mar 19, 2020

roberthdevries Mar 19, 2020 •

edited

Loading

roberthdevries commented Mar 19, 2020

WillAyd left a comment

WillAyd Mar 25, 2020

roberthdevries Mar 25, 2020

WillAyd Mar 27, 2020

WillAyd left a comment

jreback commented May 25, 2020

gfyoung commented May 25, 2020

WillAyd commented Jun 8, 2020

BUG: Add errors argument to to_csv() call to enable error handling for encoders #32702

BUG: Add errors argument to to_csv() call to enable error handling for encoders #32702

Conversation

roberthdevries commented Mar 14, 2020 • edited Loading

WillAyd commented Mar 14, 2020

roberthdevries commented Mar 15, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jreback left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

roberthdevries Mar 19, 2020 • edited Loading

Choose a reason for hiding this comment

roberthdevries commented Mar 19, 2020

WillAyd left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

WillAyd left a comment

Choose a reason for hiding this comment

jreback commented May 25, 2020

gfyoung commented May 25, 2020

WillAyd commented Jun 8, 2020

roberthdevries commented Mar 14, 2020 •

edited

Loading

roberthdevries Mar 19, 2020 •

edited

Loading