ENH: Adding engine_kwargs to DataFrame.to_excel #53220

rmhowe425 · 2023-05-14T00:34:41Z

closes ENH: add engine_kwargs to DataFrame.to_excel #52368
Tests added and passed if fixing a bug or adding a new feature
All code checks passed.
Added type annotations to new arguments/methods/functions.
Added an entry in the latest doc/source/whatsnew/vX.X.X.rst file if fixing a bug or adding a new feature.

rmhowe425 · 2023-05-14T22:55:40Z

@rhshadrach Pinging for review since the CI tests aren't all green. Looks like one test failed, and the failure was not related to my enhancement or unit tests.

mroeschke · 2023-05-15T18:32:08Z

doc/source/user_guide/io.rst

@@ -3785,6 +3785,15 @@ one can pass an :class:`~pandas.io.excel.ExcelWriter`.

 .. _io.excel_writing_buffer:

+When using the ``engine_kwargs`` parameter, pandas will pass these arguments to the
+engine.For this, it is important to know which function pandas is using internally.


Suggested change

engine.For this, it is important to know which function pandas is using internally.

engine. For this, it is important to know which function pandas is using internally.

Should be fixed with most recent push

mroeschke · 2023-05-15T18:32:25Z

doc/source/whatsnew/v2.1.0.rst

@@ -83,7 +83,7 @@ Other enhancements
 - :meth:`arrays.DatetimeArray.map`, :meth:`arrays.TimedeltaArray.map` and :meth:`arrays.PeriodArray.map` can now take a ``na_action`` argument (:issue:`51644`)
 - :meth:`arrays.SparseArray.map` now supports ``na_action`` (:issue:`52096`).
 - Add :meth:`diff()` and :meth:`round()` for :class:`Index` (:issue:`19708`)
- Add dtype of categories to ``repr`` information of :class:`CategoricalDtype` (:issue:`52179`)
+- Add dtype of categories to ``repr`` information of :class:`CategoricalDtype` (:issue:`52179`)l


Suggested change

- Add dtype of categories to ``repr`` information of :class:`CategoricalDtype` (:issue:`52179`)l

- Add dtype of categories to ``repr`` information of :class:`CategoricalDtype` (:issue:`52179`)

Should be fixed with most recent push

mroeschke · 2023-05-15T18:32:39Z

doc/source/whatsnew/v2.1.0.rst

@@ -97,8 +97,10 @@ Other enhancements
 - Let :meth:`DataFrame.to_feather` accept a non-default :class:`Index` and non-string column names (:issue:`51787`)
 - Performance improvement in :func:`read_csv` (:issue:`52632`) with ``engine="c"``
 - :meth:`Categorical.from_codes` has gotten a ``validate`` parameter (:issue:`50975`)
+- Adding ``engine_kwargs`` parameter to :meth:`DataFrame.to_excel` (:issue:`53220`)


Suggested change

- Adding ``engine_kwargs`` parameter to :meth:`DataFrame.to_excel` (:issue:`53220`)

- Added ``engine_kwargs`` parameter to :meth:`DataFrame.to_excel` (:issue:`53220`)

Should be fixed with most recent push

doc/source/whatsnew/v2.1.0.rst

mroeschke · 2023-05-15T18:33:22Z

pandas/io/excel/_xlsxwriter.py

-        self._book = Workbook(self._handles.handle, **engine_kwargs)
+        try:
+            self._book = Workbook(self._handles.handle, **engine_kwargs)
+        except:


Suggested change

except:

except TypeError:

Should be fixed with most recent push

pandas/tests/io/excel/test_writers.py

… to check for TypeError

pandas/core/generic.py

pandas/io/excel/_xlsxwriter.py

pandas/tests/io/excel/test_writers.py

…unit test

rhshadrach · 2023-05-17T21:35:16Z

pandas/tests/io/excel/test_writers.py

+            "xlsxwriter": r"__init__() got an unexpected keyword argument 'foo'",
+        }
+
+        # Handle change in error message for openpyxl (write and append mode)
+        if engine == "openpyxl" and os.path.exists(path):
+            msgs["openpyxl"] = r"__init__() got an unexpected keyword argument 'foo'"


For xlsxwriter and the append mode of openpyxl, I would not expect this error message. It appears that foo is attempting to be passed to OpenpyxlWriter.__init__ rather than stored in engine_kwargs. Is that the case?

@rhshadrach For openpyxl.load_workbook, we are getting the correct error message in my unit test of load_workbook() got an unexpected keyword argument 'foo'

For xlsxwriter (there is no append mode) and the write mode of openpyxl, both methods are passed to <engine>.Workbook().

@rhshadrach Yeah looking at the workbook class being instantiated in both _xlsxwriter.py and _openpyxl.py, the error message handling should be correct.

I see - the __init__ that raises is part of the engine. Can you check the message for e.g. Workbook.__init__() to make sure we're not raising in the wrong place.

Looks like the traceback was changed in Python 3.10; in Python 3.9 you only get __init__().... Can you import PY310 from pandas.compat._constants and check for Workbook.__init__ when true? Note this will be true whenever the Python version is 3.10 or greater.

Thanks @rhshadrach! I was going down a rabbit hole last night troubleshooting this. I was beginning to think that the issue had something to do with how we were raising the Type error in _openpyxl.py and _xlsxwriter.py without an Exception clause.

…g on the expected class

rmhowe425 · 2023-05-19T15:48:55Z

@rhshadrach @mroeschke I believe the PR should be good to go at this point. Looks like the test failure for test-arm is unrelated to my push. While writing this comment, it appears that the test is in a maintenance state.

mroeschke

LGTM. Merge when ready @rhshadrach

rhshadrach

lgtm

rhshadrach · 2023-05-20T10:36:31Z

Thanks @rmhowe425!

rmhowe425 and others added 8 commits April 8, 2023 23:11

Implementing initial logic to add engine_kwargs to DataFrame.to_excel

31af5a2

Implementing logic to add engine_kwards to to_excel. Adding unit tests

5943856

Documenting the enhancement for this GH issue

4a7994f

Merge branch 'main' into dev/to_excel

1f2ef68

Fixing formatting errors

72c5f80

Fixing formatting issues

5ad0e10

Fixing documentation errors and fixing unit test errors.

a1799a1

Fixing unit test errors

1d6677f

mroeschke reviewed May 15, 2023

View reviewed changes

doc/source/whatsnew/v2.1.0.rst Show resolved Hide resolved

mroeschke reviewed May 15, 2023

View reviewed changes

pandas/tests/io/excel/test_writers.py Show resolved Hide resolved

mroeschke added the IO Excel read_excel, to_excel label May 15, 2023

rmhowe425 added 8 commits May 15, 2023 18:48

Updating discrepancies in documentation, restricting except statement…

cca1f6c

… to check for TypeError

Adding blank line after my contribution in the whatsnew file

7d42a15

Fixing a discrepancy with my unit tests

0e86804

Fixing discrepancy with unit test

b016edd

Fixing discrepancy with unit test

7c63fc9

Fixing discrepancy with unit test

954a727

Fixing discrepancy with unit test

9938978

Fixing discrepancy with unit test

b095e10

rhshadrach requested changes May 16, 2023

View reviewed changes

pandas/core/generic.py Outdated Show resolved Hide resolved

pandas/core/generic.py Outdated Show resolved Hide resolved

pandas/io/excel/_xlsxwriter.py Show resolved Hide resolved

pandas/tests/io/excel/test_writers.py Show resolved Hide resolved

rhshadrach added Enhancement API - Consistency Internal Consistency of API/Behavior and removed API - Consistency Internal Consistency of API/Behavior labels May 16, 2023

rhshadrach added this to the 2.1 milestone May 16, 2023

Updating documentation and syntactical discrepancies based on feedback

f3dd277

rmhowe425 requested a review from rhshadrach May 17, 2023 00:14

rmhowe425 added 3 commits May 16, 2023 21:00

Removing finally statement in _xlsxwriter.py to troubleshoot failing …

bbe1d3b

…unit test

Fixing discrepancies with unit tests. All local unit tests are passing.

010e224

Fixing discrepancies with unit tests. All local unit tests are passing.

7e023ad

rmhowe425 requested a review from mroeschke May 17, 2023 04:04

rhshadrach reviewed May 17, 2023

View reviewed changes

rmhowe425 added 5 commits May 17, 2023 19:07

Updating error messages in unit tests to ensure that tests are failin…

5b76c06

…g on the expected class

Updating error messages in unit tests to ensure that tests are failin…

c014e4f

…g on the expected class

Updating error messages in unit tests to ensure that tests are failin…

e31ec6a

…g on the expected class

Updating error messages in unit tests to ensure that tests are failin…

50ef9f8

…g on the expected class

Updating error messages in unit tests to ensure that tests are failin…

cbad00f

…g on the expected class

rmhowe425 requested a review from rhshadrach May 19, 2023 15:50

mroeschke approved these changes May 19, 2023

View reviewed changes

rhshadrach approved these changes May 20, 2023

View reviewed changes

rhshadrach merged commit df8b686 into pandas-dev:main May 20, 2023

topper-123 pushed a commit to topper-123/pandas that referenced this pull request May 22, 2023

ENH: Add engine_kwargs to DataFrame.to_excel (pandas-dev#53220)

37655de

Daquisu pushed a commit to Daquisu/pandas that referenced this pull request Jul 8, 2023

ENH: Add engine_kwargs to DataFrame.to_excel (pandas-dev#53220)

001c9a4

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ENH: Adding engine_kwargs to DataFrame.to_excel #53220

ENH: Adding engine_kwargs to DataFrame.to_excel #53220

rmhowe425 commented May 14, 2023 •

edited

Loading

rmhowe425 commented May 14, 2023 •

edited

Loading

mroeschke May 15, 2023

rmhowe425 May 15, 2023

mroeschke May 15, 2023

rmhowe425 May 15, 2023

mroeschke May 15, 2023

rmhowe425 May 15, 2023

mroeschke May 15, 2023

rmhowe425 May 15, 2023

rhshadrach May 17, 2023

rmhowe425 May 17, 2023 •

edited

Loading

rmhowe425 May 17, 2023

rhshadrach May 17, 2023

rmhowe425 May 17, 2023

rhshadrach May 19, 2023

rmhowe425 May 19, 2023

rmhowe425 commented May 19, 2023 •

edited

Loading

mroeschke left a comment

rhshadrach left a comment

rhshadrach commented May 20, 2023

	engine.For this, it is important to know which function pandas is using internally.
	engine. For this, it is important to know which function pandas is using internally.

	- Add dtype of categories to ``repr`` information of :class:`CategoricalDtype` (:issue:`52179`)l
	- Add dtype of categories to ``repr`` information of :class:`CategoricalDtype` (:issue:`52179`)

	- Adding ``engine_kwargs`` parameter to :meth:`DataFrame.to_excel` (:issue:`53220`)
	- Added ``engine_kwargs`` parameter to :meth:`DataFrame.to_excel` (:issue:`53220`)

ENH: Adding engine_kwargs to DataFrame.to_excel #53220

ENH: Adding engine_kwargs to DataFrame.to_excel #53220

Conversation

rmhowe425 commented May 14, 2023 • edited Loading

rmhowe425 commented May 14, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rmhowe425 May 17, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rmhowe425 commented May 19, 2023 • edited Loading

mroeschke left a comment

Choose a reason for hiding this comment

rhshadrach left a comment

Choose a reason for hiding this comment

rhshadrach commented May 20, 2023

rmhowe425 commented May 14, 2023 •

edited

Loading

rmhowe425 commented May 14, 2023 •

edited

Loading

rmhowe425 May 17, 2023 •

edited

Loading

rmhowe425 commented May 19, 2023 •

edited

Loading