ENH: add calamine excel reader (close #50395) #54998

dimastbk · 2023-09-04T12:03:34Z

closes ENH: Adding support for calamine as Excel reader engine #50395
Tests added and passed
All code checks passed.
Added type annotations to new arguments/methods/functions.
Added an entry in the latest doc/source/whatsnew/vX.X.X.rst file if fixing a bug or adding a new feature.

Based on #50581, which was closed as stale and in favor of #53005, which was rejected. Now, calamine support datetime and sheet types, that more than pyxlsb supports, for example.

Now fail a few tests with ods:

BUG: boolean/string in OdsWriter #54994 (5 tests);
BUG (test): bad file for testing ODFReader #55045 (1 test) - bad ods file in test_reader_seconds and, I think, a bug in the OdsReader (see office:time-value in specification);
two bugs in calamine, which can be fixed in the future (one of them - problem with specification - is fixed with hack in ODFReader for example).

My English is too bad for writing documentation and I will be happy any help with it.

cc @kostyafarber

WillAyd

Nice looking change. Are you expecting to submit a conda recipe as well? Would be ideal to have this downloadable there instead of just via pip (though not a blocker)

pandas/tests/io/excel/test_readers.py

rhshadrach · 2023-09-05T20:40:06Z

doc/source/user_guide/io.rst

+files can be read using ``pyxlsb``. Also, all this formats can be read using ``python-calamine``,
+but this library has some different behavior from other libraries (mostly for ``.ods``).


Is there documentation that gives more details on this?

Below this, we document which function is used by the various engines so users can determine which engine_kwargs they pass. Can you add a line about Calamine there.

I think, now differences are not too large for explain it in documentation and I removed it:

calamine can return timedelta;

some difference from ODFReader, because calamine follows specification and extracts value from xml attributes, not from text (time and boolean, for example).

added.

I added small chapter for Calamine.

dimastbk · 2023-09-06T10:23:38Z

@WillAyd recipe was added to conda-forge.

pandas/io/excel/_calamine.py

WillAyd

This is really nice work. Kudos

doc/source/user_guide/io.rst

pandas/tests/io/excel/test_readers.py

WillAyd · 2023-09-06T16:11:57Z

pyproject.toml

@@ -66,7 +66,7 @@ computation = ['scipy>=1.8.1', 'xarray>=2022.03.0']
 fss = ['fsspec>=2022.05.0']
 aws = ['s3fs>=2022.05.0']
 gcp = ['gcsfs>=2022.05.0', 'pandas-gbq>=0.17.5']
-excel = ['odfpy>=1.4.1', 'openpyxl>=3.0.10', 'pyxlsb>=1.0.9', 'xlrd>=2.0.1', 'xlsxwriter>=3.0.3']
+excel = ['odfpy>=1.4.1', 'openpyxl>=3.0.10', 'python-calamine>=0.1.6', 'pyxlsb>=1.0.9', 'xlrd>=2.0.1', 'xlsxwriter>=3.0.3']


I think we usually wait a few cycles for a library before adding a new library to extras, but @mroeschke would know best

I think it's okay to add this to the extra right away

doc/source/user_guide/io.rst

doc/source/whatsnew/v2.2.0.rst

pandas/tests/io/excel/test_readers.py

Co-author: Kostya Farber (pandas-dev#50581)

rhshadrach · 2023-09-09T13:12:39Z

dimastbk force-pushed the issue-50395 branch from df8b523 to d6002fe
4 hours ago

This makes is quite more time consuming to review.

dimastbk · 2023-09-09T17:50:18Z

Do you mean about force-push instead of merging "main" and adding new commits? Sorry.

rhshadrach · 2023-09-09T20:59:40Z

Do you mean about force-push instead of merging "main" and adding new commits? Sorry.

Correct - not a big problem but wanted you to be aware.

rhshadrach

Just some minor comments on the docs, but the rest looks great. Only remaining issue I think is whether we're adding it to the extras just yet (#54998 (comment)).

doc/source/whatsnew/v2.2.0.rst

dimastbk · 2023-09-10T01:50:01Z

Thanks, I fixed it.

rhshadrach

lgtm; @WillAyd - you good here?

WillAyd · 2023-09-12T20:35:08Z

Thanks @dimastbk - impressive work on this

pandas-dev/pandas#54998

* Extract ReadEngine and WriteEngine types for Excel * Add calamine Excel reader engine pandas-dev/pandas#54998 * Add test for read_excel engine

dimastbk requested review from rhshadrach and mroeschke as code owners September 4, 2023 12:03

WillAyd reviewed Sep 5, 2023

View reviewed changes

pandas/tests/io/excel/test_readers.py Show resolved Hide resolved

pandas/tests/io/excel/test_readers.py Show resolved Hide resolved

WillAyd added the IO Excel read_excel, to_excel label Sep 5, 2023

dimastbk force-pushed the issue-50395 branch from bdf286b to b6701f0 Compare September 5, 2023 17:06

dimastbk mentioned this pull request Sep 5, 2023

Make exceptions more detailed dimastbk/python-calamine#36

Closed

rhshadrach requested changes Sep 5, 2023

View reviewed changes

dimastbk force-pushed the issue-50395 branch 3 times, most recently from 30288dc to fe2f6de Compare September 6, 2023 10:22

twoertwein reviewed Sep 6, 2023

View reviewed changes

pandas/io/excel/_calamine.py Show resolved Hide resolved

WillAyd approved these changes Sep 6, 2023

View reviewed changes

dimastbk force-pushed the issue-50395 branch 4 times, most recently from a0cef5b to 863c068 Compare September 7, 2023 11:32

dimastbk requested a review from rhshadrach September 7, 2023 13:38

rhshadrach requested changes Sep 7, 2023

View reviewed changes

dimastbk added 3 commits September 9, 2023 14:25

ENH: add calamine excel reader (close pandas-dev#50395)

c43a34b

Co-author: Kostya Farber (pandas-dev#50581)

suppression failed tests

48024be

fix review

d6002fe

dimastbk force-pushed the issue-50395 branch from df8b523 to d6002fe Compare September 9, 2023 08:49

expand notice in whatsnew

0c10856

rhshadrach requested changes Sep 9, 2023

View reviewed changes

doc/source/whatsnew/v2.2.0.rst Outdated Show resolved Hide resolved

doc/source/whatsnew/v2.2.0.rst Outdated Show resolved Hide resolved

doc/source/whatsnew/v2.2.0.rst Outdated Show resolved Hide resolved

doc/source/whatsnew/v2.2.0.rst Outdated Show resolved Hide resolved

dimastbk added 2 commits September 10, 2023 07:38

Merge branch 'main' into issue-50395

ca08b18

fix "whatsnew" based on review

0aace9f

Merge branch 'main' into issue-50395

a8dda74

rhshadrach added this to the 2.2 milestone Sep 12, 2023

rhshadrach added the Enhancement label Sep 12, 2023

rhshadrach approved these changes Sep 12, 2023

View reviewed changes

WillAyd merged commit 79067a7 into pandas-dev:main Sep 12, 2023

rhshadrach mentioned this pull request Sep 12, 2023

BUG: boolean/string value in OdsWriter (#54994) #54996

Merged

5 tasks

dimastbk deleted the issue-50395 branch September 13, 2023 00:58

dimastbk mentioned this pull request Dec 1, 2023

DEP: update python-calamine to 0.1.7 #56280

Merged

5 tasks

davetapley added a commit to JEFuller/pandas-stubs that referenced this pull request Mar 31, 2024

Add calamine Excel reader engine

c01c29d

pandas-dev/pandas#54998

davetapley mentioned this pull request Mar 31, 2024

Add calamine Excel reader pandas-dev/pandas-stubs#898

Merged

davetapley added a commit to JEFuller/pandas-stubs that referenced this pull request Apr 2, 2024

Add calamine Excel reader engine

8c9b133

pandas-dev/pandas#54998

davetapley added a commit to JEFuller/pandas-stubs that referenced this pull request Apr 4, 2024

Add calamine Excel reader engine

55db57a

pandas-dev/pandas#54998

davetapley added a commit to JEFuller/pandas-stubs that referenced this pull request Apr 5, 2024

Add calamine Excel reader engine

e8850b5

pandas-dev/pandas#54998

Dr-Irv pushed a commit to pandas-dev/pandas-stubs that referenced this pull request Apr 9, 2024

Add calamine Excel reader (#898)

0d54b01

* Extract ReadEngine and WriteEngine types for Excel * Add calamine Excel reader engine pandas-dev/pandas#54998 * Add test for read_excel engine

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ENH: add calamine excel reader (close #50395) #54998

ENH: add calamine excel reader (close #50395) #54998

dimastbk commented Sep 4, 2023 •

edited

Loading

WillAyd left a comment

rhshadrach Sep 5, 2023

rhshadrach Sep 5, 2023

dimastbk Sep 6, 2023 •

edited

Loading

dimastbk commented Sep 6, 2023 •

edited

Loading

WillAyd left a comment

WillAyd Sep 6, 2023

mroeschke Sep 11, 2023

rhshadrach commented Sep 9, 2023

dimastbk commented Sep 9, 2023

rhshadrach commented Sep 9, 2023

rhshadrach left a comment

dimastbk commented Sep 10, 2023

rhshadrach left a comment

WillAyd commented Sep 12, 2023

		files can be read using ``pyxlsb``. Also, all this formats can be read using ``python-calamine``,
		but this library has some different behavior from other libraries (mostly for ``.ods``).

ENH: add calamine excel reader (close #50395) #54998

ENH: add calamine excel reader (close #50395) #54998

Conversation

dimastbk commented Sep 4, 2023 • edited Loading

WillAyd left a comment

Choose a reason for hiding this comment

rhshadrach Sep 5, 2023

Choose a reason for hiding this comment

rhshadrach Sep 5, 2023

Choose a reason for hiding this comment

dimastbk Sep 6, 2023 • edited Loading

Choose a reason for hiding this comment

dimastbk commented Sep 6, 2023 • edited Loading

WillAyd left a comment

Choose a reason for hiding this comment

WillAyd Sep 6, 2023

Choose a reason for hiding this comment

mroeschke Sep 11, 2023

Choose a reason for hiding this comment

rhshadrach commented Sep 9, 2023

dimastbk commented Sep 9, 2023

rhshadrach commented Sep 9, 2023

rhshadrach left a comment

Choose a reason for hiding this comment

dimastbk commented Sep 10, 2023

rhshadrach left a comment

Choose a reason for hiding this comment

WillAyd commented Sep 12, 2023

dimastbk commented Sep 4, 2023 •

edited

Loading

dimastbk Sep 6, 2023 •

edited

Loading

dimastbk commented Sep 6, 2023 •

edited

Loading