Skip to content

REF: Consistent optional dependency handling #26802

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 10 commits into from
Jun 12, 2019

Conversation

TomAugspurger
Copy link
Contributor

@TomAugspurger TomAugspurger commented Jun 12, 2019

Standardizes how we do our optional dependency checking.

  1. Adds a new import_optional_dependency helper that handles optional importing, raising with a nice message, and warning / raising if the version is not correct.
  2. Changes inline try / except ImportError to use import_optional_dependency
  3. Documents all(?) optional dependencies with their minimum version.
  4. Documents that this is how devs should do optional dependencies.
>>> pd.read_gbq('')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/taugspurger/sandbox/pandas/pandas/io/gbq.py", line 140, in read_gbq
    pandas_gbq = _try_import()
  File "/Users/taugspurger/sandbox/pandas/pandas/io/gbq.py", line 14, in _try_import
    extra=msg,
  File "/Users/taugspurger/sandbox/pandas/pandas/compat/_optional.py", line 82, in import_optional_dependency
    raise ImportError(message.format(name=name, extra=extra)) from None
ImportError: Missing optional dependency 'pandas_gbq'. pandas-gbq is required to load data from Google BigQuery. See the docs: https://pandas-gbq.readthedocs.io. Use pip or conda to install pandas_gbq.

@TomAugspurger TomAugspurger added the Refactor Internal refactoring of code label Jun 12, 2019
@@ -275,12 +272,7 @@ def _interpolate_scipy_wrapper(x, y, new_x, method, fill_value=None,
raise ImportError("Your version of Scipy does not support "
"PCHIP interpolation.")
elif method == 'akima':
try:
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Akima1DInterpolater was added in SciPy 0.14, and we now require 0.19, so this additional check isn't needed anymore.

Copy link
Contributor

@jreback jreback left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

generally lgtm.

Dependency Minimum Version Notes
========================= ================== =============================================================
matplotlib 2.2.2 Visualization
xarray 0.8.2 pandas-like API for N-dimensional data
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

are these meant to be in some ordering?

I would maybe put them in sections: io, numeric, other i think

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I roughly grouped the I/O, HTML parsers, and just threw the others wherever.

With headers, it doesn't look as nice I think. What do you think?

Screen Shot 2019-06-12 at 8 23 47 AM

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah ok maybe alphabetize then?

@WillAyd
Copy link
Member

WillAyd commented Jun 12, 2019

This is a great idea! Just to be clear did you purposely only select a subset of optional dependencies to update? I opened #26782 which could either be included here or done as a follow up



def import_optional_dependency(
name, extra="", raise_on_missing=True, on_version="raise"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you add annotations here? Should be straightforward I think

@TomAugspurger
Copy link
Contributor Author

did you purposely only select a subset of optional dependencies to update?

I grepped for except ImportErrors and did the ones that didn't look too complex (I think sql.py was a bit strange).

I seem to have just missed the xlrd one. Do you want me to handle it here?

@TomAugspurger
Copy link
Contributor Author

Looks like I missed bottleneck too, sorry.

@WillAyd
Copy link
Member

WillAyd commented Jun 12, 2019

I seem to have just missed the xlrd one. Do you want me to handle it here?

Up to you

@codecov
Copy link

codecov bot commented Jun 12, 2019

Codecov Report

Merging #26802 into master will increase coverage by 0.08%.
The diff coverage is 91.75%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master   #26802      +/-   ##
==========================================
+ Coverage   91.76%   91.85%   +0.08%     
==========================================
  Files         178      179       +1     
  Lines       50751    50707      -44     
==========================================
+ Hits        46574    46579       +5     
+ Misses       4177     4128      -49
Flag Coverage Δ
#multiple 90.45% <91.75%> (+0.09%) ⬆️
#single 41.1% <48.45%> (-0.02%) ⬇️
Impacted Files Coverage Δ
pandas/io/formats/style.py 97.14% <100%> (+0.46%) ⬆️
pandas/core/missing.py 95.75% <100%> (+1.8%) ⬆️
pandas/io/html.py 94.29% <100%> (+1.62%) ⬆️
pandas/core/generic.py 94.19% <100%> (+0.09%) ⬆️
pandas/compat/_optional.py 100% <100%> (ø)
pandas/io/s3.py 100% <100%> (+10.52%) ⬆️
pandas/core/nanops.py 94.76% <100%> (+0.64%) ⬆️
pandas/core/window.py 96.63% <100%> (+0.23%) ⬆️
pandas/io/gcs.py 100% <100%> (+20%) ⬆️
pandas/core/arrays/sparse.py 93.71% <100%> (+0.22%) ⬆️
... and 11 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 429078b...d04425d. Read the comment docs.

@jreback jreback added this to the 0.25.0 milestone Jun 12, 2019
@jreback
Copy link
Contributor

jreback commented Jun 12, 2019

small comment, otherwise lgtm.

@WillAyd WillAyd merged commit e6d27ec into pandas-dev:master Jun 12, 2019
@WillAyd
Copy link
Member

WillAyd commented Jun 12, 2019

Great change @TomAugspurger !

# xlrd uses a capitalized attribute name
version = module.__VERSION__
version = getattr(module, '__VERSION__', None)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should we not encode the 'version' (or 'VERSION') with the list of deps directly? (or have a way to override); in fact we already do this in pandas/util/_print_versions. so actually I would update that to use this list of deps (and also to add the version accessor)

@jreback
Copy link
Contributor

jreback commented Jun 12, 2019

actually @TomAugspurger see my comments. This duplicates a ton of code in _print_versions. Can you (or others) do a followup (or create an issue) about de-duplicating.

@WillAyd
Copy link
Member

WillAyd commented Jun 12, 2019

see #26814

@TomAugspurger TomAugspurger deleted the optional-deps branch June 12, 2019 18:34
@TomAugspurger
Copy link
Contributor Author

FYI @pandas-dev/pandas-core you may want to glance through http://pandas-docs.github.io/pandas-docs-travis/development/contributing.html#optional-dependencies, when we get PRs adding a new optional dependency.

@jorisvandenbossche
Copy link
Member

Nice!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Refactor Internal refactoring of code
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants