Skip to content

CI/TYP: run pyright at manual stage in pre-commit #43799

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
Oct 3, 2021
Merged

CI/TYP: run pyright at manual stage in pre-commit #43799

merged 5 commits into from
Oct 3, 2021

Conversation

twoertwein
Copy link
Member

with numpy<1.21.0 pyright (and mypy?) cannot determine some types which leads to overlapping overloads. See #43747 (comment)

@attack68

@jreback
Copy link
Contributor

jreback commented Sep 29, 2021

we cannot bump this now. needs to be a while. (you could do it for the pyright checking i suppose) but that might be complicated.

@twoertwein
Copy link
Member Author

we cannot bump this now. needs to be a while.

Definitly not for a release but not even for development?

(you could do it for the pyright checking i suppose) but that might be complicated.

Pre-commit allows specifying additional dependencies for a hook, but it seems they need to be from the same language as the hook (pyright: node, numpy: python).

I think the best options is to run pyright at the "manual" stage to avoid annoying contributors. This would still make it possible to easily run pyright.

@twoertwein twoertwein changed the title CI/TYP: numpy>=1.21.0 for development CI/TYP: run pyright at manual stage in pre-commit Sep 29, 2021
@twoertwein
Copy link
Member Author

Mypy also needs 1.21.0, otherwise it reports many errors.

@jreback jreback added the Typing type annotations, mypy/pyright type checking label Sep 30, 2021
@@ -32,6 +32,9 @@ jobs:
with:
fetch-depth: 0

- name: numpy>=1.21.0 for type checkers
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hmm this is very fragile if we change this

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adding on to this. Can we just make two conda envs in the CI, one for typing and one for everything else?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Currently, we don't need this to force numpy>1.21.0 because pip/conda already pick numpy 1.21.*.

We could simply remove this change. If at some point the CI picks a too old numpy, we will know very quickly.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is probably best to decouple the CI-change from the pre-commit change. I will remove the CI-changes from this PR.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if you saw #43804 numpy gets installed <1.21 since that is a numba requirement when resolving dependencies.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if you saw #43804 numpy gets installed <1.21 since that is a numba requirement when resolving dependencies.

Yes, that is why I tried to add this line, but it seems the CI prefers an older numba in favor of a newer numpy :)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just to be clear, the goal of this PR has changed from "having numpy>1.21. on the CI and for pre-commit" to "not having pyright as a mandatory part of pre-commit".

Comment on lines +409 to +410
A recent version of ``numpy`` (>=1.21.0) is required for type validation.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does the [email protected] dependency include this? Because if not, as it's running as a local hook, then it won't have access to numpy, even if you have it installed in your virtual environment

Perhaps it should be repo: system?

(granted, I haven't tried running this, am just pointing out a common misconception about repo: local)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems that pre-commit does not isolate pyright from the current python environment: if numpy is installed in the currently active environment, pyright seems to have access to it.

I tried adding it as a dependency (in the pre-commit), but it seems that pre-commit does not support mixing languages (pyright: node, numpy:python) - it tries to use node to install numpy.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems that pre-commit does not isolate pyright from the current python environment: if numpy is installed in the currently active environment, pyright seems to have access to it.

I find this surprising, but if that's the case, then great 👍

Copy link
Member

@lithomas1 lithomas1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. We should pin numpy in a follow-up given that it is possible for older numpy's to be used in case of a conflict, and its better to have CI explicitly error in that case.

@lithomas1 lithomas1 added this to the 1.4 milestone Oct 3, 2021
@jreback jreback merged commit 562df33 into pandas-dev:master Oct 3, 2021
@jreback
Copy link
Contributor

jreback commented Oct 3, 2021

thanks @twoertwein yep having to manualish now is fine

gasparitiago pushed a commit to gasparitiago/pandas that referenced this pull request Oct 9, 2021
@twoertwein twoertwein deleted the numpy branch April 1, 2022 01:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Typing type annotations, mypy/pyright type checking
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants