Fix CI for Windows

[Dr-Irv]

• Closes CI failing on Windows. #240

Summary

• Uses action to install poetry
• For the test of the distribution, avoids poetry because Windows CI isn't installing it in the right place
• Adds a couple of missing PYI files that pyright said were missing

One downside (at least for me). The setting

[tool.poe.executor]
type = "virtualenv"

makes a local test with poetry not work inside of a conda environment.

But at least the CI is working.

[bashtage]

I was going to merge this since green. Fix looks fine, although the source'ing is annoying. but maybe necessary. Could you instead call poetry shell before the poe commands.

I didn't merge because looks like you accidentally got some pyi changes in that aren't needed.

[twoertwein]

Is this key correct, the CI says it is venv-Linux--?

This caches the (large) venv but I think it does not cache poetry.lock.

[Dr-Irv]

I picked this up from https://github.com/snok/install-poetry#running-on-windows

There are comments there that say that caching doesn't work on Windows.

[twoertwein]

does the file poetry.lock exist on the CI? The previous caching used pyproject.toml as the key to cache poety.lock

[Dr-Irv]

Don't know. After spending hours on this last night, I was just happy to get it working.

[twoertwein]

Why is that needed? To invalidate the cache?

This should already pick the newest wheel:

pandas-stubs/scripts/test/run.py

Line 49 in 5b70b46

path = sorted(Path("dist/").glob("pandas_stubs-*.whl"))[-1]

[Dr-Irv]

The problem here is that if I build and install the wheel from the run script, it gets installed into the wrong python environment. So now with test.yaml doing a direct build of the wheel and installation, I have to force it to install the newly built wheel, which requires a higher version number than what is available on pypi.

[twoertwein]

which requires a higher version number than what is available on pypi.

So pip is preferring the pypi.org wheel over the local wheel? There must be an option to ignore the remote wheel.

Could also simply bump the version number after each release so that the CI works but as soon as we forget it pip will install the wrong pandas-stubs - seems a bit fragile.

[Dr-Irv]

Issue here is that we need this to work for PR's, which means that contributors have to change the version number.

In the test.yaml, I am now doing this:

python -m pip install --find-links=./dist --force-reinstall pandas-stubs

The --find-links=./dist makes it so that pip will look in the dist directory. On Windows, you can't do pip install dist/*, so this is the workaround for that.

[twoertwein]

In the above steps we can use poe ..., why can't we use it here?

[Dr-Irv]

As mentioned above, if I use the run script, it installs the wheel into the "system" environment, not poetry's virtual environment.

[twoertwein]

So the issue is that we have at least two python executables and for some reason, poetry/windows CI uses both of them in an inconsistent manner?

Is it possible to remove one of them from the PATH (or delete it) so that poetry/CI is not confused by it?

[Dr-Irv]

I didn't merge because looks like you accidentally got some pyi changes in that aren't needed.

That was intentional, as pyright was reporting that some imports were missing.

[Dr-Irv]

So the issue is that we have at least two python executables and for some reason, poetry/windows CI uses both of them in an inconsistent manner?

Is it possible to remove one of them from the PATH (or delete it) so that poetry/CI is not confused by it?

I don't know. Prior to a few days ago, we always had 2 environments on Windows and things worked. With the setup-python action, it creates a "system" python, and then poetry creates a virtual one. I created an issue in the poetry repo thinking that the upgrade from poetry 1.1.15 to 1.2 was the issue, but the poetry people found a failed CI run on Windows using 1.1.15.

I'm now thinking that it is the setup-python action that messed things up.

[Dr-Irv]

I was going to merge this since green. Fix looks fine, although the source'ing is annoying. but maybe necessary. Could you instead call poetry shell before the poe commands.

See https://github.com/snok/install-poetry#running-on-windows as to why we have to do the sourcing.

[Dr-Irv]

@bashtage @twoertwein I am unsure what to do about this whole issue. @bashtage pushed a commit last night and the current CI worked just fine. So the error we are getting is a bit random.

Ideally, we'd have a failed run and a successful CI run on Windows, where the only difference was small code changes. Then we could compare the logs and see what is happening.

If you look at the successful run at https://github.com/pandas-dev/pandas-stubs/runs/8132062193?check_suite_focus=true in the past few hours, you see this:

Creating virtualenv pandas-stubs-rIU6jvyd-py3.8 in C:\Users\runneradmin\AppData\Local\pypoetry\Cache\virtualenvs
Using virtualenv: C:\Users\runneradmin\AppData\Local\pypoetry\Cache\virtualenvs\pandas-stubs-rIU6jvyd-py3.8

This says poetry is installing everything into a virtual environment.

But then when mypy is run, you see this:

2022-09-01T09:37:34.6906054Z shell: C:\Program Files\PowerShell\7\pwsh.EXE -command ". '{0}'"
2022-09-01T09:37:34.6906454Z env:
2022-09-01T09:37:34.6906856Z   pythonLocation: C:\hostedtoolcache\windows\Python\3.8.10\x64
2022-09-01T09:37:34.6907333Z   PKG_CONFIG_PATH: C:\hostedtoolcache\windows\Python\3.8.10\x64/lib/pkgconfig
2022-09-01T09:37:34.6907842Z   Python_ROOT_DIR: C:\hostedtoolcache\windows\Python\3.8.10\x64
2022-09-01T09:37:34.6908314Z   Python2_ROOT_DIR: C:\hostedtoolcache\windows\Python\3.8.10\x64
2022-09-01T09:37:34.6908791Z   Python3_ROOT_DIR: C:\hostedtoolcache\windows\Python\3.8.10\x64
2022-09-01T09:37:34.6909175Z ##[endgroup]
2022-09-01T09:37:36.2769734Z Poe => mypy
2022-09-01T09:37:57.9046815Z Success: no issues found in 217 source files

So it seems to say it will run from the python that is installed by the setup-python script.

But in a failure from here: https://github.com/pandas-dev/pandas-stubs/runs/8132162205?check_suite_focus=true, you get the same beginning, although poetry is using the cache in that case. Then when mypy fails, you get this:

2022-09-01T09:43:28.6803975Z �[36;1mpoetry run poe mypy�[0m
2022-09-01T09:43:28.6850454Z shell: C:\Program Files\PowerShell\7\pwsh.EXE -command ". '{0}'"
2022-09-01T09:43:28.6850800Z env:
2022-09-01T09:43:28.6851127Z   pythonLocation: C:\hostedtoolcache\windows\Python\3.8.10\x64
2022-09-01T09:43:28.6851567Z   PKG_CONFIG_PATH: C:\hostedtoolcache\windows\Python\3.8.10\x64/lib/pkgconfig
2022-09-01T09:43:28.6852012Z   Python_ROOT_DIR: C:\hostedtoolcache\windows\Python\3.8.10\x64
2022-09-01T09:43:28.6852446Z   Python2_ROOT_DIR: C:\hostedtoolcache\windows\Python\3.8.10\x64
2022-09-01T09:43:28.6852858Z   Python3_ROOT_DIR: C:\hostedtoolcache\windows\Python\3.8.10\x64
2022-09-01T09:43:28.6853158Z ##[endgroup]
2022-09-01T09:43:30.1866948Z Poe => mypy
2022-09-01T09:43:31.3020569Z Traceback (most recent call last):
2022-09-01T09:43:31.3029206Z   File "<string>", line 1, in <module>
2022-09-01T09:43:31.3030149Z   File "c:\hostedtoolcache\windows\python\3.8.10\x64\lib\importlib\__init__.py", line 127, in import_module
2022-09-01T09:43:31.3030741Z     return _bootstrap._gcd_import(name[level:], package, level)

Note that it tried to use the system installed version.

Maybe there is a cache file sitting around that works, and another one that doesn't, and @bashtage was lucky in one run to pick up the cache, and unlucky in another run where it didn't pick up the cache.

I don't understand how this caching works. Maybe @twoertwein has an idea.

Given the randomness we are seeing, I don't know if we should merge this PR.

[Dr-Irv]

closing in favor of #249