Configure CI to fail on skip of optional deps?

[TomAugspurger]

We have a recurring problem where tests are skipped without us realizing. See the discussion on #26852 and elsewhere.

Can we think of a way to make this more robust? Perhaps by adding an additional command line option to our pytest calls

pytest pandas --no-skip-xclip

Then we would update our calls to pytest.importorskip to use a pandas' wrapper that checks the config options. By default, it'll still skip, but if the --no-skip-{dep} flag is passed we would fail.

This would complicate the actual pytest call in the CI scripts, though it's already quite complicated.

[datapythonista]

My preference would be to have the I/O modules and their tests in a separate directory and be able to test them explicitly when needed (e.g. pytest pandas/ io/parquet io/excel). Proposed in #26804

My second preferred option would be mark tests and call them explicitly too, so we don't have the skips either (e.g. `pytest -m "parquet and excel").

What you propose seems reasonable, and less verbose, but I think more difficult to learn, understand, and to have clear what's being tested IMO. I'm ok with it, but to me sounds more like a temporary solution, better than what we have, but we'll still have similar problems I think.

[TomAugspurger]

Would your option 2 (adding marks & calling pytest -m "parquet and excel) work out of the box? I would think they would still be importorskipped if the dep isn't present.

[datapythonista]

If we just add the mark you're right, but I think we can remove the skip if we have the mark, and use -m "not excel" if we want to skip. It's surely more verbose, but I'd say it's worth, and we'd never have a problem that something is not tested without knowing.

[simonjayhawkins]

If we just add the mark you're right, but I think we can remove the skip if we have the mark, and use -m "not excel" if we want to skip. It's surely more verbose, but I'd say it's worth, and we'd never have a problem that something is not tested without knowing.

testing locally would also need to use the -m flag if optional dependencies aren't installed. so could probably do with another flag to indicate it is a CI test run and fail instead of skip.