Backport PR pandas-dev#55596 on branch 2.1.x (CI: Have NumpyDev build only test nightly numpy) (pandas-dev#55602)

Backport PR pandas-dev#55596: CI: Have NumpyDev build only test nightly numpy

Co-authored-by: Matthew Roeschke <[email protected]>

Author: meeseeksmachine

ci/deps/actions-311-numpydev.yaml

@@ -29,5 +29,4 @@ dependencies:
     - "--extra-index-url https://pypi.anaconda.org/scientific-python-nightly-wheels/simple"
     - "--pre"
     - "numpy"
-    - "scipy"
     - "tzdata>=2022.1"