CI: Don't run PANDAS_DATA_MANAGER=array test for downstream tests

[mroeschke]

This should shorten the downstream build such that pandas/tests/test_downstream.py is run instead of pandas/

[jreback]

looks fine, can you double confirm that now running the correct tests (and not running what not supposed to) and doesn't break other builds

[mroeschke]

looks fine, can you double confirm that now running the correct tests (and not running what not supposed to) and doesn't break other builds

Downstream build:

xvfb-run pytest -r fEs -n auto --dist=loadfile -s --cov=pandas --cov-report=xml --cov-append pandas/tests/test_downstream.py -m "not slow and not network and not single_cpu"
============================= test session starts ==============================
platform linux -- Python 3.8.12, pytest-7.0.1, pluggy-1.0.0
rootdir: /home/runner/work/pandas/pandas, configfile: pyproject.toml
plugins: hypothesis-6.39.2, xdist-2.5.0, forked-1.4.0, cov-3.0.0
gw0 I / gw1 I
gw0 [13] / gw1 [13]

.............
------ generated xml file: /home/runner/work/pandas/pandas/test-data.xml -------

---------- coverage: platform linux, python 3.8.12-final-0 -----------
Coverage XML written to file coverage.xml

============================= slowest 30 durations =============================
2.29s call     pandas/tests/test_downstream.py::test_oo_optimizable
1.26s call     pandas/tests/test_downstream.py::test_torch_frame_construction
0.86s call     pandas/tests/test_downstream.py::test_missing_required_dependency
0.84s call     pandas/tests/test_downstream.py::test_oo_optimized_datetime_index_unpickle
0.35s call     pandas/tests/test_downstream.py::test_geopandas
0.31s call     pandas/tests/test_downstream.py::test_scikit_learn
0.14s call     pandas/tests/test_downstream.py::test_dask
0.04s call     pandas/tests/test_downstream.py::test_yaml_dump
0.01s call     pandas/tests/test_downstream.py::test_pandas_gbq
0.01s call     pandas/tests/test_downstream.py::test_dask_ufunc
0.01s call     pandas/tests/test_downstream.py::test_pyarrow
0.01s setup    pandas/tests/test_downstream.py::test_dask

(18 durations < 0.005s hidden.  Use -vv to show these durations.)
============================= 13 passed in 24.33s ==============================

Non downstream build

xvfb-run pytest -r fEs -n auto --dist=loadfile -s --cov=pandas --cov-report=xml --cov-append pandas -m "not single_cpu"
============================= test session starts ==============================
platform linux -- Python 3.8.12, pytest-7.0.1, pluggy-1.0.0
rootdir: /home/runner/work/pandas/pandas, configfile: pyproject.toml, testpaths: pandas
plugins: hypothesis-6.39.2, xdist-2.5.0, forked-1.4.0, cov-3.0.0
gw0 I / gw1 I
gw0 [164036] / gw1 [164036]

...

= 161531 passed, 1186 skipped, 1317 xfailed, 2 xpassed, 28 warnings in 2492.74s (0:41:32) =

PANDAS_DATA_MANAGER=array pytest -n auto --dist=loadfile -s --cov=pandas --cov-report=xml --cov-append pandas -m "not single_cpu and arraymanager"
============================= test session starts ==============================
platform linux -- Python 3.8.12, pytest-7.0.1, pluggy-1.0.0
rootdir: /home/runner/work/pandas/pandas, configfile: pyproject.toml, testpaths: pandas
plugins: hypothesis-6.39.2, xdist-2.5.0, forked-1.4.0, cov-3.0.0
gw0 I / gw1 I
gw0 [7894] / gw1 [7894]

...

= 7506 passed, 77 skipped, 309 xfailed, 2 xpassed, 18 warnings in 267.15s (0:04:27) =