TST: Fix flaky import test

[TomAugspurger]

I'm not sure why, but the missing dependency test is
causing issues. Now we check that things work by running
it in a subprocess with site-packages disabled.

Closes #26952

[codecov]

Codecov Report

Merging #26953 into master will decrease coverage by <.01%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master   #26953      +/-   ##
==========================================
- Coverage   91.87%   91.86%   -0.01%     
==========================================
  Files         180      180              
  Lines       50746    50746              
==========================================
- Hits        46624    46619       -5     
- Misses       4122     4127       +5

Flag	Coverage Δ
#multiple	90.46% <ø> (ø)	⬆️
#single	41.1% <ø> (-0.08%)	⬇️

Impacted Files	Coverage Δ
pandas/io/gbq.py	88.88% <0%> (-11.12%)	⬇️
pandas/core/frame.py	96.89% <0%> (-0.12%)	⬇️
pandas/util/testing.py	90.84% <0%> (-0.11%)	⬇️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update baeb1bf...df1030b. Read the comment docs.

[codecov]

Codecov Report

Merging #26953 into master will decrease coverage by <.01%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master   #26953      +/-   ##
==========================================
- Coverage   91.87%   91.86%   -0.01%     
==========================================
  Files         180      180              
  Lines       50746    50746              
==========================================
- Hits        46624    46619       -5     
- Misses       4122     4127       +5

Flag	Coverage Δ
#multiple	90.46% <ø> (ø)	⬆️
#single	41.1% <ø> (-0.08%)	⬇️

Impacted Files	Coverage Δ
pandas/io/gbq.py	88.88% <0%> (-11.12%)	⬇️
pandas/core/frame.py	96.89% <0%> (-0.12%)	⬇️
pandas/util/testing.py	90.84% <0%> (-0.11%)	⬇️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update baeb1bf...df1030b. Read the comment docs.

[datapythonista]

I think the change here makes sense. But I'm missing something, the test I see failing is related but not the one fixed here:

=================================== FAILURES ===================================
______________________________ TestPDApi.test_api ______________________________
[gw3] darwin -- Python 3.5.6 /Users/vsts/miniconda3/envs/pandas-dev/bin/python

self = <pandas.tests.api.test_api.TestPDApi object at 0x12d4ec908>

    def test_api(self):
    
        self.check(pd,
                   self.lib + self.misc +
                   self.modules + self.deprecated_modules +
                   self.classes + self.deprecated_classes +
                   self.deprecated_classes_in_future +
                   self.funcs + self.funcs_option +
                   self.funcs_read + self.funcs_to +
                   self.deprecated_funcs_in_future +
                   self.deprecated_funcs + self.private_modules,
>                  self.ignored)

pandas/tests/api/test_api.py:113: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
pandas/tests/api/test_api.py:18: in check
    tm.assert_almost_equal(result, expected)
pandas/util/testing.py:325: in assert_almost_equal
    **kwargs)
pandas/_libs/testing.pyx:65: in pandas._libs.testing.assert_almost_equal
    cpdef assert_almost_equal(a, b,
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

>   raise_assert_detail(obj, '{0} length are different'.format(obj),
E   AssertionError: Iterable are different
E   
E   Iterable length are different
E   [left]:  132
E   [right]: 129
E   [diff]: ['dependency', 'missing_dependencies', 'hard_dependencies']

I can't understand why it's failing, if I understand it correctly, the test says that pandas got the attributes dependency,... and it shouldn't. I see that those shouldn't exist, as they are removed not long after they're created: https://github.com/pandas-dev/pandas/blob/master/pandas/__init__.py#L17

Not sure how is that possible, the import pandas shouldn't finish until __init__.py is fully executed I'd say. And the import seems just a regular import at the top of the file: https://github.com/pandas-dev/pandas/blob/master/pandas/tests/api/test_api.py#L104

What it comes to my mind is... Since we don't want those attributes to be public attributes of pandas, wouldn't make sense to prefix them with an underscore? Regardless of the test I think that would be correct (and should fix the test too).

@jreback @jorisvandenbossche I guess you want to have a look here, this is affecting few PRs.

[TomAugspurger]

My guess is that there's a strange interaction between the test / changes in ea06f8d and the existing test_api tests. See

import builtins
import pandas
import importlib


def mock_import_fail(name, *args, **kwargs):
    if name == "numpy":
        raise ImportError("cannot import name numpy")
    elif name == "pytz":
        raise ImportError("cannot import name some_dependency")
    elif name == "dateutil":
        raise ImportError("cannot import name some_other_dependency")
    else:
        return original_import(name, *args, **kwargs)


print(len(dir(pandas)))
builtins.__import__ = mock_import_fail
try:
    importlib.reload(pandas)
except:
    pass
print(len(dir(pandas)))

which outputs

141
144

So my guess is that if the missing required dep is run before the test_api, we fail.

[TomAugspurger]

Ah, here's a simpler reproducer

$ pytest pandas/tests/test_downstream.py::test_missing_required_dependency pandas/tests/api/test_api.py::TestPDApi::test_api

=================================================== FAILURES ====================================================
______________________________________________ TestPDApi.test_api _______________________________________________

self = <pandas.tests.api.test_api.TestPDApi object at 0x11f8ddcf8>

    def test_api(self):

        self.check(pd,
                   self.lib + self.misc +
                   self.modules + self.deprecated_modules +
                   self.classes + self.deprecated_classes +
                   self.deprecated_classes_in_future +
                   self.funcs + self.funcs_option +
                   self.funcs_read + self.funcs_to +
                   self.deprecated_funcs_in_future +
                   self.deprecated_funcs + self.private_modules,
>                  self.ignored)

pandas/tests/api/test_api.py:113:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
pandas/tests/api/test_api.py:18: in check
    tm.assert_almost_equal(result, expected)
pandas/util/testing.py:325: in assert_almost_equal
    **kwargs)
pandas/_libs/testing.pyx:65: in pandas._libs.testing.assert_almost_equal
    cpdef assert_almost_equal(a, b,
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

>   raise_assert_detail(obj, '{0} length are different'.format(obj),
E   AssertionError: Iterable are different
E
E   Iterable length are different
E   [left]:  132
E   [right]: 129
E   [diff]: ['dependency', 'missing_dependencies', 'hard_dependencies']

pandas/_libs/testing.pyx:163: AssertionError

I thought I tried that earlier, but apparently I did the wrong one. On my branch that passes.

[TomAugspurger]

I'm going to merge this since this is sometimes affecting other PRs. I'll be around all day to do fixups if needed.