Possibly add retry to async tests

[dshemetov]

Some of the async tests sporadically fail, due to a suspected race condition, but fail after a few more retries of the tests. We may want to add a retry condition to automatically do this.

[chinandrew]

These are the local integration tests? Could you post one of the failure messages next time you hit this?

[dshemetov]

These are the local integration tests?

Yup!

Here are some error messages:

E           aiohttp.client_exceptions.ClientResponseError: 500, message='Internal Server Error', url=URL('http://delphi_web_epidata/epidata/api.php?source=covidcast&data_source=src&signals=sig&time_type=day&geo_type=county&geo_value=11111&time_values=20200414')

/usr/local/lib/python3.8/site-packages/aiohttp/client_reqrep.py:1000: ClientResponseError
=========================== short test summary info ============================
FAILED repos/delphi/delphi-epidata/integrations/acquisition/covid_hosp/facility/test_scenarios.py::AcquisitionTests::test_acquire_dataset
FAILED repos/delphi/delphi-epidata/integrations/client/test_delphi_epidata.py::DelphiEpidataPythonClientTests::test_async_epidata
========================= 2 failed, 47 passed in 5.38s =========================

and

E                        AssertionError: Lists differ: [{'ep[273 chars]}, {'epidata': [], 'message': "database error:[7911 chars]ts'}] != [{'ep[273 chars]}, {'result': -2, 'message': 'no results'}, {'[3523 chars]ts'}]
E
E                        First differing element 1:
E                        {'epidata': [], 'message': "database error:[659 chars]: -1}
E                        {'result': -2, 'message': 'no results'}
E
E                        Diff is 13820 characters long. Set self.maxDiff to None to see it.

This is what the docker web-server log shows when it occurs

[2021-10-05 13:21:02 -0400] [42] [INFO] Invalidate connection <_mysql.connection open to 'delphi_database_epidata' at 0x564cb285d120> (reason: OperationalError:(2006, 'MySQL server has gone away'))

[melange396]

does this still happen? i cant remember the last time i saw this (though absence of evidence is not evidence of absence, and my memory can be faulty)... changes elsewhere in the server code may have resolved the issue, and im pushing to deprecate the async stuff, so this can probably be closed.

[dshemetov]

When I was working on JIT last Winter, I don't recall running into this. Maybe it got fixed, maybe not, but agree that there's little reason to hunt for ghost bugs in code that's planned to deprecate.