GHT indicator is failing on 8/20

[krivard]

GHT has not been able to deliver an update to the API since the 8/20 issue because of the error below. What's causing it? Can we repair it? If not, can we modify the pipeline to permit gaps like this?

Traceback:

Traceback (most recent call last):
  File "/home/indicators/.pyenv/versions/3.8.2/lib/python3.8/runpy.py", line 193, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/indicators/.pyenv/versions/3.8.2/lib/python3.8/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/home/indicators/runtime/google_health/delphi_google_health/__main__.py", line 11, in <module>
    run_module()  # pragma: no cover
  File "/home/indicators/runtime/google_health/delphi_google_health/run.py", line 59, in run_module
    df_hrr, df_msa = derived_counts_from_dma(df_dma, static_dir=static_dir)
  File "/home/indicators/runtime/google_health/delphi_google_health/map_values.py", line 30, in derived_counts_from_dma
    dma_mat, day_list = _dma_df_to_matrix(df_dma, static_dir)
  File "/home/indicators/runtime/google_health/delphi_google_health/map_values.py", line 97, in _dma_df_to_matrix
    raise ValueError(f"Missing values for day={day} and dma={dma}")
ValueError: Missing values for day=2020-08-20 and dma=534

[jingjtang]

It seems to be a problem on their side. I ran the pipeline just now. It was finished successfully.

I have also run into such a problem previously, usually the error disappeared when I ran it again.

[krivard]

Weird -- it's been failing every time for the last four days. Can we put in a loop to have it automatically try again? Otherwise we'll have to manually re-run the job whenever it fails.

[capnrefsmmat]

Can we put in a loop to have it automatically try again?

I'll just leave this here https://tenacity.readthedocs.io/en/latest/

[jingjtang]

PR here to get it try until successful. Cannot get it tested currently.

[jingjtang]

Switch to tenacity. Now set it to try function get_counts_states and get_counts_states 5 times until success. Before a new try, it will wait for 60s.