fix broken usafacts tests to read from the proper directories #314

sgsmob · 2020-10-15T15:49:32Z

usafacts was trying to read/write to/from receiving when run in the tests directory rather than ../receiving. The tests now pass.

…hi#182) * update cache * remove default end_date * update start_date * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cahce * Set up initial google_health-deploy branch - Add new google_health Jenkins pipeline stage scripts - Add the abilty for Ansible to write either a file or a template depending on which has been configured for the indicator - Add Ansible template directory (special tall bookshelf) - Add the ability to keep sensitive variables in `vault.yaml` - Add google_health production params template Encrypt vault.yaml * Use templates dir * Remove end date from params * Switch to midas export dir * Properly rename Jenkins pipeline stage scripts * Handle google_health's testing needs - Add: Ansible playbook for securely handling placing the `params.json` template we need for testing. This will happen during the Jenkins build stage when we are setting up the venv in the workspace on the Jenkins server. - Add: Test `params.json` template. - Add: Jenkins user variable. - Fix: Was incorrectly trying to use a file instead of a template in `ansible-deploy.yaml`. - Add: Call the small Ansible playbook from the Jenkins build wrapper. * Change to the Ansible root dir before trying to do Ansible things * Delegate to localhost * Tell Ansible we want to connect locally for this playbook * Remove set -x from Bash scripts * Add initial Jenkins/Ansible necessities for CI/CD and Automation - Add Jenkins pipeline scripts - Add production params template - Update vars and vault files with AWS secrets * Remove ght cache files * Disable pylint convention messages * Work around linter errors Co-authored-by: Addison Hu <[email protected]> Co-authored-by: Jingjing Tang <[email protected]>

Merge main into deploy-google_health

Fixing JHU deployment

Further refactoring for the geo coding utility

Propagate geomapper changes into JHU

…o main

Co-authored-by: Addison Hu <[email protected]>

Allow archive to be run as its own module

* change valid geo_res * change GEO_RESOLUTIOSN in run.py * update tests to use "county"

Fix JHU bug that renamed fips to county in receiving

Propagate fips -> county filename fix from JHU back into main

[sir-complainsalot] Add grace period and dry run

jingjtang · 2020-10-17T16:50:21Z

Hi @sgsmob, I think a better way to solve this is to add a ./receiving folder under tests, then we will stored the output from tests under usafacts/tests/receiving and check those file there.

The current solution seems weird. Since the receiving dir in usafacts/tests/params.json is stilled defined as usafacts/tests/receiving, but in test_run.py we check the output from usafacts/receiving.

sgsmob · 2020-10-19T14:38:56Z

Changed to add the ./receiving directory.

jingjtang

All unit tests passed

The base branch was changed.

krivard · 2020-10-20T14:05:25Z

Oh ffs. @korlaxxalrok this PR seems to trigger the same pylint/asteroid "recursion limit exceeded" problem as we were having in JHU. Tests seem to pass in Jingjing's environment so I think we're okay to disable them here the same way. If you agree, go ahead and do that so we can unblock Mike.

Afterwards, could you work with @benjaminysmith to figure out a more permanent solution?

sgsmob · 2020-10-20T14:17:49Z

Yes I observed the recursion limit issue. FWIW I have a pylint extension on my IDE and I corrected the mistakes there.

korlaxxalrok · 2020-10-20T14:59:41Z

I'll disable linting for now, and then spend some time looking at an actual fix.

korlaxxalrok · 2020-10-20T16:51:45Z

@krivard @sgsmob Linting is disabled.

korlaxxalrok and others added 30 commits August 5, 2020 15:39

Add missing path parameter

4183fd3

Fix variable name, reencrypt vault.yaml

4ded043

fixed geo test

e654484

try until pull the new data successfully

b566d4c

Part of fix to smoother test, not done

eabc032

Fix smoother test

58343f2

used tenacity

8db2799

Add .gitignore

c9b16a5

Update params.json.template

f969d96

Update run.py

56db650

Update setup.py

775efe1

Update test_update_sensor.py

df20f05

Update update_sensor.py

9295eac

Add constants.py

786cef4

Add static files

3391ff3

Minor changes

40091b7

removed wildcard import error

db54eb8

update params.json.template

2a9a68f

Add wip_signal key in ansible

319f038

Add missing receiving directory

17fb5ae

Merge pull request cmu-delphi#239 from cmu-delphi/main

8a40c1c

Merge main into deploy-google_health

Merge pull request cmu-delphi#230 from cmu-delphi/main

c04c722

Fixing JHU deployment

Add receiving directory

a3a582c

Update gitignore to fix merge conflict

361cd38

Update gitignore to fix merge conflict

274ac89

Update gitignore to fix merge conflict

1ad2a52

Update gitignore

78c6674

Fix merge conflict, add gitkeep to pin directory

fc25a0d

Really add gitkeep

96fa5f5

krivard and others added 11 commits October 7, 2020 13:43

Remove trailing whitespace

b830cc8

Merge pull request cmu-delphi#217 from cmu-delphi/rf_geo_refactor

8867c14

Further refactoring for the geo coding utility

Merge pull request cmu-delphi#297 from cmu-delphi/main

650c12e

Propagate geomapper changes into JHU

Merge branch 'main' of github.com:cmu-delphi/covidcast-indicators int…

547fab8

…o main

Fix docs typo in _delphi_utils_python/delphi_utils/archive.py

8d77066

Co-authored-by: Addison Hu <[email protected]>

Merge pull request cmu-delphi#295 from sgsmob/main

8678fba

Allow archive to be run as its own module

Fix JHU bug that renamed fips to county in receiving

309fb79

* change valid geo_res * change GEO_RESOLUTIOSN in run.py * update tests to use "county"

Merge pull request cmu-delphi#304 from cmu-delphi/jhu_fips_county_fix

93b5ba2

Fix JHU bug that renamed fips to county in receiving

Merge pull request cmu-delphi#305 from cmu-delphi/deploy-jhu

d357d01

Propagate fips -> county filename fix from JHU back into main

Merge pull request cmu-delphi#287 from cmu-delphi/sir-dryrun

cf6f4c1

[sir-complainsalot] Add grace period and dry run

fix broken usafacts tests to read from the proper directories

4debc31

sgsmob requested a review from jingjtang October 15, 2020 15:49

sgsmob mentioned this pull request Oct 15, 2020

Refactor usafacts to use geo utils #316

Merged

make new receiving directory in test directory

51c0c03

jingjtang previously approved these changes Oct 19, 2020

View reviewed changes

krivard changed the base branch from main to deploy-usafacts October 20, 2020 13:44

krivard previously approved these changes Oct 20, 2020

View reviewed changes

benjaminysmith mentioned this pull request Oct 20, 2020

Pylint/Astroid gives "RecursionError: maximum recursion depth exceeded" in CI tests #333

Closed

Temporarily skip linting in Jenkins

a1a1b50

korlaxxalrok dismissed krivard’s stale review via a1a1b50 October 20, 2020 16:48

krivard approved these changes Oct 20, 2020

View reviewed changes

krivard merged commit b0bb289 into cmu-delphi:deploy-usafacts Oct 20, 2020

krivard mentioned this pull request Oct 20, 2020

Propagate USAFacts paths fix to main #334

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

fix broken usafacts tests to read from the proper directories #314

fix broken usafacts tests to read from the proper directories #314

Uh oh!

sgsmob commented Oct 15, 2020

Uh oh!

jingjtang commented Oct 17, 2020

Uh oh!

sgsmob commented Oct 19, 2020 •

edited

Loading

Uh oh!

jingjtang left a comment

Uh oh!

krivard commented Oct 20, 2020 •

edited

Loading

Uh oh!

sgsmob commented Oct 20, 2020

Uh oh!

korlaxxalrok commented Oct 20, 2020

Uh oh!

korlaxxalrok commented Oct 20, 2020

Uh oh!

Uh oh!

fix broken usafacts tests to read from the proper directories #314

fix broken usafacts tests to read from the proper directories #314

Uh oh!

Conversation

sgsmob commented Oct 15, 2020

Uh oh!

jingjtang commented Oct 17, 2020

Uh oh!

sgsmob commented Oct 19, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jingjtang left a comment

Choose a reason for hiding this comment

Uh oh!

krivard commented Oct 20, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sgsmob commented Oct 20, 2020

Uh oh!

korlaxxalrok commented Oct 20, 2020

Uh oh!

korlaxxalrok commented Oct 20, 2020

Uh oh!

Uh oh!

sgsmob commented Oct 19, 2020 •

edited

Loading

krivard commented Oct 20, 2020 •

edited

Loading