Finalize production config for google_health deployment #165
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Add: Ansible playbook for securely handling placing the `params.json` template we need for testing. This will happen during the Jenkins build stage when we are setting up the venv in the workspace on the Jenkins server. - Add: Test `params.json` template. - Add: Jenkins user variable. - Fix: Was incorrectly trying to use a file instead of a template in `ansible-deploy.yaml`. - Add: Call the small Ansible playbook from the Jenkins build wrapper.
|
There are failing tests that we need to address before we can complete CI/CD and Automation for this indicator. |
|
@eujing has analyzed the code and found that the test has a bug. It depends on a fixed ordering coming out of |
|
I am going to merge this (even though tests fail), then follow up with another fix from @vishakha1812. |
krivard
pushed a commit
that referenced
this pull request
Aug 13, 2020
* Switch to midas export dir * Properly rename Jenkins pipeline stage scripts * Handle google_health's testing needs - Add: Ansible playbook for securely handling placing the `params.json` template we need for testing. This will happen during the Jenkins build stage when we are setting up the venv in the workspace on the Jenkins server. - Add: Test `params.json` template. - Add: Jenkins user variable. - Fix: Was incorrectly trying to use a file instead of a template in `ansible-deploy.yaml`. - Add: Call the small Ansible playbook from the Jenkins build wrapper. * Change to the Ansible root dir before trying to do Ansible things * Delegate to localhost * Tell Ansible we want to connect locally for this playbook * Remove set -x from Bash scripts * used set instead of list to compare dict.keys() (#186) Co-authored-by: Vishakha Srivastava <[email protected]>
korlaxxalrok
added a commit
that referenced
this pull request
Aug 13, 2020
* Fix test affected by backfill * Set target python version * Update: added new use case in safegraph * change in params.json.template * update in params.json.template: wip_prefix to wip_signal * Changed func name from epidata_signal() -- > public_signal(), signal_name() --> add_prefix() for better readability * cleaner code, * handling wip signal in python template * Minor changes in process.py, updated test cases * signal naming using wip_signal parameter * Supply missing params.json.template in tests * Switch from raw Epidata client to COVIDcast client * Move signal name constants to their own file, referenced from both run.py and process.py * Corrected logic to comply with #113 * Also converted unit test to test the process.py logic instead of the validity of the params file. * Corrected errors I'd introduced in the conversion to covidcast client * changes in compliance with #205 * Missing comma in setup.py * Linter suggested changes, added HOME_DWELL signal computation * changes in compliance with #205 * default prefix value * Apply suggestions from code review Co-authored-by: krivard <[email protected]> * Removed unnecessary comment * Updated constants.py * Updated README.md * Updated update_sensor.py * updated params.json.template * updated tests/params.json.template * New test case file for handling wip signal * Updated test_update_sensor.py * Update constants.py * Update update_sensor.py * Update test_update_sensor.py * Update test_update_sensor.py * Update update_sensor.py * Update update_sensor.py * Set up initial google_health-deploy branch - Add new google_health Jenkins pipeline stage scripts - Add the abilty for Ansible to write either a file or a template depending on which has been configured for the indicator - Add Ansible template directory (special tall bookshelf) - Add the ability to keep sensitive variables in `vault.yaml` - Add google_health production params template Encrypt vault.yaml - Use templates dir - Configure start_date and end_date * Finalize production config for google_health deployment (#165) * Switch to midas export dir * Properly rename Jenkins pipeline stage scripts * Handle google_health's testing needs - Add: Ansible playbook for securely handling placing the `params.json` template we need for testing. This will happen during the Jenkins build stage when we are setting up the venv in the workspace on the Jenkins server. - Add: Test `params.json` template. - Add: Jenkins user variable. - Fix: Was incorrectly trying to use a file instead of a template in `ansible-deploy.yaml`. - Add: Call the small Ansible playbook from the Jenkins build wrapper. * Change to the Ansible root dir before trying to do Ansible things * Delegate to localhost * Tell Ansible we want to connect locally for this playbook * Remove set -x from Bash scripts * used set instead of list to compare dict.keys() (#186) Co-authored-by: Vishakha Srivastava <[email protected]> * Use brace instead of bracket (#189) * Add path parameter (#190) * Allow `""` for `start_date` to get "latest" data (#194) - We can treat this like we do `end_date` and default to a date to some number of days in the past. The result is that we only process updates for one day, instead of of the whole shebang. - Add new cache files, a temporary necessity. * Set start date to empty string (#203) * Add a cache set that this identical to production (#211) * Sync cache * Remove end date from params * Add AWS credentials to vault Co-authored-by: Chua Eu Jing <[email protected]> Co-authored-by: Vishakha <[email protected]> Co-authored-by: Kathryn M Mazaitis <[email protected]> Co-authored-by: Addison Hu <[email protected]>
krivard
added a commit
that referenced
this pull request
Aug 19, 2020
* Fix test affected by backfill * Set target python version * Update: added new use case in safegraph * change in params.json.template * update in params.json.template: wip_prefix to wip_signal * Changed func name from epidata_signal() -- > public_signal(), signal_name() --> add_prefix() for better readability * cleaner code, * handling wip signal in python template * Minor changes in process.py, updated test cases * signal naming using wip_signal parameter * Supply missing params.json.template in tests * Switch from raw Epidata client to COVIDcast client * Move signal name constants to their own file, referenced from both run.py and process.py * Corrected logic to comply with #113 * Also converted unit test to test the process.py logic instead of the validity of the params file. * Corrected errors I'd introduced in the conversion to covidcast client * changes in compliance with #205 * Missing comma in setup.py * Linter suggested changes, added HOME_DWELL signal computation * changes in compliance with #205 * default prefix value * Apply suggestions from code review Co-authored-by: krivard <[email protected]> * Removed unnecessary comment * Updated constants.py * Updated README.md * Updated update_sensor.py * updated params.json.template * updated tests/params.json.template * New test case file for handling wip signal * Updated test_update_sensor.py * Update constants.py * Update update_sensor.py * Update test_update_sensor.py * Update test_update_sensor.py * Update update_sensor.py * Update update_sensor.py * Set up initial google_health-deploy branch - Add new google_health Jenkins pipeline stage scripts - Add the abilty for Ansible to write either a file or a template depending on which has been configured for the indicator - Add Ansible template directory (special tall bookshelf) - Add the ability to keep sensitive variables in `vault.yaml` - Add google_health production params template Encrypt vault.yaml - Use templates dir - Configure start_date and end_date * Finalize production config for google_health deployment (#165) * Switch to midas export dir * Properly rename Jenkins pipeline stage scripts * Handle google_health's testing needs - Add: Ansible playbook for securely handling placing the `params.json` template we need for testing. This will happen during the Jenkins build stage when we are setting up the venv in the workspace on the Jenkins server. - Add: Test `params.json` template. - Add: Jenkins user variable. - Fix: Was incorrectly trying to use a file instead of a template in `ansible-deploy.yaml`. - Add: Call the small Ansible playbook from the Jenkins build wrapper. * Change to the Ansible root dir before trying to do Ansible things * Delegate to localhost * Tell Ansible we want to connect locally for this playbook * Remove set -x from Bash scripts * used set instead of list to compare dict.keys() (#186) Co-authored-by: Vishakha Srivastava <[email protected]> * Use brace instead of bracket (#189) * Add path parameter (#190) * Allow `""` for `start_date` to get "latest" data (#194) - We can treat this like we do `end_date` and default to a date to some number of days in the past. The result is that we only process updates for one day, instead of of the whole shebang. - Add new cache files, a temporary necessity. * Set start date to empty string (#203) * Add a cache set that this identical to production (#211) * Sync cache * Remove end date from params * Add AWS credentials to vault Co-authored-by: Chua Eu Jing <[email protected]> Co-authored-by: Vishakha <[email protected]> Co-authored-by: Kathryn M Mazaitis <[email protected]> Co-authored-by: Addison Hu <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.