Merge the SafeGraph deployment branch updates to main #266
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* update cache * remove default end_date * update start_date * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cahce * Set up initial google_health-deploy branch - Add new google_health Jenkins pipeline stage scripts - Add the abilty for Ansible to write either a file or a template depending on which has been configured for the indicator - Add Ansible template directory (special tall bookshelf) - Add the ability to keep sensitive variables in `vault.yaml` - Add google_health production params template Encrypt vault.yaml * Use templates dir * Remove end date from params * Switch to midas export dir * Properly rename Jenkins pipeline stage scripts * Handle google_health's testing needs - Add: Ansible playbook for securely handling placing the `params.json` template we need for testing. This will happen during the Jenkins build stage when we are setting up the venv in the workspace on the Jenkins server. - Add: Test `params.json` template. - Add: Jenkins user variable. - Fix: Was incorrectly trying to use a file instead of a template in `ansible-deploy.yaml`. - Add: Call the small Ansible playbook from the Jenkins build wrapper. * Change to the Ansible root dir before trying to do Ansible things * Delegate to localhost * Tell Ansible we want to connect locally for this playbook * Remove set -x from Bash scripts * Add initial Jenkins/Ansible necessities for CI/CD and Automation - Add Jenkins pipeline scripts - Add production params template - Update vars and vault files with AWS secrets * Remove ght cache files * Disable pylint convention messages * Work around linter errors Co-authored-by: Addison Hu <[email protected]> Co-authored-by: Jingjing Tang <[email protected]>
* Add updated encrypted credentials * Add wip_signal to production params template * Add newline * Run on 12 cores * Add shell script to run the indicator
* Add updated encrypted credentials
* Add wip_signal to production params template
* Add newline
* Run on 12 cores
* Add shell script to run the indicator
* Update run script
- Use production ingestion dir
- Remove hard fail to work around an issue with `cp`.
- `cp` fails when files don't exist. I thought we could squash the
error by sending stderr to /dev/null, but `set -eo` still catches
and fails the script. Even worse, it happens silently. There should
be a better way to handle this so will add it to a future task.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This merges in the Jenkins and Ansible components needed to deploy the SafeGraph indicator.
Notable additions:
vault.yamlfor accessing SafeGraph S3 assets.Everything else is similar to what has been created for the other indicators so should not be too surprising.
The deployment branch works as expected and it should be safe enough to merge these changes, but I am happy for any review that seems necessary or warranted.