Skip to content

Commit 2fd5c72

Browse files
korlaxxalrokhuisaddisonJingjing Tang
authored
Add new configuration to set up safegraph deployment branch (#182)
* update cache * remove default end_date * update start_date * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cahce * Set up initial google_health-deploy branch - Add new google_health Jenkins pipeline stage scripts - Add the abilty for Ansible to write either a file or a template depending on which has been configured for the indicator - Add Ansible template directory (special tall bookshelf) - Add the ability to keep sensitive variables in `vault.yaml` - Add google_health production params template Encrypt vault.yaml * Use templates dir * Remove end date from params * Switch to midas export dir * Properly rename Jenkins pipeline stage scripts * Handle google_health's testing needs - Add: Ansible playbook for securely handling placing the `params.json` template we need for testing. This will happen during the Jenkins build stage when we are setting up the venv in the workspace on the Jenkins server. - Add: Test `params.json` template. - Add: Jenkins user variable. - Fix: Was incorrectly trying to use a file instead of a template in `ansible-deploy.yaml`. - Add: Call the small Ansible playbook from the Jenkins build wrapper. * Change to the Ansible root dir before trying to do Ansible things * Delegate to localhost * Tell Ansible we want to connect locally for this playbook * Remove set -x from Bash scripts * Add initial Jenkins/Ansible necessities for CI/CD and Automation - Add Jenkins pipeline scripts - Add production params template - Update vars and vault files with AWS secrets * Remove ght cache files * Disable pylint convention messages * Work around linter errors Co-authored-by: Addison Hu <[email protected]> Co-authored-by: Jingjing Tang <[email protected]>
1 parent fae9e1d commit 2fd5c72

File tree

277 files changed

+253
-34716
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

277 files changed

+253
-34716
lines changed

‎ansible/ansible-deploy.yaml

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,7 @@
22
- hosts: runtime_host
33
vars_files:
44
- vars.yaml
5+
- vault.yaml
56
tasks:
67
- name: Copy and unarchive the package into the indicators runtime host directory.
78
unarchive:
@@ -18,9 +19,26 @@
1819
group: "{{ runtime_user }}"
1920
state: link
2021

22+
- name: Check to see if we have a params file to send.
23+
local_action: stat files/{{ indicator }}-params-prod.json
24+
register: file
25+
26+
- name: Check to see if we have a params template to send.
27+
local_action: stat templates/{{ indicator }}-params-prod.json.j2
28+
register: template
29+
2130
- name: Set production params file.
2231
copy:
2332
src: files/{{ indicator }}-params-prod.json
2433
dest: "{{ indicators_runtime_dir }}/{{ indicator }}/params.json"
2534
owner: "{{ runtime_user }}"
2635
group: "{{ runtime_user }}"
36+
when: file.stat.exists
37+
38+
- name: Set production params template.
39+
template:
40+
src: templates/{{ indicator }}-params-prod.json.j2
41+
dest: "{{ indicators_runtime_dir }}/{{ indicator }}/params.json"
42+
owner: "{{ runtime_user }}"
43+
group: "{{ runtime_user }}"
44+
when: template.stat.exists

‎ansible/google_health-build.yaml

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
---
2+
- hosts: localhost
3+
connection: local
4+
vars_files:
5+
- vars.yaml
6+
- vault.yaml
7+
tasks:
8+
- name: Set test params template.
9+
template:
10+
src: templates/{{ indicator }}-params-test.json.j2
11+
dest: "{{ workspace }}/{{ indicator }}/tests/params.json"
12+
owner: "{{ jenkins_user }}"
13+
group: "{{ jenkins_user }}"
14+
mode: "0644"
15+
delegate_to: localhost

‎ansible/templates/google_health-params-prod.json.j2

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
{
2+
"static_file_dir": "./static",
3+
"export_dir": "/common/covidcast/receiving/ght",
4+
"cache_dir": "./cache",
5+
"start_date": "2020-02-01",
6+
"end_date": "",
7+
"ght_key": "{{ google_health_api_key }}"
8+
}

‎ansible/templates/google_health-params-test.json.j2

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
{
2+
"static_file_dir": "../static",
3+
"export_dir": "./receiving",
4+
"cache_dir": "../cache",
5+
"start_date": "2020-02-11",
6+
"end_date": "2020-04-30",
7+
"ght_key": "{{ google_health_api_key }}"
8+
}

‎ansible/templates/safegraph-params-prod.json.j2

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
{
2+
"static_file_dir": "./static",
3+
"raw_data_dir": "/common/safegraph",
4+
"export_dir": "./receiving",
5+
"cache_dir": "./cache",
6+
"n_core": "6",
7+
"aws_access_key_id": "{{ safegraph_aws_access_key_id }}",
8+
"aws_secret_access_key": "{{ safegraph_aws_secret_access_key }}",
9+
"aws_default_region": "us-east-1",
10+
"aws_endpoint": "https://s3.wasabisys.com"
11+
}

‎ansible/vars.yaml

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,13 @@
11
---
22
runtime_user: "indicators"
3+
jenkins_user: "jenkins"
34
jenkins_artifact_dir: "/var/lib/jenkins/artifacts"
45
indicators_runtime_dir: "/home/{{ runtime_user }}/runtime"
5-
package: "{{ indicator }}.tar.gz" # This is passed in the Ansible invocation.
6+
package: "{{ indicator }}.tar.gz" # {{ indicator }} is passed in from the Jenkins shell script wrapper.
67
python_version: "3.8.2"
78
pyenv_python_path: "/home/{{ runtime_user }}/.pyenv/versions/{{ python_version }}/bin/python"
9+
10+
# Indicators variables.
11+
google_health_api_key: "{{ vault_google_health_api_key }}"
12+
safegraph_aws_access_key_id: "{{ vault_safegraph_aws_access_key_id }}"
13+
safegraph_aws_secret_access_key: "{{ vault_aws_safegraph_secret_access_key }}"

‎ansible/vault.yaml

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
$ANSIBLE_VAULT;1.1;AES256
2+
39626439303237333632613162313237313935316232393262346461353534356537376136646336
3+
6562346664633562363466373230303931313139306163350a633130356639343830613030633839
4+
65616533653265653636613337333634333439643762376162626262653730653632643462366464
5+
6134633532303530320a636464346365373463356566353438316233333632353235313463336134
6+
30613531323236393833343462336362353734613065313364316330376635633035653366376631
7+
38373630653638643536643936313063646436663861323838313564633430613635613062383938
8+
31323932336132353938613661333531626162323163373234613830663361643830343536396432
9+
32613739366331636561656237316432653665613366393265663336343465656437333331353764
10+
33313666663439646637313466386262383332653934336233646565383833633462346137653038
11+
65666332646162323663656135343536623533363530623566353963663930633135663136666265
12+
38383761323064613432623439663732656639613264303736343538396166303363653262636538
13+
36376463613335636433326336353165666237663637666333383039353938303962366636363734
14+
64393566363730316231393362326161393063373565313133646661366562353039303965353037
15+
66386364303837666466383865333730396339353235653634623735306431383730636434306536
16+
353937393766396332646232666336343266

‎google_health/cache/Data_500_anosmia_ms.csv

Lines changed: 0 additions & 133 deletions
This file was deleted.

‎google_health/cache/Data_501_anosmia_ms.csv

Lines changed: 0 additions & 133 deletions
This file was deleted.

0 commit comments

Comments
 (0)