Fix/jhu nyc support #272

huisaddison · 2020-09-11T15:37:02Z

Resolves #256 to account for the fact that JHU is now reporting borough-level cases. (Also removes unused detritus)

Diff of old and new county incidence prop for yesterday:

< 36005,5800000.0,NA,NA
---
> 36005,4.089670971867999,NA,NA
1852c1852
< 36047,11000000.0,NA,NA
---
> 36047,4.297037817448552,NA,NA
1859c1859
< 36061,0.611744266426863,NA,NA
---
> 36061,3.1313202014359867,NA,NA
1869c1869
< 36081,9300000.0,NA,NA
---
> 36081,4.126258175980918,NA,NA
1871c1871
< 36085,2400000.0,NA,NA
---
> 36085,5.040502538103049,NA,NA

* update cache * remove default end_date * update start_date * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cache * update cahce * Set up initial google_health-deploy branch - Add new google_health Jenkins pipeline stage scripts - Add the abilty for Ansible to write either a file or a template depending on which has been configured for the indicator - Add Ansible template directory (special tall bookshelf) - Add the ability to keep sensitive variables in `vault.yaml` - Add google_health production params template Encrypt vault.yaml * Use templates dir * Remove end date from params * Switch to midas export dir * Properly rename Jenkins pipeline stage scripts * Handle google_health's testing needs - Add: Ansible playbook for securely handling placing the `params.json` template we need for testing. This will happen during the Jenkins build stage when we are setting up the venv in the workspace on the Jenkins server. - Add: Test `params.json` template. - Add: Jenkins user variable. - Fix: Was incorrectly trying to use a file instead of a template in `ansible-deploy.yaml`. - Add: Call the small Ansible playbook from the Jenkins build wrapper. * Change to the Ansible root dir before trying to do Ansible things * Delegate to localhost * Tell Ansible we want to connect locally for this playbook * Remove set -x from Bash scripts * Add initial Jenkins/Ansible necessities for CI/CD and Automation - Add Jenkins pipeline scripts - Add production params template - Update vars and vault files with AWS secrets * Remove ght cache files * Disable pylint convention messages * Work around linter errors Co-authored-by: Addison Hu <[email protected]> Co-authored-by: Jingjing Tang <[email protected]>

Merge main into deploy-google_health

Deploy jhu

* Add updated encrypted credentials * Add wip_signal to production params template * Add newline * Run on 12 cores * Add shell script to run the indicator

* Add updated encrypted credentials * Add wip_signal to production params template * Add newline * Run on 12 cores * Add shell script to run the indicator * Update run script - Use production ingestion dir - Remove hard fail to work around an issue with `cp`. - `cp` fails when files don't exist. I thought we could squash the error by sending stderr to /dev/null, but `set -eo` still catches and fails the script. Even worse, it happens silently. There should be a better way to handle this so will add it to a future task.

Merge the SafeGraph deployment branch updates to main

krivard · 2020-09-11T17:16:47Z

Details on failed check: Linting causes a recursion error starting with this stack trace:

$ env/bin/pylint delphi_jhu
Traceback (most recent call last):
  File "env/bin/pylint", line 10, in <module>
    sys.exit(run_pylint())
  File "/var/lib/jenkins/workspace/covidcast-indicators_PR-272/jhu/env/lib/python3.8/site-packages/pylint/__init__.py", line 22, in run_pylint
    PylintRun(sys.argv[1:])
  File "/var/lib/jenkins/workspace/covidcast-indicators_PR-272/jhu/env/lib/python3.8/site-packages/pylint/lint/run.py", line 349, in __init__
    linter.check(args)
  File "/var/lib/jenkins/workspace/covidcast-indicators_PR-272/jhu/env/lib/python3.8/site-packages/pylint/lint/pylinter.py", line 862, in check
    self._check_files(
  File "/var/lib/jenkins/workspace/covidcast-indicators_PR-272/jhu/env/lib/python3.8/site-packages/pylint/lint/pylinter.py", line 896, in _check_files
    self._check_file(get_ast, check_astroid_module, name, filepath, modname)
  File "/var/lib/jenkins/workspace/covidcast-indicators_PR-272/jhu/env/lib/python3.8/site-packages/pylint/lint/pylinter.py", line 922, in _check_file
    check_astroid_module(ast_node)
  File "/var/lib/jenkins/workspace/covidcast-indicators_PR-272/jhu/env/lib/python3.8/site-packages/pylint/lint/pylinter.py", line 1054, in check_astroid_module
    retval = self._check_astroid_module(
  File "/var/lib/jenkins/workspace/covidcast-indicators_PR-272/jhu/env/lib/python3.8/site-packages/pylint/lint/pylinter.py", line 1099, in _check_astroid_module
    walker.walk(ast_node)
  File "/var/lib/jenkins/workspace/covidcast-indicators_PR-272/jhu/env/lib/python3.8/site-packages/pylint/utils/ast_walker.py", line 75, in walk
    self.walk(child)
  File "/var/lib/jenkins/workspace/covidcast-indicators_PR-272/jhu/env/lib/python3.8/site-packages/pylint/utils/ast_walker.py", line 75, in walk
    self.walk(child)
  File "/var/lib/jenkins/workspace/covidcast-indicators_PR-272/jhu/env/lib/python3.8/site-packages/pylint/utils/ast_walker.py", line 72, in walk
    callback(astroid)
  File "/var/lib/jenkins/workspace/covidcast-indicators_PR-272/jhu/env/lib/python3.8/site-packages/pylint/checkers/typecheck.py", line 1037, in visit_assign
    self._check_assignment_from_function_call(node)
  File "/var/lib/jenkins/workspace/covidcast-indicators_PR-272/jhu/env/lib/python3.8/site-packages/pylint/checkers/typecheck.py", line 1047, in _check_assignment_from_function_call
    function_node = safe_infer(node.value.func)
  File "/var/lib/jenkins/workspace/covidcast-indicators_PR-272/jhu/env/lib/python3.8/site-packages/pylint/checkers/utils.py", line 1143, in safe_infer
    value = next(infer_gen)
  File "/var/lib/jenkins/workspace/covidcast-indicators_PR-272/jhu/env/lib/python3.8/site-packages/astroid/decorators.py", line 132, in raise_if_nothing_inferred
    yield next(generator)
[...]

  File "/var/lib/jenkins/workspace/covidcast-indicators_PR-272/jhu/env/lib/python3.8/site-packages/astroid/decorators.py", line 96, in wrapped
    res = next(generator)
  File "/var/lib/jenkins/workspace/covidcast-indicators_PR-272/jhu/env/lib/python3.8/site-packages/astroid/bases.py", line 136, in _infer_stmts
    for inferred in stmt.infer(context=context):
  File "/var/lib/jenkins/workspace/covidcast-indicators_PR-272/jhu/env/lib/python3.8/site-packages/astroid/util.py", line 160, in limit_inference
    yield from islice(iterator, size)
  File "/var/lib/jenkins/workspace/covidcast-indicators_PR-272/jhu/env/lib/python3.8/site-packages/astroid/context.py", line 113, in cache_generator
    for result in generator:
  File "/var/lib/jenkins/workspace/covidcast-indicators_PR-272/jhu/env/lib/python3.8/site-packages/astroid/decorators.py", line 132, in raise_if_nothing_inferred
    yield next(generator)
  File "/var/lib/jenkins/workspace/covidcast-indicators_PR-272/jhu/env/lib/python3.8/site-packages/astroid/decorators.py", line 93, in wrapped
    generator = _func(node, context, **kwargs)
  File "/var/lib/jenkins/workspace/covidcast-indicators_PR-272/jhu/env/lib/python3.8/site-packages/astroid/inference.py", line 850, in infer_assign
    stmts = list(self.assigned_stmts(context=context))
  File "/var/lib/jenkins/workspace/covidcast-indicators_PR-272/jhu/env/lib/python3.8/site-packages/astroid/protocols.py", line 309, in assend_assigned_stmts
    return self.parent.assigned_stmts(node=self, context=context)
  File "/var/lib/jenkins/workspace/covidcast-indicators_PR-272/jhu/env/lib/python3.8/site-packages/astroid/protocols.py", line 375, in arguments_assigned_stmts
    context = contextmod.copy_context(context)
  File "/var/lib/jenkins/workspace/covidcast-indicators_PR-272/jhu/env/lib/python3.8/site-packages/astroid/context.py", line 155, in copy_context
    return context.clone()
  File "/var/lib/jenkins/workspace/covidcast-indicators_PR-272/jhu/env/lib/python3.8/site-packages/astroid/context.py", line 102, in clone
    clone = InferenceContext(self.path, inferred=self.inferred)
RecursionError: maximum recursion depth exceeded
script returned exit code 1

Changing a static file should not have caused a new linting error. @huisaddison did you branch from deploy-jhu, main, or some other branch?

huisaddison · 2020-09-11T18:01:05Z

I branched off of main, not realizing that the indicator is running off of deploy-jhu. I agree that my changes should not have caused a new linting error.

My only two commits are:

dd20817 which removes an unused static file fips_pop_prop.csv, which is no longer referred to in the source code. (It does appear in the files changed under quideltest files, but that should be separate)
6fda1b2` which only changes five lines of a static file: 6fda1b2

I really don't know where to begin debugging this. Maybe @korlaxxalrok / @dshemetov can help since it seems to be jenkins related?

huisaddison · 2020-09-11T18:04:51Z

@krivard I undid my commits locally (git checkout 55a41fd4dd8a) and get the same linter failure. So I don't think that my commits introduced this linter failure. Can we try to narrow down a list of plausible people whose commits may be causing this?

krivard · 2020-09-11T18:13:38Z

@huisaddison Branching from main is the correct workflow, no worries there.

What that does tell us is that some change to the JHU code was merged to main without going through deploy-jhu first.

huisaddison · 2020-09-11T18:16:17Z

@krivard I was going to do a git bisect (@capnrefsmmat 's idea), but I am finding the same linter error with my HEAD pointed at deploy-jhu. (i.e., I did git checkout deploy-jhu, installed the module, and ran linter, in case I'm using the git jargon incorrectly).

huisaddison · 2020-09-11T18:21:03Z

I've gone to nearly the beginning of time git checkout db6a7d79a1dea16c0651702f332b3 and I'm getting the same error LOL :( maybe linter itself is having trouble? (or I'm doing this wrong)

huisaddison · 2020-09-11T18:30:23Z

FWIW, linter for google_health works OK

krivard · 2020-09-11T18:32:00Z

Oh ffs, it's apparently a common regression failure in astroid, one of the components of pylint. There must have been an inopportune version bump somewhere. Possibly an intersection with pandas as well. Some similar problems have had luck downgrading astroid; give that a shot?

huisaddison · 2020-09-11T18:38:40Z

Oh ffs, it's apparently a common regression failure in astroid, one of the components of pylint. There must have been an inopportune version bump somewhere. Possibly an intersection with pandas as well. Some similar problems have had luck downgrading astroid; give that a shot?

Not sure how...? Seems that astroid is used by pylint but not installed via pip (within the venv)?

huisaddison · 2020-09-11T18:51:00Z

@krivard Ugh so I manually installed pylint==2.3.1 and astroid==2.2.5 in the venv (which fixes it for the others that you linked) but I'm still getting the failure (but maybe due to another package that astroid doesn't like? linter seems to run for longer before failing, but maybe my mind is just playing tricks with me)

korlaxxalrok · 2020-09-11T19:22:15Z

@huisaddison Can we relatively safely disable the linter? We seem to have astroid 2.4.2 and pylint 2.6.0 in use recently for other indicators without issue, so maybe this issue is deep and tricky.

What about using another linter locally for a sanity check, and then we can disable pylint so that Jenkins is happy? Not ideal, but it would at least provide for a decent local check before we deploy new code.

dshemetov · 2020-09-11T19:48:50Z

FWIW, I just pulled this branch locally and am getting no linter errors with pylint==2.6.0 and astroid==2.4.2. Seems like an issue just on the deploy server?

korlaxxalrok · 2020-09-11T19:58:35Z

Yeah, same here. Extra wacky now. I am going to try to submit another PR with a branch based off of fix/jhu-nyc-support.

huisaddison · 2020-09-11T20:17:32Z

@dshemetov @korlaxxalrok Wow, so weird. :| Checked again and I still get those errors locally.

 addison  (e) env  ~  repos  covidcast-indicators  jhu  pylint --version
pylint 2.6.0
astroid 2.4.2
Python 3.8.3 (default, May 15 2020, 00:00:00) 
[GCC 10.1.1 20200507 (Red Hat 10.1.1-1)]

Could it be as subtle as Python / OS version?

In any case, @korlaxxalrok whether or not we should allow ourselves to disable pylint for this particular pipeline sounds like a @krivard -level decision

korlaxxalrok · 2020-09-11T20:32:08Z

Could be something like that, yeah. Probably need to dig a bit more on the Jenkins server.

Agreed, deferring to @krivard on the option to lint locally for sanity's sake and then disabling in Jenkins for the quick ⚒️

krivard · 2020-09-11T20:38:55Z

I'm happy with linting locally -- shall I merge #274 or do we need something fancier?

korlaxxalrok · 2020-09-11T20:43:00Z

I was gonna just test with #274, but if it works we can go with it. Otherwise we can make the same change in this one.

dshemetov · 2020-09-11T20:44:12Z

@huisaddison wow, yes, so I think it's the combination of Python 3.8.3 plus those other packages. Just installed a fresh virtualenv with 3.8.3 and am getting the error now.

krivard · 2020-09-11T20:47:06Z

Okay, since Dmitry was able to lint with a different Python version, I'm willing to say #274 is good. Closing this PR in favor of that one.

korlaxxalrok and others added 30 commits August 5, 2020 15:39

Add missing path parameter

4183fd3

Fix variable name, reencrypt vault.yaml

4ded043

try until pull the new data successfully

b566d4c

used tenacity

8db2799

Add .gitignore

c9b16a5

Update params.json.template

f969d96

Update run.py

56db650

Update setup.py

775efe1

Update test_update_sensor.py

df20f05

Update update_sensor.py

9295eac

Add constants.py

786cef4

Add static files

3391ff3

Minor changes

40091b7

removed wildcard import error

db54eb8

update params.json.template

2a9a68f

Add wip_signal key in ansible

319f038

Add missing receiving directory

17fb5ae

Merge pull request #239 from cmu-delphi/main

8a40c1c

Merge main into deploy-google_health

Merge branch 'deploy-google_health' into fix_ght_pulling

dd4085b

Merge pull request #252 from cmu-delphi/deploy-jhu

5a4d298

Deploy jhu

Add requirement tenacity

65490cc

fixed invalid character

d4b6f69

add prefix wip

a00c125

update test_run for wip signals

15bf81e

update signal names and export end date

6dcd1bc

fixed errors in case:no new data

df302b3

update signal names in DETAILS

f8f0bbe

update DETAILS

d883c0c

remove prefix wip_

e0a6124

korlaxxalrok and others added 5 commits September 3, 2020 14:22

Set up production necessities for deploying Safegraph (#259)

3988a3b

* Add updated encrypted credentials * Add wip_signal to production params template * Add newline * Run on 12 cores * Add shell script to run the indicator

Merge pull request #266 from cmu-delphi/deploy-safegraph

55a41fd

Merge the SafeGraph deployment branch updates to main

remove old static file that is no longer used

dd20817

hotfix for nyc boroughs

6fda1b2

huisaddison requested a review from krivard September 11, 2020 15:37

krivard changed the base branch from main to deploy-jhu September 11, 2020 15:45

krivard closed this Sep 11, 2020

krivard mentioned this pull request Sep 11, 2020

Resolve Jenkins linter regression problem in JHU #275

Closed

benjaminysmith mentioned this pull request Oct 20, 2020

Pylint/Astroid gives "RecursionError: maximum recursion depth exceeded" in CI tests #333

Closed

krivard deleted the fix/jhu-nyc-support branch October 29, 2020 18:11

Fix/jhu nyc support #272

Fix/jhu nyc support #272

Uh oh!

Conversation

huisaddison commented Sep 11, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

krivard commented Sep 11, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

huisaddison commented Sep 11, 2020

Uh oh!

huisaddison commented Sep 11, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

krivard commented Sep 11, 2020

Uh oh!

huisaddison commented Sep 11, 2020

Uh oh!

huisaddison commented Sep 11, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

huisaddison commented Sep 11, 2020

Uh oh!

krivard commented Sep 11, 2020

Uh oh!

huisaddison commented Sep 11, 2020

Uh oh!

huisaddison commented Sep 11, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

korlaxxalrok commented Sep 11, 2020

Uh oh!

dshemetov commented Sep 11, 2020

Uh oh!

korlaxxalrok commented Sep 11, 2020

Uh oh!

huisaddison commented Sep 11, 2020

Uh oh!

korlaxxalrok commented Sep 11, 2020

Uh oh!

krivard commented Sep 11, 2020

Uh oh!

korlaxxalrok commented Sep 11, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dshemetov commented Sep 11, 2020

Uh oh!

krivard commented Sep 11, 2020

Uh oh!

Uh oh!

huisaddison commented Sep 11, 2020 •

edited

Loading

krivard commented Sep 11, 2020 •

edited

Loading

huisaddison commented Sep 11, 2020 •

edited

Loading

huisaddison commented Sep 11, 2020 •

edited

Loading

huisaddison commented Sep 11, 2020 •

edited

Loading

korlaxxalrok commented Sep 11, 2020 •

edited

Loading