Adapt archive utility to run as a separate step in the pipeline

[krivard]

Currently, the archive utility is integrated into run_module(). This is problematic, because if the CSV output fails validation (either after delivery to the server, or during the imminent validation step) then we wind up saving files to the cache that do not actually reflect what data is in the API.

This is actively causing problems for #179, and could affect every indicator where the archiver is used in this way as soon as it fails ingestion once.

Once the validator is available, we will want to shift archiver duties to occur only after validation succeeds.

To prepare for that, we should adapt the archiver to run independently via the delphi_utils module. Something like

(env) $ python -m delphi_utils archive

-- the archive command will leave room for other utils commands should they come up.

This has the bonus of decoupling archiver code from the indicator implementations, reducing duplication, and making it easier to run indicators in development environments without needing S3 credentials.

[sgsmob]

Interested in jumping into this, but I don't totally understand what the desired workflow is here. If I understand correctly, what we now have is the archive running when we do (env) $ python -m $INDICATOR_PIPELINE_MODULE_NAME. After this issue is resolved, would you need to run in two steps? i.e.

(env) $ python -m $INDICATOR_PIPELINE_MODULE_NAME
(env) $ python -m delphi_utils archive

[krivard]

Essentially yes, but to be slightly more precise about the context -- the ultimate goal is a shell script like this:

     1  #!/bin/bash
       
     2  INDICATOR_MODULE=$1
       
     3  source env/bin/activate
     4  python -m $INDICATOR_MODULE
     5  if python -m delphi_validator; then
     6      python -m delphi_utils archive
     7  else
     8      # validation failed; call for help
     9  fi

...where the scope of this ticket is limited to line 6.

If this ticket gets resolved before the validator is complete, then we'll need to move from

     1  #!/bin/bash
       
     2  INDICATOR_MODULE=$1
       
     3  source env/bin/activate
     4  python -m $INDICATOR_MODULE

to

     1  #!/bin/bash
       
     2  INDICATOR_MODULE=$1
       
     3  source env/bin/activate
     4  python -m $INDICATOR_MODULE
     5  python -m delphi_utils archive

in the interim.