1078 - Refactor csv_importer.py and csv_to_database.py

[BrainIsDead]

closes 1078

• csv_to_database functionality moved to csv_importer (csv_to_database deleted)
• fixed imports
• changed mocks
• added pandas errors exceptions to pd.read_csv in the load_csv method
• changed is_header_valid method with issubset method

[krivard]

NB releasing this will need to be coordinated with configuration changes to Automation and/or Cronicle in prod.

[melange396]

great start, but i think we need to change another test file -- since we got rid of csv_to_database.py and added its content to csv_importer.py (in src/acquisition/covidcast/), we should merge test_csv_to_database.py into test_csv_importer.py (in tests/acquisition/covidcast/)

[melange396]

from this line down, all this text is a straight copy-paste from csv_to_database.py (line 15 down) with no modifications, right?

[dshemetov]

Want to point out that this is a partial fix of #1078. This PR can handle merging the files together and adding some Pandas exceptions, but I'm hoping that in another PR we can:

• rewrite load_csv to not loop over dataframe with df.itertuples and instead rely on Pandas data type-casting to do most of our validation
• all the floaty_int and maybe_apply stuff should be deleted
• extract_and_check_row is a very slow and manual way of going about the same things that can be done in Pandas column-by-column;
• the way extract_and_check_row return a (Value, Error) tuple is unnecessary, especially because a CSV file is archived as failed if a single row fails, so we should just detect the failure and return immediately (instead of parsing the rest of the CSV); it would be a bit challenging to reproduce the same error log as current code with this setup (the current code returns the string of the full row that failed; while the Pandas code would just give an error for a full column; we could use Pandas to find the offending row though, probably)

[BrainIsDead]

I implemented extract_and_check_row, validate_missing_code, validate_quantity and other validations with pandas. I found the only way to create CovidcastRows - with itertuples.
Not sure if we need to save @staticmethods it makes no sense if we not using them somewhere else

[sonarqubecloud]

Kudos, SonarCloud Quality Gate passed!   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
14 Code Smells

No Coverage information
0.0% Duplication

[dshemetov]

This should be good to go, as long as we prepare a rollback / hotfix plan in case there is a bug here undetected by our tests/integrations.

After this is merged, we should

• watch the ingestion logs in Kibana (Epidata dashboard) and make sure they look healthy
• get some queries ready to validate that the new data can be pulled from the database and that it looks fine