Partitioning Direction Update #161
Conversation
|
The new direction updater takes an argument |
Changed column names back to timestamp1, timestamp2 for testing purposes.
* Implemented the Quick Fix: Direction should be updated to NA wherever there are no historical data to compute the stdev from. * Modified the integration test to test for this behavior
* Updated column names back to value_updated_timestamp and direction_updated_timestamp.
|
To add the
We can drop the |
|
@AlSaeed The new column is added. Once merged we can do the second step. |
change signal type from VARCHAR(32) to VARCHAR(64) in temporary tables.
|
|
||
| # partition configuration | ||
| PARTITION_VARIABLE = 'geo_value' | ||
| PARTITION_SPLITS = ["'05101'", "'101'", "'13071'", "'15007'", "'17161'", "'19039'", "'20123'", "'21213'", "'24035'", |
There was a problem hiding this comment.
how did you come up with these split values?
There was a problem hiding this comment.
They were chosen so that they split a recent full issue into roughly equally-sized partitions.
| finally: | ||
| # no catch block so that an exception above will cause the program to | ||
| # fail after the following cleanup | ||
| database.disconnect(commit) |
There was a problem hiding this comment.
possibly consider moving this out of the loop so the db is consistent? i guess having partially updated directions isnt such a bad thing
There was a problem hiding this comment.
I guess there is an argument for both sides. The other side being this allows to investigate the failure and only update the remaining partitions instead of running the entire job again.
There was a problem hiding this comment.
¯\(ツ)/¯ its probably fine to leave it
There was a problem hiding this comment.
Either way someone has to go in and fix it manually, so as long as we document what the behavior is and what should be done in a failure, it's fine.
No description provided.