Optimize with dask #1981
Merged
Optimize with dask #1981
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Need to check if the memory is big enough to be able to pass the processed dataframe in memory otherwise need to look into chunking. |
minhkhul
reviewed
Jul 1, 2024
| ------- | ||
| ''' | ||
| filename = Path(filepath).name | ||
| logger.info(f"Processing {filename}") |
There was a problem hiding this comment.
nitpick: filepath might be more helpful, as it includes input dir.
Suggested change
| logger.info(f"Processing {filename}") | |
| logger.info(f"Processing {filepath}") |
minhkhul
reviewed
Jul 1, 2024
| def update_sensor( | ||
| filepath, startdate, enddate, dropdate, geo, parallel, | ||
| weekday, se, logger | ||
| data:pd.DataFrame, startdate:datetime, enddate:datetime, dropdate:datetime, geo:str, parallel: bool, |
There was a problem hiding this comment.
👍 we should start doing type specification.
There was a problem hiding this comment.
yeah but I think that should be a seperate ticket maybe? don't want to have more PR/confusion within already scoped out feature.
minhkhul
approved these changes
Jul 1, 2024
aysim319
force-pushed
the
optimize_with_dask
branch
4 times, most recently
from
e919fda to
935b7dd
Compare
aysim319
force-pushed
the
optimize_with_dask
branch
from
935b7dd to
d1ee4ce
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
doctor_visit_EDI_AGG_OUTPATIENT_26062024_1455CDT.log
doctor_visit_refactored_EDI_AGG_OUTPATIENT_26062024_1455CDT.log
doctor_visit_refactored_using_dask_7b6e0764_EDI_AGG_OUTPATIENT_26062024_1455CDT.log
!WORK IN PROGESS! need to make test and more optimization
Description
continuing optimizing for doctors_visit
Main ran: 569
doctor_visit_refactor: 3135
optimize_with_dask (this branch): 174
Changelog
Itemize code/test/documentation changes and files added/removed.
Fixes