Skip to content

[JIT Variant]: Reduce iterator stack on base signals #1049

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Feb 22, 2023
Merged

[JIT Variant]: Reduce iterator stack on base signals #1049

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
7b0addb
JIT: Small name improvement in model.py
dshemetov Dec 8, 2022
e2a2585
JIT: reduce the iterator stack on non-derived signals
dshemetov Dec 8, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 6 additions & 2 deletions src/server/endpoints/covidcast_utils/model.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -543,9 +543,13 @@ def _generate_transformed_rows(
for key, source_signal_geo_rows in groupby(parsed_rows, group_keyfunc):
base_source_name, base_signal_name, _, _ = key
# Extract the list of derived signals; if a signal is not in the dictionary, then use the identity map.
derived_signal_transform_map: SourceSignalPair = transform_dict.get(SourceSignalPair(base_source_name, [base_signal_name]), SourceSignalPair(base_source_name, [base_signal_name]))
derived_signals: SourceSignalPair = transform_dict.get(SourceSignalPair(base_source_name, [base_signal_name]), SourceSignalPair(base_source_name, [base_signal_name]))
# Speed up base signals by not transforming them.
if derived_signals.signal == [base_signal_name]:
yield from source_signal_geo_rows
continue
# Create a list of source-signal pairs along with the transformation required for the signal.
signal_names_and_transforms: List[Tuple[Tuple[str, str], Callable]] = [(derived_signal, _get_base_signal_transform((base_source_name, derived_signal))) for derived_signal in derived_signal_transform_map.signal]
signal_names_and_transforms: List[Tuple[Tuple[str, str], Callable]] = [(derived_signal, _get_base_signal_transform((base_source_name, derived_signal))) for derived_signal in derived_signals.signal]
# Put the current time series on a contiguous time index.
source_signal_geo_rows = _reindex_iterable(source_signal_geo_rows, fill_value=transform_args.get("pad_fill_value"))
# Create copies of the iterable, with smart memory usage.
Expand Down