Skip to content

FeatureGroup ingest function not utilizing all CPU cores #2111

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
alex-tang opened this issue Jan 27, 2021 · 1 comment
Closed

FeatureGroup ingest function not utilizing all CPU cores #2111

alex-tang opened this issue Jan 27, 2021 · 1 comment
Labels
contributions welcome type: feature request

Comments

@alex-tang
Copy link
Contributor

Describe the feature you'd like
Currently, the ingest function supports configuring the number of threads for parallelize ingestion. The current functionality does fully utilize machines with higher core counts (i.e. a single process is executing on a single CPU core).

The feature request is to support configuring the number of Processes for parallelizing ingestion, to increase CPU core utilization.

How would this feature be used? Please describe.
As a parameter in the ingest() API to specify the number of parallel processes.

Describe alternatives you've considered
N/A

Additional context
N/A
@ChoiByungWook ChoiByungWook mentioned this issue Mar 8, 2021
Merged
7 tasks
alex-tang pushed a commit to alex-tang/sagemaker-python-sdk that referenced this issue Apr 16, 2021
feature: support multiprocess feature group ingest (aws#2111)
9ce8f64
alex-tang pushed a commit to alex-tang/sagemaker-python-sdk that referenced this issue Apr 16, 2021
feature: support multiprocess feature group ingest (aws#2111)
71dc37b
alex-tang pushed a commit to alex-tang/sagemaker-python-sdk that referenced this issue Apr 19, 2021
feature: support multiprocess feature group ingest (aws#2111)
f2d9c2d
icywang86rui added a commit that referenced this issue Apr 20, 2021
@alex-tang @icywang86rui
feature: support multiprocess feature group ingest (#2111) (#2288)
441c53e 
Co-authored-by: Alex <[email protected]>
Co-authored-by: icywang86rui <[email protected]>
TEChopra1000 pushed a commit to TEChopra1000/sagemaker-python-sdk that referenced this issue Apr 28, 2021
@alex-tang @icywang86rui
feature: support multiprocess feature group ingest (aws#2111) (aws#2288)
97cb1be 
Co-authored-by: Alex <[email protected]>
Co-authored-by: icywang86rui <[email protected]>
@akrishna1995
Copy link
Contributor

Hi @alex-tang - seems like the fix has been merged : #2288.

Please feel free to reopen if there are any further feedback. Thank you

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
contributions welcome type: feature request
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@ajaykarpur @alex-tang @akrishna1995