Cloud Fetch download manager #146
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
mattdeekay
requested review from
susodapop,
arikfr,
yunbodeng-db and
andrefurlan-db
as code owners
yunbodeng-db
requested review from
rcypher-databricks
and removed request for
arikfr
Signed-off-by: Matthew Kim <[email protected]>
Signed-off-by: Matthew Kim <[email protected]>
Signed-off-by: Matthew Kim <[email protected]>
Signed-off-by: Matthew Kim <[email protected]>
Signed-off-by: Matthew Kim <[email protected]>
Signed-off-by: Matthew Kim <[email protected]>
Signed-off-by: Matthew Kim <[email protected]>
Signed-off-by: Matthew Kim <[email protected]>
mattdeekay
force-pushed
the
cloudfetch-download-manager
branch
from
da6260f to
f8cdfc5
Compare
…_settings Signed-off-by: Matthew Kim <[email protected]>
…port Signed-off-by: Matthew Kim <[email protected]>
Signed-off-by: Matthew Kim <[email protected]>
mattdeekay
force-pushed
the
cloudfetch-download-manager
branch
from
d8f6c77 to
0499158
Compare
Signed-off-by: Matthew Kim <[email protected]>
mattdeekay
force-pushed
the
cloudfetch-download-manager
branch
from
0499158 to
243ba57
Compare
rcypher-databricks
approved these changes
Jun 30, 2023
susodapop
approved these changes
Jul 1, 2023
susodapop
pushed a commit
to unj1m/databricks-sql-python
that referenced
this pull request
Sep 19, 2023
* Cloud Fetch download manager Signed-off-by: Matthew Kim <[email protected]> * Bug fix: submit handler.run Signed-off-by: Matthew Kim <[email protected]> * Type annotations Signed-off-by: Matthew Kim <[email protected]> * Namedtuple -> dataclass Signed-off-by: Matthew Kim <[email protected]> * Shutdown thread pool and clear handlers Signed-off-by: Matthew Kim <[email protected]> * Docstrings and comments Signed-off-by: Matthew Kim <[email protected]> * handler.run is the correct call Signed-off-by: Matthew Kim <[email protected]> * Link expiry buffer in secs Signed-off-by: Matthew Kim <[email protected]> * Adding type annotations for download_handlers and downloadable_result_settings Signed-off-by: Matthew Kim <[email protected]> * Move DownloadableResultSettings to downloader.py to avoid circular import Signed-off-by: Matthew Kim <[email protected]> * Black linting Signed-off-by: Matthew Kim <[email protected]> * Timeout is never None Signed-off-by: Matthew Kim <[email protected]> --------- Signed-off-by: Matthew Kim <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Download manager for Cloud Fetch. Maintains a thread pool for
ResultFileDownloadHandlerdownload threads.add_file_links()method is called to prepare handlers, one for each cloud fetch link.get_next_downloaded_file()will schedule relevant handlers for download, check/wait for the download status of the next file the client needs, and returns the downloaded file.