Streaming in download end point wrapper

[sharathkumaranbu]

We have a wrapper for downloading submissions and artifacts from the Submission API. Right now what is being done is the wrapper is holding the entire contents in memory and creating a buffer and later receiving entire file it is passing the buffer to the caller but this is not a feasible approach while downloading bigger files. It will throttle system memory.

https://github.com/topcoder-platform/topcoder-submission-api-wrapper/blob/develop/src/common/helper.js#L170-L190

I believe we just need to pass the stream from the API to the caller, so that the caller can directly stream it to a file, instead of taking entire buffer and writing it to the file.

This would affect Topcoder CLI which is used to download submission and artifacts.

Makes sense?

@callmekatootie @cwdcwd

[cwdcwd]

@sharathkumaranbu can we offer functions for both? Streaming and in-memory?

[sharathkumaranbu]

@cwdcwd Is in memory streaming required? Are we foreseeing any use case?

[cwdcwd]

I don't want to break any existing systems using the module. we can deprecate it over time if need be

[sharathkumaranbu]

Fine. Should we create separate function then?

[sharathkumaranbu]

I think we can reuse the same function with a parameter to switch between buffer and stream. By default it will return buffer so existing modules won't be affected and we can modify the parameter to return stream.

[cwdcwd]

I'm good with that. let's make sure all this is documented too

[dhruvit-r]

@cwdcwd @sharathkumaranbu Please go through #13 and topcoder-archive/topcoder-platform-topcoder-cli#23, and let me know if you need anything changed.