Skip to content

CRT S3 Client PutObject #2328

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Mar 18, 2021
Merged

Conversation

dagnir
Copy link
Contributor

@dagnir dagnir commented Mar 12, 2021

Description

Motivation and Context

Testing

Screenshots (if appropriate)

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)

Checklist

  • I have read the CONTRIBUTING document
  • Local run of mvn install succeeds
  • My code follows the code style of this project
  • My change requires a change to the Javadoc documentation
  • I have updated the Javadoc documentation accordingly
  • I have read the README document
  • I have added tests to cover my changes
  • All new and existing tests passed
  • A short description of the change has been added to the CHANGELOG
  • My change is to implement 1.11 parity feature and I have updated LaunchChangelog

License

  • I confirm that this pull request can be released under the Apache 2 license

@dagnir dagnir changed the base branch from master to s3-transfermanager March 12, 2021 21:58
@dagnir dagnir changed the base branch from s3-transfermanager to feature/master/transfermanager March 12, 2021 21:58
Comment on lines 30 to 31
@Override
public boolean getRequestBytes(ByteBuffer bytes) {
Copy link
Contributor Author

@dagnir dagnir Mar 12, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've doing some testing against this updated interface, and CRT doesn't seem to treat the return value the way I'd expect, given this description.

I would expect the following to work

getRequestBytes(ByteBuffer bb) {
    return false;
}

which results in getRequestBytes() being called continuously, like this:

Content length is 4 bytes
1: getRequestBytes(buffer); // returns false, buffer contains 1 byte
2: getRequestBytes(buffer); // returns false, buffer contains 1 byte
3: getRequestBytes(buffer); // returns false, buffer contains 1 byte
4: getRequestBytes(buffer); // returns true, buffer contains 1 byte

but returning false results in the request completing instead; i.e. 2-4 never happen

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Discussing with CRT team about what we can do to support the use case above. For now, the implementation fills the buffer before returning

@dagnir dagnir changed the title Tm upload CRT S3 Client PutObject Mar 16, 2021
@dagnir dagnir marked this pull request as ready for review March 16, 2021 17:11

// TODO: not volatile since it's read and written only by CRT thread(s). Need to
// ensure that CRT actually ensures consistency across their threads...
private Subscriber<? super ByteBuffer> subscriber;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this only accessed by the same CRT thread?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, in my testing I saw two different threads calling getRequestData. However, if the threads ares are synchronizing between each other, so that changes made by one thread are visible by the other thread then it should be fine. I'll check with CRT

@dagnir
Copy link
Contributor Author

dagnir commented Mar 18, 2021

Build failures due to dependency on unreleased version for CRT Java. Changes have been merged to mainline awslabs/aws-crt-java#298

@dagnir dagnir merged commit 2649249 into aws:feature/master/transfermanager Mar 18, 2021
aws-sdk-java-automation added a commit that referenced this pull request Jan 17, 2023
…f66cf7077

Pull request: release <- staging/b05e44fe-3315-40f0-a450-b03f66cf7077
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants