-
Notifications
You must be signed in to change notification settings - Fork 911
[S3] Add support for more user-friendly CopyObject source parameters #2612
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Bennett-Lynch
merged 6 commits into
aws:master
from
Bennett-Lynch:s3-copy-source-url-encoding
Jul 21, 2021
Merged
Changes from 2 commits
Commits
Show all changes
6 commits
Select commit
Hold shift + click to select a range
0c1a10a
[S3] Add support for more user-friendly CopyObject source parameters
a1513c0
[S3] Add support for more user-friendly CopyObject source parameters
cf7f797
[S3] Add support for more user-friendly CopyObject source parameters
0f2e575
[S3] Add support for more user-friendly CopyObject source parameters
ff4ac00
[S3] Add support for more user-friendly CopyObject source parameters
e3dc764
Merge branch 'master' into s3-copy-source-url-encoding
Bennett-Lynch File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,6 @@ | ||
| { | ||
| "category": "Amazon S3", | ||
| "contributor": "", | ||
| "type": "feature", | ||
| "description": "Add support for more user-friendly CopyObject source parameters" | ||
| } |
148 changes: 148 additions & 0 deletions
148
services/s3/src/it/java/software/amazon/awssdk/services/s3/CopySourceIntegrationTest.java
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,148 @@ | ||
| /* | ||
| * Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. | ||
| * | ||
| * Licensed under the Apache License, Version 2.0 (the "License"). | ||
| * You may not use this file except in compliance with the License. | ||
| * A copy of the License is located at | ||
| * | ||
| * http://aws.amazon.com/apache2.0 | ||
| * | ||
| * or in the "license" file accompanying this file. This file is distributed | ||
| * on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either | ||
| * express or implied. See the License for the specific language governing | ||
| * permissions and limitations under the License. | ||
| */ | ||
|
|
||
| package software.amazon.awssdk.services.s3; | ||
|
|
||
| import static org.hamcrest.MatcherAssert.assertThat; | ||
| import static org.hamcrest.Matchers.is; | ||
| import static software.amazon.awssdk.testutils.service.S3BucketUtils.temporaryBucketName; | ||
|
|
||
| import java.nio.charset.StandardCharsets; | ||
| import java.util.Arrays; | ||
| import java.util.Collection; | ||
| import java.util.HashMap; | ||
| import java.util.Map; | ||
| import java.util.UUID; | ||
| import org.junit.After; | ||
| import org.junit.Before; | ||
| import org.junit.Test; | ||
| import org.junit.runner.RunWith; | ||
| import org.junit.runners.Parameterized; | ||
| import org.junit.runners.Parameterized.Parameters; | ||
| import software.amazon.awssdk.core.sync.RequestBody; | ||
| import software.amazon.awssdk.services.s3.internal.handlers.CopySourceInterceptor; | ||
| import software.amazon.awssdk.services.s3.model.BucketVersioningStatus; | ||
| import software.amazon.awssdk.services.s3.model.CopyObjectRequest; | ||
| import software.amazon.awssdk.services.s3.model.GetObjectRequest; | ||
| import software.amazon.awssdk.services.s3.model.PutObjectRequest; | ||
| import software.amazon.awssdk.services.s3.model.PutObjectResponse; | ||
|
|
||
| /** | ||
| * Integration tests for the {@code sourceBucket}, {@code sourceKey}, and {@code sourceVersionId} parameters for | ||
| * {@link CopyObjectRequest}. Specifically, we ensure that users are able to seamlessly use the same input for both the | ||
| * {@link PutObjectRequest} key and the {@link CopyObjectRequest} source key (and not be required to manually URL encode the | ||
| * COPY source key). This also effectively tests for parity with the SDK v1 behavior. | ||
| * | ||
| * @see CopySourceInterceptor | ||
| */ | ||
| @RunWith(Parameterized.class) | ||
| public class CopySourceIntegrationTest extends S3IntegrationTestBase { | ||
|
|
||
| private static final String SOURCE_BUCKET_NAME = temporaryBucketName("copy-source-integ-test-src"); | ||
| private static final String DESTINATION_BUCKET_NAME = temporaryBucketName("copy-source-integ-test-dest"); | ||
|
|
||
| @Before | ||
| public void initializeTestData() throws Exception { | ||
| createBucket(SOURCE_BUCKET_NAME); | ||
| createBucket(DESTINATION_BUCKET_NAME); | ||
| } | ||
|
|
||
| @After | ||
| public void tearDown() { | ||
| deleteBucketAndAllContents(SOURCE_BUCKET_NAME); | ||
| deleteBucketAndAllContents(DESTINATION_BUCKET_NAME); | ||
| } | ||
|
|
||
| @Parameters | ||
| public static Collection parameters() throws Exception { | ||
| return Arrays.asList( | ||
| "simpleKey", | ||
| "key/with/slashes", | ||
| "\uD83E\uDEA3", | ||
| "specialChars/ +!#$&'()*,:;=?@\"", | ||
| "%20" | ||
| ); | ||
| } | ||
|
|
||
| private final String key; | ||
|
|
||
| public CopySourceIntegrationTest(String key) { | ||
| this.key = key; | ||
| } | ||
|
|
||
| @Test | ||
| public void copyObject_WithoutExplicitVersion_AcceptsSameKeyAsPut() throws Exception { | ||
Bennett-Lynch marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| String originalContent = UUID.randomUUID().toString(); | ||
|
|
||
| s3.putObject(PutObjectRequest.builder() | ||
| .bucket(SOURCE_BUCKET_NAME) | ||
| .key(key) | ||
| .build(), RequestBody.fromString(originalContent, StandardCharsets.UTF_8)); | ||
|
|
||
| s3.copyObject(CopyObjectRequest.builder() | ||
| .sourceBucket(SOURCE_BUCKET_NAME) | ||
| .sourceKey(key) | ||
| .destinationBucket(DESTINATION_BUCKET_NAME) | ||
| .destinationKey(key) | ||
| .build()); | ||
|
|
||
| String copiedContent = s3.getObjectAsBytes(GetObjectRequest.builder() | ||
| .bucket(DESTINATION_BUCKET_NAME) | ||
| .key(key) | ||
| .build()).asUtf8String(); | ||
|
|
||
| assertThat(copiedContent, is(originalContent)); | ||
| } | ||
|
|
||
| /** | ||
| * Test that we can correctly copy versioned source objects. | ||
| * <p> | ||
| * Motivated by: https://github.com/aws/aws-sdk-js/issues/727 | ||
| */ | ||
| @Test | ||
| public void copyObject_WithVersioning_AcceptsSameKeyAsPut() throws Exception { | ||
| s3.putBucketVersioning(r -> r | ||
| .bucket(SOURCE_BUCKET_NAME) | ||
| .versioningConfiguration(v -> v.status(BucketVersioningStatus.ENABLED))); | ||
|
|
||
| Map<String, String> versionToContentMap = new HashMap<>(); | ||
| int numVersionsToCreate = 3; | ||
| for (int i = 0; i < numVersionsToCreate; i++) { | ||
| String originalContent = UUID.randomUUID().toString(); | ||
| PutObjectResponse response = s3.putObject(PutObjectRequest.builder() | ||
| .bucket(SOURCE_BUCKET_NAME) | ||
| .key(key) | ||
| .build(), | ||
| RequestBody.fromString(originalContent, StandardCharsets.UTF_8)); | ||
| versionToContentMap.put(response.versionId(), originalContent); | ||
| } | ||
|
|
||
| versionToContentMap.forEach((versionId, originalContent) -> { | ||
| s3.copyObject(CopyObjectRequest.builder() | ||
| .sourceBucket(SOURCE_BUCKET_NAME) | ||
| .sourceKey(key) | ||
| .sourceVersionId(versionId) | ||
| .destinationBucket(DESTINATION_BUCKET_NAME) | ||
| .destinationKey(key) | ||
| .build()); | ||
|
|
||
| String copiedContent = s3.getObjectAsBytes(GetObjectRequest.builder() | ||
| .bucket(DESTINATION_BUCKET_NAME) | ||
| .key(key) | ||
| .build()).asUtf8String(); | ||
| assertThat(copiedContent, is(originalContent)); | ||
| }); | ||
| } | ||
| } | ||
127 changes: 127 additions & 0 deletions
127
...main/java/software/amazon/awssdk/services/s3/internal/handlers/CopySourceInterceptor.java
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,127 @@ | ||
| /* | ||
| * Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. | ||
| * | ||
| * Licensed under the Apache License, Version 2.0 (the "License"). | ||
| * You may not use this file except in compliance with the License. | ||
| * A copy of the License is located at | ||
| * | ||
| * http://aws.amazon.com/apache2.0 | ||
| * | ||
| * or in the "license" file accompanying this file. This file is distributed | ||
| * on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either | ||
| * express or implied. See the License for the specific language governing | ||
| * permissions and limitations under the License. | ||
| */ | ||
|
|
||
| package software.amazon.awssdk.services.s3.internal.handlers; | ||
|
|
||
| import static software.amazon.awssdk.services.s3.internal.resource.S3ArnUtils.isArnFor; | ||
| import static software.amazon.awssdk.utils.http.SdkHttpUtils.urlEncodeIgnoreSlashes; | ||
|
|
||
| import software.amazon.awssdk.annotations.SdkInternalApi; | ||
| import software.amazon.awssdk.core.SdkRequest; | ||
| import software.amazon.awssdk.core.interceptor.Context.ModifyRequest; | ||
| import software.amazon.awssdk.core.interceptor.ExecutionAttributes; | ||
| import software.amazon.awssdk.core.interceptor.ExecutionInterceptor; | ||
| import software.amazon.awssdk.services.s3.internal.resource.S3ResourceType; | ||
| import software.amazon.awssdk.services.s3.model.CopyObjectRequest; | ||
| import software.amazon.awssdk.services.s3.model.PutObjectRequest; | ||
| import software.amazon.awssdk.services.s3.model.UploadPartCopyRequest; | ||
|
|
||
| /** | ||
| * This interceptor transforms the {@code sourceBucket}, {@code sourceKey}, and {@code sourceVersionId} parameters for | ||
| * {@link CopyObjectRequest} and {@link UploadPartCopyRequest} into a {@code copySource} parameter. The logic needed to | ||
| * construct a {@code copySource} can be considered non-trivial, so this interceptor facilitates allowing users to | ||
| * use higher-level constructs that more closely match other APIs, like {@link PutObjectRequest}. Additionally, this | ||
| * interceptor is responsible for URL encoding the relevant portions of the {@code copySource} value. | ||
| * <p> | ||
| * <a href="https://docs.aws.amazon.com/AmazonS3/latest/API/API_CopyObject.html#API_CopyObject_RequestParameters">API_CopyObject_RequestParameters</a> | ||
| * <p> | ||
| * <a href="https://docs.aws.amazon.com/AmazonS3/latest/API/API_UploadPartCopy.html#API_UploadPartCopy_RequestParameters">API_UploadPartCopy_RequestParameters</a> | ||
| */ | ||
| @SdkInternalApi | ||
| public final class CopySourceInterceptor implements ExecutionInterceptor { | ||
|
|
||
| @Override | ||
| public SdkRequest modifyRequest(ModifyRequest context, ExecutionAttributes executionAttributes) { | ||
| SdkRequest request = context.request(); | ||
| if (request instanceof CopyObjectRequest) { | ||
| return modifyCopyObjectRequest((CopyObjectRequest) request); | ||
| } | ||
| if (request instanceof UploadPartCopyRequest) { | ||
| return modifyUploadPartCopyRequest((UploadPartCopyRequest) request); | ||
| } | ||
| return request; | ||
| } | ||
|
|
||
| private static SdkRequest modifyCopyObjectRequest(CopyObjectRequest request) { | ||
| if (request.copySource() != null) { | ||
| requireNotSet(request.sourceBucket(), "sourceBucket"); | ||
| requireNotSet(request.sourceKey(), "sourceKey"); | ||
| requireNotSet(request.sourceVersionId(), "sourceVersionId"); | ||
| return request; | ||
| } | ||
| String copySource = constructCopySource( | ||
| requireSet(request.sourceBucket(), "sourceBucket"), | ||
| requireSet(request.sourceKey(), "sourceKey"), | ||
| request.sourceVersionId() | ||
| ); | ||
| return request.toBuilder() | ||
| .sourceBucket(null) | ||
| .sourceKey(null) | ||
| .sourceVersionId(null) | ||
| .copySource(copySource) | ||
| .build(); | ||
| } | ||
|
|
||
| private static SdkRequest modifyUploadPartCopyRequest(UploadPartCopyRequest request) { | ||
| if (request.copySource() != null) { | ||
| requireNotSet(request.sourceBucket(), "sourceBucket"); | ||
| requireNotSet(request.sourceKey(), "sourceKey"); | ||
| requireNotSet(request.sourceVersionId(), "sourceVersionId"); | ||
| return request; | ||
| } | ||
| String copySource = constructCopySource( | ||
| requireSet(request.sourceBucket(), "sourceBucket"), | ||
| requireSet(request.sourceKey(), "sourceKey"), | ||
| request.sourceVersionId() | ||
| ); | ||
| return request.toBuilder() | ||
| .sourceBucket(null) | ||
| .sourceKey(null) | ||
| .sourceVersionId(null) | ||
| .copySource(copySource) | ||
| .build(); | ||
| } | ||
|
|
||
| private static String constructCopySource(String sourceBucket, String sourceKey, String sourceVersionId) { | ||
| StringBuilder copySource = new StringBuilder(); | ||
| copySource.append("/"); | ||
| copySource.append(urlEncodeIgnoreSlashes(sourceBucket)); | ||
| if (isArnFor(S3ResourceType.ACCESS_POINT, sourceBucket) || isArnFor(S3ResourceType.OUTPOST, sourceBucket)) { | ||
| copySource.append("/object"); | ||
| } | ||
Bennett-Lynch marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| copySource.append("/"); | ||
| copySource.append(urlEncodeIgnoreSlashes(sourceKey)); | ||
| if (sourceVersionId != null) { | ||
| copySource.append("?versionId="); | ||
| copySource.append(urlEncodeIgnoreSlashes(sourceVersionId)); | ||
| } | ||
| return copySource.toString(); | ||
| } | ||
|
|
||
| private static void requireNotSet(Object value, String paramName) { | ||
| if (value != null) { | ||
| throw new IllegalArgumentException(String.format("Parameter 'copySource' must not be used in conjunction with '%s'", | ||
| paramName)); | ||
| } | ||
| } | ||
|
|
||
| private static <T> T requireSet(T value, String paramName) { | ||
| if (value == null) { | ||
| throw new IllegalArgumentException(String.format("Parameter '%s' must not be null", | ||
| paramName)); | ||
| } | ||
| return value; | ||
Bennett-Lynch marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| } | ||
| } | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.