Skip to content

Commit 0c1a10a

Browse files
author
Bennett Lynch
committed
[S3] Add support for more user-friendly CopyObject source parameters
## Motivation and Context The current S3Client interface has a cumbersome API for invoking CopyObjectRequests. We require users to define the bucket name, key name, and version ID in a raw string format. We require that the string conform to the S3 API, which forces users to know the intricate details for how to join these values together. Additionally, portions (but not all) of the value must be URL encoded, further increasing the burden. https://docs.aws.amazon.com/AmazonS3/latest/API/API_CopyObject.html#API_CopyObject_RequestParameters In the Java SDK v1, users are given explicit parameters for the different copy source attributes. But in v2, parity for this support is clearly lacking. E.g., v1: ``` s3.copyObject(new CopyObjectRequest() .withSourceBucketName(SOURCE_BUCKET) .withSourceKey(key) .withSourceVersionId(versionId) .withDestinationBucketName(DESTINATION_BUCKET) .withDestinationKey(key)); ``` v2: ``` s3.copyObject(CopyObjectRequest.builder() .copySource(SOURCE_BUCKET + "/" + key + "?versionId=" + versionId) .destinationBucket(DESTINATION_BUCKET) .destinationKey(key) .build()); ``` The v1 SDK will also URL encode on the user's behalf, allowing users to use the same input values as they would for a PutObjectRequest. The v2 code snippet above may appear to work for most users until they run into unexpected source keys that require URL encoding, at which point they will typically be given `NoSuchKey` errors. This API deficiency has been called out by users in at least the following issues: * aws#1313 * aws#1452 * aws#1656 * awsdocs/aws-doc-sdk-examples#740 ## Description * For both CopyObjectRequest and UploadPartCopyRequest, add explicit parameters for: SourceBucket, SourceKey, SourceVersionId * If specified, these values will be used to construct a CopySource on the user's behalf, including URL encoding the relevant portions. * These values are introduced in a backwards compatible fashion. Users who are already using CopySource today will see no change in behavior, but these new fields may not be used in conjunction with CopySource. * A follow-up PR will be submitted to propose deprecating the current CopySource parameter. It is excluded from this PR since our current code gen configuration lacks appropriate support. * Add support for "DestinationBucket" & "DestinationKey" to UploadPartCopyRequest (this support already existed for CopyObjectRequest) * Utility function added to detect if an ARN is for a particular S3 resource type. This is to conform with the S3 API requirements of inserting "/object" in the path of these requests. ## Testing * New unit tests added * New integration tests added
1 parent ef15b89 commit 0c1a10a

File tree

8 files changed

+534
-1
lines changed

8 files changed

+534
-1
lines changed

‎services/s3/src/it/java/software/amazon/awssdk/services/s3/CopySourceIntegrationTest.java

Lines changed: 148 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,148 @@
1+
/*
2+
* Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
3+
*
4+
* Licensed under the Apache License, Version 2.0 (the "License").
5+
* You may not use this file except in compliance with the License.
6+
* A copy of the License is located at
7+
*
8+
* http://aws.amazon.com/apache2.0
9+
*
10+
* or in the "license" file accompanying this file. This file is distributed
11+
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
12+
* express or implied. See the License for the specific language governing
13+
* permissions and limitations under the License.
14+
*/
15+
16+
package software.amazon.awssdk.services.s3;
17+
18+
import static org.hamcrest.MatcherAssert.assertThat;
19+
import static org.hamcrest.Matchers.is;
20+
import static software.amazon.awssdk.testutils.service.S3BucketUtils.temporaryBucketName;
21+
22+
import java.nio.charset.StandardCharsets;
23+
import java.util.Arrays;
24+
import java.util.Collection;
25+
import java.util.HashMap;
26+
import java.util.Map;
27+
import java.util.UUID;
28+
import org.junit.After;
29+
import org.junit.Before;
30+
import org.junit.Test;
31+
import org.junit.runner.RunWith;
32+
import org.junit.runners.Parameterized;
33+
import org.junit.runners.Parameterized.Parameters;
34+
import software.amazon.awssdk.core.sync.RequestBody;
35+
import software.amazon.awssdk.services.s3.internal.handlers.CopySourceInterceptor;
36+
import software.amazon.awssdk.services.s3.model.BucketVersioningStatus;
37+
import software.amazon.awssdk.services.s3.model.CopyObjectRequest;
38+
import software.amazon.awssdk.services.s3.model.GetObjectRequest;
39+
import software.amazon.awssdk.services.s3.model.PutObjectRequest;
40+
import software.amazon.awssdk.services.s3.model.PutObjectResponse;
41+
42+
/**
43+
* Integration tests for the {@code sourceBucket}, {@code sourceKey}, and {@code sourceVersionId} parameters for
44+
* {@link CopyObjectRequest}. Specifically, we ensure that users are able to seamlessly use the same input for both the
45+
* {@link PutObjectRequest} key and the {@link CopyObjectRequest} source key (and not be required to manually URL encode the
46+
* COPY source key). This also effectively tests for parity with the SDK v1 behavior.
47+
*
48+
* @see CopySourceInterceptor
49+
*/
50+
@RunWith(Parameterized.class)
51+
public class CopySourceIntegrationTest extends S3IntegrationTestBase {
52+
53+
private static final String SOURCE_BUCKET_NAME = temporaryBucketName("copy-source-integ-test-src");
54+
private static final String DESTINATION_BUCKET_NAME = temporaryBucketName("copy-source-integ-test-dest");
55+
56+
@Before
57+
public void initializeTestData() throws Exception {
58+
createBucket(SOURCE_BUCKET_NAME);
59+
createBucket(DESTINATION_BUCKET_NAME);
60+
}
61+
62+
@After
63+
public void tearDown() {
64+
deleteBucketAndAllContents(SOURCE_BUCKET_NAME);
65+
deleteBucketAndAllContents(DESTINATION_BUCKET_NAME);
66+
}
67+
68+
@Parameters
69+
public static Collection parameters() throws Exception {
70+
return Arrays.asList(
71+
"simpleKey",
72+
"key/with/slashes",
73+
"\uD83E\uDEA3",
74+
"specialChars/ +!#$&'()*,:;=?@\"",
75+
"%20"
76+
);
77+
}
78+
79+
private final String key;
80+
81+
public CopySourceIntegrationTest(String key) {
82+
this.key = key;
83+
}
84+
85+
@Test
86+
public void copyObject_WithoutExplicitVersion_AcceptsSameKeyAsPut() throws Exception {
87+
String originalContent = UUID.randomUUID().toString();
88+
89+
s3.putObject(PutObjectRequest.builder()
90+
.bucket(SOURCE_BUCKET_NAME)
91+
.key(key)
92+
.build(), RequestBody.fromString(originalContent, StandardCharsets.UTF_8));
93+
94+
s3.copyObject(CopyObjectRequest.builder()
95+
.sourceBucket(SOURCE_BUCKET_NAME)
96+
.sourceKey(key)
97+
.destinationBucket(DESTINATION_BUCKET_NAME)
98+
.destinationKey(key)
99+
.build());
100+
101+
String copiedContent = s3.getObjectAsBytes(GetObjectRequest.builder()
102+
.bucket(DESTINATION_BUCKET_NAME)
103+
.key(key)
104+
.build()).asUtf8String();
105+
106+
assertThat(copiedContent, is(originalContent));
107+
}
108+
109+
/**
110+
* Test that we can correctly copy versioned source objects.
111+
* <p>
112+
* Motivated by: https://github.com/aws/aws-sdk-js/issues/727
113+
*/
114+
@Test
115+
public void copyObject_WithVersioning_AcceptsSameKeyAsPut() throws Exception {
116+
s3.putBucketVersioning(r -> r
117+
.bucket(SOURCE_BUCKET_NAME)
118+
.versioningConfiguration(v -> v.status(BucketVersioningStatus.ENABLED)));
119+
120+
Map<String, String> versionToContentMap = new HashMap<>();
121+
int numVersionsToCreate = 3;
122+
for (int i = 0; i < numVersionsToCreate; i++) {
123+
String originalContent = UUID.randomUUID().toString();
124+
PutObjectResponse response = s3.putObject(PutObjectRequest.builder()
125+
.bucket(SOURCE_BUCKET_NAME)
126+
.key(key)
127+
.build(),
128+
RequestBody.fromString(originalContent, StandardCharsets.UTF_8));
129+
versionToContentMap.put(response.versionId(), originalContent);
130+
}
131+
132+
versionToContentMap.forEach((versionId, originalContent) -> {
133+
s3.copyObject(CopyObjectRequest.builder()
134+
.sourceBucket(SOURCE_BUCKET_NAME)
135+
.sourceKey(key)
136+
.sourceVersionId(versionId)
137+
.destinationBucket(DESTINATION_BUCKET_NAME)
138+
.destinationKey(key)
139+
.build());
140+
141+
String copiedContent = s3.getObjectAsBytes(GetObjectRequest.builder()
142+
.bucket(DESTINATION_BUCKET_NAME)
143+
.key(key)
144+
.build()).asUtf8String();
145+
assertThat(copiedContent, is(originalContent));
146+
});
147+
}
148+
}

‎services/s3/src/main/java/software/amazon/awssdk/services/s3/internal/handlers/CopySourceInterceptor.java

Lines changed: 127 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,127 @@
1+
/*
2+
* Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
3+
*
4+
* Licensed under the Apache License, Version 2.0 (the "License").
5+
* You may not use this file except in compliance with the License.
6+
* A copy of the License is located at
7+
*
8+
* http://aws.amazon.com/apache2.0
9+
*
10+
* or in the "license" file accompanying this file. This file is distributed
11+
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
12+
* express or implied. See the License for the specific language governing
13+
* permissions and limitations under the License.
14+
*/
15+
16+
package software.amazon.awssdk.services.s3.internal.handlers;
17+
18+
import static software.amazon.awssdk.services.s3.internal.resource.S3ArnUtils.isArnFor;
19+
import static software.amazon.awssdk.utils.http.SdkHttpUtils.urlEncodeIgnoreSlashes;
20+
21+
import software.amazon.awssdk.annotations.SdkInternalApi;
22+
import software.amazon.awssdk.core.SdkRequest;
23+
import software.amazon.awssdk.core.interceptor.Context.ModifyRequest;
24+
import software.amazon.awssdk.core.interceptor.ExecutionAttributes;
25+
import software.amazon.awssdk.core.interceptor.ExecutionInterceptor;
26+
import software.amazon.awssdk.services.s3.internal.resource.S3ResourceType;
27+
import software.amazon.awssdk.services.s3.model.CopyObjectRequest;
28+
import software.amazon.awssdk.services.s3.model.PutObjectRequest;
29+
import software.amazon.awssdk.services.s3.model.UploadPartCopyRequest;
30+
31+
/**
32+
* This interceptor transforms the {@code sourceBucket}, {@code sourceKey}, and {@code sourceVersionId} parameters for
33+
* {@link CopyObjectRequest} and {@link UploadPartCopyRequest} into a {@code copySource} parameter. The logic needed to
34+
* construct a {@code copySource} can be considered non-trivial, so this interceptor facilitates allowing users to
35+
* use higher-level constructs that more closely match other APIs, like {@link PutObjectRequest}. Additionally, this
36+
* interceptor is responsible for URL encoding the relevant portions of the {@code copySource} value.
37+
* <p>
38+
* <a href="https://docs.aws.amazon.com/AmazonS3/latest/API/API_CopyObject.html#API_CopyObject_RequestParameters">API_CopyObject_RequestParameters</a>
39+
* <p>
40+
* <a href="https://docs.aws.amazon.com/AmazonS3/latest/API/API_UploadPartCopy.html#API_UploadPartCopy_RequestParameters">API_UploadPartCopy_RequestParameters</a>
41+
*/
42+
@SdkInternalApi
43+
public final class CopySourceInterceptor implements ExecutionInterceptor {
44+
45+
@Override
46+
public SdkRequest modifyRequest(ModifyRequest context, ExecutionAttributes executionAttributes) {
47+
SdkRequest request = context.request();
48+
if (request instanceof CopyObjectRequest) {
49+
return modifyCopyObjectRequest((CopyObjectRequest) request);
50+
}
51+
if (request instanceof UploadPartCopyRequest) {
52+
return modifyUploadPartCopyRequest((UploadPartCopyRequest) request);
53+
}
54+
return request;
55+
}
56+
57+
private static SdkRequest modifyCopyObjectRequest(CopyObjectRequest request) {
58+
if (request.copySource() != null) {
59+
requireNotSet(request.sourceBucket(), "sourceBucket");
60+
requireNotSet(request.sourceKey(), "sourceKey");
61+
requireNotSet(request.sourceVersionId(), "sourceVersionId");
62+
return request;
63+
}
64+
String copySource = constructCopySource(
65+
requireSet(request.sourceBucket(), "sourceBucket"),
66+
requireSet(request.sourceKey(), "sourceKey"),
67+
request.sourceVersionId()
68+
);
69+
return request.toBuilder()
70+
.sourceBucket(null)
71+
.sourceKey(null)
72+
.sourceVersionId(null)
73+
.copySource(copySource)
74+
.build();
75+
}
76+
77+
private static SdkRequest modifyUploadPartCopyRequest(UploadPartCopyRequest request) {
78+
if (request.copySource() != null) {
79+
requireNotSet(request.sourceBucket(), "sourceBucket");
80+
requireNotSet(request.sourceKey(), "sourceKey");
81+
requireNotSet(request.sourceVersionId(), "sourceVersionId");
82+
return request;
83+
}
84+
String copySource = constructCopySource(
85+
requireSet(request.sourceBucket(), "sourceBucket"),
86+
requireSet(request.sourceKey(), "sourceKey"),
87+
request.sourceVersionId()
88+
);
89+
return request.toBuilder()
90+
.sourceBucket(null)
91+
.sourceKey(null)
92+
.sourceVersionId(null)
93+
.copySource(copySource)
94+
.build();
95+
}
96+
97+
private static String constructCopySource(String sourceBucket, String sourceKey, String sourceVersionId) {
98+
StringBuilder copySource = new StringBuilder();
99+
copySource.append("/");
100+
copySource.append(urlEncodeIgnoreSlashes(sourceBucket));
101+
if (isArnFor(S3ResourceType.ACCESS_POINT, sourceBucket) || isArnFor(S3ResourceType.OUTPOST, sourceBucket)) {
102+
copySource.append("/object");
103+
}
104+
copySource.append("/");
105+
copySource.append(urlEncodeIgnoreSlashes(sourceKey));
106+
if (sourceVersionId != null) {
107+
copySource.append("?versionId=");
108+
copySource.append(urlEncodeIgnoreSlashes(sourceVersionId));
109+
}
110+
return copySource.toString();
111+
}
112+
113+
private static void requireNotSet(Object value, String paramName) {
114+
if (value != null) {
115+
throw new IllegalArgumentException(String.format("Parameter 'copySource' must not be used in conjunction with '%s'",
116+
paramName));
117+
}
118+
}
119+
120+
private static <T> T requireSet(T value, String paramName) {
121+
if (value == null) {
122+
throw new IllegalArgumentException(String.format("Parameter '%s' must not be null",
123+
paramName));
124+
}
125+
return value;
126+
}
127+
}

‎services/s3/src/main/java/software/amazon/awssdk/services/s3/internal/resource/S3ArnUtils.java

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -72,4 +72,14 @@ public static IntermediateOutpostResource parseOutpostArn(Arn arn) {
7272
.outpostSubresource(ArnResource.fromString(subresource))
7373
.build();
7474
}
75+
76+
public static boolean isArnFor(S3ResourceType s3ResourceType, String arnString) {
77+
try {
78+
Arn arn = Arn.fromString(arnString);
79+
String parsedResourceType = arn.resource().resourceType().get();
80+
return S3ResourceType.fromValue(parsedResourceType) == s3ResourceType;
81+
} catch (Exception ignored) {
82+
return false;
83+
}
84+
}
7585
}

‎services/s3/src/main/resources/META-INF/native-image/software.amazon.awssdk/s3/reflect-config.json

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -115,5 +115,14 @@
115115
"parameterTypes": []
116116
}
117117
]
118+
},
119+
{
120+
"name": "software.amazon.awssdk.services.s3.internal.handlers.CopySourceInterceptor",
121+
"methods": [
122+
{
123+
"name": "<init>",
124+
"parameterTypes": []
125+
}
126+
]
118127
}
119128
]

‎services/s3/src/main/resources/codegen-resources/customization.config

Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,52 @@
1717
]
1818
},
1919
"CopyObjectRequest": {
20+
"inject": [
21+
{
22+
"SourceBucket": {
23+
"shape": "BucketName",
24+
"documentation": "The name of the bucket containing the object to copy. The provided input will be URL encoded. The {@code sourceBucket}, {@code sourceKey}, and {@code sourceVersionId} parameters must not be used in conjunction with the {@code copySource} parameter."
25+
},
26+
"SourceKey": {
27+
"shape": "ObjectKey",
28+
"documentation": "The key of the object to copy. The provided input will be URL encoded. The {@code sourceBucket}, {@code sourceKey}, and {@code sourceVersionId} parameters must not be used in conjunction with the {@code copySource} parameter."
29+
},
30+
"SourceVersionId": {
31+
"shape": "ObjectVersionId",
32+
"documentation": "Specifies a particular version of the source object to copy. By default the latest version is copied. The {@code sourceBucket}, {@code sourceKey}, and {@code sourceVersionId} parameters must not be used in conjunction with the {@code copySource} parameter."
33+
}
34+
}
35+
],
36+
"modify": [
37+
{
38+
"Bucket": {
39+
"emitPropertyName": "DestinationBucket",
40+
"existingNameDeprecated": true
41+
},
42+
"Key": {
43+
"emitPropertyName": "DestinationKey",
44+
"existingNameDeprecated": true
45+
}
46+
}
47+
]
48+
},
49+
"UploadPartCopyRequest": {
50+
"inject": [
51+
{
52+
"SourceBucket": {
53+
"shape": "BucketName",
54+
"documentation": "The name of the bucket containing the object to copy. The provided input will be URL encoded. The {@code sourceBucket}, {@code sourceKey}, and {@code sourceVersionId} parameters must not be used in conjunction with the {@code copySource} parameter."
55+
},
56+
"SourceKey": {
57+
"shape": "ObjectKey",
58+
"documentation": "The key of the object to copy. The provided input will be URL encoded. The {@code sourceBucket}, {@code sourceKey}, and {@code sourceVersionId} parameters must not be used in conjunction with the {@code copySource} parameter."
59+
},
60+
"SourceVersionId": {
61+
"shape": "ObjectVersionId",
62+
"documentation": "Specifies a particular version of the source object to copy. By default the latest version is copied. The {@code sourceBucket}, {@code sourceKey}, and {@code sourceVersionId} parameters must not be used in conjunction with the {@code copySource} parameter."
63+
}
64+
}
65+
],
2066
"modify": [
2167
{
2268
"Bucket": {

‎services/s3/src/main/resources/software/amazon/awssdk/services/s3/execution.interceptors

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -10,4 +10,5 @@ software.amazon.awssdk.services.s3.internal.handlers.AsyncChecksumValidationInte
1010
software.amazon.awssdk.services.s3.internal.handlers.SyncChecksumValidationInterceptor
1111
software.amazon.awssdk.services.s3.internal.handlers.EnableTrailingChecksumInterceptor
1212
software.amazon.awssdk.services.s3.internal.handlers.ExceptionTranslationInterceptor
13-
software.amazon.awssdk.services.s3.internal.handlers.GetObjectInterceptor
13+
software.amazon.awssdk.services.s3.internal.handlers.GetObjectInterceptor
14+
software.amazon.awssdk.services.s3.internal.handlers.CopySourceInterceptor

0 commit comments

Comments
 (0)