Skip to content

Commit 28607f7

Browse files
Bennett-LynchBennett-Lynch
authored
[S3] Add support for more user-friendly CopyObject source parameters (#2612)
* [S3] Add support for more user-friendly CopyObject source parameters ## Motivation and Context The current S3Client interface has a cumbersome API for invoking CopyObjectRequests. We require users to define the bucket name, key name, and version ID in a raw string format. We require that the string conform to the S3 API, which forces users to know the intricate details for how to join these values together. Additionally, portions (but not all) of the value must be URL encoded, further increasing the burden. https://docs.aws.amazon.com/AmazonS3/latest/API/API_CopyObject.html#API_CopyObject_RequestParameters In the Java SDK v1, users are given explicit parameters for the different copy source attributes. But in v2, parity for this support is clearly lacking. E.g., v1: ``` s3.copyObject(new CopyObjectRequest() .withSourceBucketName(SOURCE_BUCKET) .withSourceKey(key) .withSourceVersionId(versionId) .withDestinationBucketName(DESTINATION_BUCKET) .withDestinationKey(key)); ``` v2: ``` s3.copyObject(CopyObjectRequest.builder() .copySource(SOURCE_BUCKET + "/" + key + "?versionId=" + versionId) .destinationBucket(DESTINATION_BUCKET) .destinationKey(key) .build()); ``` The v1 SDK will also URL encode on the user's behalf, allowing users to use the same input values as they would for a PutObjectRequest. The v2 code snippet above may appear to work for most users until they run into unexpected source keys that require URL encoding, at which point they will typically be given `NoSuchKey` errors. This API deficiency has been called out by users in at least the following issues: * #1313 * #1452 * #1656 * awsdocs/aws-doc-sdk-examples#740 ## Description * For both CopyObjectRequest and UploadPartCopyRequest, add explicit parameters for: SourceBucket, SourceKey, SourceVersionId * If specified, these values will be used to construct a CopySource on the user's behalf, including URL encoding the relevant portions. * These values are introduced in a backwards compatible fashion. Users who are already using CopySource today will see no change in behavior, but these new fields may not be used in conjunction with CopySource. * A follow-up PR will be submitted to propose deprecating the current CopySource parameter. It is excluded from this PR since our current code gen configuration lacks appropriate support. * Add support for "DestinationBucket" & "DestinationKey" to UploadPartCopyRequest (this support already existed for CopyObjectRequest) * Utility function added to detect if an ARN is for a particular S3 resource type. This is to conform with the S3 API requirements of inserting "/object" in the path of these requests. ## Testing * New unit tests added * New integration tests added
1 parent a848bdc commit 28607f7

File tree

9 files changed

+543
-1
lines changed

9 files changed

+543
-1
lines changed

‎.changes/next-release/feature-AmazonS3-9b198d0.json

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
{
2+
"category": "Amazon S3",
3+
"contributor": "",
4+
"type": "feature",
5+
"description": "Add support for more user-friendly CopyObject source parameters"
6+
}

‎services/s3/src/it/java/software/amazon/awssdk/services/s3/CopySourceIntegrationTest.java

Lines changed: 150 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,150 @@
1+
/*
2+
* Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
3+
*
4+
* Licensed under the Apache License, Version 2.0 (the "License").
5+
* You may not use this file except in compliance with the License.
6+
* A copy of the License is located at
7+
*
8+
* http://aws.amazon.com/apache2.0
9+
*
10+
* or in the "license" file accompanying this file. This file is distributed
11+
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
12+
* express or implied. See the License for the specific language governing
13+
* permissions and limitations under the License.
14+
*/
15+
16+
package software.amazon.awssdk.services.s3;
17+
18+
import static org.hamcrest.MatcherAssert.assertThat;
19+
import static org.hamcrest.Matchers.is;
20+
import static software.amazon.awssdk.testutils.service.S3BucketUtils.temporaryBucketName;
21+
22+
import java.nio.charset.StandardCharsets;
23+
import java.util.Arrays;
24+
import java.util.Collection;
25+
import java.util.HashMap;
26+
import java.util.Map;
27+
import java.util.UUID;
28+
import org.junit.AfterClass;
29+
import org.junit.BeforeClass;
30+
import org.junit.Test;
31+
import org.junit.runner.RunWith;
32+
import org.junit.runners.Parameterized;
33+
import org.junit.runners.Parameterized.Parameters;
34+
import software.amazon.awssdk.core.sync.RequestBody;
35+
import software.amazon.awssdk.services.s3.internal.handlers.CopySourceInterceptor;
36+
import software.amazon.awssdk.services.s3.model.BucketVersioningStatus;
37+
import software.amazon.awssdk.services.s3.model.CopyObjectRequest;
38+
import software.amazon.awssdk.services.s3.model.GetObjectRequest;
39+
import software.amazon.awssdk.services.s3.model.PutObjectRequest;
40+
import software.amazon.awssdk.services.s3.model.PutObjectResponse;
41+
42+
/**
43+
* Integration tests for the {@code sourceBucket}, {@code sourceKey}, and {@code sourceVersionId} parameters for
44+
* {@link CopyObjectRequest}. Specifically, we ensure that users are able to seamlessly use the same input for both the
45+
* {@link PutObjectRequest} key and the {@link CopyObjectRequest} source key (and not be required to manually URL encode the
46+
* COPY source key). This also effectively tests for parity with the SDK v1 behavior.
47+
*
48+
* @see CopySourceInterceptor
49+
*/
50+
@RunWith(Parameterized.class)
51+
public class CopySourceIntegrationTest extends S3IntegrationTestBase {
52+
53+
private static final String SOURCE_UNVERSIONED_BUCKET_NAME = temporaryBucketName("copy-source-integ-test-src");
54+
private static final String SOURCE_VERSIONED_BUCKET_NAME = temporaryBucketName("copy-source-integ-test-versioned-src");
55+
private static final String DESTINATION_BUCKET_NAME = temporaryBucketName("copy-source-integ-test-dest");
56+
57+
@BeforeClass
58+
public static void initializeTestData() throws Exception {
59+
createBucket(SOURCE_UNVERSIONED_BUCKET_NAME);
60+
createBucket(SOURCE_VERSIONED_BUCKET_NAME);
61+
s3.putBucketVersioning(r -> r
62+
.bucket(SOURCE_VERSIONED_BUCKET_NAME)
63+
.versioningConfiguration(v -> v.status(BucketVersioningStatus.ENABLED)));
64+
createBucket(DESTINATION_BUCKET_NAME);
65+
}
66+
67+
@AfterClass
68+
public static void tearDown() {
69+
deleteBucketAndAllContents(SOURCE_UNVERSIONED_BUCKET_NAME);
70+
deleteBucketAndAllContents(SOURCE_VERSIONED_BUCKET_NAME);
71+
deleteBucketAndAllContents(DESTINATION_BUCKET_NAME);
72+
}
73+
74+
@Parameters
75+
public static Collection<String> parameters() throws Exception {
76+
return Arrays.asList(
77+
"simpleKey",
78+
"key/with/slashes",
79+
"\uD83E\uDEA3",
80+
"specialChars/ +!#$&'()*,:;=?@\"",
81+
"%20"
82+
);
83+
}
84+
85+
private final String key;
86+
87+
public CopySourceIntegrationTest(String key) {
88+
this.key = key;
89+
}
90+
91+
@Test
92+
public void copyObject_WithoutVersion_AcceptsSameKeyAsPut() throws Exception {
93+
String originalContent = UUID.randomUUID().toString();
94+
95+
s3.putObject(PutObjectRequest.builder()
96+
.bucket(SOURCE_UNVERSIONED_BUCKET_NAME)
97+
.key(key)
98+
.build(), RequestBody.fromString(originalContent, StandardCharsets.UTF_8));
99+
100+
s3.copyObject(CopyObjectRequest.builder()
101+
.sourceBucket(SOURCE_UNVERSIONED_BUCKET_NAME)
102+
.sourceKey(key)
103+
.destinationBucket(DESTINATION_BUCKET_NAME)
104+
.destinationKey(key)
105+
.build());
106+
107+
String copiedContent = s3.getObjectAsBytes(GetObjectRequest.builder()
108+
.bucket(DESTINATION_BUCKET_NAME)
109+
.key(key)
110+
.build()).asUtf8String();
111+
112+
assertThat(copiedContent, is(originalContent));
113+
}
114+
115+
/**
116+
* Test that we can correctly copy versioned source objects.
117+
* <p>
118+
* Motivated by: https://github.com/aws/aws-sdk-js/issues/727
119+
*/
120+
@Test
121+
public void copyObject_WithVersion_AcceptsSameKeyAsPut() throws Exception {
122+
Map<String, String> versionToContentMap = new HashMap<>();
123+
int numVersionsToCreate = 3;
124+
for (int i = 0; i < numVersionsToCreate; i++) {
125+
String originalContent = UUID.randomUUID().toString();
126+
PutObjectResponse response = s3.putObject(PutObjectRequest.builder()
127+
.bucket(SOURCE_VERSIONED_BUCKET_NAME)
128+
.key(key)
129+
.build(),
130+
RequestBody.fromString(originalContent, StandardCharsets.UTF_8));
131+
versionToContentMap.put(response.versionId(), originalContent);
132+
}
133+
134+
versionToContentMap.forEach((versionId, originalContent) -> {
135+
s3.copyObject(CopyObjectRequest.builder()
136+
.sourceBucket(SOURCE_VERSIONED_BUCKET_NAME)
137+
.sourceKey(key)
138+
.sourceVersionId(versionId)
139+
.destinationBucket(DESTINATION_BUCKET_NAME)
140+
.destinationKey(key)
141+
.build());
142+
143+
String copiedContent = s3.getObjectAsBytes(GetObjectRequest.builder()
144+
.bucket(DESTINATION_BUCKET_NAME)
145+
.key(key)
146+
.build()).asUtf8String();
147+
assertThat(copiedContent, is(originalContent));
148+
});
149+
}
150+
}

‎services/s3/src/main/java/software/amazon/awssdk/services/s3/internal/handlers/CopySourceInterceptor.java

Lines changed: 126 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,126 @@
1+
/*
2+
* Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
3+
*
4+
* Licensed under the Apache License, Version 2.0 (the "License").
5+
* You may not use this file except in compliance with the License.
6+
* A copy of the License is located at
7+
*
8+
* http://aws.amazon.com/apache2.0
9+
*
10+
* or in the "license" file accompanying this file. This file is distributed
11+
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
12+
* express or implied. See the License for the specific language governing
13+
* permissions and limitations under the License.
14+
*/
15+
16+
package software.amazon.awssdk.services.s3.internal.handlers;
17+
18+
import static software.amazon.awssdk.utils.http.SdkHttpUtils.urlEncodeIgnoreSlashes;
19+
20+
import software.amazon.awssdk.annotations.SdkInternalApi;
21+
import software.amazon.awssdk.core.SdkRequest;
22+
import software.amazon.awssdk.core.interceptor.Context.ModifyRequest;
23+
import software.amazon.awssdk.core.interceptor.ExecutionAttributes;
24+
import software.amazon.awssdk.core.interceptor.ExecutionInterceptor;
25+
import software.amazon.awssdk.services.s3.internal.resource.S3ArnUtils;
26+
import software.amazon.awssdk.services.s3.internal.resource.S3ResourceType;
27+
import software.amazon.awssdk.services.s3.model.CopyObjectRequest;
28+
import software.amazon.awssdk.services.s3.model.PutObjectRequest;
29+
import software.amazon.awssdk.services.s3.model.UploadPartCopyRequest;
30+
import software.amazon.awssdk.utils.Validate;
31+
32+
/**
33+
* This interceptor transforms the {@code sourceBucket}, {@code sourceKey}, and {@code sourceVersionId} parameters for
34+
* {@link CopyObjectRequest} and {@link UploadPartCopyRequest} into a {@code copySource} parameter. The logic needed to
35+
* construct a {@code copySource} can be considered non-trivial, so this interceptor facilitates allowing users to
36+
* use higher-level constructs that more closely match other APIs, like {@link PutObjectRequest}. Additionally, this
37+
* interceptor is responsible for URL encoding the relevant portions of the {@code copySource} value.
38+
* <p>
39+
* <a href="https://docs.aws.amazon.com/AmazonS3/latest/API/API_CopyObject.html#API_CopyObject_RequestParameters">API_CopyObject_RequestParameters</a>
40+
* <p>
41+
* <a href="https://docs.aws.amazon.com/AmazonS3/latest/API/API_UploadPartCopy.html#API_UploadPartCopy_RequestParameters">API_UploadPartCopy_RequestParameters</a>
42+
*/
43+
@SdkInternalApi
44+
public final class CopySourceInterceptor implements ExecutionInterceptor {
45+
46+
@Override
47+
public SdkRequest modifyRequest(ModifyRequest context, ExecutionAttributes executionAttributes) {
48+
SdkRequest request = context.request();
49+
if (request instanceof CopyObjectRequest) {
50+
return modifyCopyObjectRequest((CopyObjectRequest) request);
51+
}
52+
if (request instanceof UploadPartCopyRequest) {
53+
return modifyUploadPartCopyRequest((UploadPartCopyRequest) request);
54+
}
55+
return request;
56+
}
57+
58+
private static SdkRequest modifyCopyObjectRequest(CopyObjectRequest request) {
59+
if (request.copySource() != null) {
60+
requireNotSet(request.sourceBucket(), "sourceBucket");
61+
requireNotSet(request.sourceKey(), "sourceKey");
62+
requireNotSet(request.sourceVersionId(), "sourceVersionId");
63+
return request;
64+
}
65+
String copySource = constructCopySource(
66+
requireSet(request.sourceBucket(), "sourceBucket"),
67+
requireSet(request.sourceKey(), "sourceKey"),
68+
request.sourceVersionId()
69+
);
70+
return request.toBuilder()
71+
.sourceBucket(null)
72+
.sourceKey(null)
73+
.sourceVersionId(null)
74+
.copySource(copySource)
75+
.build();
76+
}
77+
78+
private static SdkRequest modifyUploadPartCopyRequest(UploadPartCopyRequest request) {
79+
if (request.copySource() != null) {
80+
requireNotSet(request.sourceBucket(), "sourceBucket");
81+
requireNotSet(request.sourceKey(), "sourceKey");
82+
requireNotSet(request.sourceVersionId(), "sourceVersionId");
83+
return request;
84+
}
85+
String copySource = constructCopySource(
86+
requireSet(request.sourceBucket(), "sourceBucket"),
87+
requireSet(request.sourceKey(), "sourceKey"),
88+
request.sourceVersionId()
89+
);
90+
return request.toBuilder()
91+
.sourceBucket(null)
92+
.sourceKey(null)
93+
.sourceVersionId(null)
94+
.copySource(copySource)
95+
.build();
96+
}
97+
98+
private static String constructCopySource(String sourceBucket, String sourceKey, String sourceVersionId) {
99+
StringBuilder copySource = new StringBuilder();
100+
copySource.append("/");
101+
copySource.append(urlEncodeIgnoreSlashes(sourceBucket));
102+
S3ArnUtils.getArnType(sourceBucket).ifPresent(arnType -> {
103+
if (arnType == S3ResourceType.ACCESS_POINT || arnType == S3ResourceType.OUTPOST) {
104+
copySource.append("/object");
105+
}
106+
});
107+
copySource.append("/");
108+
copySource.append(urlEncodeIgnoreSlashes(sourceKey));
109+
if (sourceVersionId != null) {
110+
copySource.append("?versionId=");
111+
copySource.append(urlEncodeIgnoreSlashes(sourceVersionId));
112+
}
113+
return copySource.toString();
114+
}
115+
116+
private static void requireNotSet(Object value, String paramName) {
117+
Validate.isTrue(value == null, "Parameter 'copySource' must not be used in conjunction with '%s'",
118+
paramName);
119+
}
120+
121+
private static <T> T requireSet(T value, String paramName) {
122+
Validate.isTrue(value != null, "Parameter '%s' must not be null",
123+
paramName);
124+
return value;
125+
}
126+
}

‎services/s3/src/main/java/software/amazon/awssdk/services/s3/internal/resource/S3ArnUtils.java

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,7 @@
1616
package software.amazon.awssdk.services.s3.internal.resource;
1717

1818

19+
import java.util.Optional;
1920
import software.amazon.awssdk.annotations.SdkInternalApi;
2021
import software.amazon.awssdk.arns.Arn;
2122
import software.amazon.awssdk.arns.ArnResource;
@@ -72,4 +73,15 @@ public static IntermediateOutpostResource parseOutpostArn(Arn arn) {
7273
.outpostSubresource(ArnResource.fromString(subresource))
7374
.build();
7475
}
76+
77+
public static Optional<S3ResourceType> getArnType(String arnString) {
78+
try {
79+
Arn arn = Arn.fromString(arnString);
80+
String resourceType = arn.resource().resourceType().get();
81+
S3ResourceType s3ResourceType = S3ResourceType.fromValue(resourceType);
82+
return Optional.of(s3ResourceType);
83+
} catch (Exception ignored) {
84+
return Optional.empty();
85+
}
86+
}
7587
}

‎services/s3/src/main/resources/META-INF/native-image/software.amazon.awssdk/s3/reflect-config.json

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -115,5 +115,14 @@
115115
"parameterTypes": []
116116
}
117117
]
118+
},
119+
{
120+
"name": "software.amazon.awssdk.services.s3.internal.handlers.CopySourceInterceptor",
121+
"methods": [
122+
{
123+
"name": "<init>",
124+
"parameterTypes": []
125+
}
126+
]
118127
}
119128
]

‎services/s3/src/main/resources/codegen-resources/customization.config

Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,52 @@
1717
]
1818
},
1919
"CopyObjectRequest": {
20+
"inject": [
21+
{
22+
"SourceBucket": {
23+
"shape": "BucketName",
24+
"documentation": "The name of the bucket containing the object to copy. The provided input will be URL encoded. The {@code sourceBucket}, {@code sourceKey}, and {@code sourceVersionId} parameters must not be used in conjunction with the {@code copySource} parameter."
25+
},
26+
"SourceKey": {
27+
"shape": "ObjectKey",
28+
"documentation": "The key of the object to copy. The provided input will be URL encoded. The {@code sourceBucket}, {@code sourceKey}, and {@code sourceVersionId} parameters must not be used in conjunction with the {@code copySource} parameter."
29+
},
30+
"SourceVersionId": {
31+
"shape": "ObjectVersionId",
32+
"documentation": "Specifies a particular version of the source object to copy. By default the latest version is copied. The {@code sourceBucket}, {@code sourceKey}, and {@code sourceVersionId} parameters must not be used in conjunction with the {@code copySource} parameter."
33+
}
34+
}
35+
],
36+
"modify": [
37+
{
38+
"Bucket": {
39+
"emitPropertyName": "DestinationBucket",
40+
"existingNameDeprecated": true
41+
},
42+
"Key": {
43+
"emitPropertyName": "DestinationKey",
44+
"existingNameDeprecated": true
45+
}
46+
}
47+
]
48+
},
49+
"UploadPartCopyRequest": {
50+
"inject": [
51+
{
52+
"SourceBucket": {
53+
"shape": "BucketName",
54+
"documentation": "The name of the bucket containing the object to copy. The provided input will be URL encoded. The {@code sourceBucket}, {@code sourceKey}, and {@code sourceVersionId} parameters must not be used in conjunction with the {@code copySource} parameter."
55+
},
56+
"SourceKey": {
57+
"shape": "ObjectKey",
58+
"documentation": "The key of the object to copy. The provided input will be URL encoded. The {@code sourceBucket}, {@code sourceKey}, and {@code sourceVersionId} parameters must not be used in conjunction with the {@code copySource} parameter."
59+
},
60+
"SourceVersionId": {
61+
"shape": "ObjectVersionId",
62+
"documentation": "Specifies a particular version of the source object to copy. By default the latest version is copied. The {@code sourceBucket}, {@code sourceKey}, and {@code sourceVersionId} parameters must not be used in conjunction with the {@code copySource} parameter."
63+
}
64+
}
65+
],
2066
"modify": [
2167
{
2268
"Bucket": {

‎services/s3/src/main/resources/software/amazon/awssdk/services/s3/execution.interceptors

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -10,4 +10,5 @@ software.amazon.awssdk.services.s3.internal.handlers.AsyncChecksumValidationInte
1010
software.amazon.awssdk.services.s3.internal.handlers.SyncChecksumValidationInterceptor
1111
software.amazon.awssdk.services.s3.internal.handlers.EnableTrailingChecksumInterceptor
1212
software.amazon.awssdk.services.s3.internal.handlers.ExceptionTranslationInterceptor
13-
software.amazon.awssdk.services.s3.internal.handlers.GetObjectInterceptor
13+
software.amazon.awssdk.services.s3.internal.handlers.GetObjectInterceptor
14+
software.amazon.awssdk.services.s3.internal.handlers.CopySourceInterceptor

0 commit comments

Comments
 (0)