Skip to content

Commit 63ade31

Browse files
zoewanggjoviegasmillemsdavidh44
authored
Merge TM feature branch to mainline (#3619)
* Replacing S3TransferManager interfaces that allowed builder methods of S3ClientConfiguration with builder methods of S3AsyncClient (#3247) * Added customization in codegen to generate additional builder methods (#3252) * S3Object based DownloadFilter and removing DownloadFileContext as destination based filter is removed (#3258) * Moved tm POJO classes to model pckage and tm config classes to config package. Added integration tests for s3 select using S3CrtAsyncClient (#3289) * Fix broken integ test (#3301) * S3 Transfer manager renamings based on feedback: (#3297) 1. Rename destinationDirectory to destination. 2. Move DownloadDirectoryRequest.prefix and delimiter to just rely on modifying the list requests. 3. Remove upload directory recursive option in favor of using maxDepth(1). 4. Rename UploadDirectoryRequest's prefix and delimiter to s3Prefix and s3Delimiter. 5. Rename ResumableFileDownload's to* and writeTo* methods to serializeTo*. Remove charsets from write/read methods, and just use UTF-8. 6. Do not base64 encode when writing ResumableFileDownload to disk. * Allow pausing a resumed download even when the download hasn't already started. (#3300) * Add POJO classes for upload pause/resume (#3337) * Refactoring of Transfer manager APIs (#3374) * Refactoring of Transfer manager APIs * Merging the integ test failure Pr 2119 from stagging branch * Add flexible checksum support and update perf tests (#3376) * Fix flexiblechecksum implementation (#3391) * [TM upload pause/resume Part 2] Implement pause and resume for uploadFile (#3357) * Implement pause and resume for uploadFile * Update Javadocs * address feedback * Implement automatic multipart copy functionality in S3 CRT async client (#3403) * Implement automatic multipart copy functionality in S3 CRT async client * Add more tests * fix cancellation logic * Refactor CopyRequestProvider, fix request conversion and add more tests * Fix checkstyle * Transfer Manager tests refactoring (#3420) * Remove use of Junit4, clean up and consolidate tests in tm module * Ignoring the test if unicode can't be used as directory name * Add serialization and deserialization support for ResumableFileUpload (#3432) * Support serialization and deserialization of ResumableFileUpload * Address feedback * Empty json should be unmarshalled to empty map * Errors should not be wrapped - S3 Transfer Manager (#3433) * Errors should not be wrapped * update handleException() * Changelog entry * Resolve comments Update changelog description, refactor handleException(), add test * Add failed message to SdkException * Refactor handleException() and format changelog (#3461) * Fixed an issue where SSEC params were not correctly passed in copy operation (#3464) * Replace inline snippets with external compilable snippets (#3465) * Replace inline snippets with external compilable snippets * Fix build and address feedback * Fix build * Only enable CRT checksum for getObject and putObject (#3477) * Only use CRT flexible checksum for getObject and putObject * Fix build * Fix integ tests set up and tear down steps (#3485) * Enable backpressure in TM (#3533) * integrate with crt s3 flow control * Update benchmark code * Add backpressure config * Change window size * Update initial window size * Change intial window size * Use heap max memory for initial window size * Give some buffer * change window size * Make read buffer size configurable * Log result to a file * Various updates * Various updates * Add CRT benchmark * Various updates * Fix checkstyle errors and tests * Fix flaky test * Fix checkstyle errors * Add validation * Add tests * For copy operation, always forward multipart copy exception from one … (#3549) * For copy operation, always forward multipart copy exception from one request to other multipart copy requests * Minor refactoring in CopyObjectHelper (#3552) * Add benchmarks for copy, uploadDirectory and downloadDirectory (#3551) * Add benchmarks for copy, uploadDirectory and downloadDirectory * Update sample code and fix snippet path (#3567) * Update sample code and fix snippet path * Fix link * Integrate with CRT checksum fix (#3566) * Integrate with CRT checksum fix * Rename sourceDirectory to source and add S3AsycncClient#crtCreate (#3572) * Rname sourceDirectory to source and add S3AsycncClient#crtCreate * Use ByteBufferStoringSubscriber (#3581) * Use ByteBufferStoringSubscriber * Add a comment * Create constant for bytes bufferred * Increase chunk size for file upload (#3583) * Rename S3TransferManager.build().maxDepth to uploadDirectoryMaxDepth, rename S3TransferManager.builder().s3AsyncClient to .s3Client (#3584) * Fixed an issue where sdkRepsonse is not present in the ProgressSnapshot for upload and copy (#3585) * Throw UnsupportedOperationException if a user tries to pause a upload… (#3586) * Throw UnsupportedOperationException if a user tries to pause a upload with non CRT-based S3 client * Use SimplePublisher (#3594) * Update documentation for Transfer Manager (#3592) * Update javadoc * Integrate with latest CRT pause/resume fix (#3588) * Integrate with latest CRT pause/resume fix * Bump CRT version * Fixed an issue that could result in uncompletable future when headObject request threw exception in copy (#3609) * Make crt dependency optional in transfer manager module (#3613) * Make aws-crt an optional dependency in s3-transfer-manager module. * Update README * Fix category for changelog entries Co-authored-by: John Viegas <[email protected]> Co-authored-by: Matthew Miller <[email protected]> Co-authored-by: David Ho <[email protected]>
1 parent 98c4188 commit 63ade31

File tree

217 files changed

+9803
-5661
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

217 files changed

+9803
-5661
lines changed

‎.changes/next-release/bugfix-AWSS3TransferManager-6d575a5.json

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
{
2+
"type": "bugfix",
3+
"category": "Amazon S3 Transfer Manager",
4+
"contributor": "",
5+
"description": "Require setting the bytes transferred on transfer progress snapshots. This prevents programming bugs where the caller forgets to set the value and it gets defaulted to 0."
6+
}

‎.changes/next-release/bugfix-AWSS3TransferManager-e849d27.json

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
{
2+
"type": "bugfix",
3+
"category": "Amazon S3 Transfer Manager",
4+
"contributor": "",
5+
"description": "Allow pausing a resumed download, even if the resumed download hasn't started."
6+
}

‎.changes/next-release/bugfix-S3TransferManager-f1c2208.json

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
{
2+
"category": "Amazon S3 Transfer Manager",
3+
"contributor": "",
4+
"type": "bugfix",
5+
"description": "Fixed issues in S3 Transfer Manager resumeDownloadFile API where errors were being wrapped by SdkClientException"
6+
}

‎.changes/next-release/feature-AWSSDKTransferMananger-85d49ff.json

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
{
2+
"type": "feature",
3+
"category": "Amazon S3 Transfer Manager",
4+
"contributor": "",
5+
"description": "Flattened and removed OverrideConfigurations such that these can be configured directly to S3TransferManagerBuilder, upload and transfer Requests. Also refactored certain attributes and API names to represent better meaningful names."
6+
}

‎.changes/next-release/feature-AmazonS3TransferManager-0639145.json

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
{
2+
"type": "feature",
3+
"category": "Amazon S3 Transfer Manager",
4+
"contributor": "",
5+
"description": "Rename `UploadDirectoryRequest.prefix` and `delimiter` to `s3Prefix` and `s3Delimiter` to make it clear these are S3 parameters, not file-system parameters."
6+
}

‎.changes/next-release/feature-AmazonS3TransferManager-345c5d4.json

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
{
2+
"type": "feature",
3+
"category": "Amazon S3 Transfer Manager",
4+
"contributor": "",
5+
"description": "Rename `DownloadDirectoryRequest.destinationDirectory` to `destination`."
6+
}

‎.changes/next-release/feature-AmazonS3TransferManager-b2e7839.json

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
{
2+
"type": "feature",
3+
"category": "Amazon S3 Transfer Manager",
4+
"contributor": "",
5+
"description": "Rename `ResumableFileDownload`'s `to` and `writeTo` methods to `serializeTo` to make it clear that these aren't affecting the download."
6+
}

‎.changes/next-release/feature-AmazonS3TransferManager-fe2d57a.json

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
{
2+
"type": "feature",
3+
"category": "Amazon S3 Transfer Manager",
4+
"contributor": "",
5+
"description": "Do not base 64 encode when writing `ResumableFileDownload` to disk."
6+
}

‎.changes/next-release/feature-S3TransferManagerDeveloperPreview-f9efc42.json

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
{
2+
"type": "feature",
3+
"category": "Amazon S3 Transfer Manager",
4+
"contributor": "",
5+
"description": "Moved POJO classes to `software.amazon.awssdk.transfer.s3.model` and moved configuration classes to `software.amazon.awssdk.transfer.s3.config`"
6+
}

‎.changes/next-release/removal-AWSSDKTransferMananger-a684fae.json

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
{
2+
"type": "removal",
3+
"category": "Amazon S3 Transfer Manager",
4+
"contributor": "",
5+
"description": "Backward incompatible changes after removing s3ClientConfiguration() from builder API of S3TransferManager Interface.\n- Added crtBuilder in S3AsyncClient Interface.\n- Backward incompatible changes after changing args of test() API in DownloadFilter Interface from DownloadFileContext to S3Object."
6+
}

‎.changes/next-release/removal-AmazonS3TransferManager-20eecbb.json

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
{
2+
"type": "removal",
3+
"category": "Amazon S3 Transfer Manager",
4+
"contributor": "",
5+
"description": "Remove `DownloadDirectoryRequest.prefix` and `delimiter`. The same functionality is already available via the `listObjectsV2RequestTransformer`."
6+
}

‎.changes/next-release/removal-AmazonS3TransferManager-68fc4a5.json

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
{
2+
"type": "removal",
3+
"category": "Amazon S3 Transfer Manager",
4+
"contributor": "",
5+
"description": "Remove UploadDirectoryRequest's override configuration's `recursive` option. The same functionality can be achieved with `UploadDirectoryRequest.builder().maxDepth(1)` or S3TransferManager.builder().maxUploadDirectoryDepth(1)."
6+
}

‎.changes/next-release/removal-AmazonS3TransferManager-7e62a30.json

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
{
2+
"type": "removal",
3+
"category": "Amazon S3 Transfer Manager",
4+
"contributor": "",
5+
"description": "Removed charset options from `ResumableFileDownload`'s string options, because there's no reason to choose something other than UTF-8."
6+
}

‎.changes/next-release/removal-S3TransferManager-2c2c795.json

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
{
2+
"type": "removal",
3+
"category": "Amazon S3 Transfer Manager",
4+
"contributor": "",
5+
"description": "Make `aws-crt` an optional dependency in `s3-transfer-manager` module. Customers need to explicitly add `aws-crt` dependency if they want to use CRT-based Transfer Manager"
6+
}

‎build-tools/src/main/resources/software/amazon/awssdk/spotbugs-suppressions.xml

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -252,4 +252,18 @@
252252
</Or>
253253
<Bug pattern="SDK_BAD_METHOD_CALL"/>
254254
</Match>
255+
256+
<!-- False positive on JDK 11+ -->
257+
<Match>
258+
<Class name="software.amazon.awssdk.transfer.s3.model.ResumableFileDownload"/>
259+
<Method name="fromFile"/>
260+
<Bug pattern="RCN_REDUNDANT_NULLCHECK_OF_NONNULL_VALUE"/>
261+
</Match>
262+
263+
<!-- False positive on JDK 11+ -->
264+
<Match>
265+
<Class name="software.amazon.awssdk.transfer.s3.model.ResumableFileUpload"/>
266+
<Method name="fromFile"/>
267+
<Bug pattern="RCN_REDUNDANT_NULLCHECK_OF_NONNULL_VALUE"/>
268+
</Match>
255269
</FindBugsFilter>

‎codegen/src/main/java/software/amazon/awssdk/codegen/model/config/customization/AdditionalBuilderMethod.java

Lines changed: 111 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,111 @@
1+
/*
2+
* Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
3+
*
4+
* Licensed under the Apache License, Version 2.0 (the "License").
5+
* You may not use this file except in compliance with the License.
6+
* A copy of the License is located at
7+
*
8+
* http://aws.amazon.com/apache2.0
9+
*
10+
* or in the "license" file accompanying this file. This file is distributed
11+
* on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
12+
* express or implied. See the License for the specific language governing
13+
* permissions and limitations under the License.
14+
*/
15+
16+
package software.amazon.awssdk.codegen.model.config.customization;
17+
18+
import java.util.Locale;
19+
import software.amazon.awssdk.core.ClientType;
20+
21+
/**
22+
* Config required to generate the additional static methods in Client interface.
23+
*/
24+
public class AdditionalBuilderMethod {
25+
26+
/**
27+
* Name of the additional static method.
28+
*/
29+
private String methodName;
30+
31+
/**
32+
* Fqcn of the return type
33+
*/
34+
private String returnType;
35+
36+
/**
37+
* Fqcn of the class that will delegate the static method call
38+
*/
39+
private String instanceType;
40+
41+
/**
42+
* JavaDoc for the method
43+
*/
44+
private String javaDoc;
45+
46+
/**
47+
* Method body
48+
*/
49+
private String statement;
50+
51+
/**
52+
* The clientType for which the builder needs to be added.
53+
*/
54+
private ClientType clientType;
55+
56+
public String getReturnType() {
57+
return returnType;
58+
}
59+
60+
public void setReturnType(String returnType) {
61+
this.returnType = returnType;
62+
}
63+
64+
public String getJavaDoc() {
65+
return javaDoc;
66+
}
67+
68+
public void setJavaDoc(String javaDoc) {
69+
this.javaDoc = javaDoc;
70+
}
71+
72+
public String getMethodName() {
73+
return methodName;
74+
}
75+
76+
public void setMethodName(String methodName) {
77+
this.methodName = methodName;
78+
}
79+
80+
public String getInstanceType() {
81+
return instanceType;
82+
}
83+
84+
public void setInstanceType(String instanceType) {
85+
this.instanceType = instanceType;
86+
}
87+
88+
public String getClientType() {
89+
return clientType != null ? clientType.toString() : null;
90+
}
91+
92+
public void setClientType(String clientType) {
93+
this.clientType = clientType != null ? ClientType.valueOf(clientType.toUpperCase(Locale.US)) : null;
94+
}
95+
96+
public ClientType getClientTypeEnum() {
97+
return clientType;
98+
}
99+
100+
public void setClientTypeEnum(ClientType clientType) {
101+
this.clientType = clientType;
102+
}
103+
104+
public String getStatement() {
105+
return statement;
106+
}
107+
108+
public void setStatement(String statement) {
109+
this.statement = statement;
110+
}
111+
}

‎codegen/src/main/java/software/amazon/awssdk/codegen/model/config/customization/CustomizationConfig.java

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -160,6 +160,11 @@ public class CustomizationConfig {
160160
*/
161161
private UtilitiesMethod utilitiesMethod;
162162

163+
/**
164+
* Config to generate a additional Builder methods in the client interface.
165+
*/
166+
private List<AdditionalBuilderMethod> additionalBuilderMethods;
167+
163168
/**
164169
* Force generation of deprecated client builder method 'enableEndpointDiscovery'. Only services that already had
165170
* this method when it was deprecated require this flag to be set.
@@ -444,6 +449,15 @@ public void setUtilitiesMethod(UtilitiesMethod utilitiesMethod) {
444449
this.utilitiesMethod = utilitiesMethod;
445450
}
446451

452+
453+
public List<AdditionalBuilderMethod> getAdditionalBuilderMethods() {
454+
return additionalBuilderMethods;
455+
}
456+
457+
public void setAdditionalBuilderMethods(List<AdditionalBuilderMethod> additionalBuilderMethods) {
458+
this.additionalBuilderMethods = additionalBuilderMethods;
459+
}
460+
447461
public boolean isEnableEndpointDiscoveryMethodRequired() {
448462
return enableEndpointDiscoveryMethodRequired;
449463
}

‎codegen/src/main/java/software/amazon/awssdk/codegen/poet/client/AsyncClientInterface.java

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -42,6 +42,7 @@
4242
import software.amazon.awssdk.codegen.docs.DocConfiguration;
4343
import software.amazon.awssdk.codegen.docs.SimpleMethodOverload;
4444
import software.amazon.awssdk.codegen.docs.WaiterDocs;
45+
import software.amazon.awssdk.codegen.model.config.customization.AdditionalBuilderMethod;
4546
import software.amazon.awssdk.codegen.model.config.customization.UtilitiesMethod;
4647
import software.amazon.awssdk.codegen.model.intermediate.IntermediateModel;
4748
import software.amazon.awssdk.codegen.model.intermediate.OperationModel;
@@ -56,6 +57,7 @@
5657
import software.amazon.awssdk.core.async.AsyncResponseTransformer;
5758
import software.amazon.awssdk.regions.ServiceMetadataProvider;
5859
import software.amazon.awssdk.regions.providers.DefaultAwsRegionProviderChain;
60+
import software.amazon.awssdk.utils.Validate;
5961

6062
public class AsyncClientInterface implements ClassSpec {
6163

@@ -107,6 +109,13 @@ public TypeSpec poetSpec() {
107109
result.addMethod(utilitiesMethod());
108110
}
109111

112+
List<AdditionalBuilderMethod> additionaBuilders = model.getCustomizationConfig().getAdditionalBuilderMethods();
113+
if (additionaBuilders != null && !additionaBuilders.isEmpty()) {
114+
additionaBuilders.stream()
115+
.filter(builder -> software.amazon.awssdk.core.ClientType.ASYNC.equals(builder.getClientTypeEnum()))
116+
.forEach(builders -> result.addMethod(additionalBuilders(builders)));
117+
}
118+
110119
if (model.hasWaiters()) {
111120
result.addMethod(waiterMethod());
112121
}
@@ -462,6 +471,23 @@ protected MethodSpec utilitiesMethod() {
462471
return utilitiesOperationBody(builder).build();
463472
}
464473

474+
private MethodSpec additionalBuilders(AdditionalBuilderMethod additionalMethod) {
475+
476+
String methodName = Validate.paramNotNull(additionalMethod.getMethodName(), "methodName");
477+
ClassName returnType = PoetUtils.classNameFromFqcn(
478+
Validate.paramNotNull(additionalMethod.getReturnType(), "returnType"));
479+
ClassName instanceType = PoetUtils.classNameFromFqcn(
480+
Validate.paramNotNull(additionalMethod.getInstanceType(), "instanceType"));
481+
482+
MethodSpec.Builder builder = MethodSpec.methodBuilder(methodName)
483+
.returns(returnType)
484+
.addModifiers(Modifier.STATIC, Modifier.PUBLIC)
485+
.addJavadoc(additionalMethod.getJavaDoc())
486+
.addStatement("return $T.$L", instanceType, additionalMethod.getStatement());
487+
488+
return builder.build();
489+
}
490+
465491
protected MethodSpec.Builder utilitiesOperationBody(MethodSpec.Builder builder) {
466492
return builder.addModifiers(Modifier.DEFAULT).addStatement("throw new $T()", UnsupportedOperationException.class);
467493
}

‎codegen/src/main/java/software/amazon/awssdk/codegen/poet/client/DelegatingAsyncClientClass.java

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -56,7 +56,7 @@ public TypeSpec poetSpec() {
5656
.addAnnotation(SdkInternalApi.class)
5757
.addModifiers(Modifier.ABSTRACT, Modifier.PUBLIC)
5858
.addField(FieldSpec.builder(interfaceClass, "delegate")
59-
.addModifiers(Modifier.PRIVATE, Modifier.FINAL)
59+
.addModifiers(Modifier.PROTECTED, Modifier.FINAL)
6060
.build())
6161
.addMethods(operations())
6262
.addMethod(closeMethod());

‎codegen/src/test/resources/software/amazon/awssdk/codegen/poet/client/c2j/rest-json/customization.config

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,40 @@
2222
"returnType": "software.amazon.awssdk.services.json.JsonUtilities",
2323
"createMethodParams": ["param1", "param2", "param3"]
2424
},
25+
"additionalBuilderMethods": [
26+
{
27+
"methodName": "builderOne",
28+
"clientType": "ASYNC",
29+
"instanceType": "software.amazon.awssdk.services.builder.DefaultBuilder",
30+
"returnType": "software.amazon.awssdk.services.builder.CustomBuilder",
31+
"statement": "builder().build()",
32+
"javaDoc": "Create a default builder"
33+
},
34+
{
35+
"methodName": "builderTwo",
36+
"clientType": "ASYNC",
37+
"instanceType": "software.amazon.awssdk.services.builder.DefaultBuilderTwo",
38+
"returnType": "software.amazon.awssdk.services.builder.Builder",
39+
"statement": "builder2().build()",
40+
"javaDoc": "Create a default builder two"
41+
},
42+
43+
{
44+
"methodName": "builderThree",
45+
"clientType": "SYNC",
46+
"instanceType": "software.amazon.awssdk.services.builder.DefaultBuilder",
47+
"returnType": "software.amazon.awssdk.services.builder.CustomBuilder",
48+
"statement": "builder().build()",
49+
"javaDoc": "Create a default builder"
50+
},
51+
{
52+
"methodName": "builderFour",
53+
"instanceType": "software.amazon.awssdk.services.builder.DefaultBuilder",
54+
"returnType": "software.amazon.awssdk.services.builder.CustomBuilder",
55+
"statement": "builder().build()",
56+
"javaDoc": "Create a default builder"
57+
}
58+
],
2559
"useLegacyEventGenerationScheme": {
2660
"EventStream": ["EventOne", "event-two", "eventThree"]
2761
},

‎codegen/src/test/resources/software/amazon/awssdk/codegen/poet/client/test-abstract-async-client-class.java

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -46,7 +46,7 @@
4646
@Generated("software.amazon.awssdk:codegen")
4747
@SdkInternalApi
4848
public abstract class DelegatingJsonAsyncClient implements JsonAsyncClient {
49-
private final JsonAsyncClient delegate;
49+
protected final JsonAsyncClient delegate;
5050

5151
public DelegatingJsonAsyncClient(JsonAsyncClient delegate) {
5252
this.delegate = delegate;

0 commit comments

Comments
 (0)