Add support for AwsResponseMetadata

[zoewangg]

Description

Add AwsResponseMetadata support to allow customers to retrieve metadata information such as requestId, hostId for debugging purpose.

see #670

streaming operation looks like the following:

    public <ReturnT> ReturnT getObject(GetObjectRequest getObjectRequest,
                                       ResponseTransformer<GetObjectResponse, ReturnT> responseTransformer) throws NoSuchKeyException, AwsServiceException,
                                                                                                                   SdkClientException, S3Exception {
        HttpResponseHandler<GetObjectResponse> responseHandler = new S3ResponseHandler<>(new GetObjectResponseUnmarshaller(),
                                                                                           new StaxOperationMetadata().withHasStreamingSuccessResponse(true));
        DefaultErrorResponseHandler errorResponseHandler = new DefaultErrorResponseHandler(exceptionUnmarshallers);
        return clientHandler.execute(
            new ClientExecutionParams<GetObjectRequest, GetObjectResponse>().withResponseHandler(responseHandler)
                                                                            .withErrorResponseHandler(errorResponseHandler).withInput(getObjectRequest)
                                                                            .withMarshaller(new GetObjectRequestMarshaller()), responseTransformer);
    }

non-streaming operation:

    public GetObjectAclResponse getObjectAcl(GetObjectAclRequest getObjectAclRequest) throws NoSuchKeyException,
                                                                                             AwsServiceException, SdkClientException, S3Exception {
        S3ResponseHandler<GetObjectAclResponse> responseHandler = new S3ResponseHandler<>(
            new GetObjectAclResponseUnmarshaller(), new StaxOperationMetadata().withHasStreamingSuccessResponse(false));
        DefaultErrorResponseHandler errorResponseHandler = new DefaultErrorResponseHandler(exceptionUnmarshallers);
        return clientHandler.execute(new ClientExecutionParams<GetObjectAclRequest, GetObjectAclResponse>()
                                         .withResponseHandler(responseHandler).withErrorResponseHandler(errorResponseHandler)
                                         .withInput(getObjectAclRequest).withMarshaller(new GetObjectAclRequestMarshaller()));
    }

Pending

• Codegen changes are pending.

I've include the DefaultS3Client, S3ResponseMetadata, S3ResponseHanlder in the PR.

Proposed Codegen Changes:

    "customResponseMetadata": {
        "EXTENDED_REQUEST_ID" : "x-amz-id-2",
        "CLOUD_FRONT_ID" : "X-Amz-Cf-Id",
        "REQUEST_ID" : "x-amz-request-id"
    }

customResponseMetadata is a map of metadata key to header key. S3ResponseMetadata and S3ResponseHandler will be generated based on it.

Currently it only supports header, let me know if there are other places we can retrieve additional metadata.

• Integ tests

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)

Checklist

• I have read the CONTRIBUTING document
• Local run of mvn install succeeds
• My code follows the code style of this project
• My change requires a change to the Javadoc documentation
• I have updated the Javadoc documentation accordingly
• I have read the README document
• I have added tests to cover my changes
• All new and existing tests passed
• A short description of the change has been added to the CHANGELOG

License

• I confirm that this pull request can be released under the Apache 2 license

[millems]

What does the customer experience look like, accessing the data?

I'd expect response.metadata().extendedRequestId() or something similar.

[zoewangg]

Here's the test: https://github.com/aws/aws-sdk-java-v2/pull/701/files#diff-c598a7406fc83ad8b0cc5ff552173b69

Basically,

    private void verifyResponseMetadata(S3Response response) {
        S3ResponseMetadata s3ResponseMetadata = response.responseMetadata();
        assertThat(s3ResponseMetadata).isNotNull();
        assertThat(s3ResponseMetadata.metadata().size()).isEqualTo(3);
        assertThat(s3ResponseMetadata.requestId()).isNotEqualTo("UNKNOWN");
        assertThat(s3ResponseMetadata.extendedRequestId()).isNotEqualTo("UNKNOWN");
    }

yeah, responseMetadata() seems a bit verbose, will update it to metadata()

[millems]

response.metadata().metadata() seems like stuttering. Is there a reason we need to expose a map view of the metadata?

[zoewangg]

Unfortunately, we cannot use metadata() because it conflicts with some response models that have metadata fields, such as GetObjectResponse.metadata(), I will keep responseMetadata for now unless anyone has better name.

Is there a reason we need to expose a map view of the metadata?

I thought it would be useful for customers who want to retrieve all metadata without doing multiple method calls. As to the naming, how about response.responseMetadata().rawData()? I guess we can also remove it for now and add it back if they ask.

[zoewangg]

Removed the logic of subclassing of responsehandler to extract metadata since simpleDb is the only services has additional metadata and we no longer support simpledb in v2.

All headers are being put to the AwsResponseMetadata.metadata map.

• Services with responsemetadata customization: a service specific ResponseMetadata that subclasses AwsResponseMetadata(abstract class) will be generated to expose specific metadata methods.

• No customized response metadata all service responses are going to use DefaultAwsResponseMetadata which only has requestId() defined.

[shorea]

Is this still needed if we put all headers into the map?

[zoewangg]

Yeah, this is still needed for ec2 because the requestId for it is extracted from xml.

[shorea]

Kinesis actually has an id-2 like S3. Can we customize for that?

[zoewangg]

+1

[shorea]

Should we move the unmarshaller into this param like class? Or do you wanna keep it separate to avoid including type params?

[zoewangg]

I made it like this to follow the same pattern as json protocol.

aws-sdk-java-v2/core/aws-core/src/main/java/software/amazon/awssdk/awscore/protocol/json/AwsJsonProtocolFactory.java

Lines 68 to 71 in 2fa64ca

	public <T> JsonResponseHandler<T> createResponseHandler(
	JsonOperationMetadata operationMetadata,
	Unmarshaller<T, JsonUnmarshallerContext> responseUnmarshaller) {
	return getSdkFactory().createResponseHandler(operationMetadata, responseUnmarshaller);

But we can move the unmarshaller into this class. Don't have strong opinions.

[shorea]

I think I like the current pattern as it keeps the metadata object a simple POJO with no type params.