Missing dependency leads to opaque failover with DefaultCredentialsProvider

[ascheja]

Describe the issue

When using the DefaultCredentialsProvider with the intent to have WebIdentityTokenFileCredentialsProvider provide the credentials in a production kubernetes (EKS) environment without having the sts library on the classpath the DefaultCredentialsProvider silently fails over to the next CredentialsProvider in the chain.

The missing sts dependency leads to a ClassNotFoundException being rethrown as an IllegalStateException (in WebIdentityCredentialsUtils.factory()), but is immediately caught and assigned to a field. When the AwsCredentialsProviderChain later calls resolveCredentials on the WebIdentityTokenFileCredentialsProvider that exception is rethrown and immediately being caught again by AwsCredentialsProviderChain, moving to the next CredentialsProvider in the chain.

In my case the next successful provider in the chain was the instance profile of the underlying worker node, discovered via ec2 metadata service, not having the required policies attached to call the API in question - in this case s3 - leading to a 403 Forbidden, putting myself on a wild goose chase trying to find out why my (perfectly well configured) kubernetes service account doesn't seem to have the required permission to call s3.

In my opinion the ClassNotFoundException should probably not be suppressed, as when reaching that point it's rather obvious (given that two environment variables or properties with the correct names exist) that it's the developer's intention to use web identity and the missing library being a mistake of said developer rather than a intended fallback to another CredentialProvider.

Steps to Reproduce

import software.amazon.awssdk.services.s3.S3Client;
import software.amazon.awssdk.services.s3.model.GetObjectRequest;
import software.amazon.awssdk.regions.Region;

public class App {
    public static void main(String[] args) {
         var s3Client = S3Client.builder().region(Region.EU_WEST_1).build();
         var request = GetObjectRequest.builder().bucket("my-bucket").key("/my-key").build();
         s3Client.getObject(request); // Exception
    }
}

For simplicity's sake have a configured aws-cli profile (with no permissions to GetObject from my-bucket) lying in your home directory (~/.aws).
Run with s3 library (and transitional dependencies, but not sts) on classpath and both of these environment variables set to some string:

• AWS_WEB_IDENTITY_TOKEN_FILE
• AWS_ROLE_ARN

Your Environment

• AWS Java SDK version used: 2.13.41
• JDK version used: 11

[erikhh]

I just spent the best part of this day on this exact same issue. The experience was rather puzzling.

To debug I made the CLI tools print the IAM role it operates under and it showed up with the correct STS role. But my java app kept just insisting that it was under a different IAM role (it fell back to the default instance role silently).

I would be good to throw an Exception when AWS_WEB_IDENTITY_TOKEN_FILE and AWS_ROLE_ARN have (sensible?) values but the STS module is not on the classpath. You generally need to put in quite some effort to get those Environment variables filled, it's not something that happens by accident, not adding the STS module to the classpath is much easier to do by accident.

Or maybe better yet, just make sure the STS module is automatically added to the classpath by the modules that use it? Is there a specific reason to not add it to the classpath by default?

Anyhow, just my two cents.

[debora-ito]

@ascheja Apologies for the super long silence in here. This seems a reasonable ask, I'm marking as a feature request.

You probably know this, but you can you set the client to specifically use the WebIdentityTokenFileCredentialsProvider instead of relying in the credential chain.

[dagnir]

Unfortunately, it's difficult to infer intent just from the presence or absence of the environment variables or system properties; the situation described where the Web Identity provider failed so the chain moved on to the next one in the precedence list is the intended behavior of the chain.

In this case, since the SDK is expected to use the WebIdentityTokenFileCredentialsProviderr and nothing else, I would suggest that that provider be set explicitly on the client, so that any failures are thrown up to the caller instead of being swallowed as they are in the default credentials chain.

        S3Client s3 = S3Client.builder()
                .region(Region.EU_WEST_1)
                .credentialsProvider(WebIdentityTokenFileCredentialsProvider.create())
                .build();

        // will throw if WebIdentityTokenFileCredentialsProvider fails to initialize
        s3.listBuckets();

[ascheja]

It is true that this would fix the problem in a production environment, but for example when developing locally it's nice to have an automatic fallback to for example the cli profile.

What about adding at least a higher level log message (maybe warn or even error?) in case of the ClassNotFoundException? Due to how WebIdentityTokenFileCredentialsProvider is coded that higher log level message would only pop up in if either the properties or the environment variables exist, so it wouldn't even be logged for people with apparently no intent to use web identity credentials.

[dagnir]

What about adding at least a higher level log message (maybe warn or even error?) in case of the ClassNotFoundException?

Sure! I think this is reasonable. Would you like to submit a PR for this?

[ascheja]

Sure.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

[ascheja]

Thanks!