Help wanted with DynamoDbItemEncryptorException: Configuration mismatch partition or sort key does not exist in item.

[motica1]

Hi,
I am observing an error software.amazon.cryptography.dbencryptionsdk.dynamodb.itemencryptor.model.DynamoDbItemEncryptorException: Configuration mismatch partition or sort key does not exist in item. during reading from a global secondary index. The query operation returns successfully, but during reading the returned response the above mentioned error occurs.

Here is everything that i verified:

• Table config includes plaintextOverride(PlaintextOverride.FORBID_PLAINTEXT_WRITE_ALLOW_PLAINTEXT_READ)
• Table description looks accurate

{
    "Table": {
        "AttributeDefinitions": [
            {
                "AttributeName": "pk",
                "AttributeType": "S"
            },
            {
                "AttributeName": "sk",
                "AttributeType": "S"
            },
            {
                "AttributeName": "gsi1_sk",
                "AttributeType": "S"
            },
            {
                "AttributeName": "gsi2_sk",
                "AttributeType": "S"
            }
        ],
        "TableName": "pii-details",
        "KeySchema": [
            {
                "AttributeName": "pk",
                "KeyType": "HASH"
            },
            {
                "AttributeName": "sk",
                "KeyType": "RANGE"
            }
        ],
        "TableStatus": "ACTIVE",
        "CreationDateTime": "2023-09-07T14:48:44.776000-04:00",
        "ProvisionedThroughput": {
            "LastIncreaseDateTime": "1969-12-31T19:00:00-05:00",
            "LastDecreaseDateTime": "1969-12-31T19:00:00-05:00",
            "NumberOfDecreasesToday": 0,
            "ReadCapacityUnits": 0,
            "WriteCapacityUnits": 0
        },
        "TableSizeBytes": 4524,
        "ItemCount": 4,
        "TableArn": "arn:aws:dynamodb:us-west-2:000000000000:table/pii-details",
        "TableId": "958b7ee0-df96-419d-a378-83fc8e70fca1",
        "BillingModeSummary": {
            "BillingMode": "PAY_PER_REQUEST",
            "LastUpdateToPayPerRequestDateTime": "2023-09-07T14:48:44.776000-04:00"
        },
        "GlobalSecondaryIndexes": [
            {
                "IndexName": "gsi1",
                "KeySchema": [
                    {
                        "AttributeName": "pk",
                        "KeyType": "HASH"
                    },
                    {
                        "AttributeName": "gsi1_sk",
                        "KeyType": "RANGE"
                    }
                ],
                "Projection": {
                    "ProjectionType": "INCLUDE",
                    "NonKeyAttributes": [
                        "identityId",
                        "createdAt",
                        "nickName"
                    ]
                },
                "IndexStatus": "ACTIVE",
                "ProvisionedThroughput": {
                    "ReadCapacityUnits": 0,
                    "WriteCapacityUnits": 0
                },
                "IndexSizeBytes": 4524,
                "ItemCount": 4,
                "IndexArn": "arn:aws:dynamodb:ddblocal:000000000000:table/pii-details/index/gsi1"
            },
            {
                "IndexName": "gsi2",
                "KeySchema": [
                    {
                        "AttributeName": "pk",
                        "KeyType": "HASH"
                    },
                    {
                        "AttributeName": "gsi2_sk",
                        "KeyType": "RANGE"
                    }
                ],
                "Projection": {
                    "ProjectionType": "INCLUDE",
                    "NonKeyAttributes": [
                        "identityId",
                        "createdAt",
                        "nickName"
                    ]
                },
                "IndexStatus": "ACTIVE",
                "ProvisionedThroughput": {
                    "ReadCapacityUnits": 0,
                    "WriteCapacityUnits": 0
                },
                "IndexSizeBytes": 4524,
                "ItemCount": 4,
                "IndexArn": "arn:aws:dynamodb:ddblocal:000000000000:table/pii-details/index/gsi2"
            }
        ],
        "Replicas": []
    }
}

• Items are correctly populated in the index, snippet below

"Items": [
        {
            "sk": {
                "S": "LATEST#1ff1b931-4a24-46ec-97ec-5ccd54f8ef2a"
            },
            "createdAt": {
                "S": "2023-09-07T18:51:13.634084084Z"
            },
            "pk": {
                "S": "<tenantId>"
            },
            "gsi1_sk": {
                "S": "LATEST#BENEFICIARY"
            },
            "identityId": {
                "S": "1ff1b931-4a24-46ec-97ec-5ccd54f8ef2a"
            },
            "nickName": {
                "S": "nickName"
            }
        },

• Using 2.20.86 version of AWS SDK (more context @DynamoDbSecondaryPartitionKey and @DynamoDbSecondarySortKey annotations have no effect on table.createTable() aws-sdk-java-v2#3923 (comment))
• All the setters for the attributes are public
• Also read this class where the error is thrown, but i am not sure what is the correct approach here.

Can you help me with some direction? Also it would be worth adding some examples for Query and Scan operations for this new encryption client. TIA :)

[lucasmcdonald3]

Hi @motica1,

This error would occur when an attribute configured as your partition key or sort key in your DynamoDbTableEncryptionConfig object is not present in the item returned by the query.
Could you provide the code you use to create your DynamoDbTableEncryptionConfig object? An example is here.
These should be set to the partition key and sort key for the table, not for the GSI.

With respect to Query and Scan examples:
Queries and scans with the AWS SDK for Java using the Database Encryption SDK for DynamoDB are performed the same way as you would without the DB-ESDK.
For examples using these actions, see this link.
Our examples primarily focus on how to configure the DB-ESDK in your code to work with the AWS SDK for Java's DynamoDB clients.
We do not explicitly maintain examples demonstrating Queries and Scans because, once you configure your DynamoDB client to use the DB-ESDK, you can perform these actions in the same way as described in AWS SDK for Java documentation.

That said: we have a handful of examples where we create QueryRequests and use those to query our table.
We create these in the same way as the examples for the AWS SDK for Java.
These queries exist in the context of examples for our searchable encryption feature since this feature slightly modifies Query behavior.
You can reference the construction and usage of these queries for your own implementation without setting up searchable encryption.
See here for a basic example, or here for a directory with more of these examples.

You are correct that we do not have examples demonstrating Scan operations.
You can reference the AWS SDK for Java's examples demonstrating Scan operations.

Please provide us with your DynamoDbTableEncryptionConfig config and let us know if we can help further.

Best,
Lucas

[motica1]

Thank you so much @lucasmcdonald3 for your detailed response. I understand that the config should define partition key and sort key accurately and this is where i think there might be an issue.

• I am using DynamoDbEnhancedTableEncryptionConfig , referencing the EnhancedPutGetExample. This is how i define my encryption config and it doesn't have an option to define partition and sort key explicitly in the config.

final Map<String, DynamoDbEnhancedTableEncryptionConfig> tableConfigs = new HashMap<>();
tableConfigs.put(
        DDB_TABLE_NAME,
        DynamoDbEnhancedTableEncryptionConfig.builder()
            .logicalTableName(DDB_TABLE_NAME)
            .keyring(kmsKeyRing)
            .allowedUnsignedAttributePrefix(UNSIGN_ATTR_PREFIX)
            .schemaOnEncrypt(TableSchema.fromBean(DetailsDynamoDbBean.class))
            .build());

• My schema is such that I share partition key from the base table with GSIs. So essentially i always use the same partition key, with different sort keys (as you would have probably seen from my describeTable output above). My hunch is that my DynamoDbBean class is either missing something or i can't use @DynamoDbPartitionKey and @DynamoDbSecondaryPartitionKey(indexNames = {"gsi1", "gsi2"}) on the same attribute. Here is how i define my schema.

@DynamoDbBean
@Data
public class DetailsDynamoDbBean {

  private String pk; // tenantId
  private String sk; // version#identityId
  private String gsi1_sk; // version#identityType
  private String gsi2_sk; // version#nickName
  private String identityId;
  private String createdAt;
  private String nickName;

  @DynamoDbPartitionKey
  @DynamoDbAttribute(value = "pk")
  @DynamoDbSecondaryPartitionKey(indexNames = {"gsi1", "gsi2"})
  public String getPk() {
    return pk;
  }

  @DynamoDbSortKey
  @DynamoDbAttribute(value = "sk")
  public String getSk() {
    return sk;
  }

  @DynamoDbAttribute(value = "gsi1_sk")
  @DynamoDbSecondarySortKey(indexNames = {"gsi1"})
  @DynamoDbEncryptionSignOnly
  public String getGsi1_sk() {
    return gsi1_sk;
  }

  @DynamoDbAttribute(value = "gsi2_sk")
  @DynamoDbSecondarySortKey(indexNames = {"gsi2"})
  @DynamoDbEncryptionSignOnly
  public String getGsi2_sk() {
    return gsi2_sk;
  }

  @DynamoDbAttribute(value = "identityId")
  @DynamoDbEncryptionSignOnly
  public String getIdentityId() {
    return identityId;
  }

  @DynamoDbAttribute(value = "createdAt")
  @DynamoDbEncryptionSignOnly
  public String getCreatedAt() {
    return createdAt;
  }

  @DynamoDbAttribute(value = "nickName")
  @DynamoDbEncryptionSignOnly
  public String getNickName() {
    return nickName;
  }

 
  public void setPk(String pk) {
    this.pk = pk;
  }

  public void setSk(String sk) {
    this.sk = sk;
  }

  public void setGsi1_sk(String gsi1_sk) {
    this.gsi1_sk = gsi1_sk;
  }

  public void setGsi2_sk(String gsi2_sk) {
    this.gsi2_sk = gsi2_sk;
  }

  public void setIdentityId(String identityId) {
    this.identityId = identityId;
  }

  public void setCreatedAt(String createdAt) {
    this.createdAt = createdAt;
  }

  public void setNickName(String nickName) {
    this.nickName = nickName;
  }
}

[lucasmcdonald3]

Hi @motica1,

Thank you for the information!

We released a new version of this library (3.1.0) today that improves error messaging in this situation.
This should help us debug your issue.
Could you update to the latest version (3.1.0) and let us know what error message you see?

Thank you,
Lucas

[motica1]

Thank you for the detailed error handling, i see more descriptive error now.
DynamoDbItemEncryptorException: On Decrypt : Partition key 'pk' does not exist in item. Sort key 'sk' does not exist in item. Item contains these attributes : createdAt identityId nickName.

This is strange since the index is getting correctly populated.

➜  ~ aws dynamodb scan --endpoint-url http://localhost:4566 --table-name pii-details --index-name gsi1

{
    "Items": [
        {
            "sk": {
                "S": "LATEST#6f80ac88-5de6-40d5-a5ae-b252f7430f3c"
            },
            "createdAt": {
                "S": "2023-09-12T12:38:45.360871881Z"
            },
            "pk": {
                "S": "<>"
            },
            "gsi1_sk": {
                "S": "LATEST#BENEFICIARY"
            },
            "identityId": {
                "S": "6f80ac88-5de6-40d5-a5ae-b252f7430f3c"
            },
            "nickName": {
                "S": "nickName"
            }
        },
        {
            "sk": {
                "S": "LATEST#33367aa4-4539-4086-99bb-5a1a89c9b82e"
            },
            "createdAt": {
                "S": "2023-09-12T12:38:43.557326672Z"
            },
            "pk": {
                "S": "<>"
            },
            "gsi1_sk": {
                "S": "LATEST#SENDER"
            },
            "identityId": {
                "S": "33367aa4-4539-4086-99bb-5a1a89c9b82e"
            },
            "nickName": {
                "S": "nickName"
            }
        },
        {
            "sk": {
                "S": "v1#6f80ac88-5de6-40d5-a5ae-b252f7430f3c"
            },
            "createdAt": {
                "S": "2023-09-12T12:38:44.969909548Z"
            },
            "pk": {
                "S": "<>"
            },
            "gsi1_sk": {
                "S": "v1#BENEFICIARY"
            },
            "identityId": {
                "S": "6f80ac88-5de6-40d5-a5ae-b252f7430f3c"
            },
            "nickName": {
                "S": "nickName"
            }
        },
        {
            "sk": {
                "S": "v1#33367aa4-4539-4086-99bb-5a1a89c9b82e"
            },
            "createdAt": {
                "S": "2023-09-12T12:38:40.726121129Z"
            },
            "pk": {
                "S": "<>"
            },
            "gsi1_sk": {
                "S": "v1#SENDER"
            },
            "identityId": {
                "S": "33367aa4-4539-4086-99bb-5a1a89c9b82e"
            },
            "nickName": {
                "S": "nickName"
            }
        }
    ],
    "Count": 4,
    "ScannedCount": 4,
    "ConsumedCapacity": null
}
(END)

[motica1]

I think i have to include the partition key and sort key in my attributesToProject. It worked for me. I was under the impression that table schema would be included by default, but that is the case when a GSI is created, not when querying the GSI it seems. Thank you for your assistance, the improved error message helped.