diff --git a/framework/test-vectors/README.md b/framework/test-vectors/README.md index 6a421b5b..f480d05c 100644 --- a/framework/test-vectors/README.md +++ b/framework/test-vectors/README.md @@ -97,7 +97,7 @@ looking at how test vectors are created helps reason about the edge cases of a c ## Drawbacks Not every configuration can be practically tested. -But we can expand on the test cases. +See [test vector enumeration](test-vector-enumeration.md#selecting-a-representative-input-value) for details. We will need to write minimal clients in every language with which we want to use these test vectors to understand the manifests described in subsequent features. diff --git a/framework/test-vectors/complete-vectors/encryption-context.md b/framework/test-vectors/complete-vectors/encryption-context.md index 901136b9..047e85ec 100644 --- a/framework/test-vectors/complete-vectors/encryption-context.md +++ b/framework/test-vectors/complete-vectors/encryption-context.md @@ -13,10 +13,13 @@ This is a description of the standard encryption contexts to test. ## Reference-level Explanation -### Standard Encryption Contexts - -MUST have an empty map. -MUST have a small map. -MUST have a large map. -MUST have multibyte UTF8 characters in both the key and value. -MUST have multibyte non-BMP characters in both the key and value. +### Standard Encryption Contexts Constraints + +- MUST have an empty map. + - The number of the items in the empty map MUST equal 0. +- MUST have a small map. + - The number of the items in the small map MUST be between 1 and 10. +- MUST have a large map. + - The number of the items in the large map MUST be greater than 10. +- MUST have multibyte UTF8 characters in both the key and value. +- MUST have multibyte non-BMP characters in both the key and value. diff --git a/framework/test-vectors/complete-vectors/hierarchy.md b/framework/test-vectors/complete-vectors/hierarchy.md index 75ff87ec..83785c5c 100644 --- a/framework/test-vectors/complete-vectors/hierarchy.md +++ b/framework/test-vectors/complete-vectors/hierarchy.md @@ -21,3 +21,59 @@ with every available [algorithm suite](../../algorithm-suites.md#algorithm-suite For every available algorithm suite and static branch key, a test MUST attempt to encrypt and decrypt with every [standard encryption context](./encryption-context.md#standard-encryption-contexts). + +### Input dimensions + +#### Encrypt + +- key description + - key: range of [representative branch keys](#representative-branch-keys) + +#### Decrypt + +- key description + - key: range of [representative branch keys](#representative-branch-keys) + - logicalKeyStore: + - "Default", + - Represents the logical key store name on encrypt + - "Other" + - Represents any other logical key store name + +### Representative branch keys + +- `"static-branch-key-1"` + - MUST be some valid branch key. +- `"static-branch-key-2"` + - MUST be some valid branch key other than `static-branch-key-1`. +- `"branch-key-no-permissions"` + - MUST be some valid branch key where the test vector runner does not have permissions for the KMS key. +- `"branch-key-not-in-table"` + - MUST be some branch key ID not present in the keystore table., +- `"branch-key-no-version"` + - MUST be some branch key that is in the table, but the configured version is not in the table. +- `"invalid-branch-key-material"` + - MUST be some branch key with illegally mutated invalid branch key material. + +### Evaluation rules + +- If encrypting key type is anything other than `"aws-kms-hierarchy"` + and decrypting key type is `"aws-kms-hierarchy"`, + the result MUST be `"negative-decrypt"`. +- If encrypting key type is `"aws-kms-hierarchy"` + and decrypting key type is anything other than `"aws-kms-hierarchy"`, + the result MUST be `"negative-decrypt"`. +- If the logical key store is "Other", + the result MUST be `"negative-decrypt"`. +- If `"key"` is different on encrypt and decrypt, + the result MUST be `"negative-decrypt"`. + (i.e. no key specified here is interoperable with any other key.) +- If `"key"` is any of: + - `"branch-key-no-permissions"` + - `"branch-key-not-in-table"` + - `"branch-key-no-version"` + - `"invalid-branch-key"` + (i.e. an "invalid" key with a particular invalid condition) + on either encrypt or decrypt, + the result MUST be either `"negative-encrypt"` or `"negative-decrypt"`, + depending on whether the invalid key was specified on encrypt or decrypt. +- In all other cases, the result MUST be `"positive"`. diff --git a/framework/test-vectors/complete-vectors/required-encryption-context-cmm.md b/framework/test-vectors/complete-vectors/required-encryption-context-cmm.md index ba48b3a4..3ed57aa5 100644 --- a/framework/test-vectors/complete-vectors/required-encryption-context-cmm.md +++ b/framework/test-vectors/complete-vectors/required-encryption-context-cmm.md @@ -77,4 +77,55 @@ For example: - Given the encryption context `{a:a, b:b}` with the `requiredEncryptionContextKeys` set to `{a}`, the only success case for a message to successfully decrypt will be - to supply the reproducedEncryptionContext `{a}`. + to supply the reproducedEncryptionContext `{a:a}`. + +## Input dimensions and ranges + +### Encrypt + +- cmm: Adds a `"RequiredEncryptionContext"` allowed value + - MUST add a `"RequiredEncryptionContext"` value to the `"cmm"` input dimension. +- required encryption context keys: Range is every [representative required encryption context key](#representative-required-encryption-context-keys) + - MUST test the full range of representative required encryption context keys. +- reproduced encryption context: Range is every [representative reproduced encryption context](#representative-reproduced-encryption-context) + - MUST test the full range of representative reproduced encryption context. + +### Decrypt + +These are the same as [Encrypt](#encrypt), but are specified separately so Duvet can link to unique lines for decrypt configuration. + +- cmm: Adds a `"RequiredEncryptionContext"` allowed value + - MUST add a `"RequiredEncryptionContext"` value to the `"cmm"` input dimension. +- required encryption context keys: Range is every [representative required encryption context key](#representative-required-encryption-context-keys) + - MUST test the full range of representative required encryption context keys. +- reproduced encryption context: Range is every [representative reproduced encryption context](#representative-reproduced-encryption-context) + - MUST test the full range of representative reproduced encryption context. + +### Representative values + +#### Representative required encryption context keys + +- Every subset of keys in the provided [encryption context](../../structures.md#encryption-context) + - MUST test every subset of keys in the provided encryption context. +- Any key NOT in the provided encryption context. + - MUST test with some additional key that is not in the provided encryption context. + +#### Representative reproduced encryption context + +- Every subset of items in the provided [encryption context](../../structures.md#encryption-context) + - MUST test every subset of items in the provided encryption context. +- Any item NOT in the provided encryption context. + - MUST test with some additional item that is not in the provided encryption context. + +## Test vector evaluation rules + +- If any of the `requiredEncryptionContextKeys` do not exist in the + supplied encryption context on encrypt + then the test result MUST be `negative-encrypt-keyring`. [source](#required-encryption-context-cmm-failures-on-encrypt) +- If the set of keys in `reproducedEncryptionContext` on decrypt + does not match the set of `requiredEncryptionContextKeys`, + then the test result MUST be `negative-decrypt-keyring`. [source] +- If the the value for any key in `reproducedEncryptionContext` on decrypt + does not match the value provided for that key on encrypt, + then the test result MUST be `negative-decrypt-keyring`. [source](#required-encryption-context-cmm-failures-on-decrypt) +- In all other cases, the test result MUST be `positive-keyring`. diff --git a/framework/test-vectors/esdk-test-vector-enumeration.md b/framework/test-vectors/esdk-test-vector-enumeration.md new file mode 100644 index 00000000..dbc89990 --- /dev/null +++ b/framework/test-vectors/esdk-test-vector-enumeration.md @@ -0,0 +1,115 @@ +[//]: # "Copyright Amazon.com Inc. or its affiliates. All Rights Reserved." +[//]: # "SPDX-License-Identifier: CC-BY-SA-4.0" + +# ESDK Test Vector Enumeration + +This document performs the [test vector enumeration](test-vector-enumeration.md) process for the ESDK +to construct the full suite of test vectors +for the ESDK. + +The full suite of test vectors can be constructed +by [enumerating all input configurations](test-vector-enumeration.md#enumerating-input-configurations) from this spec's input dimensions +and [evaluating each configuration's expected result](test-vector-enumeration.md#determining-expected-results) from this spec's evaluation rules. + +## Input dimensions + +- Every [MPL input dimension](mpl-test-vector-enumeration.md#input-dimensions) is an input dimension for ESDK. +- plaintext: Range of [representative plaintext values](#representative-plaintext-constraints) +- commitment policy: Range of allowed [commitment policies](../../client-apis/client.md#commitment-policy) +- frame size: Range of [representative frame sizes](#representative-frame-sizes) +- maximum encrypted data keys: Range of [representative number of maximum encrypted data keys](#representative-number-of-maximum-encrypted-data-keys-edks) + +## Evaluation rules + +- Every [MPL evaluation rule](mpl-test-vector-enumeration.md#evaluation-rules) is an evaluation rule for ESDK. + +## Representative values + +### Representative frame sizes + +- Non-framed data: `frame size = 0` + - The representative frame size value for non-framed data + MUST have frame size = 0. +- Small frame: `0 < frame size < 4096` + - The representative value for a small frame + MUST have frame size between 0 and 4096. +- Default frame: `frame size = 4096` + - The representative value for the default frame size + MUST have frame size = 4096. +- Large frame: `frame size > 4096` + - The representative value for a large frame + MUST have frame size greater than 4096. + +### Representative plaintext constraints + +#### Framed data + +These MUST only be used if `frame size > 0`. + +- Empty: `length = 0` + - The representative plaintext value for empty framed data + MUST have length = 0. +- Small: `0 < length < 10` + - The representative plaintext value for small framed data + MUST have length between 0 and 10. + - If `frame size < 10`, omit this. +- Medium: `10 ≤ length < 1000` + - The representative plaintext value for medium framed data + MUST have length of at least 10 and less than 1000. + - If `frame size < 1000`, omit this. +- Large: `1000 ≤ length <` [frame size](#representative-frame-sizes) + - The representative plaintext value for large framed data + MUST have length of at least 1000 and less than the configured frame size. +- Largest frame: `length = frame size` + - The representative plaintext value for the largest frame + MUST have length equal to the configured frame size. +- Largest frame + partial frame: `frame size < length < 2\*(frame size)` + - The representative plaintext value for a largest frame plus partial frame + MUST have length between the configured frame size + and twice the configured frame size. +- Two largest frames: `length = 2\*(frame size)` + - The representative plaintext value for two largest frames + MUST have length equal to twice the configured frame size. +- Many frames: `2*(frame size) < length < (frame size)\*(maximum # of frames)` + - The representative plaintext value for many frames + MUST have length between twice the configured frame size + and ((the configured maximum number of frames) times (the configured frame size)). +- Maximum frames: `length = (frame size)\*(maximum # of frames)` + - The representative plaintext value for maximum frames + MUST have length equal to the configured maximum number of frames times the configured frame size. + +#### Non-framed data + +These MUST only be used if `frame length = 0`. + +- Empty: `length = 0` + - The representative plaintext value for empty non-framed data + MUST have length equal to 0. +- Small: `0 < length < 10` + - The representative plaintext value for small non-framed data + MUST have length between 0 and 10. +- Medium: `10 ≤ length < 1000` + - The representative plaintext value for empty non-framed data + MUST have length of at least 10 and less than 1000. +- Large: `1000 < length ≤ 2^32` + - The representative plaintext value for empty non-framed data + MUST have length of at least 1000 and less than 2^32. + +### Representative number of maximum encrypted data keys (EDKs) + +- No configured value + - The representative value for no configured maxiumum number of EDKs + MUST be some unset value + that is interpreted as "no maximum value" by the ESDK implementation. +- Zero: `max EDKs = 0` + - The representative value for zero maximum EDKs + MUST have length = 0. +- One: `max EDKs = 1` + - The representative value for one maximum EDK + MUST have length = 1. +- Few: `1 < max EDKs < 10` + - The representative value for few maximum EDKs + MUST have length between 1 and 10. +- Many: `10 ≤ max EDKs` + - The representative value for many maximum EDKs + MUST have length of at least 10. diff --git a/framework/test-vectors/mpl-test-vector-enumeration.md b/framework/test-vectors/mpl-test-vector-enumeration.md new file mode 100644 index 00000000..86368b9a --- /dev/null +++ b/framework/test-vectors/mpl-test-vector-enumeration.md @@ -0,0 +1,24 @@ +[//]: # "Copyright Amazon.com Inc. or its affiliates. All Rights Reserved." +[//]: # "SPDX-License-Identifier: CC-BY-SA-4.0" + +# MPL Test Vector Enumeration + +This document performs the [test vector enumeration](test-vector-enumeration.md) process for the MPL +to construct the full suite of test vectors +for the MPL. + +The full suite of test vectors can be constructed +by [enumerating all input configurations](test-vector-enumeration.md#enumerating-input-configurations) from this spec's input dimensions +and [evaluating each configuration's expected result](test-vector-enumeration.md#determining-expected-results) from this spec's evaluation rules. + +## Input dimensions + +This section enumerates the [input dimensions](../test-vectors/test-vector-enumeration.md#input-dimensions) +for MPL test vectors: + +- algorithm suite ID: Range of supported [Algorithm IDs](../algorithm-suites.md#algorithm-suite-id) +- encryption context: Range of [representative encryption context values](./complete-vectors/encryption-context.md) +- required encryption context keys: Range of every [representative required encryption context key](#representative-required-encryption-context-keys) +- reproduced encryption context: Range is every [representative reproduced encryption context](#representative-reproduced-encryption-context) +- encrypt key description: Range of all [key descriptions](./key-description.md) used to request encrypt materials +- decrypt key description: Range of all [key descriptions](./key-description.md) used to decrypt diff --git a/framework/test-vectors/test-vector-enumeration.md b/framework/test-vectors/test-vector-enumeration.md new file mode 100644 index 00000000..630190c7 --- /dev/null +++ b/framework/test-vectors/test-vector-enumeration.md @@ -0,0 +1,284 @@ +[//]: # "Copyright Amazon.com Inc. or its affiliates. All Rights Reserved." +[//]: # "SPDX-License-Identifier: CC-BY-SA-4.0" + +# Test Vector Enumeration + +## Overview + +Test vectors are a suite of known value tests. +Successfully running test vectors asserts that +given some set of input configurations to encrypt and/or decrypt, +when these configurations are encrypted and/or decrypted, +then the expected result is achieved on encrypt and/or decrypt. + +This document outlines a framework +to enumerate a set of pairings +of input configurations to encryption and/or decryption +and expected results of encryption and/or decryption. + +The framework does not enumerate +every possible input configuration for two reasons: + +1. It is impractical to enumerate every possible input value + for some attributes in an input configuration + (see ["representative values"](#selecting-a-representative-input-value)). +2. Some input configurations with `negative` expected results + are not as critical of test cases as others + and are not a priority to test at this time + (see ["filtering input configurations"](#filtering-input-configurations)) + +These reasons are both motivated by test efficiency. +It is infeasible to enumerate every possible plaintext, +or test every invalid encryption configuration. +By strategically identifying test cases to enumerate, +the test vector framework employs orders of magnitude of leverage +to have comprehensive test coverage +with a concise suite of tests. + +As a result, +the framework does not actually enumerate +all possible input configurations and expected results. +However, by carefully reasoning about +the selection of representative values +and the filtered test cases, +the framework constructs a set of pairings +that is _representative_ of all _relevant_ +input configurations and expected results. + +## Input configuration + +An input configuration is a list of key-value pairs +where the key represents some [input dimension](#input-dimensions) +and the value is one allowed value in that input dimension. + +Input configurations for encrypt and decrypt SHOULD be specified separately. +In "positive" test cases, a given input dimension +will likely have the same value in the encrypt and decrypt configurations. +However, in "negative" test cases, a given input dimension +will likely have different values between encrypt and decrypt. +Specifying encrypt and decrypt configurations separately provides flexibility +and cleanly supports negative test cases. + +### Input dimensions + +An input dimension describes the range of possible values +an attribute of the [input configuration](#input-configuration) can take on. + +For example, the input dimension for `"algorithmSuiteId"` +in the ESDK can take on any of the [supported algorithm suite values](../algorithm-suites.md#supported-algorithm-suites-enum). + +The spec for an input dimension SHOULD define +its range of possible values +or how to construct its range of possible values. + +### Selecting a representative input value + +Not every input value in an [input dimension](#input-dimensions) can be practically tested, +as the number of allowed configurations as inputs to encryption is massive. +For example, the number of possible plaintexts that can be encrypted is massive; +it is impractical to enumerate and test all possible plaintexts. + +In cases where there are too many inputs to enumerate and test, +we instead test "representative" values. + +To select representative values, +we first partition the set of all possible inputs +into smaller sets, +where all members of a smaller set share some characteristic of interest. +Then, we choose a concrete "representative" value of the smaller set, +and test it in the test vectors. + +For example, we might partition the set of all possible plaintexts +based on the plaintext length. +Our hypothetical "smaller sets" might be: + +- Empty: length = 0 +- Small: all plaintexts where (1 < length ≤ 10) +- Medium: all plaintexts where (10 < length ≤ 1000) +- Large: all plaintexts where (1000 < length ≤ 2^32-1) +- Largest frame: all plaintexts where (length = 2^32-1) +- Largest frame + partial frame: all plaintexts where (length = 2^32-1 + [1 .. 2^32-1)) +- Two largest frames: all plaintexts where (length = 2\*(2^32-1)) +- Many frames: all plaintexts where (length = 2*(2^32-1) + [1 .. (2^32-1)*(2^32-3)]) + +(The motivation for partitioning this set based on lengths of 2^32-1 comes from the ESDK message [frame size](https://docs.aws.amazon.com/encryption-sdk/latest/developer-guide/message-format.html#body-framing).) + +From each of these smaller sets, we select one (or more) +concrete "representative" values that are tested in test vectors. +For example, a representative "small" plaintext could be `"abc"`. + +This framework of representative values +reduces any impractically large set of test inputs +to a practically testable size +while maintaining reasonable test coverage. +Its specification here documents reasoning for this framework +for readers or users of a test vector library +to understand the motivation for this concept. + +### Modifying representative values + +It is expected that a representative value could be modified +within its constraints +without loss in test coverage. + +For example, +given a representative small plaintext of `"abc"`, +if one were to change this to `"wxyz"`, +this should have no impact on test coverage. +This is still a small plaintext +according to the constraints on small plaintexts +and should be an allowed change. + +On the other hand, +given a representative small plaintext of `"abc"`, +if one were to change this to `""`, +this removes coverage for small plaintexts. +This violates the constraint for small plaintexts +and should not be an allowed change. + +The property that a representative value may be modified within its constraints +acts as an assertion that the representative value is not special, +and is truly "representative" of other values in its constrained set. + +### Filtering input configurations + +Input configurations may be "filtered" +and not be tested. + +This is a mechanism to identify test cases that +may not be as important as others +and avoid writing them to a test vector manifest. + +#### Motivation + +Some input configurations produced as a result of the enumeration above +will not be tested +as they are not important enough +to warrant the time spent testing them. + +For example, +consider all cases where a hierarchical keyring is encrypting, +but some raw AES keyring is decrypting. +This will have an [expected result](#expected-results) of `negative-decrypt`, +since a hierarchical keyring's encrypt result is incompatible with a Raw AES Keyring's decryption. +This may be worth validating once with test vectors to ensure this scenario +fails with the expected result. + +However, consider that the enumeration process above +would result in a large number of test scenarios +with an encrypting hierarchical keyring and a decrypting raw AES keyring, +but with many combinations of [input dimension](#input-dimensions) values. +(Consider that a scenario will be written with every combination of [representative plaintext length](esdk-test-vector-enumeration.md#representative-plaintext-constraints), +every [commitment policy](../../client-apis/client.md#commitment-policy), every [algorithm suite ID](../algorithm-suites.md#algorithm-suite-id), etc.) +In each case, the expected result is still `negative-decrypt`, +but the test manifest now has a large number of test scenarios +that fail for the same reason. + +Testing that hierarchical keyring-encrypted messages +cannot be decrypted by raw AES keyrings +across all possible input configurations +is simply not as important +as testing one of these input configurations. + +Another motivation for filtering test vectors +comes from their usage as a code testing tool. +Developers run test vectors against their code changes +to ensure their changes +do not break interoperability with other libraries. +These tests should finish in a reasonable amount of time, and +running unimportant tests wastes time. + +As a result of these considerations, +determining which enumerated input configurations +should be tested +becomes a matter of priorities +for a library developer. +This spec explicitly does not define +what an "important" test case is, +nor set a standard for how long test vectors should take to execute. +Test vector manifest generation library authors +should determine which input configurations are "important" +and determine a reasonable execution duration for these tests. + +#### Guidance + +Some guidance for filtering input configurations: + +- Prune invalid configurations as early as possible. + Input configurations can be filtered at any point in the vector enumeration process, + though it is beneficial to filter as early as possible + and prune large numbers of filtered configurations + to make input configuration enumeration as fast as possible. + For example, if the configuration under consideration + has a hierarchy keyring on encrypt and a raw AES keyring on decrypt, + the manifest generator might generate only one of these cases + (as a smoke test), + then skip considering all other configurations. + In contrast, a naïve approach would consider all of these other configurations + and deem them invalid. + This guidance suggests short-circuiting that evaluation as much as possible + to save time when generating vectors. + +### Enumerating input configurations + +One test vector input configuration can be constructed by selecting one possible value (or possible [representative value](test-vector-enumeration.md#selecting-a-representative-input-value)) from all relevant [input dimensions](test-vector-enumeration.md#input-dimensions). + +All test vector input configurations can be enumerated by constructing all unique input configurations. +If a test should be [filtered](#filtering-input-configurations), it should not be enumerated. + +## Expected results + +An expected result is a categorical value +and an optional error description. + +An expected result categorical value is one of: + +- Positive (successful test scenario) + - `"positive-keyring"` +- Negative encrypt (failure on encrypt) + - `"negative-encrypt-keyring"` +- Negative decrypt (failure on decrypt) + - `"negative-decrypt-keyring"` + +The error description for negative scenarios +describes the reason for the scenario's failure. + +### Determining expected results + +In contrast to [input configuration enumeration](#enumerating-input-configurations), +which is a constructive multiplicative process, +the expected result of a test +is deterministic from the input configuration. + +The expected result of an [input configuration](#input-configuration) can be determined +by evaluating all relevant [evaluation rules](#expected-result-evaluation-rules) for that input configuration. +If no evaluation rules fail, the test scenario result should be `"positive-keyring"`. +If one evaluation rule fails, the test scenario result should be the result of that evaluation rule. +If more than one evaluation rule fails with a different result +(i.e. input configuration can result in both `"negative-encrypt"` and `"negative-decrypt"`), +the test scenario result should be `"negative-encrypt"`. +(Encryption happens first; failure on encrypt will be encountered first.) + +#### Expected result evaluation rules + +An evaluation rule reads the input configuration [input configuration](#input-configuration), +and determines whether the configuration is valid +under the conditions the rule evaluates. + +For example, +the [required encryption context CMM component](complete-vectors/required-encryption-context-cmm.md#required-encryption-context-cmm-failures-on-encrypt) +defines the conditions for encryption to fail. +This can be interpreted as an evaluation rule, +and the test manifest generator should implement this +when generating test vectors using the required encryption context CMM +to determine when these vectors should fail. + +These rules can be relatively informal in the spec, +but ideally should be written programmatically, +so a developer can implement these rules in test manifest generation code. + +Evaluation rules are a useful abstraction to determine a given test scenario's expected result. +A list of independent rules is flexible and maintainable. +A list of rules that maintain the same interface +(reads input configuration; outputs success/fail) +makes evaluating these rules programmable.