From e846ff5a73f7ed8f91e630612a82486bfbb8f3ed Mon Sep 17 00:00:00 2001 From: Lucas McDonald Date: Mon, 15 Apr 2024 14:57:34 -0700 Subject: [PATCH 01/19] changes --- framework/test-vectors/README.md | 2 +- .../complete-vectors/encryption-context.md | 8 +- .../esdk-test-vector-enumeration.md | 20 ++ .../mpl-test-vector-enumeration.md | 54 ++++++ .../test-vectors/test-vector-enumeration.md | 183 ++++++++++++++++++ 5 files changed, 265 insertions(+), 2 deletions(-) create mode 100644 framework/test-vectors/esdk-test-vector-enumeration.md create mode 100644 framework/test-vectors/mpl-test-vector-enumeration.md create mode 100644 framework/test-vectors/test-vector-enumeration.md diff --git a/framework/test-vectors/README.md b/framework/test-vectors/README.md index 6a421b5b..f480d05c 100644 --- a/framework/test-vectors/README.md +++ b/framework/test-vectors/README.md @@ -97,7 +97,7 @@ looking at how test vectors are created helps reason about the edge cases of a c ## Drawbacks Not every configuration can be practically tested. -But we can expand on the test cases. +See [test vector enumeration](test-vector-enumeration.md#selecting-a-representative-input-value) for details. We will need to write minimal clients in every language with which we want to use these test vectors to understand the manifests described in subsequent features. diff --git a/framework/test-vectors/complete-vectors/encryption-context.md b/framework/test-vectors/complete-vectors/encryption-context.md index 901136b9..d99d46b7 100644 --- a/framework/test-vectors/complete-vectors/encryption-context.md +++ b/framework/test-vectors/complete-vectors/encryption-context.md @@ -13,10 +13,16 @@ This is a description of the standard encryption contexts to test. ## Reference-level Explanation -### Standard Encryption Contexts +### Standard Encryption Contexts Constraints MUST have an empty map. +The number of the items in the empty map MUST equal 0. MUST have a small map. +The number of the items in the small map MUST be between 1 and 10. MUST have a large map. +The number of the items in the small map MUST be greater than 10. MUST have multibyte UTF8 characters in both the key and value. MUST have multibyte non-BMP characters in both the key and value. + +(Note: Constraints on representative values should go in the spec. +Concrete representative values probably do not need to go here.) \ No newline at end of file diff --git a/framework/test-vectors/esdk-test-vector-enumeration.md b/framework/test-vectors/esdk-test-vector-enumeration.md new file mode 100644 index 00000000..c5980a9e --- /dev/null +++ b/framework/test-vectors/esdk-test-vector-enumeration.md @@ -0,0 +1,20 @@ +[//]: # "Copyright Amazon.com Inc. or its affiliates. All Rights Reserved." +[//]: # "SPDX-License-Identifier: CC-BY-SA-4.0" + +# ESDK Test Vector Enumeration + +This document performs the [test vector enumeration](test-vector-enumeration.md) process for the ESDK +to construct the full suite of test vectors +for the ESDK. + +The full suite of test vectors can be constructed +by [enumerating all input configurations](test-vector-enumeration.md#enumerating-input-configurations) from this spec's input dimensions +and [evaluating each configuration's expected result](test-vector-enumeration.md#determining-expected-results) from this spec's evaluation rules. + +## Input dimensions + +* Every [MPL input dimension](mpl-test-vector-enumeration.md#input-dimensions) is an input dimension for ESDK. + +## Evaluation rules + +* Every [MPL evaluation rule](mpl-test-vector-enumeration.md#evaluation-rules) is an evaluation rule for ESDK. diff --git a/framework/test-vectors/mpl-test-vector-enumeration.md b/framework/test-vectors/mpl-test-vector-enumeration.md new file mode 100644 index 00000000..aac11dac --- /dev/null +++ b/framework/test-vectors/mpl-test-vector-enumeration.md @@ -0,0 +1,54 @@ +[//]: # "Copyright Amazon.com Inc. or its affiliates. All Rights Reserved." +[//]: # "SPDX-License-Identifier: CC-BY-SA-4.0" + +# ESDK Test Vector Enumeration + +This document performs the [test vector enumeration](test-vector-enumeration.md) process for the MPL +to construct the full suite of test vectors +for the MPL. + +The full suite of test vectors can be constructed +by [enumerating all input configurations](test-vector-enumeration.md#enumerating-input-configurations) from this spec's input dimensions +and [evaluating each configuration's expected result](test-vector-enumeration.md#determining-expected-results) from this spec's evaluation rules. + +## Input dimensions + +This section enumerates the [input dimensions](../test-vectors/test-vector-enumeration.md#input-dimensions) +for MPL test vectors: + +- algorithm suite ID: Range of supported [Algorithm IDs](../algorithm-suites.md#algorithm-suite-id) +- encryption context: Range of [representative encryption context values](./complete-vectors/encryption-context.md) +- required encryption context keys: (TODO: write rule for dtermining a representative required EC input range) +- reproduced encryption context: Permutations of encryption context and required encryption context keys as described in the [required encryption context CMM test vectors spec](./complete-vectors/required-encryption-context-cmm.md) +- plaintext: Range of [representative plaintext values](#representative-plaintext-constraints) +- encrypt key description: Range of all [key descriptions](./key-description.md) used to request encrypt materials +- decrypt key description: Range of all [key description](./key-description.md) used to decrypt + +TODO: maxplaintextlength +TODO: filtering? like, do we need to test all pairings of encrypt/decrypt key description? Lots of failed vectors... + +## Evaluation rules + +TODO: Link to MPL. A lot of these will be shared across the MPL. + +# TODO MOVEME: representative values + +## Representative plaintext constraints + +* Empty: length = 0 +* Small: all plaintexts where (1 < length ≤ 10) +* Medium: all plaintexts where (10 < length ≤ 1000) +* Large: all plaintexts where (1000 < length ≤ 2^32-1) +* Largest frame: all plaintexts where (length = 2^32-1) +* Largest frame + partial frame: all plaintexts where (length = 2^32-1 + [1 .. 2^32-1)) +* Two largest frames: all plaintexts where (length = 2*(2^32-1)) +* Many frames: all plaintexts where (length = 2*(2^32-1) + [1 .. (2^32-1)*(2^32-3)]) + +## Concrete representative plaintext values + +(Should these even go in the spec?) + +* Empty: "" +* Small: "abc" +* Medium: "abcdefg12345678910" +TODO diff --git a/framework/test-vectors/test-vector-enumeration.md b/framework/test-vectors/test-vector-enumeration.md new file mode 100644 index 00000000..988fab80 --- /dev/null +++ b/framework/test-vectors/test-vector-enumeration.md @@ -0,0 +1,183 @@ +[//]: # "Copyright Amazon.com Inc. or its affiliates. All Rights Reserved." +[//]: # "SPDX-License-Identifier: CC-BY-SA-4.0" + +# Test Vector Enumeration + +## Overview + +Test vectors are a suite of known value tests. +Successfully running test vectors asserts that +given some set of input configurations to encrypt and/or decrypt, +when these configurations are encrypted and/or decrypted, +then the expected result is achieved on encrypt and/or decrypt. + +This document outlines a framework +to enumerate a set of pairings +of input configurations to encryption and/or decryption +and expected results of encryption and/or decryption. +The framework constructs this set of pairings +to be representative of all possible +input configurations and expected results. +As a result, executing all enumerated pairings' input configurations +and receiving the pairing's expected result +is representative of all possible input and output configurations. + +## Input configuration + +An input configuration is a list of key-value pairs +where the key represents some [input dimension](#input-dimensions) +and the value is one allowed value in that input dimension. + +### Input dimensions + +An input dimension describes the range of possible values +an attribute of the [input configuration](#input-configuration) can take on. + +For example, the input dimension for `"algorithmSuiteId"` +in the ESDK can take on any of the [supported algorithm suite values](../algorithm-suites.md#supported-algorithm-suites-enum). + +The spec for an input dimension SHOULD define +its range of possible values +or how to construct its range of possible values. + +### Selecting a representative input value + +Not every input value in an [input dimension](#input-dimensions) can be practically tested, +as the number of allowed configurations as inputs to encryption is massive. +For example, the number of possible plaintexts that can be encrypted is massive; +it is impractical to enumerate and test all possible plaintexts. + +In cases where there are too many inputs to enumerate and test, +we instead test "representative" values. + +To select representative values, +we first partition the set of all possible inputs +into smaller sets, +where all members of a smaller set share some characteristic of interest. +Then, we choose a concrete "representative" value of the smaller set, +and test it in the test vectors. + +For example, we might partition the set of all possible plaintexts +based on the plaintext length. +Our hypothetical "smaller sets" might be: +* Empty: length = 0 +* Small: all plaintexts where (1 < length ≤ 10) +* Medium: all plaintexts where (10 < length ≤ 1000) +* Large: all plaintexts where (1000 < length ≤ 2^32-1) +* Largest frame: all plaintexts where (length = 2^32-1) +* Largest frame + partial frame: all plaintexts where (length = 2^32-1 + [1 .. 2^32-1)) +* Two largest frames: all plaintexts where (length = 2*(2^32-1)) +* Many frames: all plaintexts where (length = 2*(2^32-1) + [1 .. (2^32-1)*(2^32-3)]) + +(The motivation for partitioning this set based on lengths of 2^32-1 comes from the ESDK message [frame size](https://docs.aws.amazon.com/encryption-sdk/latest/developer-guide/message-format.html#body-framing).) + +From each of these smaller sets, we select one (or more) +concrete "representative" values that are tested in test vectors. +For example, a representative "small" plaintext could be `"abc"`. + +This framework of representative values +reduces any impractically large set of test inputs +to a practically testable size +while maintaining reasonable test coverage. + +### Modifying representative values + +It is expected that a representative value could be modified +within its constraints +without loss in test coverage. + +For example, +given a representative small plaintext of `"abc"`, +if one were to change this to `"wxyz"`, +this should have no impact on test coverage. +This is still a small plaintext +according to the constraints on small plaintexts +and should be an allowed change. + +On the other hand, +given a representative small plaintext of `"abc"`, +if one were to change this to `""`, +this removes coverage for small plaintexts. +This violates the constraint for small plaintexts +and should not be an allowed change. + +The property that a representative value may be modified within its constraints +acts as an assertion that the representative value is not special, +and is truly "representative" of other values in its constrained set. + +### Enumerating input configurations + +One test vector input configuration can be constructed by selecting one possible value (or possible [representative value](test-vector-enumeration.md#selecting-a-representative-input-value)) from all relevant [input dimensions](test-vector-enumeration.md#input-dimensions). + +All test vector input configurations can be enumerated by constructing all unique input configurations. + +## Expected results + +An expected result is a categorical value +and an optional error description. + +An expected result categorical value is one of: +* Positive (successful test scenario) + * `"positive-keyring"` +* Negative encrypt (failure on encrypt) + * `"negative-encrypt-keyring"` +* Negative decrypt (failure on decrypt) + * `"negative-decrypt-keyring"` + +The error description for negative scenarios +describes the reason for the scenario's failure. + +### Determining expected results + +In contrast to [input configuration enumeration](#enumerating-input-configurations), +which is a constructive multiplicative process, +the expected result of a test +is deterministic from the input configuration +and is a categorical result. + +The expected result of an [input configuration](#input-configuration) can be determined +by evaluating all relevant [evaluation rules](#expected-result-evaluation-rules) for that input configuration. +If no evaluation rules fail, the test scenario result should be `"positive-keyring"`. +If one evaluation rule fails, the test scenario result should be the result of that evaluation rule. +If multiple evaluation rules fail, the test scenario should not be written. See [below](#multiple-errors-todo). + +#### Expected result evaluation rules + +An evaluation rule reads the input configuration [input configuration](#input-configuration), +and determines whether the configuration is valid +under the conditions the rule evaluates. + +An evaluation rule should be as specific as possible. + +For example, +one evaluation rule might be that +every key in `"requiredEncryptionContextKeys"`, +MUST be present in the `"reproducedEncryptionContext"`, +otherwise the scenario results in `"negative-decrypt-keyring"`. + +these rules can be relatively informal in the spec, +but should be written somewhat programmatically, +so a developer can write these rules in code. + +for example, some failure rules: +* reproduced encryption context does not match encryption context for requried EC CMM => fail +* key does not match on decrypt => fail + +this is a good way to organize test vectors expected result. +* maintainable: essentially a list of these rules; easy to update/delete rules. +* every rule reads the input configuration and outputs a success/fail; evaluating rules can be programmatic + +### Multiple errors (TODO) + +An input configuration SHOULD only be deterministic of a single error. +If a single input configuration can result in multiple errors, +it SHOULD NOT be written. +For example, +if a test scenario specifies that `decrypt` provides incorrect `reproducedEncryptionContext` to a requried encryption context CMM +but also specifies a key that cannot decrypt the encrypted value, +that scenario should not be specified. + +(Is the above accurate? +Do we need some method to determine precedence of errors? +ex. if the decrypt key is wrong, but the reproduced EC is also wrong, +should that be allowed and we should just pick the first error?) \ No newline at end of file From d0b48cfbf673644d5541fb642910053b9715bed5 Mon Sep 17 00:00:00 2001 From: Lucas McDonald Date: Mon, 15 Apr 2024 16:22:31 -0700 Subject: [PATCH 02/19] cleanup --- .../esdk-test-vector-enumeration.md | 27 +++++++++++++++++-- .../mpl-test-vector-enumeration.md | 22 --------------- 2 files changed, 25 insertions(+), 24 deletions(-) diff --git a/framework/test-vectors/esdk-test-vector-enumeration.md b/framework/test-vectors/esdk-test-vector-enumeration.md index c5980a9e..5ff227b2 100644 --- a/framework/test-vectors/esdk-test-vector-enumeration.md +++ b/framework/test-vectors/esdk-test-vector-enumeration.md @@ -13,8 +13,31 @@ and [evaluating each configuration's expected result](test-vector-enumeration.md ## Input dimensions -* Every [MPL input dimension](mpl-test-vector-enumeration.md#input-dimensions) is an input dimension for ESDK. +- Every [MPL input dimension](mpl-test-vector-enumeration.md#input-dimensions) is an input dimension for ESDK. +- plaintext: Range of [representative plaintext values](#representative-plaintext-constraints) ## Evaluation rules -* Every [MPL evaluation rule](mpl-test-vector-enumeration.md#evaluation-rules) is an evaluation rule for ESDK. +- Every [MPL evaluation rule](mpl-test-vector-enumeration.md#evaluation-rules) is an evaluation rule for ESDK. + +# TODO MOVEME: representative values + +## Representative plaintext constraints + +* Empty: length = 0 +* Small: all plaintexts where (1 < length ≤ 10) +* Medium: all plaintexts where (10 < length ≤ 1000) +* Large: all plaintexts where (1000 < length ≤ 2^32-1) +* Largest frame: all plaintexts where (length = 2^32-1) +* Largest frame + partial frame: all plaintexts where (length = 2^32-1 + [1 .. 2^32-1)) +* Two largest frames: all plaintexts where (length = 2*(2^32-1)) +* Many frames: all plaintexts where (length = 2*(2^32-1) + [1 .. (2^32-1)*(2^32-3)]) + +## Concrete representative plaintext values + +(Should these even go in the spec?) + +* Empty: "" +* Small: "abc" +* Medium: "abcdefg12345678910" +TODO diff --git a/framework/test-vectors/mpl-test-vector-enumeration.md b/framework/test-vectors/mpl-test-vector-enumeration.md index aac11dac..0a49eb0f 100644 --- a/framework/test-vectors/mpl-test-vector-enumeration.md +++ b/framework/test-vectors/mpl-test-vector-enumeration.md @@ -20,7 +20,6 @@ for MPL test vectors: - encryption context: Range of [representative encryption context values](./complete-vectors/encryption-context.md) - required encryption context keys: (TODO: write rule for dtermining a representative required EC input range) - reproduced encryption context: Permutations of encryption context and required encryption context keys as described in the [required encryption context CMM test vectors spec](./complete-vectors/required-encryption-context-cmm.md) -- plaintext: Range of [representative plaintext values](#representative-plaintext-constraints) - encrypt key description: Range of all [key descriptions](./key-description.md) used to request encrypt materials - decrypt key description: Range of all [key description](./key-description.md) used to decrypt @@ -31,24 +30,3 @@ TODO: filtering? like, do we need to test all pairings of encrypt/decrypt key de TODO: Link to MPL. A lot of these will be shared across the MPL. -# TODO MOVEME: representative values - -## Representative plaintext constraints - -* Empty: length = 0 -* Small: all plaintexts where (1 < length ≤ 10) -* Medium: all plaintexts where (10 < length ≤ 1000) -* Large: all plaintexts where (1000 < length ≤ 2^32-1) -* Largest frame: all plaintexts where (length = 2^32-1) -* Largest frame + partial frame: all plaintexts where (length = 2^32-1 + [1 .. 2^32-1)) -* Two largest frames: all plaintexts where (length = 2*(2^32-1)) -* Many frames: all plaintexts where (length = 2*(2^32-1) + [1 .. (2^32-1)*(2^32-3)]) - -## Concrete representative plaintext values - -(Should these even go in the spec?) - -* Empty: "" -* Small: "abc" -* Medium: "abcdefg12345678910" -TODO From dbb9193490a31e6759d1ba8f33b6a801a67f5e67 Mon Sep 17 00:00:00 2001 From: Lucas McDonald Date: Tue, 16 Apr 2024 11:06:28 -0700 Subject: [PATCH 03/19] cleanup --- .../test-vectors/test-vector-enumeration.md | 31 +++++++------------ 1 file changed, 11 insertions(+), 20 deletions(-) diff --git a/framework/test-vectors/test-vector-enumeration.md b/framework/test-vectors/test-vector-enumeration.md index 988fab80..ddfcf3ed 100644 --- a/framework/test-vectors/test-vector-enumeration.md +++ b/framework/test-vectors/test-vector-enumeration.md @@ -132,8 +132,7 @@ describes the reason for the scenario's failure. In contrast to [input configuration enumeration](#enumerating-input-configurations), which is a constructive multiplicative process, the expected result of a test -is deterministic from the input configuration -and is a categorical result. +is deterministic from the input configuration. The expected result of an [input configuration](#input-configuration) can be determined by evaluating all relevant [evaluation rules](#expected-result-evaluation-rules) for that input configuration. @@ -147,25 +146,17 @@ An evaluation rule reads the input configuration [input configuration](#input-co and determines whether the configuration is valid under the conditions the rule evaluates. -An evaluation rule should be as specific as possible. - For example, -one evaluation rule might be that -every key in `"requiredEncryptionContextKeys"`, -MUST be present in the `"reproducedEncryptionContext"`, -otherwise the scenario results in `"negative-decrypt-keyring"`. - -these rules can be relatively informal in the spec, -but should be written somewhat programmatically, -so a developer can write these rules in code. - -for example, some failure rules: -* reproduced encryption context does not match encryption context for requried EC CMM => fail -* key does not match on decrypt => fail - -this is a good way to organize test vectors expected result. -* maintainable: essentially a list of these rules; easy to update/delete rules. -* every rule reads the input configuration and outputs a success/fail; evaluating rules can be programmatic +the [required encryption context CMM component](complete-vectors/required-encryption-context-cmm.md#required-encryption-context-cmm-failures-on-encrypt) +defines the conditions for encryption to fail. +This can be interpreted as an evaluation rule, +and the test manifest generator should implement this. + +These rules can be relatively informal in the spec, +but ideally should be written programmatically, +so a developer can write implement these rules in test manifest generation code. + +Evaluation rules are a useful abstraction to determine a given test scenario's expected result. A list of independent rules is flexible and maintainable. A list of rules that maintain the same interface (reads input configuration; outputs success/fail) makes evaluating these rules programmable. ### Multiple errors (TODO) From 2e8ce825bb4a90b05d1f8d3a1b748f69ad081580 Mon Sep 17 00:00:00 2001 From: Lucas McDonald Date: Tue, 16 Apr 2024 11:55:39 -0700 Subject: [PATCH 04/19] reccmm --- .../required-encryption-context-cmm.md | 18 +++++++++++++++++- .../test-vectors/test-vector-enumeration.md | 4 +++- 2 files changed, 20 insertions(+), 2 deletions(-) diff --git a/framework/test-vectors/complete-vectors/required-encryption-context-cmm.md b/framework/test-vectors/complete-vectors/required-encryption-context-cmm.md index ba48b3a4..f2f5019d 100644 --- a/framework/test-vectors/complete-vectors/required-encryption-context-cmm.md +++ b/framework/test-vectors/complete-vectors/required-encryption-context-cmm.md @@ -77,4 +77,20 @@ For example: - Given the encryption context `{a:a, b:b}` with the `requiredEncryptionContextKeys` set to `{a}`, the only success case for a message to successfully decrypt will be - to supply the reproducedEncryptionContext `{a}`. + to supply the reproducedEncryptionContext `{a:a}`. + +## Test vector input dimensions and ranges + +- required encryption context keys: Range is every subset of keys in a given [encryption context](../../structures.md#encryption-context). +- reproduced encryption context: Range is every subset of of the encryption context (TODO: plus representative nonsense values?) + +## Test vector evaluation rules + +- If any of the `requiredEncryptionContextKeys` do not exist in the +supplied encryption context, +then the test result MUST be `negative-encrypt-keyring`. [source](#required-encryption-context-cmm-failures-on-encrypt) +- If the `reproducedEncryptionContext` does not match +the supplied encryption context on encrypt +for every key in `requiredEncryptionContextKeys`, +then the test result MUST be `negative-decrypt-keyring`. [source](#required-encryption-context-cmm-failures-on-decrypt) +- In all other cases, the test result MUST be `positive-keyring`. \ No newline at end of file diff --git a/framework/test-vectors/test-vector-enumeration.md b/framework/test-vectors/test-vector-enumeration.md index ddfcf3ed..6d940b62 100644 --- a/framework/test-vectors/test-vector-enumeration.md +++ b/framework/test-vectors/test-vector-enumeration.md @@ -150,7 +150,9 @@ For example, the [required encryption context CMM component](complete-vectors/required-encryption-context-cmm.md#required-encryption-context-cmm-failures-on-encrypt) defines the conditions for encryption to fail. This can be interpreted as an evaluation rule, -and the test manifest generator should implement this. +and the test manifest generator should implement this +when generating test vectors using the required encryption context CMM +to determine when these vectors should fail. These rules can be relatively informal in the spec, but ideally should be written programmatically, From 80194d5ee4917ab02e13a57795e937838e923236 Mon Sep 17 00:00:00 2001 From: Lucas McDonald Date: Tue, 16 Apr 2024 12:05:43 -0700 Subject: [PATCH 05/19] reccmm --- .../complete-vectors/required-encryption-context-cmm.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/framework/test-vectors/complete-vectors/required-encryption-context-cmm.md b/framework/test-vectors/complete-vectors/required-encryption-context-cmm.md index f2f5019d..7010b598 100644 --- a/framework/test-vectors/complete-vectors/required-encryption-context-cmm.md +++ b/framework/test-vectors/complete-vectors/required-encryption-context-cmm.md @@ -87,10 +87,10 @@ For example: ## Test vector evaluation rules - If any of the `requiredEncryptionContextKeys` do not exist in the -supplied encryption context, +supplied encryption context on encrypt then the test result MUST be `negative-encrypt-keyring`. [source](#required-encryption-context-cmm-failures-on-encrypt) -- If the `reproducedEncryptionContext` does not match -the supplied encryption context on encrypt +- If the value of `reproducedEncryptionContext` on decrypt does not match +the value of the supplied encryption context on encrypt for every key in `requiredEncryptionContextKeys`, then the test result MUST be `negative-decrypt-keyring`. [source](#required-encryption-context-cmm-failures-on-decrypt) - In all other cases, the test result MUST be `positive-keyring`. \ No newline at end of file From a46455f8bd53b47f0738f1a9c0446b7251fa8bb3 Mon Sep 17 00:00:00 2001 From: Lucas McDonald Date: Thu, 18 Apr 2024 11:00:57 -0700 Subject: [PATCH 06/19] updates --- .../complete-vectors/encryption-context.md | 7 +- .../complete-vectors/hierarchy.md | 22 ++++ .../required-encryption-context-cmm.md | 17 ++- .../esdk-test-vector-enumeration.md | 11 +- .../mpl-test-vector-enumeration.md | 2 +- .../test-vectors/test-vector-enumeration.md | 113 ++++++++++++++---- 6 files changed, 132 insertions(+), 40 deletions(-) diff --git a/framework/test-vectors/complete-vectors/encryption-context.md b/framework/test-vectors/complete-vectors/encryption-context.md index d99d46b7..80220997 100644 --- a/framework/test-vectors/complete-vectors/encryption-context.md +++ b/framework/test-vectors/complete-vectors/encryption-context.md @@ -20,9 +20,6 @@ The number of the items in the empty map MUST equal 0. MUST have a small map. The number of the items in the small map MUST be between 1 and 10. MUST have a large map. -The number of the items in the small map MUST be greater than 10. +The number of the items in the large map MUST be greater than 10. MUST have multibyte UTF8 characters in both the key and value. -MUST have multibyte non-BMP characters in both the key and value. - -(Note: Constraints on representative values should go in the spec. -Concrete representative values probably do not need to go here.) \ No newline at end of file +MUST have multibyte non-BMP characters in both the key and value. \ No newline at end of file diff --git a/framework/test-vectors/complete-vectors/hierarchy.md b/framework/test-vectors/complete-vectors/hierarchy.md index 75ff87ec..6fb210e9 100644 --- a/framework/test-vectors/complete-vectors/hierarchy.md +++ b/framework/test-vectors/complete-vectors/hierarchy.md @@ -21,3 +21,25 @@ with every available [algorithm suite](../../algorithm-suites.md#algorithm-suite For every available algorithm suite and static branch key, a test MUST attempt to encrypt and decrypt with every [standard encryption context](./encryption-context.md#standard-encryption-contexts). + +### Input dimensions + +- encrypting key type: "aws-kms-hierarchy" +- decrypting key type: "aws-kms-hierarchy" +- key: [ + "static-branch-key-1", + "static-branch-key-2" +] + +### Evaluation rules + +- If encrypting key type is anything other than `"aws-kms-hierarchy"` +and decrypting key type is `"aws-kms-hierarchy"`, +the result should be `"negative-decrypt"`. +- If encrypting key type is `"aws-kms-hierarchy"` +and decrypting key type is anything other than `"aws-kms-hierarchy"`, +the result should be `"negative-decrypt"`. +- If encrypting and decrypting key type are both `"aws-kms-hierarchy"` +and the `"key"` is different on encrypt and decrypt, +the result should be `"negative-decrypt"`. +- In all other cases, the result should be `"positive"`. \ No newline at end of file diff --git a/framework/test-vectors/complete-vectors/required-encryption-context-cmm.md b/framework/test-vectors/complete-vectors/required-encryption-context-cmm.md index 7010b598..efd760ff 100644 --- a/framework/test-vectors/complete-vectors/required-encryption-context-cmm.md +++ b/framework/test-vectors/complete-vectors/required-encryption-context-cmm.md @@ -81,16 +81,25 @@ For example: ## Test vector input dimensions and ranges -- required encryption context keys: Range is every subset of keys in a given [encryption context](../../structures.md#encryption-context). +- required encryption context keys: Range is (TODO: plus representative nonsense values?) - reproduced encryption context: Range is every subset of of the encryption context (TODO: plus representative nonsense values?) +### Representative value constraints + +#### Required encryption context keys + +* Every subset of keys in the provided [encryption context](../../structures.md#encryption-context) +* Any value NOT in the provided encryption context. + ## Test vector evaluation rules - If any of the `requiredEncryptionContextKeys` do not exist in the supplied encryption context on encrypt then the test result MUST be `negative-encrypt-keyring`. [source](#required-encryption-context-cmm-failures-on-encrypt) -- If the value of `reproducedEncryptionContext` on decrypt does not match -the value of the supplied encryption context on encrypt -for every key in `requiredEncryptionContextKeys`, +- If the set of keys in `reproducedEncryptionContext` on decrypt +does not match the set of `requiredEncryptionContextKeys`, +then the test result MUST be `negative-decrypt-keyring`. [source] +- If the the value for any key in `reproducedEncryptionContext` on decrypt +does not match the value provided for that key on encrypt, then the test result MUST be `negative-decrypt-keyring`. [source](#required-encryption-context-cmm-failures-on-decrypt) - In all other cases, the test result MUST be `positive-keyring`. \ No newline at end of file diff --git a/framework/test-vectors/esdk-test-vector-enumeration.md b/framework/test-vectors/esdk-test-vector-enumeration.md index 5ff227b2..f60292e9 100644 --- a/framework/test-vectors/esdk-test-vector-enumeration.md +++ b/framework/test-vectors/esdk-test-vector-enumeration.md @@ -15,6 +15,8 @@ and [evaluating each configuration's expected result](test-vector-enumeration.md - Every [MPL input dimension](mpl-test-vector-enumeration.md#input-dimensions) is an input dimension for ESDK. - plaintext: Range of [representative plaintext values](#representative-plaintext-constraints) +- commitment policy: Range of allowed [commitment policies](../../client-apis/client.md#commitment-policy) +- frame size: Range of representative [frame sizes](TODO) ## Evaluation rules @@ -32,12 +34,3 @@ and [evaluating each configuration's expected result](test-vector-enumeration.md * Largest frame + partial frame: all plaintexts where (length = 2^32-1 + [1 .. 2^32-1)) * Two largest frames: all plaintexts where (length = 2*(2^32-1)) * Many frames: all plaintexts where (length = 2*(2^32-1) + [1 .. (2^32-1)*(2^32-3)]) - -## Concrete representative plaintext values - -(Should these even go in the spec?) - -* Empty: "" -* Small: "abc" -* Medium: "abcdefg12345678910" -TODO diff --git a/framework/test-vectors/mpl-test-vector-enumeration.md b/framework/test-vectors/mpl-test-vector-enumeration.md index 0a49eb0f..4810cb1e 100644 --- a/framework/test-vectors/mpl-test-vector-enumeration.md +++ b/framework/test-vectors/mpl-test-vector-enumeration.md @@ -18,7 +18,7 @@ for MPL test vectors: - algorithm suite ID: Range of supported [Algorithm IDs](../algorithm-suites.md#algorithm-suite-id) - encryption context: Range of [representative encryption context values](./complete-vectors/encryption-context.md) -- required encryption context keys: (TODO: write rule for dtermining a representative required EC input range) +- required encryption context keys: (TODO: write rule for determining a representative required EC input range; probably all possible subsets of EC?) - reproduced encryption context: Permutations of encryption context and required encryption context keys as described in the [required encryption context CMM test vectors spec](./complete-vectors/required-encryption-context-cmm.md) - encrypt key description: Range of all [key descriptions](./key-description.md) used to request encrypt materials - decrypt key description: Range of all [key description](./key-description.md) used to decrypt diff --git a/framework/test-vectors/test-vector-enumeration.md b/framework/test-vectors/test-vector-enumeration.md index 6d940b62..0e0f3fbb 100644 --- a/framework/test-vectors/test-vector-enumeration.md +++ b/framework/test-vectors/test-vector-enumeration.md @@ -15,12 +15,26 @@ This document outlines a framework to enumerate a set of pairings of input configurations to encryption and/or decryption and expected results of encryption and/or decryption. -The framework constructs this set of pairings -to be representative of all possible + +The framework does not enumerate +every possible input configuration for two reasons: +1. It is impractical to enumerate every possible input value +for some attributes in an input configuration +(see ["representative values"](#selecting-a-representative-input-value)). +2. Some input configurations with `negative` expected results +are not as critical of test cases as others +and are not a priority to test at this time +(see ["filtering input configurations"](#filtering-input-configurations)) + +As a result, +the framework does not actually enumerate +all possible input configurations and expected results. +However, by carefully reasoning about +the selection of representative values +and the filtered test cases, +the framework constructs a set of pairings +that is *representative* of all *relevant* input configurations and expected results. -As a result, executing all enumerated pairings' input configurations -and receiving the pairing's expected result -is representative of all possible input and output configurations. ## Input configuration @@ -79,6 +93,9 @@ This framework of representative values reduces any impractically large set of test inputs to a practically testable size while maintaining reasonable test coverage. +Its specification here documents reasoning for this framework +for readers or users of a test vector library +to understand the motivation for this concept. ### Modifying representative values @@ -105,11 +122,76 @@ The property that a representative value may be modified within its constraints acts as an assertion that the representative value is not special, and is truly "representative" of other values in its constrained set. +### Filtering input configurations + +Input configurations may be "filtered" +and not be tested. + +This is a mechanism to identify test cases that +may not be as important as others +and avoid writing them to a test vector manifest. + +Input configurations can be filtered at any point in the vector enumeration process, +though it is beneficial to filter as early as possible +to make enumeration as fast as possible. + +#### Motivation + +Some input configurations produced as a result of the enumeration above +will not be tested +as they are not important enough +to warrant the time spent testing them. + +For example, +consider all cases where a hierarchical keyring is encrypting, +but some raw AES key is decrypting. +This is expected have an [expected result](#expected-results) of `negative-decrypt`, +since a hierarchical keyring's encrypt result is incompatible with AES decryption. +This may be worth validating once with test vectors to ensure this scenario +fails with the expected result. + +However, consider that the enumeration process above +would result in a large number of test scenarios +with an encrypting hierarchical keyring and a decrypting raw AES key, +but with many combinations of [input dimension](#input-dimensions) values. +(Consider that a scenario will be written with every combination of [representative plaintext length](esdk-test-vector-enumeration.md#representative-plaintext-constraints), +every [commitment policy](../../client-apis/client.md#commitment-policy), every [algorithm suite ID](../algorithm-suites.md#algorithm-suite-id), etc.) +In each case, the expected result is still `negative-decrypt`, +but the test manifest now has a large number of test scenarios +that fail for the same reason. + +Testing that hierarchical keyring-encrypted messages +cannot be decrypted by raw AES keyrings +across all possible input configurations +is simply not as important +as testing one of these input configurations. + +Another motivation for filtering test vectors +comes from their usage as a code testing tool. +Developers run test vectors against their code changes +to ensure their changes +do not break interoperability with other libraries. +These tests should finish in a reasonable amount of time, and +running unimportant tests wastes time. + +As a result of these considerations, +determining which enumerated input configurations +should be tested +becomes a matter of priorities +for a library developer. +This spec explicitly does not define +what an "important" test case is, +nor set a standard for how long test vectors should take to execute. +Test vector manifest generation library authors +should determine which input configurations are "important" +and determine a reasonable execution duration for these tests. + ### Enumerating input configurations One test vector input configuration can be constructed by selecting one possible value (or possible [representative value](test-vector-enumeration.md#selecting-a-representative-input-value)) from all relevant [input dimensions](test-vector-enumeration.md#input-dimensions). All test vector input configurations can be enumerated by constructing all unique input configurations. +If a test should be [filtered](#filtering-input-configurations), it should not be enumerated. ## Expected results @@ -158,19 +240,8 @@ These rules can be relatively informal in the spec, but ideally should be written programmatically, so a developer can write implement these rules in test manifest generation code. -Evaluation rules are a useful abstraction to determine a given test scenario's expected result. A list of independent rules is flexible and maintainable. A list of rules that maintain the same interface (reads input configuration; outputs success/fail) makes evaluating these rules programmable. - -### Multiple errors (TODO) - -An input configuration SHOULD only be deterministic of a single error. -If a single input configuration can result in multiple errors, -it SHOULD NOT be written. -For example, -if a test scenario specifies that `decrypt` provides incorrect `reproducedEncryptionContext` to a requried encryption context CMM -but also specifies a key that cannot decrypt the encrypted value, -that scenario should not be specified. - -(Is the above accurate? -Do we need some method to determine precedence of errors? -ex. if the decrypt key is wrong, but the reproduced EC is also wrong, -should that be allowed and we should just pick the first error?) \ No newline at end of file +Evaluation rules are a useful abstraction to determine a given test scenario's expected result. +A list of independent rules is flexible and maintainable. +A list of rules that maintain the same interface +(reads input configuration; outputs success/fail) +makes evaluating these rules programmable. From 9f14ca245e00e646c6665181d80267dc8cfac4bd Mon Sep 17 00:00:00 2001 From: Lucas McDonald Date: Thu, 18 Apr 2024 11:14:45 -0700 Subject: [PATCH 07/19] negative hierarchy --- .../complete-vectors/hierarchy.md | 50 ++++++++++++++++--- 1 file changed, 44 insertions(+), 6 deletions(-) diff --git a/framework/test-vectors/complete-vectors/hierarchy.md b/framework/test-vectors/complete-vectors/hierarchy.md index 6fb210e9..20c702f8 100644 --- a/framework/test-vectors/complete-vectors/hierarchy.md +++ b/framework/test-vectors/complete-vectors/hierarchy.md @@ -24,11 +24,32 @@ a test MUST attempt to encrypt and decrypt with every [standard encryption conte ### Input dimensions -- encrypting key type: "aws-kms-hierarchy" -- decrypting key type: "aws-kms-hierarchy" +- encrypt key description: + { + type: "aws-kms-hierarchy" + key: range of [representative branch keys](#representative-branch-keys) + } +- decrypt key description: + { + type: "aws-kms-hierarchy" + key: range of [representative branch keys](#representative-branch-keys) + logicalKeyStore: [ + "Default", + "Other", # any other logical key store name + ] + } + +### Representative branch keys - key: [ - "static-branch-key-1", - "static-branch-key-2" + "static-branch-key-1", # any valid branch key + "static-branch-key-2", # any other valid branch key + "branch-key-no-permissions", # any valid branch key where + the test vector runner does not have permissions + for the KMS key + "branch-key-not-in-table", # any branch key ID not + in the keystore table + "branch-key-no-version", # any branch key without a version + "invalid-branch-key", # any illegally mutated invalid branch key ] ### Evaluation rules @@ -39,7 +60,24 @@ the result should be `"negative-decrypt"`. - If encrypting key type is `"aws-kms-hierarchy"` and decrypting key type is anything other than `"aws-kms-hierarchy"`, the result should be `"negative-decrypt"`. -- If encrypting and decrypting key type are both `"aws-kms-hierarchy"` -and the `"key"` is different on encrypt and decrypt, +- If the logical key store is "Other", +the result should be `"negative-decrypt"`. +- If `"key"` is different on encrypt and decrypt, +the result should be `"negative-decrypt"`. +- If `"key"` is `"branch-key-no-permissions"` on encrypt, +the result should be `"negative-encrypt"`. +- If `"key"` is `"branch-key-no-permissions"` on decrypt, +the result should be `"negative-decrypt"`. +- If `"key"` is `"branch-key-not-in-table"` on encrypt, +the result should be `"negative-encrypt"`. +- If `"key"` is `"branch-key-not-in-table"` on decrypt, +the result should be `"negative-decrypt"`. +- If `"key"` is `"branch-key-no-version"` on encrypt, +the result should be `"negative-encrypt"`. +- If `"key"` is `"branch-key-no-version"` on decrypt, +the result should be `"negative-decrypt"`. +- If `"key"` is `"invalid-branch-key"` on encrypt, +the result should be `"negative-encrypt"`. +- If `"key"` is `"invalid-branch-key"` on decrypt, the result should be `"negative-decrypt"`. - In all other cases, the result should be `"positive"`. \ No newline at end of file From da9cd4078d88ecf00f3a0980bd38b1d3da163dbc Mon Sep 17 00:00:00 2001 From: Lucas McDonald Date: Thu, 18 Apr 2024 11:30:50 -0700 Subject: [PATCH 08/19] guidance --- .../complete-vectors/hierarchy.md | 26 +++++++------------ .../test-vectors/test-vector-enumeration.md | 23 +++++++++++++--- 2 files changed, 29 insertions(+), 20 deletions(-) diff --git a/framework/test-vectors/complete-vectors/hierarchy.md b/framework/test-vectors/complete-vectors/hierarchy.md index 20c702f8..e54c37be 100644 --- a/framework/test-vectors/complete-vectors/hierarchy.md +++ b/framework/test-vectors/complete-vectors/hierarchy.md @@ -64,20 +64,14 @@ the result should be `"negative-decrypt"`. the result should be `"negative-decrypt"`. - If `"key"` is different on encrypt and decrypt, the result should be `"negative-decrypt"`. -- If `"key"` is `"branch-key-no-permissions"` on encrypt, -the result should be `"negative-encrypt"`. -- If `"key"` is `"branch-key-no-permissions"` on decrypt, -the result should be `"negative-decrypt"`. -- If `"key"` is `"branch-key-not-in-table"` on encrypt, -the result should be `"negative-encrypt"`. -- If `"key"` is `"branch-key-not-in-table"` on decrypt, -the result should be `"negative-decrypt"`. -- If `"key"` is `"branch-key-no-version"` on encrypt, -the result should be `"negative-encrypt"`. -- If `"key"` is `"branch-key-no-version"` on decrypt, -the result should be `"negative-decrypt"`. -- If `"key"` is `"invalid-branch-key"` on encrypt, -the result should be `"negative-encrypt"`. -- If `"key"` is `"invalid-branch-key"` on decrypt, -the result should be `"negative-decrypt"`. +(i.e. no key specified here is interoperable with any other key.) +- If `"key"` is any of: + - `"branch-key-no-permissions"` + - `"branch-key-not-in-table"` + - `"branch-key-no-version"` + - `"invalid-branch-key"` +(i.e. an "invalid" key with a particular invalid condition) +on either encrypt or decrypt, +the result should be either `"negative-encrypt"` or `"negative-decrypt"`, +depending on whether the invalid key was specified on encrypt or decrypt. - In all other cases, the result should be `"positive"`. \ No newline at end of file diff --git a/framework/test-vectors/test-vector-enumeration.md b/framework/test-vectors/test-vector-enumeration.md index 0e0f3fbb..2a8a6b8b 100644 --- a/framework/test-vectors/test-vector-enumeration.md +++ b/framework/test-vectors/test-vector-enumeration.md @@ -131,10 +131,6 @@ This is a mechanism to identify test cases that may not be as important as others and avoid writing them to a test vector manifest. -Input configurations can be filtered at any point in the vector enumeration process, -though it is beneficial to filter as early as possible -to make enumeration as fast as possible. - #### Motivation Some input configurations produced as a result of the enumeration above @@ -186,6 +182,25 @@ Test vector manifest generation library authors should determine which input configurations are "important" and determine a reasonable execution duration for these tests. +#### Guidance + +Some guidance for filtering input configurations: + +* Prune invalid configurations as early as possible. +Input configurations can be filtered at any point in the vector enumeration process, +though it is beneficial to filter as early as possible +and prune large numbers of filtered configurations +to make input configuration enumeration as fast as possible. +For example, if the configuration under consideration +has a hierarchy keyring on encrypt and a raw AES keyring on decrypt, +the manifest generator might generate only one of these cases +(as a smoke test), +then skip considering all other configurations. +In contrast, a naïve approach would consider all of these other configurations +and deem them invalid. +This guidance suggests short-circuiting that evaluation as much as possible +to save time when generating vectors. + ### Enumerating input configurations One test vector input configuration can be constructed by selecting one possible value (or possible [representative value](test-vector-enumeration.md#selecting-a-representative-input-value)) from all relevant [input dimensions](test-vector-enumeration.md#input-dimensions). From 167f64908d7d48292bdbb4fdc68a2286df454689 Mon Sep 17 00:00:00 2001 From: Lucas McDonald Date: Thu, 18 Apr 2024 11:35:13 -0700 Subject: [PATCH 09/19] fix --- .../complete-vectors/hierarchy.md | 26 +++++++++---------- 1 file changed, 13 insertions(+), 13 deletions(-) diff --git a/framework/test-vectors/complete-vectors/hierarchy.md b/framework/test-vectors/complete-vectors/hierarchy.md index e54c37be..e5dd90f9 100644 --- a/framework/test-vectors/complete-vectors/hierarchy.md +++ b/framework/test-vectors/complete-vectors/hierarchy.md @@ -24,12 +24,13 @@ a test MUST attempt to encrypt and decrypt with every [standard encryption conte ### Input dimensions -- encrypt key description: +``` +encrypt key description: { type: "aws-kms-hierarchy" key: range of [representative branch keys](#representative-branch-keys) } -- decrypt key description: +decrypt key description: { type: "aws-kms-hierarchy" key: range of [representative branch keys](#representative-branch-keys) @@ -38,19 +39,18 @@ a test MUST attempt to encrypt and decrypt with every [standard encryption conte "Other", # any other logical key store name ] } +``` ### Representative branch keys -- key: [ - "static-branch-key-1", # any valid branch key - "static-branch-key-2", # any other valid branch key - "branch-key-no-permissions", # any valid branch key where - the test vector runner does not have permissions - for the KMS key - "branch-key-not-in-table", # any branch key ID not - in the keystore table - "branch-key-no-version", # any branch key without a version - "invalid-branch-key", # any illegally mutated invalid branch key -] +* "static-branch-key-1", # any valid branch key +* "static-branch-key-2", # any other valid branch key +* "branch-key-no-permissions", # any valid branch key where +the test vector runner does not have permissions +for the KMS key +* "branch-key-not-in-table", # any branch key ID not +in the keystore table +* "branch-key-no-version", # any branch key without a version +* "invalid-branch-key", # any illegally mutated invalid branch key ### Evaluation rules From a93a6ba7ebb8ea8bc7959ef64a70f1d4daf3dd02 Mon Sep 17 00:00:00 2001 From: Lucas McDonald Date: Thu, 18 Apr 2024 11:36:34 -0700 Subject: [PATCH 10/19] fix --- .../complete-vectors/hierarchy.md | 19 ++++++++++++------- 1 file changed, 12 insertions(+), 7 deletions(-) diff --git a/framework/test-vectors/complete-vectors/hierarchy.md b/framework/test-vectors/complete-vectors/hierarchy.md index e5dd90f9..5e1c3ada 100644 --- a/framework/test-vectors/complete-vectors/hierarchy.md +++ b/framework/test-vectors/complete-vectors/hierarchy.md @@ -42,15 +42,20 @@ decrypt key description: ``` ### Representative branch keys -* "static-branch-key-1", # any valid branch key -* "static-branch-key-2", # any other valid branch key -* "branch-key-no-permissions", # any valid branch key where +* `"static-branch-key-1"` + * Any valid branch key. +* "static-branch-key-2" + * Any other valid branch key +* "branch-key-no-permissions" + * Any valid branch key where the test vector runner does not have permissions for the KMS key -* "branch-key-not-in-table", # any branch key ID not -in the keystore table -* "branch-key-no-version", # any branch key without a version -* "invalid-branch-key", # any illegally mutated invalid branch key +* "branch-key-not-in-table" + * Any branch key ID not in the keystore table +* "branch-key-no-version" + * Any branch key without a version +* "invalid-branch-key" + * Any illegally mutated invalid branch key ### Evaluation rules From 63daa0bae69f4b080d09667e8d4c2c64b452dedf Mon Sep 17 00:00:00 2001 From: Lucas McDonald Date: Thu, 18 Apr 2024 13:24:00 -0700 Subject: [PATCH 11/19] add --- .../complete-vectors/encryption-context.md | 16 ++++---- .../complete-vectors/hierarchy.md | 10 ++--- .../required-encryption-context-cmm.md | 15 ++++--- .../esdk-test-vector-enumeration.md | 41 +++++++++++++++---- .../mpl-test-vector-enumeration.md | 14 ++----- .../test-vectors/test-vector-enumeration.md | 5 ++- 6 files changed, 62 insertions(+), 39 deletions(-) diff --git a/framework/test-vectors/complete-vectors/encryption-context.md b/framework/test-vectors/complete-vectors/encryption-context.md index 80220997..e3f70d24 100644 --- a/framework/test-vectors/complete-vectors/encryption-context.md +++ b/framework/test-vectors/complete-vectors/encryption-context.md @@ -15,11 +15,11 @@ This is a description of the standard encryption contexts to test. ### Standard Encryption Contexts Constraints -MUST have an empty map. -The number of the items in the empty map MUST equal 0. -MUST have a small map. -The number of the items in the small map MUST be between 1 and 10. -MUST have a large map. -The number of the items in the large map MUST be greater than 10. -MUST have multibyte UTF8 characters in both the key and value. -MUST have multibyte non-BMP characters in both the key and value. \ No newline at end of file +* MUST have an empty map. + * The number of the items in the empty map MUST equal 0. +* MUST have a small map. + * The number of the items in the small map MUST be between 1 and 10. +* MUST have a large map. + * The number of the items in the large map MUST be greater than 10. +* MUST have multibyte UTF8 characters in both the key and value. +* MUST have multibyte non-BMP characters in both the key and value. \ No newline at end of file diff --git a/framework/test-vectors/complete-vectors/hierarchy.md b/framework/test-vectors/complete-vectors/hierarchy.md index 5e1c3ada..476b1dc6 100644 --- a/framework/test-vectors/complete-vectors/hierarchy.md +++ b/framework/test-vectors/complete-vectors/hierarchy.md @@ -44,17 +44,17 @@ decrypt key description: ### Representative branch keys * `"static-branch-key-1"` * Any valid branch key. -* "static-branch-key-2" +* `"static-branch-key-2"` * Any other valid branch key -* "branch-key-no-permissions" +* `"branch-key-no-permissions"` * Any valid branch key where the test vector runner does not have permissions for the KMS key -* "branch-key-not-in-table" +* `"branch-key-not-in-table"` * Any branch key ID not in the keystore table -* "branch-key-no-version" +* `"branch-key-no-version"` * Any branch key without a version -* "invalid-branch-key" +* `"invalid-branch-key"` * Any illegally mutated invalid branch key ### Evaluation rules diff --git a/framework/test-vectors/complete-vectors/required-encryption-context-cmm.md b/framework/test-vectors/complete-vectors/required-encryption-context-cmm.md index efd760ff..86340120 100644 --- a/framework/test-vectors/complete-vectors/required-encryption-context-cmm.md +++ b/framework/test-vectors/complete-vectors/required-encryption-context-cmm.md @@ -81,15 +81,20 @@ For example: ## Test vector input dimensions and ranges -- required encryption context keys: Range is (TODO: plus representative nonsense values?) -- reproduced encryption context: Range is every subset of of the encryption context (TODO: plus representative nonsense values?) +- required encryption context keys: Range is every [representative required encryption context key](#representative-required-encryption-context-keys) +- reproduced encryption context: Range is every [representative reproduced encryption context](#representative-reproduced-encryption-context) -### Representative value constraints +### Representative values -#### Required encryption context keys +#### Representative required encryption context keys * Every subset of keys in the provided [encryption context](../../structures.md#encryption-context) -* Any value NOT in the provided encryption context. +* Any key NOT in the provided encryption context. + +#### Representative reproduced encryption context + +* Every subset of items in the provided [encryption context](../../structures.md#encryption-context) +* Any item NOT in the provided encryption context. ## Test vector evaluation rules diff --git a/framework/test-vectors/esdk-test-vector-enumeration.md b/framework/test-vectors/esdk-test-vector-enumeration.md index f60292e9..84080966 100644 --- a/framework/test-vectors/esdk-test-vector-enumeration.md +++ b/framework/test-vectors/esdk-test-vector-enumeration.md @@ -16,21 +16,44 @@ and [evaluating each configuration's expected result](test-vector-enumeration.md - Every [MPL input dimension](mpl-test-vector-enumeration.md#input-dimensions) is an input dimension for ESDK. - plaintext: Range of [representative plaintext values](#representative-plaintext-constraints) - commitment policy: Range of allowed [commitment policies](../../client-apis/client.md#commitment-policy) -- frame size: Range of representative [frame sizes](TODO) +- frame size: Range of [representative frame sizes](#representative-frame-sizes) ## Evaluation rules - Every [MPL evaluation rule](mpl-test-vector-enumeration.md#evaluation-rules) is an evaluation rule for ESDK. -# TODO MOVEME: representative values +## Representative values -## Representative plaintext constraints +### Representative frame sizes -* Empty: length = 0 +* Non-framed data: `length = 0` +* Small frame: `0 < length < 4096` +* Default frame: `length = 4096` +* Large frame: `length > 4096` + +### Representative plaintext constraints + +#### Framed data + +These should ONLY be used if `frame length > 0`. + +* Empty: `length = 0` +* Small: all plaintexts where (1 < length ≤ 10) + * If `frame size < 10`, omit this. +* Medium: all plaintexts where (10 < length ≤ 1000) + * If `frame size < 1000`, omit this. +* Large: all plaintexts where (1000 < length ≤ [frame size](#representative-frame-sizes)) +* Largest frame: all plaintexts where (length = frame size) +* Largest frame + partial frame: all plaintexts where (length = frame size + [1 .. frame size)) +* Two largest frames: all plaintexts where (length = 2*(frame size)) +* Many frames: all plaintexts where (length = 2*(frame size) + [1 .. (frame size)*(maximum # of frames-2)]) +* Maximum frames: all plaintexts where (length = (frame size)*(maximum # of frames)) + +#### Non-framed data + +These should ONLY be used if `frame length = 0`. + +* Empty: `length = 0` * Small: all plaintexts where (1 < length ≤ 10) * Medium: all plaintexts where (10 < length ≤ 1000) -* Large: all plaintexts where (1000 < length ≤ 2^32-1) -* Largest frame: all plaintexts where (length = 2^32-1) -* Largest frame + partial frame: all plaintexts where (length = 2^32-1 + [1 .. 2^32-1)) -* Two largest frames: all plaintexts where (length = 2*(2^32-1)) -* Many frames: all plaintexts where (length = 2*(2^32-1) + [1 .. (2^32-1)*(2^32-3)]) +* Large: all plaintexts where (1000 < length ≤ 2^32) \ No newline at end of file diff --git a/framework/test-vectors/mpl-test-vector-enumeration.md b/framework/test-vectors/mpl-test-vector-enumeration.md index 4810cb1e..e5b95bb6 100644 --- a/framework/test-vectors/mpl-test-vector-enumeration.md +++ b/framework/test-vectors/mpl-test-vector-enumeration.md @@ -18,15 +18,7 @@ for MPL test vectors: - algorithm suite ID: Range of supported [Algorithm IDs](../algorithm-suites.md#algorithm-suite-id) - encryption context: Range of [representative encryption context values](./complete-vectors/encryption-context.md) -- required encryption context keys: (TODO: write rule for determining a representative required EC input range; probably all possible subsets of EC?) -- reproduced encryption context: Permutations of encryption context and required encryption context keys as described in the [required encryption context CMM test vectors spec](./complete-vectors/required-encryption-context-cmm.md) +- required encryption context keys: Range of every [representative required encryption context key](#representative-required-encryption-context-keys) +- reproduced encryption context: Range is every [representative reproduced encryption context](#representative-reproduced-encryption-context) - encrypt key description: Range of all [key descriptions](./key-description.md) used to request encrypt materials -- decrypt key description: Range of all [key description](./key-description.md) used to decrypt - -TODO: maxplaintextlength -TODO: filtering? like, do we need to test all pairings of encrypt/decrypt key description? Lots of failed vectors... - -## Evaluation rules - -TODO: Link to MPL. A lot of these will be shared across the MPL. - +- decrypt key description: Range of all [key descriptions](./key-description.md) used to decrypt diff --git a/framework/test-vectors/test-vector-enumeration.md b/framework/test-vectors/test-vector-enumeration.md index 2a8a6b8b..60095dbd 100644 --- a/framework/test-vectors/test-vector-enumeration.md +++ b/framework/test-vectors/test-vector-enumeration.md @@ -235,7 +235,10 @@ The expected result of an [input configuration](#input-configuration) can be det by evaluating all relevant [evaluation rules](#expected-result-evaluation-rules) for that input configuration. If no evaluation rules fail, the test scenario result should be `"positive-keyring"`. If one evaluation rule fails, the test scenario result should be the result of that evaluation rule. -If multiple evaluation rules fail, the test scenario should not be written. See [below](#multiple-errors-todo). +If more than one evaluation rule fails with a different result +(i.e. input configuration can result in both `"negative-encrypt"` and `"negative-decrypt"`), +the test scenario result should be `"negative-encrypt"`. +(Encryption happens first; failure on encrypt will be encountered first.) #### Expected result evaluation rules From 0f56efdf3c7a365f8d9d405202a48b70ab0e637f Mon Sep 17 00:00:00 2001 From: Lucas McDonald Date: Thu, 18 Apr 2024 13:27:19 -0700 Subject: [PATCH 12/19] add --- .../complete-vectors/hierarchy.md | 23 ++++++------------- 1 file changed, 7 insertions(+), 16 deletions(-) diff --git a/framework/test-vectors/complete-vectors/hierarchy.md b/framework/test-vectors/complete-vectors/hierarchy.md index 476b1dc6..bb12325e 100644 --- a/framework/test-vectors/complete-vectors/hierarchy.md +++ b/framework/test-vectors/complete-vectors/hierarchy.md @@ -24,22 +24,13 @@ a test MUST attempt to encrypt and decrypt with every [standard encryption conte ### Input dimensions -``` -encrypt key description: - { - type: "aws-kms-hierarchy" - key: range of [representative branch keys](#representative-branch-keys) - } -decrypt key description: - { - type: "aws-kms-hierarchy" - key: range of [representative branch keys](#representative-branch-keys) - logicalKeyStore: [ - "Default", - "Other", # any other logical key store name - ] - } -``` +- encrypt key description.key: range of [representative branch keys](#representative-branch-keys) +- decrypt key description.key: range of [representative branch keys](#representative-branch-keys) +- decrypt key description.logicalKeyStore: + - "Default", + - Represents the logical key store name on encrypt + - "Other" + - Represents any other logical key store name ### Representative branch keys * `"static-branch-key-1"` From 583247290d56460845c609883d4705676bb7f6ba Mon Sep 17 00:00:00 2001 From: Lucas McDonald Date: Thu, 18 Apr 2024 13:29:22 -0700 Subject: [PATCH 13/19] prettier --- .github/workflows/repo-sync.yml | 30 +++---- .../complete-vectors/encryption-context.md | 16 ++-- .../complete-vectors/hierarchy.md | 53 +++++++------ .../required-encryption-context-cmm.md | 26 +++--- .../esdk-test-vector-enumeration.md | 44 +++++------ .../mpl-test-vector-enumeration.md | 2 +- .../test-vectors/test-vector-enumeration.md | 79 ++++++++++--------- util/extract.js | 44 ++++++----- util/report.js | 4 +- util/test_conditions | 50 ++++++------ 10 files changed, 176 insertions(+), 172 deletions(-) diff --git a/.github/workflows/repo-sync.yml b/.github/workflows/repo-sync.yml index b7605354..adaff49f 100644 --- a/.github/workflows/repo-sync.yml +++ b/.github/workflows/repo-sync.yml @@ -1,7 +1,7 @@ name: Repo Sync on: - workflow_dispatch: # allows triggering this manually through the Actions UI + workflow_dispatch: # allows triggering this manually through the Actions UI jobs: repo-sync: @@ -9,17 +9,17 @@ jobs: environment: repo-sync runs-on: ubuntu-latest steps: - - uses: actions/checkout@v2 - - uses: repo-sync/github-sync@v2 - name: Sync repo to branch - with: - source_repo: ${{ secrets.SOURCE_REPO }} - source_branch: master - destination_branch: ${{ secrets.INTERMEDIATE_BRANCH }} - github_token: ${{ secrets.GITHUB_TOKEN }} - - uses: repo-sync/pull-request@v2 - name: Create pull request - with: - source_branch: ${{ secrets.INTERMEDIATE_BRANCH }} - destination_branch: master - github_token: ${{ secrets.GITHUB_TOKEN }} + - uses: actions/checkout@v2 + - uses: repo-sync/github-sync@v2 + name: Sync repo to branch + with: + source_repo: ${{ secrets.SOURCE_REPO }} + source_branch: master + destination_branch: ${{ secrets.INTERMEDIATE_BRANCH }} + github_token: ${{ secrets.GITHUB_TOKEN }} + - uses: repo-sync/pull-request@v2 + name: Create pull request + with: + source_branch: ${{ secrets.INTERMEDIATE_BRANCH }} + destination_branch: master + github_token: ${{ secrets.GITHUB_TOKEN }} diff --git a/framework/test-vectors/complete-vectors/encryption-context.md b/framework/test-vectors/complete-vectors/encryption-context.md index e3f70d24..047e85ec 100644 --- a/framework/test-vectors/complete-vectors/encryption-context.md +++ b/framework/test-vectors/complete-vectors/encryption-context.md @@ -15,11 +15,11 @@ This is a description of the standard encryption contexts to test. ### Standard Encryption Contexts Constraints -* MUST have an empty map. - * The number of the items in the empty map MUST equal 0. -* MUST have a small map. - * The number of the items in the small map MUST be between 1 and 10. -* MUST have a large map. - * The number of the items in the large map MUST be greater than 10. -* MUST have multibyte UTF8 characters in both the key and value. -* MUST have multibyte non-BMP characters in both the key and value. \ No newline at end of file +- MUST have an empty map. + - The number of the items in the empty map MUST equal 0. +- MUST have a small map. + - The number of the items in the small map MUST be between 1 and 10. +- MUST have a large map. + - The number of the items in the large map MUST be greater than 10. +- MUST have multibyte UTF8 characters in both the key and value. +- MUST have multibyte non-BMP characters in both the key and value. diff --git a/framework/test-vectors/complete-vectors/hierarchy.md b/framework/test-vectors/complete-vectors/hierarchy.md index bb12325e..52f4ead1 100644 --- a/framework/test-vectors/complete-vectors/hierarchy.md +++ b/framework/test-vectors/complete-vectors/hierarchy.md @@ -33,41 +33,42 @@ a test MUST attempt to encrypt and decrypt with every [standard encryption conte - Represents any other logical key store name ### Representative branch keys -* `"static-branch-key-1"` - * Any valid branch key. -* `"static-branch-key-2"` - * Any other valid branch key -* `"branch-key-no-permissions"` - * Any valid branch key where -the test vector runner does not have permissions -for the KMS key -* `"branch-key-not-in-table"` - * Any branch key ID not in the keystore table -* `"branch-key-no-version"` - * Any branch key without a version -* `"invalid-branch-key"` - * Any illegally mutated invalid branch key + +- `"static-branch-key-1"` + - Any valid branch key. +- `"static-branch-key-2"` + - Any other valid branch key +- `"branch-key-no-permissions"` + - Any valid branch key where + the test vector runner does not have permissions + for the KMS key +- `"branch-key-not-in-table"` + - Any branch key ID not in the keystore table +- `"branch-key-no-version"` + - Any branch key without a version +- `"invalid-branch-key"` + - Any illegally mutated invalid branch key ### Evaluation rules - If encrypting key type is anything other than `"aws-kms-hierarchy"` -and decrypting key type is `"aws-kms-hierarchy"`, -the result should be `"negative-decrypt"`. + and decrypting key type is `"aws-kms-hierarchy"`, + the result should be `"negative-decrypt"`. - If encrypting key type is `"aws-kms-hierarchy"` -and decrypting key type is anything other than `"aws-kms-hierarchy"`, -the result should be `"negative-decrypt"`. + and decrypting key type is anything other than `"aws-kms-hierarchy"`, + the result should be `"negative-decrypt"`. - If the logical key store is "Other", -the result should be `"negative-decrypt"`. + the result should be `"negative-decrypt"`. - If `"key"` is different on encrypt and decrypt, -the result should be `"negative-decrypt"`. -(i.e. no key specified here is interoperable with any other key.) + the result should be `"negative-decrypt"`. + (i.e. no key specified here is interoperable with any other key.) - If `"key"` is any of: - `"branch-key-no-permissions"` - `"branch-key-not-in-table"` - `"branch-key-no-version"` - `"invalid-branch-key"` -(i.e. an "invalid" key with a particular invalid condition) -on either encrypt or decrypt, -the result should be either `"negative-encrypt"` or `"negative-decrypt"`, -depending on whether the invalid key was specified on encrypt or decrypt. -- In all other cases, the result should be `"positive"`. \ No newline at end of file + (i.e. an "invalid" key with a particular invalid condition) + on either encrypt or decrypt, + the result should be either `"negative-encrypt"` or `"negative-decrypt"`, + depending on whether the invalid key was specified on encrypt or decrypt. +- In all other cases, the result should be `"positive"`. diff --git a/framework/test-vectors/complete-vectors/required-encryption-context-cmm.md b/framework/test-vectors/complete-vectors/required-encryption-context-cmm.md index 86340120..bb777fc6 100644 --- a/framework/test-vectors/complete-vectors/required-encryption-context-cmm.md +++ b/framework/test-vectors/complete-vectors/required-encryption-context-cmm.md @@ -88,23 +88,23 @@ For example: #### Representative required encryption context keys -* Every subset of keys in the provided [encryption context](../../structures.md#encryption-context) -* Any key NOT in the provided encryption context. +- Every subset of keys in the provided [encryption context](../../structures.md#encryption-context) +- Any key NOT in the provided encryption context. #### Representative reproduced encryption context -* Every subset of items in the provided [encryption context](../../structures.md#encryption-context) -* Any item NOT in the provided encryption context. +- Every subset of items in the provided [encryption context](../../structures.md#encryption-context) +- Any item NOT in the provided encryption context. ## Test vector evaluation rules - If any of the `requiredEncryptionContextKeys` do not exist in the -supplied encryption context on encrypt -then the test result MUST be `negative-encrypt-keyring`. [source](#required-encryption-context-cmm-failures-on-encrypt) -- If the set of keys in `reproducedEncryptionContext` on decrypt -does not match the set of `requiredEncryptionContextKeys`, -then the test result MUST be `negative-decrypt-keyring`. [source] -- If the the value for any key in `reproducedEncryptionContext` on decrypt -does not match the value provided for that key on encrypt, -then the test result MUST be `negative-decrypt-keyring`. [source](#required-encryption-context-cmm-failures-on-decrypt) -- In all other cases, the test result MUST be `positive-keyring`. \ No newline at end of file + supplied encryption context on encrypt + then the test result MUST be `negative-encrypt-keyring`. [source](#required-encryption-context-cmm-failures-on-encrypt) +- If the set of keys in `reproducedEncryptionContext` on decrypt + does not match the set of `requiredEncryptionContextKeys`, + then the test result MUST be `negative-decrypt-keyring`. [source] +- If the the value for any key in `reproducedEncryptionContext` on decrypt + does not match the value provided for that key on encrypt, + then the test result MUST be `negative-decrypt-keyring`. [source](#required-encryption-context-cmm-failures-on-decrypt) +- In all other cases, the test result MUST be `positive-keyring`. diff --git a/framework/test-vectors/esdk-test-vector-enumeration.md b/framework/test-vectors/esdk-test-vector-enumeration.md index 84080966..37efcfd2 100644 --- a/framework/test-vectors/esdk-test-vector-enumeration.md +++ b/framework/test-vectors/esdk-test-vector-enumeration.md @@ -1,7 +1,7 @@ [//]: # "Copyright Amazon.com Inc. or its affiliates. All Rights Reserved." [//]: # "SPDX-License-Identifier: CC-BY-SA-4.0" -# ESDK Test Vector Enumeration +# ESDK Test Vector Enumeration This document performs the [test vector enumeration](test-vector-enumeration.md) process for the ESDK to construct the full suite of test vectors @@ -26,34 +26,34 @@ and [evaluating each configuration's expected result](test-vector-enumeration.md ### Representative frame sizes -* Non-framed data: `length = 0` -* Small frame: `0 < length < 4096` -* Default frame: `length = 4096` -* Large frame: `length > 4096` +- Non-framed data: `length = 0` +- Small frame: `0 < length < 4096` +- Default frame: `length = 4096` +- Large frame: `length > 4096` ### Representative plaintext constraints -#### Framed data +#### Framed data These should ONLY be used if `frame length > 0`. -* Empty: `length = 0` -* Small: all plaintexts where (1 < length ≤ 10) - * If `frame size < 10`, omit this. -* Medium: all plaintexts where (10 < length ≤ 1000) - * If `frame size < 1000`, omit this. -* Large: all plaintexts where (1000 < length ≤ [frame size](#representative-frame-sizes)) -* Largest frame: all plaintexts where (length = frame size) -* Largest frame + partial frame: all plaintexts where (length = frame size + [1 .. frame size)) -* Two largest frames: all plaintexts where (length = 2*(frame size)) -* Many frames: all plaintexts where (length = 2*(frame size) + [1 .. (frame size)*(maximum # of frames-2)]) -* Maximum frames: all plaintexts where (length = (frame size)*(maximum # of frames)) +- Empty: `length = 0` +- Small: all plaintexts where (1 < length ≤ 10) + - If `frame size < 10`, omit this. +- Medium: all plaintexts where (10 < length ≤ 1000) + - If `frame size < 1000`, omit this. +- Large: all plaintexts where (1000 < length ≤ [frame size](#representative-frame-sizes)) +- Largest frame: all plaintexts where (length = frame size) +- Largest frame + partial frame: all plaintexts where (length = frame size + [1 .. frame size)) +- Two largest frames: all plaintexts where (length = 2\*(frame size)) +- Many frames: all plaintexts where (length = 2*(frame size) + [1 .. (frame size)*(maximum # of frames-2)]) +- Maximum frames: all plaintexts where (length = (frame size)\*(maximum # of frames)) -#### Non-framed data +#### Non-framed data These should ONLY be used if `frame length = 0`. -* Empty: `length = 0` -* Small: all plaintexts where (1 < length ≤ 10) -* Medium: all plaintexts where (10 < length ≤ 1000) -* Large: all plaintexts where (1000 < length ≤ 2^32) \ No newline at end of file +- Empty: `length = 0` +- Small: all plaintexts where (1 < length ≤ 10) +- Medium: all plaintexts where (10 < length ≤ 1000) +- Large: all plaintexts where (1000 < length ≤ 2^32) diff --git a/framework/test-vectors/mpl-test-vector-enumeration.md b/framework/test-vectors/mpl-test-vector-enumeration.md index e5b95bb6..dbb63641 100644 --- a/framework/test-vectors/mpl-test-vector-enumeration.md +++ b/framework/test-vectors/mpl-test-vector-enumeration.md @@ -1,7 +1,7 @@ [//]: # "Copyright Amazon.com Inc. or its affiliates. All Rights Reserved." [//]: # "SPDX-License-Identifier: CC-BY-SA-4.0" -# ESDK Test Vector Enumeration +# ESDK Test Vector Enumeration This document performs the [test vector enumeration](test-vector-enumeration.md) process for the MPL to construct the full suite of test vectors diff --git a/framework/test-vectors/test-vector-enumeration.md b/framework/test-vectors/test-vector-enumeration.md index 60095dbd..4208617e 100644 --- a/framework/test-vectors/test-vector-enumeration.md +++ b/framework/test-vectors/test-vector-enumeration.md @@ -1,7 +1,7 @@ [//]: # "Copyright Amazon.com Inc. or its affiliates. All Rights Reserved." [//]: # "SPDX-License-Identifier: CC-BY-SA-4.0" -# Test Vector Enumeration +# Test Vector Enumeration ## Overview @@ -18,13 +18,14 @@ and expected results of encryption and/or decryption. The framework does not enumerate every possible input configuration for two reasons: -1. It is impractical to enumerate every possible input value -for some attributes in an input configuration -(see ["representative values"](#selecting-a-representative-input-value)). + +1. It is impractical to enumerate every possible input value + for some attributes in an input configuration + (see ["representative values"](#selecting-a-representative-input-value)). 2. Some input configurations with `negative` expected results -are not as critical of test cases as others -and are not a priority to test at this time -(see ["filtering input configurations"](#filtering-input-configurations)) + are not as critical of test cases as others + and are not a priority to test at this time + (see ["filtering input configurations"](#filtering-input-configurations)) As a result, the framework does not actually enumerate @@ -33,7 +34,7 @@ However, by carefully reasoning about the selection of representative values and the filtered test cases, the framework constructs a set of pairings -that is *representative* of all *relevant* +that is _representative_ of all _relevant_ input configurations and expected results. ## Input configuration @@ -56,7 +57,7 @@ or how to construct its range of possible values. ### Selecting a representative input value -Not every input value in an [input dimension](#input-dimensions) can be practically tested, +Not every input value in an [input dimension](#input-dimensions) can be practically tested, as the number of allowed configurations as inputs to encryption is massive. For example, the number of possible plaintexts that can be encrypted is massive; it is impractical to enumerate and test all possible plaintexts. @@ -74,18 +75,19 @@ and test it in the test vectors. For example, we might partition the set of all possible plaintexts based on the plaintext length. Our hypothetical "smaller sets" might be: -* Empty: length = 0 -* Small: all plaintexts where (1 < length ≤ 10) -* Medium: all plaintexts where (10 < length ≤ 1000) -* Large: all plaintexts where (1000 < length ≤ 2^32-1) -* Largest frame: all plaintexts where (length = 2^32-1) -* Largest frame + partial frame: all plaintexts where (length = 2^32-1 + [1 .. 2^32-1)) -* Two largest frames: all plaintexts where (length = 2*(2^32-1)) -* Many frames: all plaintexts where (length = 2*(2^32-1) + [1 .. (2^32-1)*(2^32-3)]) + +- Empty: length = 0 +- Small: all plaintexts where (1 < length ≤ 10) +- Medium: all plaintexts where (10 < length ≤ 1000) +- Large: all plaintexts where (1000 < length ≤ 2^32-1) +- Largest frame: all plaintexts where (length = 2^32-1) +- Largest frame + partial frame: all plaintexts where (length = 2^32-1 + [1 .. 2^32-1)) +- Two largest frames: all plaintexts where (length = 2\*(2^32-1)) +- Many frames: all plaintexts where (length = 2*(2^32-1) + [1 .. (2^32-1)*(2^32-3)]) (The motivation for partitioning this set based on lengths of 2^32-1 comes from the ESDK message [frame size](https://docs.aws.amazon.com/encryption-sdk/latest/developer-guide/message-format.html#body-framing).) -From each of these smaller sets, we select one (or more) +From each of these smaller sets, we select one (or more) concrete "representative" values that are tested in test vectors. For example, a representative "small" plaintext could be `"abc"`. @@ -186,20 +188,20 @@ and determine a reasonable execution duration for these tests. Some guidance for filtering input configurations: -* Prune invalid configurations as early as possible. -Input configurations can be filtered at any point in the vector enumeration process, -though it is beneficial to filter as early as possible -and prune large numbers of filtered configurations -to make input configuration enumeration as fast as possible. -For example, if the configuration under consideration -has a hierarchy keyring on encrypt and a raw AES keyring on decrypt, -the manifest generator might generate only one of these cases -(as a smoke test), -then skip considering all other configurations. -In contrast, a naïve approach would consider all of these other configurations -and deem them invalid. -This guidance suggests short-circuiting that evaluation as much as possible -to save time when generating vectors. +- Prune invalid configurations as early as possible. + Input configurations can be filtered at any point in the vector enumeration process, + though it is beneficial to filter as early as possible + and prune large numbers of filtered configurations + to make input configuration enumeration as fast as possible. + For example, if the configuration under consideration + has a hierarchy keyring on encrypt and a raw AES keyring on decrypt, + the manifest generator might generate only one of these cases + (as a smoke test), + then skip considering all other configurations. + In contrast, a naïve approach would consider all of these other configurations + and deem them invalid. + This guidance suggests short-circuiting that evaluation as much as possible + to save time when generating vectors. ### Enumerating input configurations @@ -214,12 +216,13 @@ An expected result is a categorical value and an optional error description. An expected result categorical value is one of: -* Positive (successful test scenario) - * `"positive-keyring"` -* Negative encrypt (failure on encrypt) - * `"negative-encrypt-keyring"` -* Negative decrypt (failure on decrypt) - * `"negative-decrypt-keyring"` + +- Positive (successful test scenario) + - `"positive-keyring"` +- Negative encrypt (failure on encrypt) + - `"negative-encrypt-keyring"` +- Negative decrypt (failure on decrypt) + - `"negative-decrypt-keyring"` The error description for negative scenarios describes the reason for the scenario's failure. diff --git a/util/extract.js b/util/extract.js index 161b12ea..b65d5a15 100755 --- a/util/extract.js +++ b/util/extract.js @@ -10,27 +10,21 @@ const { extname, basename, resolve, dirname, join, relative } = require("path"); const { execSync } = require("child_process"); -const { - readFileSync, - writeFileSync, - statSync, - constants, - mkdirSync, -} = require("fs"); +const { readFileSync, writeFileSync, statSync, constants, mkdirSync } = require("fs"); const ext = ".md"; const pathToComplianceRoot = `${relative(process.cwd(), `${__dirname}/../compliance`)}`; needs( () => execSync("which kramdown-rfc2629"), - "kramdown-rfc2629 needs to be installed, try `gem install kramdown-rfc2629 -v 1.5.21`" + "kramdown-rfc2629 needs to be installed, try `gem install kramdown-rfc2629 -v 1.5.21`", ); needs( () => execSync("which xml2rfc"), - "xml2rfc needs to be installed, try `pip install xml2rfc==3.5.0`" + "xml2rfc needs to be installed, try `pip install xml2rfc==3.5.0`", ); needs( () => execSync("which duvet"), - "duvet needs to be installed, try `util/install-duvet`" + "duvet needs to be installed, try `util/install-duvet`", ); /* May need to change this to a better parser... @@ -70,11 +64,13 @@ function extract(filePath) { writeFileSync( markdownSpecFile, [header(fileName), readFileSync(filePath, { encoding: "utf8" }), footer()].join("\n"), - { encoding: "utf8" } + { encoding: "utf8" }, ); // Convert the markdown file from RFC XML - execSync(["kramdown-rfc2629", "-3", markdownSpecFile, ">", xmlRfcFile].join(" "), {stdio: 'inherit'}); + execSync(["kramdown-rfc2629", "-3", markdownSpecFile, ">", xmlRfcFile].join(" "), { + stdio: "inherit", + }); // An existing spec may exists, clean up first try { @@ -91,23 +87,31 @@ function extract(filePath) { mkdirSync(complianceDir, { recursive: true }); // Convert the RFC XML to a ietf rfc - execSync(["xml2rfc", - "-P", xmlRfcFile, - "-s", "'Too long line found'", // Suppress warnings about long table line length - "-s", "'Total table width'", // Suppress warnings about overall table width - "-o", complianceSpec - ].join(" "), {stdio: 'inherit'}); + execSync( + [ + "xml2rfc", + "-P", + xmlRfcFile, + "-s", + "'Too long line found'", // Suppress warnings about long table line length + "-s", + "'Total table width'", // Suppress warnings about overall table width + "-o", + complianceSpec, + ].join(" "), + { stdio: "inherit" }, + ); const args = ["duvet", "extract", `${complianceSpec}`, "-o", "compliance"]; // extract the specification - execSync(args.join(" "), { encoding: 'utf8', stdio: 'inherit'}); + execSync(args.join(" "), { encoding: "utf8", stdio: "inherit" }); } function extract_needs(filePath) { needs(extname(filePath) === ext, `Unsupported ext ${ext}`); needs( () => statSync(filePath).isFile(), - `Specification file ${filePath} does not exist.` + `Specification file ${filePath} does not exist.`, ); return filePath; } diff --git a/util/report.js b/util/report.js index b1b26f02..b31c9f5b 100755 --- a/util/report.js +++ b/util/report.js @@ -5,7 +5,7 @@ const { execSync } = require("child_process"); needs( () => execSync("which duvet"), - "duvet needs to be installed try `util/install-duvet`" + "duvet needs to be installed try `util/install-duvet`", ); const data = execSync("git remote -v") @@ -21,7 +21,7 @@ const data = execSync("git remote -v") .map((line) => line.startsWith("https://") ? line - : line.replace("git@github.com:", "https://github.com/") + : line.replace("git@github.com:", "https://github.com/"), ) /* Drop the `.git` because this is not in GitHub blob or issue urls */ .map((line) => line.replace(".git", "")) diff --git a/util/test_conditions b/util/test_conditions index b64456d4..cb7c5526 100755 --- a/util/test_conditions +++ b/util/test_conditions @@ -47,23 +47,19 @@ const fileSettings = { }, // C files ".c:": { - conditionRemove: - /(\*\/)|(\/\*)/g, + conditionRemove: /(\*\/)|(\/\*)/g, complianceConfig: standardComplianceParams, }, ".h:": { - conditionRemove: - /(\*\/)|(\/\*)/g, + conditionRemove: /(\*\/)|(\/\*)/g, complianceConfig: standardComplianceParams, }, ".cpp:": { - conditionRemove: - /(\*\/)|(\/\*)/g, + conditionRemove: /(\*\/)|(\/\*)/g, complianceConfig: standardComplianceParams, }, ".txt:": { - conditionRemove: - /(\*\/)|(\/\*)/g, + conditionRemove: /(\*\/)|(\/\*)/g, complianceConfig: standardComplianceParams, }, }; @@ -98,7 +94,7 @@ const [sourceGlobs, testGlobs] = process.argv.slice(2).reduce( throw new Error(`Unknown argument for ${arg}`); } }, - [[], []] + [[], []], ); const conditionGrep = `grep -E 'Precondition:|Postcondition(:|\\):)'`; @@ -107,19 +103,19 @@ const complianceGrep = `grep -E '//[=#]'`; Promise.all([ executeAndClean( sourceGlobs.map((g) => `${conditionGrep} ${g}`), - clean + clean, ), executeAndClean( testGlobs.map((g) => `${conditionGrep} ${g}`), - clean + clean, ), executeAndClean( sourceGlobs.map((g) => `${complianceGrep} ${g}`), - cleanCompliance + cleanCompliance, ), executeAndClean( testGlobs.map((g) => `${complianceGrep} ${g}`), - cleanCompliance + cleanCompliance, ), ]).then(([conditionSource, conditionTests, complianceCitations, complianceTests]) => { let issues = 0; @@ -132,12 +128,12 @@ Promise.all([ issues += noDuplicates( conditionSource, conditionSourceSet, - "Duplicate source conditions" + "Duplicate source conditions", ); issues += noDuplicates( conditionTestArray, conditionTestSet, - "Duplicate test conditions" + "Duplicate test conditions", ); issues += [...complianceCitationsMap.values()] @@ -151,23 +147,23 @@ Promise.all([ issues += noDuplicates( complianceCitations, complianceCitationsMap, - "Duplicate compliance citation" + "Duplicate compliance citation", ); issues += noDuplicates( complianceTests, complianceTestsMap, - "Duplicate compliance test" + "Duplicate compliance test", ); issues += missingRight(conditionSourceSet, conditionTestSet).map((line) => - console.log(`Missing test condition for \n${line}`) + console.log(`Missing test condition for \n${line}`), ).length; issues += missingRight(conditionTestSet, conditionSourceSet).map((line) => - console.log(`Update or change test condition \n${line}`) + console.log(`Update or change test condition \n${line}`), ).length; issues += missingRight(complianceCitationsMap, complianceTestsMap).map((line) => - console.log(`Missing compliance test for \n${complianceCitationsMap.get(line)}`) + console.log(`Missing compliance test for \n${complianceCitationsMap.get(line)}`), ).length; issues += missingRight(complianceTestsMap, complianceCitationsMap).map((line) => { console.log(`Update or change compliance test \n${complianceTestsMap.get(line)}`); @@ -198,12 +194,12 @@ function clean({ stdout }) { .map((rawLine) => { if (!rawLine) return ""; const [match] = rawLine.match(filePathMatch); - try{ + try { const { conditionRemove } = fileSettings[match]; - return rawLine.split(match).pop().replace(conditionRemove, "").trim(); - } catch(ex) { - console.log(filePathMatch, match) - throw ex + return rawLine.split(match).pop().replace(conditionRemove, "").trim(); + } catch (ex) { + console.log(filePathMatch, match); + throw ex; } }) .filter((line) => line !== ""); @@ -312,8 +308,8 @@ async function executeAndClean(commands, cleanFn) { } throw ex; }) - .then(cleanFn) - ) + .then(cleanFn), + ), ); return [].concat(...results); From 24ef6e163e31baeea590d04b0504c1409fec5071 Mon Sep 17 00:00:00 2001 From: Lucas McDonald Date: Thu, 18 Apr 2024 13:31:33 -0700 Subject: [PATCH 14/19] unprettier --- .github/workflows/repo-sync.yml | 30 ++++++++++---------- util/extract.js | 44 +++++++++++++---------------- util/report.js | 4 +-- util/test_conditions | 50 ++++++++++++++++++--------------- 4 files changed, 64 insertions(+), 64 deletions(-) diff --git a/.github/workflows/repo-sync.yml b/.github/workflows/repo-sync.yml index adaff49f..b7605354 100644 --- a/.github/workflows/repo-sync.yml +++ b/.github/workflows/repo-sync.yml @@ -1,7 +1,7 @@ name: Repo Sync on: - workflow_dispatch: # allows triggering this manually through the Actions UI + workflow_dispatch: # allows triggering this manually through the Actions UI jobs: repo-sync: @@ -9,17 +9,17 @@ jobs: environment: repo-sync runs-on: ubuntu-latest steps: - - uses: actions/checkout@v2 - - uses: repo-sync/github-sync@v2 - name: Sync repo to branch - with: - source_repo: ${{ secrets.SOURCE_REPO }} - source_branch: master - destination_branch: ${{ secrets.INTERMEDIATE_BRANCH }} - github_token: ${{ secrets.GITHUB_TOKEN }} - - uses: repo-sync/pull-request@v2 - name: Create pull request - with: - source_branch: ${{ secrets.INTERMEDIATE_BRANCH }} - destination_branch: master - github_token: ${{ secrets.GITHUB_TOKEN }} + - uses: actions/checkout@v2 + - uses: repo-sync/github-sync@v2 + name: Sync repo to branch + with: + source_repo: ${{ secrets.SOURCE_REPO }} + source_branch: master + destination_branch: ${{ secrets.INTERMEDIATE_BRANCH }} + github_token: ${{ secrets.GITHUB_TOKEN }} + - uses: repo-sync/pull-request@v2 + name: Create pull request + with: + source_branch: ${{ secrets.INTERMEDIATE_BRANCH }} + destination_branch: master + github_token: ${{ secrets.GITHUB_TOKEN }} diff --git a/util/extract.js b/util/extract.js index b65d5a15..161b12ea 100755 --- a/util/extract.js +++ b/util/extract.js @@ -10,21 +10,27 @@ const { extname, basename, resolve, dirname, join, relative } = require("path"); const { execSync } = require("child_process"); -const { readFileSync, writeFileSync, statSync, constants, mkdirSync } = require("fs"); +const { + readFileSync, + writeFileSync, + statSync, + constants, + mkdirSync, +} = require("fs"); const ext = ".md"; const pathToComplianceRoot = `${relative(process.cwd(), `${__dirname}/../compliance`)}`; needs( () => execSync("which kramdown-rfc2629"), - "kramdown-rfc2629 needs to be installed, try `gem install kramdown-rfc2629 -v 1.5.21`", + "kramdown-rfc2629 needs to be installed, try `gem install kramdown-rfc2629 -v 1.5.21`" ); needs( () => execSync("which xml2rfc"), - "xml2rfc needs to be installed, try `pip install xml2rfc==3.5.0`", + "xml2rfc needs to be installed, try `pip install xml2rfc==3.5.0`" ); needs( () => execSync("which duvet"), - "duvet needs to be installed, try `util/install-duvet`", + "duvet needs to be installed, try `util/install-duvet`" ); /* May need to change this to a better parser... @@ -64,13 +70,11 @@ function extract(filePath) { writeFileSync( markdownSpecFile, [header(fileName), readFileSync(filePath, { encoding: "utf8" }), footer()].join("\n"), - { encoding: "utf8" }, + { encoding: "utf8" } ); // Convert the markdown file from RFC XML - execSync(["kramdown-rfc2629", "-3", markdownSpecFile, ">", xmlRfcFile].join(" "), { - stdio: "inherit", - }); + execSync(["kramdown-rfc2629", "-3", markdownSpecFile, ">", xmlRfcFile].join(" "), {stdio: 'inherit'}); // An existing spec may exists, clean up first try { @@ -87,31 +91,23 @@ function extract(filePath) { mkdirSync(complianceDir, { recursive: true }); // Convert the RFC XML to a ietf rfc - execSync( - [ - "xml2rfc", - "-P", - xmlRfcFile, - "-s", - "'Too long line found'", // Suppress warnings about long table line length - "-s", - "'Total table width'", // Suppress warnings about overall table width - "-o", - complianceSpec, - ].join(" "), - { stdio: "inherit" }, - ); + execSync(["xml2rfc", + "-P", xmlRfcFile, + "-s", "'Too long line found'", // Suppress warnings about long table line length + "-s", "'Total table width'", // Suppress warnings about overall table width + "-o", complianceSpec + ].join(" "), {stdio: 'inherit'}); const args = ["duvet", "extract", `${complianceSpec}`, "-o", "compliance"]; // extract the specification - execSync(args.join(" "), { encoding: "utf8", stdio: "inherit" }); + execSync(args.join(" "), { encoding: 'utf8', stdio: 'inherit'}); } function extract_needs(filePath) { needs(extname(filePath) === ext, `Unsupported ext ${ext}`); needs( () => statSync(filePath).isFile(), - `Specification file ${filePath} does not exist.`, + `Specification file ${filePath} does not exist.` ); return filePath; } diff --git a/util/report.js b/util/report.js index b31c9f5b..b1b26f02 100755 --- a/util/report.js +++ b/util/report.js @@ -5,7 +5,7 @@ const { execSync } = require("child_process"); needs( () => execSync("which duvet"), - "duvet needs to be installed try `util/install-duvet`", + "duvet needs to be installed try `util/install-duvet`" ); const data = execSync("git remote -v") @@ -21,7 +21,7 @@ const data = execSync("git remote -v") .map((line) => line.startsWith("https://") ? line - : line.replace("git@github.com:", "https://github.com/"), + : line.replace("git@github.com:", "https://github.com/") ) /* Drop the `.git` because this is not in GitHub blob or issue urls */ .map((line) => line.replace(".git", "")) diff --git a/util/test_conditions b/util/test_conditions index cb7c5526..b64456d4 100755 --- a/util/test_conditions +++ b/util/test_conditions @@ -47,19 +47,23 @@ const fileSettings = { }, // C files ".c:": { - conditionRemove: /(\*\/)|(\/\*)/g, + conditionRemove: + /(\*\/)|(\/\*)/g, complianceConfig: standardComplianceParams, }, ".h:": { - conditionRemove: /(\*\/)|(\/\*)/g, + conditionRemove: + /(\*\/)|(\/\*)/g, complianceConfig: standardComplianceParams, }, ".cpp:": { - conditionRemove: /(\*\/)|(\/\*)/g, + conditionRemove: + /(\*\/)|(\/\*)/g, complianceConfig: standardComplianceParams, }, ".txt:": { - conditionRemove: /(\*\/)|(\/\*)/g, + conditionRemove: + /(\*\/)|(\/\*)/g, complianceConfig: standardComplianceParams, }, }; @@ -94,7 +98,7 @@ const [sourceGlobs, testGlobs] = process.argv.slice(2).reduce( throw new Error(`Unknown argument for ${arg}`); } }, - [[], []], + [[], []] ); const conditionGrep = `grep -E 'Precondition:|Postcondition(:|\\):)'`; @@ -103,19 +107,19 @@ const complianceGrep = `grep -E '//[=#]'`; Promise.all([ executeAndClean( sourceGlobs.map((g) => `${conditionGrep} ${g}`), - clean, + clean ), executeAndClean( testGlobs.map((g) => `${conditionGrep} ${g}`), - clean, + clean ), executeAndClean( sourceGlobs.map((g) => `${complianceGrep} ${g}`), - cleanCompliance, + cleanCompliance ), executeAndClean( testGlobs.map((g) => `${complianceGrep} ${g}`), - cleanCompliance, + cleanCompliance ), ]).then(([conditionSource, conditionTests, complianceCitations, complianceTests]) => { let issues = 0; @@ -128,12 +132,12 @@ Promise.all([ issues += noDuplicates( conditionSource, conditionSourceSet, - "Duplicate source conditions", + "Duplicate source conditions" ); issues += noDuplicates( conditionTestArray, conditionTestSet, - "Duplicate test conditions", + "Duplicate test conditions" ); issues += [...complianceCitationsMap.values()] @@ -147,23 +151,23 @@ Promise.all([ issues += noDuplicates( complianceCitations, complianceCitationsMap, - "Duplicate compliance citation", + "Duplicate compliance citation" ); issues += noDuplicates( complianceTests, complianceTestsMap, - "Duplicate compliance test", + "Duplicate compliance test" ); issues += missingRight(conditionSourceSet, conditionTestSet).map((line) => - console.log(`Missing test condition for \n${line}`), + console.log(`Missing test condition for \n${line}`) ).length; issues += missingRight(conditionTestSet, conditionSourceSet).map((line) => - console.log(`Update or change test condition \n${line}`), + console.log(`Update or change test condition \n${line}`) ).length; issues += missingRight(complianceCitationsMap, complianceTestsMap).map((line) => - console.log(`Missing compliance test for \n${complianceCitationsMap.get(line)}`), + console.log(`Missing compliance test for \n${complianceCitationsMap.get(line)}`) ).length; issues += missingRight(complianceTestsMap, complianceCitationsMap).map((line) => { console.log(`Update or change compliance test \n${complianceTestsMap.get(line)}`); @@ -194,12 +198,12 @@ function clean({ stdout }) { .map((rawLine) => { if (!rawLine) return ""; const [match] = rawLine.match(filePathMatch); - try { + try{ const { conditionRemove } = fileSettings[match]; - return rawLine.split(match).pop().replace(conditionRemove, "").trim(); - } catch (ex) { - console.log(filePathMatch, match); - throw ex; + return rawLine.split(match).pop().replace(conditionRemove, "").trim(); + } catch(ex) { + console.log(filePathMatch, match) + throw ex } }) .filter((line) => line !== ""); @@ -308,8 +312,8 @@ async function executeAndClean(commands, cleanFn) { } throw ex; }) - .then(cleanFn), - ), + .then(cleanFn) + ) ); return [].concat(...results); From a97d48241eeeab70e8b45dd06a905962422ffc32 Mon Sep 17 00:00:00 2001 From: Lucas McDonald Date: Tue, 7 May 2024 16:00:39 -0700 Subject: [PATCH 15/19] fixes --- .../complete-vectors/hierarchy.md | 27 +++++----- .../esdk-test-vector-enumeration.md | 53 +++++++++++++++---- .../mpl-test-vector-enumeration.md | 2 +- .../test-vectors/test-vector-enumeration.md | 8 +-- 4 files changed, 61 insertions(+), 29 deletions(-) diff --git a/framework/test-vectors/complete-vectors/hierarchy.md b/framework/test-vectors/complete-vectors/hierarchy.md index 52f4ead1..bb625e8b 100644 --- a/framework/test-vectors/complete-vectors/hierarchy.md +++ b/framework/test-vectors/complete-vectors/hierarchy.md @@ -25,12 +25,13 @@ a test MUST attempt to encrypt and decrypt with every [standard encryption conte ### Input dimensions - encrypt key description.key: range of [representative branch keys](#representative-branch-keys) -- decrypt key description.key: range of [representative branch keys](#representative-branch-keys) -- decrypt key description.logicalKeyStore: - - "Default", - - Represents the logical key store name on encrypt - - "Other" - - Represents any other logical key store name +- decrypt key description + - key: range of [representative branch keys](#representative-branch-keys) + - logicalKeyStore: + - "Default", + - Represents the logical key store name on encrypt + - "Other" + - Represents any other logical key store name ### Representative branch keys @@ -45,7 +46,7 @@ a test MUST attempt to encrypt and decrypt with every [standard encryption conte - `"branch-key-not-in-table"` - Any branch key ID not in the keystore table - `"branch-key-no-version"` - - Any branch key without a version + - A Version of a branch key which does not exist - `"invalid-branch-key"` - Any illegally mutated invalid branch key @@ -53,14 +54,14 @@ a test MUST attempt to encrypt and decrypt with every [standard encryption conte - If encrypting key type is anything other than `"aws-kms-hierarchy"` and decrypting key type is `"aws-kms-hierarchy"`, - the result should be `"negative-decrypt"`. + the result MUST be `"negative-decrypt"`. - If encrypting key type is `"aws-kms-hierarchy"` and decrypting key type is anything other than `"aws-kms-hierarchy"`, - the result should be `"negative-decrypt"`. + the result MUST be `"negative-decrypt"`. - If the logical key store is "Other", - the result should be `"negative-decrypt"`. + the result MUST be `"negative-decrypt"`. - If `"key"` is different on encrypt and decrypt, - the result should be `"negative-decrypt"`. + the result MUST be `"negative-decrypt"`. (i.e. no key specified here is interoperable with any other key.) - If `"key"` is any of: - `"branch-key-no-permissions"` @@ -69,6 +70,6 @@ a test MUST attempt to encrypt and decrypt with every [standard encryption conte - `"invalid-branch-key"` (i.e. an "invalid" key with a particular invalid condition) on either encrypt or decrypt, - the result should be either `"negative-encrypt"` or `"negative-decrypt"`, + the result MUST be either `"negative-encrypt"` or `"negative-decrypt"`, depending on whether the invalid key was specified on encrypt or decrypt. -- In all other cases, the result should be `"positive"`. +- In all other cases, the result MUST be `"positive"`. diff --git a/framework/test-vectors/esdk-test-vector-enumeration.md b/framework/test-vectors/esdk-test-vector-enumeration.md index 37efcfd2..f3c1828c 100644 --- a/framework/test-vectors/esdk-test-vector-enumeration.md +++ b/framework/test-vectors/esdk-test-vector-enumeration.md @@ -17,6 +17,7 @@ and [evaluating each configuration's expected result](test-vector-enumeration.md - plaintext: Range of [representative plaintext values](#representative-plaintext-constraints) - commitment policy: Range of allowed [commitment policies](../../client-apis/client.md#commitment-policy) - frame size: Range of [representative frame sizes](#representative-frame-sizes) +- maximum encrypted data keys: Range of [representative number of maximum encrypted data keys](#representative-number-of-maximum-encrypted-data-keys-edks) ## Evaluation rules @@ -27,9 +28,26 @@ and [evaluating each configuration's expected result](test-vector-enumeration.md ### Representative frame sizes - Non-framed data: `length = 0` + - The representative value for non-framed data MUST have length = 0. - Small frame: `0 < length < 4096` + - The representative value for a small frame MUST have length between 0 and 4096. - Default frame: `length = 4096` + - The representative value for non-framed data MUST have length = 4096. - Large frame: `length > 4096` + - The representative value for non-framed data MUST have length greater than 4096. + +### Representative number of maximum encrypted data keys (EDKs) + +- None: `max EDKs unset` + - The representative value for no configured maxiumum number of EDKs MUST be an unset value. +- Zero: `max EDKs = 0` + - The representative value for zero maximum EDKs MUST have length = 0. +- One: `max EDKs = 1` + - The representative value for one maximum EDK MUST have length = 1. +- Few: `1 < max EDKs < 10` + - The representative value for few maximum EDKs MUST have length between 1 and 10. +- Many: `10 ≤ max EDKs` + - The representative value for many maximum EDKs MUST have length of at least 10. ### Representative plaintext constraints @@ -38,22 +56,35 @@ and [evaluating each configuration's expected result](test-vector-enumeration.md These should ONLY be used if `frame length > 0`. - Empty: `length = 0` -- Small: all plaintexts where (1 < length ≤ 10) + - The representative value for empty framed data MUST have length = 0. +- Small: `0 < length < 10` + - The representative value for small framed data MUST have length between 0 and 10. - If `frame size < 10`, omit this. -- Medium: all plaintexts where (10 < length ≤ 1000) +- Medium: `10 ≤ length < 1000` + - The representative value for medium framed data MUST have length of at least 10 and less than 1000. - If `frame size < 1000`, omit this. -- Large: all plaintexts where (1000 < length ≤ [frame size](#representative-frame-sizes)) -- Largest frame: all plaintexts where (length = frame size) -- Largest frame + partial frame: all plaintexts where (length = frame size + [1 .. frame size)) -- Two largest frames: all plaintexts where (length = 2\*(frame size)) -- Many frames: all plaintexts where (length = 2*(frame size) + [1 .. (frame size)*(maximum # of frames-2)]) -- Maximum frames: all plaintexts where (length = (frame size)\*(maximum # of frames)) +- Large: `1000 ≤ length <` [frame size](#representative-frame-sizes) + - The representative value for large framed data MUST have length of at least 1000 and less than the configured frame size. +- Largest frame: `length = frame size` + - The representative value for the largest frame a MUST have length equal to the configured frame size. +- Largest frame + partial frame: `frame size < length < 2\*(frame size)` + - The representative value for a largest frame plus partial frame a MUST have length between the configured frame size and ((twice the configured frame size) minus one). +- Two largest frames: `length = 2\*(frame size)` + - The representative value for two largest frame a MUST have length equal to twice the configured frame size. +- Many frames: `2*(frame size) < length < (frame size)\*(maximum # of frames)` + - The representative value for many frames a MUST have length between twice the configured frame size and ((the configured maximum number of frames) times (the configured frame size)). +- Maximum frames: `length = (frame size)\*(maximum # of frames)` + - The representative value for maximum frames a MUST have length equal to the configured maximum number of frames times the configured frame size. #### Non-framed data These should ONLY be used if `frame length = 0`. - Empty: `length = 0` -- Small: all plaintexts where (1 < length ≤ 10) -- Medium: all plaintexts where (10 < length ≤ 1000) -- Large: all plaintexts where (1000 < length ≤ 2^32) + - The representative value for empty non-framed data MUST have length equal to 0. +- Small: `0 < length < 10` + - The representative value for small non-framed data MUST have length between 0 and 10. +- Medium: `10 ≤ length < 1000` + - The representative value for empty non-framed data MUST have length of at least 10 and less than 1000. +- Large: `1000 < length ≤ 2^32` + - The representative value for empty non-framed data MUST have length of at least 1000 and less than 2^32. \ No newline at end of file diff --git a/framework/test-vectors/mpl-test-vector-enumeration.md b/framework/test-vectors/mpl-test-vector-enumeration.md index dbb63641..86368b9a 100644 --- a/framework/test-vectors/mpl-test-vector-enumeration.md +++ b/framework/test-vectors/mpl-test-vector-enumeration.md @@ -1,7 +1,7 @@ [//]: # "Copyright Amazon.com Inc. or its affiliates. All Rights Reserved." [//]: # "SPDX-License-Identifier: CC-BY-SA-4.0" -# ESDK Test Vector Enumeration +# MPL Test Vector Enumeration This document performs the [test vector enumeration](test-vector-enumeration.md) process for the MPL to construct the full suite of test vectors diff --git a/framework/test-vectors/test-vector-enumeration.md b/framework/test-vectors/test-vector-enumeration.md index 4208617e..b20c3fb7 100644 --- a/framework/test-vectors/test-vector-enumeration.md +++ b/framework/test-vectors/test-vector-enumeration.md @@ -142,15 +142,15 @@ to warrant the time spent testing them. For example, consider all cases where a hierarchical keyring is encrypting, -but some raw AES key is decrypting. -This is expected have an [expected result](#expected-results) of `negative-decrypt`, -since a hierarchical keyring's encrypt result is incompatible with AES decryption. +but some raw AES keyring is decrypting. +This will have an [expected result](#expected-results) of `negative-decrypt`, +since a hierarchical keyring's encrypt result is incompatible with an AES meyring's decryption. This may be worth validating once with test vectors to ensure this scenario fails with the expected result. However, consider that the enumeration process above would result in a large number of test scenarios -with an encrypting hierarchical keyring and a decrypting raw AES key, +with an encrypting hierarchical keyring and a decrypting raw AES keyring, but with many combinations of [input dimension](#input-dimensions) values. (Consider that a scenario will be written with every combination of [representative plaintext length](esdk-test-vector-enumeration.md#representative-plaintext-constraints), every [commitment policy](../../client-apis/client.md#commitment-policy), every [algorithm suite ID](../algorithm-suites.md#algorithm-suite-id), etc.) From 4ce6d2b9153511ad7aec8b4271dbc4818a92fe6d Mon Sep 17 00:00:00 2001 From: Lucas McDonald Date: Tue, 7 May 2024 16:13:53 -0700 Subject: [PATCH 16/19] cleanup --- .../complete-vectors/hierarchy.md | 16 ++- .../esdk-test-vector-enumeration.md | 97 ++++++++++++------- 2 files changed, 68 insertions(+), 45 deletions(-) diff --git a/framework/test-vectors/complete-vectors/hierarchy.md b/framework/test-vectors/complete-vectors/hierarchy.md index bb625e8b..2ed0db97 100644 --- a/framework/test-vectors/complete-vectors/hierarchy.md +++ b/framework/test-vectors/complete-vectors/hierarchy.md @@ -36,19 +36,17 @@ a test MUST attempt to encrypt and decrypt with every [standard encryption conte ### Representative branch keys - `"static-branch-key-1"` - - Any valid branch key. + - MUST be some valid branch key. - `"static-branch-key-2"` - - Any other valid branch key + - MUST be some valid branch key other than `static-branch-key-1`. - `"branch-key-no-permissions"` - - Any valid branch key where - the test vector runner does not have permissions - for the KMS key + - MUST be some valid branch key where the test vector runner does not have permissions for the KMS key - `"branch-key-not-in-table"` - - Any branch key ID not in the keystore table + - MUST be some branch key ID not in the keystore table - `"branch-key-no-version"` - - A Version of a branch key which does not exist -- `"invalid-branch-key"` - - Any illegally mutated invalid branch key + - MUST be some branch key with an invalid version that does not exist +- `"invalid-branch-key-material"` + - MUST be some branch key with illegally mutated invalid branch key material ### Evaluation rules diff --git a/framework/test-vectors/esdk-test-vector-enumeration.md b/framework/test-vectors/esdk-test-vector-enumeration.md index f3c1828c..93799af5 100644 --- a/framework/test-vectors/esdk-test-vector-enumeration.md +++ b/framework/test-vectors/esdk-test-vector-enumeration.md @@ -27,64 +27,89 @@ and [evaluating each configuration's expected result](test-vector-enumeration.md ### Representative frame sizes -- Non-framed data: `length = 0` - - The representative value for non-framed data MUST have length = 0. -- Small frame: `0 < length < 4096` - - The representative value for a small frame MUST have length between 0 and 4096. -- Default frame: `length = 4096` - - The representative value for non-framed data MUST have length = 4096. -- Large frame: `length > 4096` - - The representative value for non-framed data MUST have length greater than 4096. - -### Representative number of maximum encrypted data keys (EDKs) - -- None: `max EDKs unset` - - The representative value for no configured maxiumum number of EDKs MUST be an unset value. -- Zero: `max EDKs = 0` - - The representative value for zero maximum EDKs MUST have length = 0. -- One: `max EDKs = 1` - - The representative value for one maximum EDK MUST have length = 1. -- Few: `1 < max EDKs < 10` - - The representative value for few maximum EDKs MUST have length between 1 and 10. -- Many: `10 ≤ max EDKs` - - The representative value for many maximum EDKs MUST have length of at least 10. +- Non-framed data: `frame size = 0` + - The representative frame size value for non-framed data + MUST have frame size = 0. +- Small frame: `0 < frame size < 4096` + - The representative value for a small frame + MUST have frame size between 0 and 4096. +- Default frame: `frame size = 4096` + - The representative value for the default frame size + MUST have frame size = 4096. +- Large frame: `frame size > 4096` + - The representative value for a large frame + MUST have frame size greater than 4096. ### Representative plaintext constraints #### Framed data -These should ONLY be used if `frame length > 0`. +These MUST only be used if `frame length > 0`. - Empty: `length = 0` - - The representative value for empty framed data MUST have length = 0. + - The representative plaintext value for empty framed data + MUST have length = 0. - Small: `0 < length < 10` - - The representative value for small framed data MUST have length between 0 and 10. + - The representative plaintext value for small framed data + MUST have length between 0 and 10. - If `frame size < 10`, omit this. - Medium: `10 ≤ length < 1000` - - The representative value for medium framed data MUST have length of at least 10 and less than 1000. + - The representative plaintext value for medium framed data + MUST have length of at least 10 and less than 1000. - If `frame size < 1000`, omit this. - Large: `1000 ≤ length <` [frame size](#representative-frame-sizes) - - The representative value for large framed data MUST have length of at least 1000 and less than the configured frame size. + - The representative plaintext value for large framed data + MUST have length of at least 1000 and less than the configured frame size. - Largest frame: `length = frame size` - - The representative value for the largest frame a MUST have length equal to the configured frame size. + - The representative plaintext value for the largest frame + MUST have length equal to the configured frame size. - Largest frame + partial frame: `frame size < length < 2\*(frame size)` - - The representative value for a largest frame plus partial frame a MUST have length between the configured frame size and ((twice the configured frame size) minus one). + - The representative plaintext value for a largest frame plus partial frame + MUST have length between the configured frame size + and twice the configured frame size. - Two largest frames: `length = 2\*(frame size)` - - The representative value for two largest frame a MUST have length equal to twice the configured frame size. + - The representative plaintext value for two largest frames + MUST have length equal to twice the configured frame size. - Many frames: `2*(frame size) < length < (frame size)\*(maximum # of frames)` - - The representative value for many frames a MUST have length between twice the configured frame size and ((the configured maximum number of frames) times (the configured frame size)). + - The representative plaintext value for many frames + MUST have length between twice the configured frame size + and ((the configured maximum number of frames) times (the configured frame size)). - Maximum frames: `length = (frame size)\*(maximum # of frames)` - - The representative value for maximum frames a MUST have length equal to the configured maximum number of frames times the configured frame size. + - The representative plaintext value for maximum frames + MUST have length equal to the configured maximum number of frames times the configured frame size. #### Non-framed data -These should ONLY be used if `frame length = 0`. +These MUST only be used if `frame length = 0`. - Empty: `length = 0` - - The representative value for empty non-framed data MUST have length equal to 0. + - The representative plaintext value for empty non-framed data + MUST have length equal to 0. - Small: `0 < length < 10` - - The representative value for small non-framed data MUST have length between 0 and 10. + - The representative plaintext value for small non-framed data + MUST have length between 0 and 10. - Medium: `10 ≤ length < 1000` - - The representative value for empty non-framed data MUST have length of at least 10 and less than 1000. + - The representative plaintext value for empty non-framed data + MUST have length of at least 10 and less than 1000. - Large: `1000 < length ≤ 2^32` - - The representative value for empty non-framed data MUST have length of at least 1000 and less than 2^32. \ No newline at end of file + - The representative plaintext value for empty non-framed data + MUST have length of at least 1000 and less than 2^32. + +### Representative number of maximum encrypted data keys (EDKs) + +- No configured value + - The representative value for no configured maxiumum number of EDKs + MUST be some unset value + that is interpreted as "no maximum value" by the ESDK implementation. +- Zero: `max EDKs = 0` + - The representative value for zero maximum EDKs + MUST have length = 0. +- One: `max EDKs = 1` + - The representative value for one maximum EDK + MUST have length = 1. +- Few: `1 < max EDKs < 10` + - The representative value for few maximum EDKs + MUST have length between 1 and 10. +- Many: `10 ≤ max EDKs` + - The representative value for many maximum EDKs + MUST have length of at least 10. \ No newline at end of file From 1e26bb82fa2ecd4bda40bc2847e654524877eaeb Mon Sep 17 00:00:00 2001 From: Lucas McDonald Date: Wed, 8 May 2024 10:24:15 -0700 Subject: [PATCH 17/19] fixes --- .../complete-vectors/hierarchy.md | 18 +++++++++++------ .../required-encryption-context-cmm.md | 12 ++++++++++- .../esdk-test-vector-enumeration.md | 2 +- .../test-vectors/test-vector-enumeration.md | 20 +++++++++++++++++-- 4 files changed, 42 insertions(+), 10 deletions(-) diff --git a/framework/test-vectors/complete-vectors/hierarchy.md b/framework/test-vectors/complete-vectors/hierarchy.md index 2ed0db97..83785c5c 100644 --- a/framework/test-vectors/complete-vectors/hierarchy.md +++ b/framework/test-vectors/complete-vectors/hierarchy.md @@ -24,8 +24,14 @@ a test MUST attempt to encrypt and decrypt with every [standard encryption conte ### Input dimensions -- encrypt key description.key: range of [representative branch keys](#representative-branch-keys) -- decrypt key description +#### Encrypt + +- key description + - key: range of [representative branch keys](#representative-branch-keys) + +#### Decrypt + +- key description - key: range of [representative branch keys](#representative-branch-keys) - logicalKeyStore: - "Default", @@ -40,13 +46,13 @@ a test MUST attempt to encrypt and decrypt with every [standard encryption conte - `"static-branch-key-2"` - MUST be some valid branch key other than `static-branch-key-1`. - `"branch-key-no-permissions"` - - MUST be some valid branch key where the test vector runner does not have permissions for the KMS key + - MUST be some valid branch key where the test vector runner does not have permissions for the KMS key. - `"branch-key-not-in-table"` - - MUST be some branch key ID not in the keystore table + - MUST be some branch key ID not present in the keystore table., - `"branch-key-no-version"` - - MUST be some branch key with an invalid version that does not exist + - MUST be some branch key that is in the table, but the configured version is not in the table. - `"invalid-branch-key-material"` - - MUST be some branch key with illegally mutated invalid branch key material + - MUST be some branch key with illegally mutated invalid branch key material. ### Evaluation rules diff --git a/framework/test-vectors/complete-vectors/required-encryption-context-cmm.md b/framework/test-vectors/complete-vectors/required-encryption-context-cmm.md index bb777fc6..c172aea1 100644 --- a/framework/test-vectors/complete-vectors/required-encryption-context-cmm.md +++ b/framework/test-vectors/complete-vectors/required-encryption-context-cmm.md @@ -79,22 +79,32 @@ For example: set to `{a}`, the only success case for a message to successfully decrypt will be to supply the reproducedEncryptionContext `{a:a}`. -## Test vector input dimensions and ranges +## Input dimensions and ranges +### Encrypt + +- cmm: Adds a `"RequiredEncryptionContext"` allowed value + - MUST add a `"RequiredEncryptionContext"` value to the `"cmm"` input dimension. - required encryption context keys: Range is every [representative required encryption context key](#representative-required-encryption-context-keys) + - MUST test the full range of representative required encryption context keys. - reproduced encryption context: Range is every [representative reproduced encryption context](#representative-reproduced-encryption-context) + - MUST test the full range of representative reproduced encryption context. ### Representative values #### Representative required encryption context keys - Every subset of keys in the provided [encryption context](../../structures.md#encryption-context) + - MUST test every subset of keys in the provided encryption context. - Any key NOT in the provided encryption context. + - MUST test with some additional key that is not in the provided encryption context. #### Representative reproduced encryption context - Every subset of items in the provided [encryption context](../../structures.md#encryption-context) + - MUST test every subset of items in the provided encryption context. - Any item NOT in the provided encryption context. + - MUST test with some additional item that is not in the provided encryption context. ## Test vector evaluation rules diff --git a/framework/test-vectors/esdk-test-vector-enumeration.md b/framework/test-vectors/esdk-test-vector-enumeration.md index 93799af5..0fa8dbfb 100644 --- a/framework/test-vectors/esdk-test-vector-enumeration.md +++ b/framework/test-vectors/esdk-test-vector-enumeration.md @@ -44,7 +44,7 @@ and [evaluating each configuration's expected result](test-vector-enumeration.md #### Framed data -These MUST only be used if `frame length > 0`. +These MUST only be used if `frame size > 0`. - Empty: `length = 0` - The representative plaintext value for empty framed data diff --git a/framework/test-vectors/test-vector-enumeration.md b/framework/test-vectors/test-vector-enumeration.md index b20c3fb7..630190c7 100644 --- a/framework/test-vectors/test-vector-enumeration.md +++ b/framework/test-vectors/test-vector-enumeration.md @@ -27,6 +27,14 @@ every possible input configuration for two reasons: and are not a priority to test at this time (see ["filtering input configurations"](#filtering-input-configurations)) +These reasons are both motivated by test efficiency. +It is infeasible to enumerate every possible plaintext, +or test every invalid encryption configuration. +By strategically identifying test cases to enumerate, +the test vector framework employs orders of magnitude of leverage +to have comprehensive test coverage +with a concise suite of tests. + As a result, the framework does not actually enumerate all possible input configurations and expected results. @@ -43,6 +51,14 @@ An input configuration is a list of key-value pairs where the key represents some [input dimension](#input-dimensions) and the value is one allowed value in that input dimension. +Input configurations for encrypt and decrypt SHOULD be specified separately. +In "positive" test cases, a given input dimension +will likely have the same value in the encrypt and decrypt configurations. +However, in "negative" test cases, a given input dimension +will likely have different values between encrypt and decrypt. +Specifying encrypt and decrypt configurations separately provides flexibility +and cleanly supports negative test cases. + ### Input dimensions An input dimension describes the range of possible values @@ -144,7 +160,7 @@ For example, consider all cases where a hierarchical keyring is encrypting, but some raw AES keyring is decrypting. This will have an [expected result](#expected-results) of `negative-decrypt`, -since a hierarchical keyring's encrypt result is incompatible with an AES meyring's decryption. +since a hierarchical keyring's encrypt result is incompatible with a Raw AES Keyring's decryption. This may be worth validating once with test vectors to ensure this scenario fails with the expected result. @@ -259,7 +275,7 @@ to determine when these vectors should fail. These rules can be relatively informal in the spec, but ideally should be written programmatically, -so a developer can write implement these rules in test manifest generation code. +so a developer can implement these rules in test manifest generation code. Evaluation rules are a useful abstraction to determine a given test scenario's expected result. A list of independent rules is flexible and maintainable. From 4561e45e61c362d013b1197f3b7e5589770404b6 Mon Sep 17 00:00:00 2001 From: Lucas McDonald Date: Wed, 8 May 2024 10:29:35 -0700 Subject: [PATCH 18/19] fix --- .../required-encryption-context-cmm.md | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/framework/test-vectors/complete-vectors/required-encryption-context-cmm.md b/framework/test-vectors/complete-vectors/required-encryption-context-cmm.md index c172aea1..3ed57aa5 100644 --- a/framework/test-vectors/complete-vectors/required-encryption-context-cmm.md +++ b/framework/test-vectors/complete-vectors/required-encryption-context-cmm.md @@ -83,6 +83,17 @@ For example: ### Encrypt +- cmm: Adds a `"RequiredEncryptionContext"` allowed value + - MUST add a `"RequiredEncryptionContext"` value to the `"cmm"` input dimension. +- required encryption context keys: Range is every [representative required encryption context key](#representative-required-encryption-context-keys) + - MUST test the full range of representative required encryption context keys. +- reproduced encryption context: Range is every [representative reproduced encryption context](#representative-reproduced-encryption-context) + - MUST test the full range of representative reproduced encryption context. + +### Decrypt + +These are the same as [Encrypt](#encrypt), but are specified separately so Duvet can link to unique lines for decrypt configuration. + - cmm: Adds a `"RequiredEncryptionContext"` allowed value - MUST add a `"RequiredEncryptionContext"` value to the `"cmm"` input dimension. - required encryption context keys: Range is every [representative required encryption context key](#representative-required-encryption-context-keys) From 061befc4a36009ed3ac714daa2fb4a664a3afb1c Mon Sep 17 00:00:00 2001 From: Lucas McDonald Date: Wed, 8 May 2024 10:29:58 -0700 Subject: [PATCH 19/19] prettier --- framework/test-vectors/esdk-test-vector-enumeration.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/framework/test-vectors/esdk-test-vector-enumeration.md b/framework/test-vectors/esdk-test-vector-enumeration.md index 0fa8dbfb..dbc89990 100644 --- a/framework/test-vectors/esdk-test-vector-enumeration.md +++ b/framework/test-vectors/esdk-test-vector-enumeration.md @@ -112,4 +112,4 @@ These MUST only be used if `frame length = 0`. MUST have length between 1 and 10. - Many: `10 ≤ max EDKs` - The representative value for many maximum EDKs - MUST have length of at least 10. \ No newline at end of file + MUST have length of at least 10.