chore: performance tests for ESDK-python

.gitignore

@@ -33,6 +33,10 @@ __pycache__
 .pytest_cache
 # Ignore key materials generated by examples or tests
 test_keyrings/
+# Ignore results of performance test
+performance_tests/results/*
+# Ignore the memory profile logs
+mprofile_*
 
 # PyCharm
 .idea/

performance_tests/README.md

@@ -0,0 +1,134 @@
+# Performance Tests for ESDK Python
+
+## License
+
+This project is licensed under the Apache-2.0 License.
+
+## Overview
+
+Here are the keyrings / master key-providers that we are testing:
+
+1. KMS Keyring / KMS Master Key Provider
+2. Raw AES Keyring / AES Master Key Provider
+3. HKeyring / caching CMM example ("old" caching solution vs the (current) "new" caching solution)
+4. Raw RSA Keyring / RSA Master Key Provider
+
+For each test on the above keyrings / master key-providers, we measure the execution time and memory consumption in each test.
+
+For each keyring / master key-provider, we test the execution time and memory consumption time for three operations:
+1. Create keyring / master key-provider
+2. Encrypt
+3. Decrypt
+
+We demonstrate the usage of the performance tests through an [AWS KMS Keyring](https://docs.aws.amazon.com/encryption-sdk/latest/developer-guide/use-kms-keyring.html). However, the procedure is the same for any keyring / master key-provider, with slight change in the input arguments.
+
+The results for the performance test will be available in the results folder in the performance_tests directory.
+
+## Usage: Execution Time
+
+### Create Keyring
+To run the performance test for execution time, please use the following commands in the performance_tests directory
+```
+python test/keyrings/test_aws_kms_keyring.py create
+```
+
+#### Optional Arguments
+* kms_key_id: The KMS key ID you want to use
+* n_iters: Number of iterations you want to run the test for. For instance, if n_iters = 100, this performance test script will run the create_keyring method 100 times and report the execution time of each of the calls.
+* output_file: The output file for execution times for each function call, default='kms_keyring_create' in the results folder
+
+#### Consolidate Results
+
+In order to find the minimum, maximum and average times from the n_iters runs, please use the following script from the performance_tests directory:
+```
+python consolidate_results.py results/kms_keyring_create.csv
+```
+
+### Encrypt
+To run the performance test for execution time, please use the following commands in the performance_tests directory
+```
+python test/keyrings/test_aws_kms_keyring.py encrypt
+```
+
+Here, you will receive a prompt on the terminal to specify the plaintext file you want to encrypt. Some example plaintext data files are present in the 'test/resources' directory.
+
+Alternatively, if you want to provide the arguments as flags without using the interactive CLI, you can run the command in the following manner:
+
+```
+python test/keyrings/test_aws_kms_keyring.py encrypt --plaintext_data_filename test/resources/plaintext-data-medium.dat
+```
+
+You can choose to use any other plaintext file as well.
+
+#### Arguments
+* plaintext_data_filename: Filename containing plaintext data you want to encrypt
+
+#### Optional Arguments
+* kms_key_id: The KMS key ID you want to use to encrypt the data
+* n_iters: Number of iterations you want to run the test for. For instance, if n_iters = 100, this performance test script will run the encrypt method 100 times and report the execution time of each of the calls.
+* output_file: The output file for execution times for each function call, default='kms_keyring_encrypt'
+
+#### Consolidate Results
+
+In order to find the minimum, maximum and average times from the n_iters runs, please use the following script from the performance_tests directory:
+```
+python consolidate_results.py results/kms_keyring_encrypt.csv
+```
+
+### Decrypt
+To run the performance test for execution time, please use the following commands in the performance_tests directory
+```
+python test/keyrings/test_aws_kms_keyring.py decrypt
+```
+
+Here, you will receive a prompt on the terminal to specify the ciphertext file you want to decrypt. Some example ciphertext data files are present in the 'test/resources' directory.
+
+Alternatively, if you want to provide the arguments as flags without using the interactive CLI, you can run the command in the following manner:
+
+```
+python test/keyrings/test_aws_kms_keyring.py decrypt --ciphertext_data_filename test/resources/ciphertext-data-medium.ct
+```
+
+You can choose to use any other ciphertext file as well.
+
+#### Arguments
+* ciphertext_data_filename: Filename containing ciphertext data you want to decrypt
+
+#### Optional Arguments
+* kms_key_id: The KMS key ID you want to use to decrypt the data
+* n_iters: Number of iterations you want to run the test for. For instance, if n_iters = 100, this performance test script will run the decrypt method 100 times and report the execution time of each of the calls.
+* output_file: The output file for execution times for each function call, default='kms_keyring_decrypt'
+
+#### Consolidate Results
+
+In order to find the minimum, maximum and average times from the n_iters runs, please use the following script from the performance_tests directory:
+```
+python consolidate_results.py results/kms_keyring_decrypt.csv
+```
+
+## Usage: Memory Consumption
+To get the memory consumption, simply use 'mprof run' instead of 'python' in the previously mentioned commands.
+
+For example, if you want to calculate the memory consumption of the encrypt function of a AWS KMS Keyring, simply write:
+```
+mprof run test/keyrings/test_aws_kms_keyring.py encrypt --plaintext_data_filename test/resources/plaintext-data-medium.dat
+```
+
+This should generate an mprofile log file in your current directory. To plot the memory consumption with time, please use the following command from the same directory
+```
+mprof plot
+```
+
+This 'mprof plot' command will plot the most recent mprofile log file.
+
+## Usage: Performance Graph
+To generate a performance graph, please use the following command to generate the pstats log file by specifying the output pstats file path. Here, we use 'results/kms_keyring_create.pstats' as the output file.
+
+```
+python -m cProfile -o results/kms_keyring_create.pstats test/keyrings/test_aws_kms_keyring.py create
+```
+
+After generating the pstats file, please run the following command to generate the performance graph. The output performance graph will be a .png file that you specify. Here, we use 'results/kms_keyring_create.png' as the output file.
+```
+gprof2dot -f pstats results/kms_keyring_create.pstats | dot -Tpng -o results/kms_keyring_create.png && eog results/kms_keyring_create.png 
+```

performance_tests/__init__.py

@@ -0,0 +1,3 @@
+# Copyright Amazon.com Inc. or its affiliates. All Rights Reserved.
+# SPDX-License-Identifier: Apache-2.0
+"""Stub module indicator to make linter configuration simpler."""

performance_tests/consolidate_results.py

@@ -0,0 +1,47 @@
+# Copyright Amazon.com Inc. or its affiliates. All Rights Reserved.
+# SPDX-License-Identifier: Apache-2.0
+"""Script for consolidating results for execution times"""
+
+import csv
+import argparse
+import numpy as np
+
+
+def calculate_statistics(_csv_file):
+    """Calculate min, max, average and p99 statistics for execution times in a CSV file."""
+    with open(_csv_file, 'r', encoding='utf-8') as file:
+        reader = csv.reader(file)
+        data = [float(row[0]) for row in reader]
+
+    # Calculate statistics
+    if data:
+        data = np.sort(data)
+        _total_entries = len(data)
+        _average = sum(data) / _total_entries
+        _trimmed_average_99 = np.mean(data[0:int(0.99 * len(data))])
+        _minimum = min(data)
+        _maximum = max(data)
+        _perc_99 = np.percentile(data, 99)
+        return _total_entries, _average, _trimmed_average_99, _minimum, _maximum, _perc_99
+
+    return None
+
+
+if __name__ == "__main__":
+    parser = argparse.ArgumentParser()
+    parser.add_argument('csv_file',
+                        help='csv file containing the outputs of execution times for n_iter iterations')
+    args = parser.parse_args()
+
+    statistics = calculate_statistics(args.csv_file)
+    if statistics:
+        total_entries, average, trimmend_average_99, minimum, maximum, perc_99 = statistics
+        print("CSV File:", args.csv_file)
+        print("Total Entries:", total_entries)
+        print("Average:", average)
+        print("99th percentile trimmed average:", trimmend_average_99)
+        print("Minimum:", minimum)
+        print("Maximum:", maximum)
+        print("99th percentile:", perc_99)
+    else:
+        print("No data found in the CSV file.")

performance_tests/pylintrc

@@ -0,0 +1,45 @@
+[MESSAGE CONTROL]
+# Disabling messages that either we don't care about we intentionally break.
+disable =
+    import-error, # ignore mpl import errors
+    invalid-name,  # we prefer long, descriptive, names for examples
+    bad-continuation,  # we let black handle this
+    ungrouped-imports,  # we let isort handle this
+    no-member,  # breaks with attrs
+    no-self-use,  # interesting to keep in mind for later refactoring, but not blocking
+    useless-object-inheritance,  # we need to support Python 2, so no, not useless
+    duplicate-code,  # some examples may be similar
+    too-few-public-methods,  # does not allow value stores
+    too-many-locals,  # examples may sometimes have more locals defined for clarity than would be appropriate in code
+    no-else-return,  # we omit this on purpose for brevity where it would add no value
+    attribute-defined-outside-init,  # breaks with attrs_post_init
+    abstract-method,  # throws false positives on io.BaseIO grandchildren
+    redefined-outer-name,  # we do this on purpose in multiple places
+    consider-using-f-string # disable until 2022-05-05; 6 months after 3.5 deprecation
+
+[BASIC]
+# Allow function names up to 50 characters
+function-rgx = [a-z_][a-z0-9_]{2,50}$
+# Allow method names up to 50 characters
+method-rgx = [a-z_][a-z0-9_]{2,50}$
+# Allow class attribute names up to 50 characters
+# Whitelist class attribute names: iv
+class-attribute-rgx = (([A-Za-z_][A-Za-z0-9_]{2,50}|(__.*__))$)|(^iv$)
+# Whitelist attribute names: iv
+attr-rgx = ([a-z_][a-z0-9_]{2,30}$)|(^iv$)
+# Whitelist argument names: iv, b
+argument-rgx = ([a-z_][a-z0-9_]{2,30}$)|(^iv$)|(^b$)
+# Whitelist variable names: iv, b, _b, x, y, r, s
+variable-rgx = ([a-z_][a-z0-9_]{2,30}$)|(^iv$)|(^b$)|(^_b$)|(^x$)|(^y$)|(^r$)|(^s$)
+
+[VARIABLES]
+additional-builtins = raw_input
+
+[DESIGN]
+max-args = 10
+
+[FORMAT]
+max-line-length = 120
+
+[REPORTS]
+msg-template = {path}:{line}: [{msg_id}({symbol}), {obj}] {msg}

performance_tests/requirements.txt

@@ -0,0 +1,5 @@
+attrs >= 17.4.0
+aws-encryption-sdk>=2.3.0
+pytest>=3.3.1
+tqdm
+click

performance_tests/requirements_mpl.txt

@@ -0,0 +1 @@
+aws-cryptographic-material-providers @ git+https://github.com/aws/aws-cryptographic-material-providers-library.git@lucmcdon/python-mpl#subdirectory=AwsCryptographicMaterialProviders/runtimes/python

performance_tests/setup.cfg

@@ -0,0 +1,41 @@
+[wheel]
+universal = 1
+
+[metadata]
+license_file = LICENSE
+
+[coverage:run]
+branch = True
+
+[coverage:report]
+show_missing = True
+
+[mypy]
+ignore_missing_imports = True
+
+[flake8]
+max_complexity = 10
+max_line_length = 120
+import_order_style = google
+application_import_names = aws_encryption_sdk_cli
+builtins = raw_input
+ignore =
+    # Ignoring D205 and D400 because of false positives
+    D205, D400,
+    # E203 is not PEP8 compliant https://github.com/ambv/black#slices
+    E203,
+    # W503 is not PEP8 compliant https://github.com/ambv/black#line-breaks--binary-operators
+    W503
+
+[doc8]
+max-line-length = 120
+
+[isort]
+line_length = 120
+# https://github.com/timothycrosley/isort#multi-line-output-modes
+multi_line_output = 3
+include_trailing_comma = True
+force_grid_wrap = 0
+combine_as_imports = True
+not_skip = __init__.py
+known_third_party = attr,aws_encryption_sdk,pytest,setuptools,six

performance_tests/setup.py

@@ -0,0 +1,34 @@
+"""Performance test for the AWS Encryption SDK for Python."""
+import os
+import re
+
+from setuptools import find_packages, setup
+
+VERSION_RE = re.compile(r"""__version__ = ['"]([0-9.]+)['"]""")
+HERE = os.path.abspath(os.path.dirname(__file__))
+
+
+def read(*args):
+    """Read complete file contents."""
+    return open(os.path.join(HERE, *args), encoding="utf-8").read()  # pylint: disable=consider-using-with
+
+
+def get_version():
+    """Read the version from this module."""
+    init = read("src", "aws_encryption_sdk_performance_tests", "__init__.py")
+    return VERSION_RE.search(init).group(1)
+
+
+setup(
+    name="aws-encryption-sdk-performance-tests",
+    packages=find_packages("src"),
+    package_dir={"": "src"},
+    author="Amazon Web Services",
+    maintainer="Amazon Web Services",
+    author_email="[email protected]",
+    url="https://github.com/awslabs/aws-encryption-sdk-python",
+    description="Performance tests for the AWS Encryption SDK for Python",
+    keywords="aws-encryption-sdk aws kms encryption",
+    license="Apache License 2.0",
+    version=get_version(),
+)

performance_tests/src/aws_encryption_sdk_performance_tests/__init__.py

@@ -0,0 +1,4 @@
+# Copyright Amazon.com Inc. or its affiliates. All Rights Reserved.
+# SPDX-License-Identifier: Apache-2.0
+"""Stub module indicator to make linter configuration simpler."""
+__version__ = "0.0.0"

performance_tests/src/aws_encryption_sdk_performance_tests/keyrings/__init__.py

@@ -0,0 +1,3 @@
+# Copyright Amazon.com Inc. or its affiliates. All Rights Reserved.
+# SPDX-License-Identifier: Apache-2.0
+"""Stub module indicator to make linter configuration simpler."""