Skip to content

Commit 8fba0cc

Browse files
RitvikKapilaRitvikKapila
committed
updated README; refactoring
1 parent 457d20d commit 8fba0cc

File tree

8 files changed

+263
-132
lines changed

8 files changed

+263
-132
lines changed

‎.gitignore

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -34,9 +34,9 @@ __pycache__
3434
# Ignore key materials generated by examples or tests
3535
test_keyrings/
3636
# Ignore results of performance test
37-
performance_tests/results/*
38-
# To track the results folder, keep the __init__.py file
39-
!performance_tests/results/__init__.py
37+
performance_tests/results/*.csv
38+
performance_tests/results/*.pstats
39+
performance_tests/results/*.png
4040
# Ignore the memory profile logs
4141
mprofile_*
4242

‎performance_tests/README.rst

Lines changed: 185 additions & 88 deletions
Original file line numberDiff line numberDiff line change
@@ -1,135 +1,232 @@
1-
# aws-encryption-sdk performance tests
1+
#####################################
2+
aws-encryption-sdk performance tests
3+
#####################################
24

3-
## License
5+
This module runs performance tests for the `AWS Encryption SDK Python`_.
46

5-
This project is licensed under the Apache-2.0 License.
7+
********
8+
Overview
9+
********
610

7-
## Overview
8-
9-
This library tests the following keyrings / master key-providers:
11+
This module tests the following keyrings / master key-providers:
1012

1113
1. KMS Keyring / KMS Master Key Provider
1214
2. Raw AES Keyring / AES Master Key Provider
1315
3. Raw RSA Keyring / RSA Master Key Provider
1416
4. Hierarchy Keyring
1517
5. Caching CMM
1618

17-
For each test on the above keyrings / master key-providers, this package measures the execution time and memory consumption.
19+
For each test on the above keyrings / master key-providers, this package measures:
20+
21+
1. Execution time
22+
2. Total memory consumption
23+
24+
For each keyring / master key-provider, the execution time and memory consumption
25+
is measured for three operations:
1826

19-
For each keyring / master key-provider, the execution time and memory consumption time is measured for three operations:
2027
1. Create keyring / master key-provider
2128
2. Encrypt
2229
3. Decrypt
2330

24-
The usage of the performance tests is demonstrated through an [AWS KMS Keyring](https://docs.aws.amazon.com/encryption-sdk/latest/developer-guide/use-kms-keyring.html). However, the procedure is the same for any keyring / master key-provider, with slight change in the input arguments.
31+
The usage of the performance tests is demonstrated through an `AWS KMS Keyring`_.
32+
However, the procedure is the same for any keyring / master key-provider, with slight
33+
changes in the input arguments.
34+
35+
The results for the performance test will be available in the results folder in the
36+
performance_tests directory.
37+
38+
**********************
39+
Required Prerequisites
40+
**********************
41+
42+
* Python 3.11+
43+
* aws-encryption-sdk
44+
45+
*****
46+
Usage
47+
*****
48+
49+
Execution Time
50+
==============
51+
52+
Create Keyring
53+
--------------
54+
To run the performance test for execution time, please use the
55+
following commands in the performance_tests directory.
56+
57+
.. code::
58+
59+
usage: python test/keyrings/test_aws_kms_keyring.py create
60+
61+
Create a keyring to use for encryption and decryption.
62+
63+
optional arguments:
64+
-h, --help show this help message and exit.
65+
--kms_key_id KMS_KEY_ID The KMS key ID you want to use.
66+
--n_iters N_ITERS Number of iterations you want to
67+
run the test for. For instance,
68+
if n_iters = 100, this performance
69+
test script will run the create_keyring
70+
method 100 times and report the
71+
execution time of each of the calls.
72+
--output_file OUTPUT_FILE The output file for execution times
73+
for each function call,
74+
default='kms_keyring_create' in the
75+
results folder.
76+
77+
78+
Consolidate Results
79+
~~~~~~~~~~~~~~~~~~~
80+
81+
In order to find the minimum, maximum, average, 99th percentile and bottom
82+
99th percentile trimmed average times from the n_iters runs, please use the
83+
following script from the performance_tests directory:
2584

26-
The results for the performance test will be available in the results folder in the performance_tests directory.
85+
.. code::
2786
28-
## Usage: Execution Time
87+
usage: python consolidate_results.py results/kms_keyring_create.csv
2988
30-
### Create Keyring
31-
To run the performance test for execution time, please use the following commands in the performance_tests directory
32-
```
33-
python test/keyrings/test_aws_kms_keyring.py create
34-
```
3589
36-
#### Optional Arguments
37-
* kms_key_id: The KMS key ID you want to use
38-
* n_iters: Number of iterations you want to run the test for. For instance, if n_iters = 100, this performance test script will run the create_keyring method 100 times and report the execution time of each of the calls.
39-
* output_file: The output file for execution times for each function call, default='kms_keyring_create' in the results folder
90+
Encrypt
91+
-------
4092

41-
#### Consolidate Results
93+
To run the performance test for execution time, please use the following
94+
commands in the performance_tests directory:
4295

43-
In order to find the minimum, maximum and average times from the n_iters runs, please use the following script from the performance_tests directory:
44-
```
45-
python consolidate_results.py results/kms_keyring_create.csv
46-
```
96+
.. code::
4797
48-
### Encrypt
49-
To run the performance test for execution time, please use the following commands in the performance_tests directory
50-
```
51-
python test/keyrings/test_aws_kms_keyring.py encrypt
52-
```
98+
usage: python test/keyrings/test_aws_kms_keyring.py encrypt
5399
54-
Here, you will receive a prompt on the terminal to specify the plaintext file you want to encrypt. Some example plaintext data files are present in the 'test/resources' directory.
100+
optional arguments:
101+
-h, --help show this help message and exit.
102+
--plaintext_data_filename PLAINTEXT_DATA_FILENAME Filename containing plaintext data
103+
you want to encrypt.
104+
default='test/resources/plaintext/plaintext-data-medium.dat'.
105+
You can choose to use any other plaintext
106+
file as well. Some example plaintext
107+
data files are present in the
108+
'test/resources' directory.
109+
--kms_key_id KMS_KEY_ID The KMS key ID you want to use.
110+
--n_iters N_ITERS Number of iterations you want to
111+
run the test for. For instance,
112+
if n_iters = 100, this performance
113+
test script will run the create_keyring
114+
method 100 times and report the
115+
execution time of each of the calls.
116+
--output_file OUTPUT_FILE The output file for execution times
117+
for each function call,
118+
default='kms_keyring_create' in the
119+
results folder.
55120
56-
Alternatively, if you want to provide the arguments as flags without using the interactive CLI, you can run the command in the following manner:
121+
Consolidate Results
122+
~~~~~~~~~~~~~~~~~~~
57123

58-
```
59-
python test/keyrings/test_aws_kms_keyring.py encrypt --plaintext_data_filename test/resources/plaintext-data-medium.dat
60-
```
124+
In order to find the minimum, maximum, average, 99th percentile and bottom
125+
99th percentile trimmed average times from the n_iters runs, please use the
126+
following script from the performance_tests directory:
61127

62-
You can choose to use any other plaintext file as well.
128+
.. code::
63129
64-
#### Arguments
65-
* plaintext_data_filename: Filename containing plaintext data you want to encrypt
130+
usage: python consolidate_results.py results/kms_keyring_encrypt.csv
66131
67-
#### Optional Arguments
68-
* kms_key_id: The KMS key ID you want to use to encrypt the data
69-
* n_iters: Number of iterations you want to run the test for. For instance, if n_iters = 100, this performance test script will run the encrypt method 100 times and report the execution time of each of the calls.
70-
* output_file: The output file for execution times for each function call, default='kms_keyring_encrypt'
71132
72-
#### Consolidate Results
133+
Decrypt
134+
-------
73135

74-
In order to find the minimum, maximum and average times from the n_iters runs, please use the following script from the performance_tests directory:
75-
```
76-
python consolidate_results.py results/kms_keyring_encrypt.csv
77-
```
136+
To run the performance test for execution time, please use the
137+
following commands in the performance_tests directory
78138

79-
### Decrypt
80-
To run the performance test for execution time, please use the following commands in the performance_tests directory
81-
```
82-
python test/keyrings/test_aws_kms_keyring.py decrypt
83-
```
139+
.. code::
84140
85-
Here, you will receive a prompt on the terminal to specify the ciphertext file you want to decrypt. Some example ciphertext data files are present in the 'test/resources' directory.
141+
usage: python test/keyrings/test_aws_kms_keyring.py decrypt
86142
87-
Alternatively, if you want to provide the arguments as flags without using the interactive CLI, you can run the command in the following manner:
143+
optional arguments:
144+
-h, --help show this help message and exit.
145+
--ciphertext_data_filename CIPHERTEXT_DATA_FILENAME Filename containing ciphertext data
146+
you want to decrypt.
147+
default='test/resources/ciphertext/kms/ciphertext-data-medium.ct'.
148+
You can choose to use any other
149+
ciphertext file as well. Some example
150+
ciphertext data files are present in
151+
the 'test/resources' directory.
152+
--kms_key_id KMS_KEY_ID The KMS key ID you want to use.
153+
--n_iters N_ITERS Number of iterations you want to
154+
run the test for. For instance,
155+
if n_iters = 100, this performance
156+
test script will run the create_keyring
157+
method 100 times and report the
158+
execution time of each of the calls.
159+
--output_file OUTPUT_FILE The output file for execution times
160+
for each function call,
161+
default='kms_keyring_create' in the
162+
results folder.
88163
89-
```
90-
python test/keyrings/test_aws_kms_keyring.py decrypt --ciphertext_data_filename test/resources/ciphertext-data-medium.ct
91-
```
164+
Consolidate Results
165+
~~~~~~~~~~~~~~~~~~~
92166

93-
You can choose to use any other ciphertext file as well.
167+
In order to find the minimum, maximum, average, 99th percentile and bottom
168+
99th percentile trimmed average times from the n_iters runs, please use the
169+
following script from the performance_tests directory:
94170

95-
#### Arguments
96-
* ciphertext_data_filename: Filename containing ciphertext data you want to decrypt
171+
.. code::
97172
98-
#### Optional Arguments
99-
* kms_key_id: The KMS key ID you want to use to decrypt the data
100-
* n_iters: Number of iterations you want to run the test for. For instance, if n_iters = 100, this performance test script will run the decrypt method 100 times and report the execution time of each of the calls.
101-
* output_file: The output file for execution times for each function call, default='kms_keyring_decrypt'
173+
usage: python consolidate_results.py results/kms_keyring_decrypt.csv
102174
103-
#### Consolidate Results
175+
Memory Consumption
176+
==================
104177

105-
In order to find the minimum, maximum and average times from the n_iters runs, please use the following script from the performance_tests directory:
106-
```
107-
python consolidate_results.py results/kms_keyring_decrypt.csv
108-
```
178+
To get the memory consumption, simply replace 'python'
179+
with 'mprof run' in the previously mentioned commands.
109180

110-
## Usage: Memory Consumption
111-
To get the memory consumption, simply use 'mprof run' instead of 'python' in the previously mentioned commands.
181+
For example, if you want to calculate the memory consumption
182+
of the encrypt function of a AWS KMS Keyring, simply write:
112183

113-
For example, if you want to calculate the memory consumption of the encrypt function of a AWS KMS Keyring, simply write:
114-
```
115-
mprof run test/keyrings/test_aws_kms_keyring.py encrypt --plaintext_data_filename test/resources/plaintext-data-medium.dat
116-
```
184+
.. code::
185+
186+
usage: mprof run test/keyrings/test_aws_kms_keyring.py encrypt
187+
188+
189+
This should generate an mprofile log file in your current directory.
190+
This mprofile log file contains the total memory consumed by the program
191+
with respect to time elapsed.
192+
To plot the memory consumption with respect to time, please use the following
193+
command from the same directory
194+
195+
.. code::
196+
197+
usage: mprof plot
117198
118-
This should generate an mprofile log file in your current directory. To plot the memory consumption with time, please use the following command from the same directory
119-
```
120-
mprof plot
121-
```
122199
123200
This 'mprof plot' command will plot the most recent mprofile log file.
124201

125-
## Usage: Performance Graph
126-
To generate a performance graph, please use the following command to generate the pstats log file by specifying the output pstats file path. Here, 'results/kms_keyring_create.pstats' is set as the default output file.
127202

128-
```
129-
python -m cProfile -o results/kms_keyring_create.pstats test/keyrings/test_aws_kms_keyring.py create
130-
```
203+
Performance Graph
204+
=================
205+
206+
To generate a performance graph, please use the following command
207+
to generate the pstats log file by specifying the output pstats file
208+
path. Here, 'results/kms_keyring_create.pstats' is set as the default
209+
output file.
210+
211+
.. code::
212+
213+
usage: python -m cProfile -o results/kms_keyring_create.pstats test/keyrings/test_aws_kms_keyring.py create
214+
215+
216+
After generating the pstats file, please run the following command
217+
to generate the performance graph. The output performance graph will
218+
be a .png file that you specify. Here, 'results/kms_keyring_create.png'
219+
is set as the default output file.
220+
221+
.. code::
222+
223+
usage: gprof2dot -f pstats results/kms_keyring_create.pstats | dot -Tpng -o results/kms_keyring_create.png && eog results/kms_keyring_create.png
224+
225+
226+
Note: This project does not adhere to semantic versioning; as such it
227+
makes no guarantees that functionality will persist across major,
228+
minor, or patch versions.
229+
**DO NOT** take a standalone dependency on this library.
131230

132-
After generating the pstats file, please run the following command to generate the performance graph. The output performance graph will be a .png file that you specify. Here, 'results/kms_keyring_create.png' is set as the default output file.
133-
```
134-
gprof2dot -f pstats results/kms_keyring_create.pstats | dot -Tpng -o results/kms_keyring_create.png && eog results/kms_keyring_create.png
135-
```
231+
.. _AWS Encryption SDK Python: https://github.com/aws/aws-encryption-sdk-python/
232+
.. _AWS KMS Keyring: https://docs.aws.amazon.com/encryption-sdk/latest/developer-guide/use-kms-keyring.html

‎performance_tests/results/.gitkeep

Whitespace-only changes.

‎performance_tests/results/__init__.py

Lines changed: 0 additions & 3 deletions
This file was deleted.

‎performance_tests/src/aws_encryption_sdk_performance_tests/__init__.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
11
# Copyright Amazon.com Inc. or its affiliates. All Rights Reserved.
22
# SPDX-License-Identifier: Apache-2.0
33
"""Stub module indicator to make linter configuration simpler."""
4-
__version__ = "0.0.0"
4+
__version__ = "0.1.0"

‎performance_tests/src/aws_encryption_sdk_performance_tests/keyrings/aws_kms_keyring.py

Lines changed: 6 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -23,7 +23,7 @@ def create_keyring(
2323
https://docs.aws.amazon.com/kms/latest/developerguide/concepts.html#key-id
2424
"""
2525
# Create a boto3 client for KMS.
26-
kms_client = boto3.client('kms', region_name="us-west-2")
26+
kms_client = create_kms_client()
2727

2828
# Create a KMS keyring
2929
mat_prov: AwsCryptographicMaterialProviders = AwsCryptographicMaterialProviders(
@@ -42,13 +42,15 @@ def create_keyring(
4242
return keyring
4343

4444

45-
def create_kms_client():
45+
def create_kms_client(aws_region="us-west-2"):
4646
"""Create an AWS KMS client.
4747
48-
Usage: create_kms_client()
48+
Usage: create_kms_client(aws_region)
49+
:param aws_region: AWS region to use for KMS client.
50+
:type aws_region: string
4951
"""
5052
# Create a boto3 client for KMS.
51-
kms_client = boto3.client('kms', region_name="us-west-2")
53+
kms_client = boto3.client('kms', region_name=aws_region)
5254

5355
return kms_client
5456

0 commit comments

Comments
 (0)