Skip to content

Make the experience running tests outside of tox friendlier #46

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 3 commits into from
Apr 4, 2018
Merged

Make the experience running tests outside of tox friendlier #46

merged 3 commits into from
Apr 4, 2018

Conversation

mattsb42-aws
Copy link
Member

@mattsb42-aws mattsb42-aws commented Mar 23, 2018 •
edited
Loading

Seeing this[1], I realized two things:

  1. It is not currently very easy to run our tests outside of tox.
  2. Some of our integration tests look for the CMK ARN environment variable during setup rather than execution. This can cause problems when attempting to run non-integ tests using pytest markers because the setup code is run whether you want those markers or not.

This change fixes both of these issues.

[1] https://github.com/pyca/cryptography/blob/master/Jenkinsfile#L151-L165

@mattsb42-aws mattsb42-aws requested a review from a team March 23, 2018 22:13
@mattsb42-aws
tests that do not need to talk to KMS should not need the CMK environ…
730b25d 
…ment variable to be set: some of the integ tests were looking for this in setup instead of execution, triggering failures even if the pytest markers for those tests were not requested
@mattsb42-aws
* consolidate tox testenv command boilerplate
de517dd 
* add testenv that does not pass through or set any environment variables to ensure that local tests work without them
arealjim
arealjim approved these changes Apr 4, 2018
View reviewed changes
Copy link

@arealjim arealjim left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good. Ship it!

@mattsb42-aws mattsb42-aws merged commit cdf85a7 into aws:master Apr 4, 2018
@mattsb42-aws mattsb42-aws deleted the friendlier-tests branch April 4, 2018 19:56
mattsb42-aws added a commit to mattsb42-aws/cryptography that referenced this pull request Apr 5, 2018
@mattsb42-aws
simplify Jenkins test run taking advantage of aws/aws-encryption-sdk-…
76e1a7d 
…python#46
@mattsb42-aws mattsb42-aws mentioned this pull request Apr 5, 2018
Merged
reaperhulk pushed a commit to pyca/cryptography that referenced this pull request Apr 7, 2018
@mattsb42-aws @reaperhulk
simplify Jenkins test run taking advantage of aws/aws-encryption-sdk-…
2be4390 
…python#46 (#4185)
@mattsb42-aws mattsb42-aws mentioned this pull request Apr 19, 2018
Merged
clrpackages pushed a commit to clearlinux-pkgs/cryptography that referenced this pull request Jul 19, 2018
@fenrus75 @djklimes
[update to 2.3] fix encoding BMPString in x509 name entries (#4321)
964f734 
Alex Gaynor (36):
      Revert "don't allow GeneralNames to be an empty list (#4128)" (#4161)
      forward port 2.2.1 changelog (#4167)
      updated link to sphinx docs (#4181)
      Fixed links to sphinx docs (#4182)
      Remove version number from an error message so that it makes sense (#4188)
      temporarily fix tests (#4195)
      Update URLs for new pypi! (#4194)
      Remove CDN purging, the new PyPI does the right thing (#4198)
      Revert #4195 (#4201)
      Remove setup.py branch (#4203)
      bump openssl version in travis (#4204)
      Remove cffi branch for pypy that's not needed (#4209)
      update link to draft rfc (#4214)
      Use a checklist for bumping openssl version (#4221)
      fixed variable name to be correct (#4226)
      Fixed some confusing type descriptions in docs (#4231)
      Run no longer used debugging output from travis (#4233)
      Remove macOS travis stuff, we don't intend to reenable it (#4244)
      satisfy shellcheck (#4247)
      Only send coverage for tox builds, in prep for #4228 (#4248)
      Fixes #4228 -- move downstream builders to travis (#4250)
      Removes branches in bindings for various OpenSSL 1.1.0 prereleases (#4269)
      Test certbot with OpenSSL 1.1.0 (#4278)
      Correct pass bytes; refs #4289 (#4291)
      Correctly pass bytes; refs #4289 (#4290)
      Perform an OPENSSL_cleanup before checking the heap in our memleak tests (#4293)
      Fixes #4242 -- added an additional assert to make this test more resillient (#4308)
      Don't change the cwd in travis scripts (#4309)
      Make the docs clearer on why truncated tags are a bad idea (#4312)
      Make our locking setup function compy with our style guide (#4316)
      Convert a pair of asserts to use openssl_assert (#4318)
      bump to latest libressl versions (#4329)
      Change the exception we raise in keywrap unwrapping on invalid length (#4337)
      Refs #3331 -- added initial wycheproof integration, starting with x25519, rsa, and keywrap (#4310)
      Refs #3331 -- integrated wycheproof ECDSA tests (#4341)
      Refs #3331 -- integrated wycheproof ECDH tests (#4354)

Amaury Forgeot d'Arc (1):
      Add Session functions, necessary to implement new features in Python 3.6. (#4205)

David Benjamin (12):
      Remove unused BN bindings. (#4219)
      Fix some callback type signatures. (#4227)
      Check for CMAC_Init errors. (#4232)
      Remove unused BIO bindings. (#4220)
      Clean up unused EC bindings. (#4225)
      Remove some unused RAND bindings. (#4239)
      Fix some stuttering. (#4240)
      Validate the public/private halves of EC keys on import. (#4241)
      Remove some unused RSA bindings. (#4243)
      Remove ECDSA_sign_setup and *sign_ex bindings. (#4245)
      Cleanup unused err bindings. (#4246)
      Revert the const bits of #4220. (#4276)

Denis Lila (1):
      add custom extensions functions for openssl >=1.0.2 (#4202)

Eric Brown (2):
      Future proofing use of the six python version constants (#4238)
      Add testing support of Python 3.7 (#4305)

Jon Dufresne (1):
      Document project as stable and ready for use in production (#4284)

Joshua Crowgey (4):
      Added badtime.pem vector (#4179)
      Raise ve on bad gt (#4180)
      add X509_NAME_print_ex (#4174)
      adding name so that 1.3.6.1.4.1.11129.2.4.2 is no longer and 'Unknown OID' (#4218)

Justin Holmes (1):
      Updated pip wheel option in installation script. (#4212)

Marti Raudsepp (2):
      Add OID for RSASSA-PSS X.509 signature algorithm (RFC 4055) (#4294)
      Make RelativeDistinguishedName preserve attribtue order (#4306)

Matt Bullock (2):
      simplify Jenkins test run taking advantage of aws/aws-encryption-sdk-python#46 (#4185)
      add downstream tests for awslabs/aws-dynamodb-encryption-python (#4280)

Paul Kehrer (42):
      open master for 2.3 (#4151)
      add botan's AESKWP vectors reformatted for our NIST loader (#4159)
      fix bug with n % 8 length wrapping on AESKWP (#4160)
      Add urllib3 to downstream tests (#4165)
      port changelog for 2.2.2 (#4172)
      update the NIST keywrap vectors (#4191)
      add SHA3 and SHAKE vectors (#4213)
      Add support for extracting timestamp from a Fernet token (#4229)
      switch to py3 on docs job (#4230)
      Add SHA512/224 and SHA512/256 test vectors from NIST CAVP (#4237)
      remove block size as a required part of HashAlgorithm (#4249)
      build and test libre on travis (#4256)
      deprecate pythons without hmac.compare_digest (#4261)
      parametrize a few things in test_ec (#4268)
      simplify and parametrize DSA tests (#4267)
      LibreSSL 2.7.x support (#4270)
      Add py37 (#4298)
      6 and 7 are right next to each other (#4302)
      since the generator order is 570 bits this should be 570 (#4307)
      set an OPENSSL_API_COMPAT level (#4313)
      reduce number of deprecated signer/verifier calls in test_rsa (#4314)
      document that an ECPublicNumbers object has some unexpected properties (#4319)
      fix encoding BMPString in x509 name entries (#4321)
      fix a memory leak when calling X25519PrivateKey.public_key() (#4326)
      we don't actually care about the errstack here, it's an invalid signature (#4325)
      don't install docs when we build openssls...and do it parallel (#4327)
      try compiling with asm for our custom openssl (#4328)
      raise valueerror for null x25519 derived keys (#4332)
      switch cryptography wheel builders back to pip wheel (#4334)
      document one shot AEAD length restrictions (#4322)
      add crl.get_revoked_certificate method (#4331)
      add DSA wycheproof tests (#4346)
      add aes cbc pkcs5 wycheproof tests (#4347)
      disallow implicit tag truncation with finalize_with_tag (#4342)
      add wycheproof tests for AES CMAC (#4344)
      add chacha20poly1305 wycheproof tests (#4345)
      raise ValueError on zero length GCM IV (#4348)
      also check iv length for GCM nonce in AEAD (#4350)
      min_tag_length is an int (#4351)
      add wycheproof gcm tests (#4349)
      improve skip msg when skipping an ECDH test in test_ec (#4355)
      bump version and changelog for 2.3 release (#4356)

Quinten Stokkink (1):
      Exposed OpenSSL prime methods (#4292)

Thierry Bastian (1):
      OpenSSL 1.0.2o has switched to winsock2 (#4184)

Thom Dixon (1):
      Make AuthorityKeyIdentifier docs reflect reality (#4252)

Tim D. Smith (1):
      Fix typo (#4178)

Tux (1):
      Expose OpenSSL constant time bignum arithmetic (#4200)

Ville Skyttä (1):
      Use pytest instead of py.test per upstream recommendation, #dropthedot (#4236)

Vladyslav Moisieienkov (1):
      Add clearer message when key type is not bytes (#4289)

dumol (1):
      Fixed build errors on HP-UX. (#4259)

Коренберг Марк (1):
      Add serialisation output examples (#4286)
clrpackages pushed a commit to clearlinux-pkgs/cryptography_vectors that referenced this pull request Jul 19, 2018
@fenrus75 @djklimes
[update to 2.3] fix encoding BMPString in x509 name entries (#4321)
a2a45ef 
Alex Gaynor (36):
      Revert "don't allow GeneralNames to be an empty list (#4128)" (#4161)
      forward port 2.2.1 changelog (#4167)
      updated link to sphinx docs (#4181)
      Fixed links to sphinx docs (#4182)
      Remove version number from an error message so that it makes sense (#4188)
      temporarily fix tests (#4195)
      Update URLs for new pypi! (#4194)
      Remove CDN purging, the new PyPI does the right thing (#4198)
      Revert #4195 (#4201)
      Remove setup.py branch (#4203)
      bump openssl version in travis (#4204)
      Remove cffi branch for pypy that's not needed (#4209)
      update link to draft rfc (#4214)
      Use a checklist for bumping openssl version (#4221)
      fixed variable name to be correct (#4226)
      Fixed some confusing type descriptions in docs (#4231)
      Run no longer used debugging output from travis (#4233)
      Remove macOS travis stuff, we don't intend to reenable it (#4244)
      satisfy shellcheck (#4247)
      Only send coverage for tox builds, in prep for #4228 (#4248)
      Fixes #4228 -- move downstream builders to travis (#4250)
      Removes branches in bindings for various OpenSSL 1.1.0 prereleases (#4269)
      Test certbot with OpenSSL 1.1.0 (#4278)
      Correct pass bytes; refs #4289 (#4291)
      Correctly pass bytes; refs #4289 (#4290)
      Perform an OPENSSL_cleanup before checking the heap in our memleak tests (#4293)
      Fixes #4242 -- added an additional assert to make this test more resillient (#4308)
      Don't change the cwd in travis scripts (#4309)
      Make the docs clearer on why truncated tags are a bad idea (#4312)
      Make our locking setup function compy with our style guide (#4316)
      Convert a pair of asserts to use openssl_assert (#4318)
      bump to latest libressl versions (#4329)
      Change the exception we raise in keywrap unwrapping on invalid length (#4337)
      Refs #3331 -- added initial wycheproof integration, starting with x25519, rsa, and keywrap (#4310)
      Refs #3331 -- integrated wycheproof ECDSA tests (#4341)
      Refs #3331 -- integrated wycheproof ECDH tests (#4354)

Amaury Forgeot d'Arc (1):
      Add Session functions, necessary to implement new features in Python 3.6. (#4205)

David Benjamin (12):
      Remove unused BN bindings. (#4219)
      Fix some callback type signatures. (#4227)
      Check for CMAC_Init errors. (#4232)
      Remove unused BIO bindings. (#4220)
      Clean up unused EC bindings. (#4225)
      Remove some unused RAND bindings. (#4239)
      Fix some stuttering. (#4240)
      Validate the public/private halves of EC keys on import. (#4241)
      Remove some unused RSA bindings. (#4243)
      Remove ECDSA_sign_setup and *sign_ex bindings. (#4245)
      Cleanup unused err bindings. (#4246)
      Revert the const bits of #4220. (#4276)

Denis Lila (1):
      add custom extensions functions for openssl >=1.0.2 (#4202)

Eric Brown (2):
      Future proofing use of the six python version constants (#4238)
      Add testing support of Python 3.7 (#4305)

Jon Dufresne (1):
      Document project as stable and ready for use in production (#4284)

Joshua Crowgey (4):
      Added badtime.pem vector (#4179)
      Raise ve on bad gt (#4180)
      add X509_NAME_print_ex (#4174)
      adding name so that 1.3.6.1.4.1.11129.2.4.2 is no longer and 'Unknown OID' (#4218)

Justin Holmes (1):
      Updated pip wheel option in installation script. (#4212)

Marti Raudsepp (2):
      Add OID for RSASSA-PSS X.509 signature algorithm (RFC 4055) (#4294)
      Make RelativeDistinguishedName preserve attribtue order (#4306)

Matt Bullock (2):
      simplify Jenkins test run taking advantage of aws/aws-encryption-sdk-python#46 (#4185)
      add downstream tests for awslabs/aws-dynamodb-encryption-python (#4280)

Paul Kehrer (42):
      open master for 2.3 (#4151)
      add botan's AESKWP vectors reformatted for our NIST loader (#4159)
      fix bug with n % 8 length wrapping on AESKWP (#4160)
      Add urllib3 to downstream tests (#4165)
      port changelog for 2.2.2 (#4172)
      update the NIST keywrap vectors (#4191)
      add SHA3 and SHAKE vectors (#4213)
      Add support for extracting timestamp from a Fernet token (#4229)
      switch to py3 on docs job (#4230)
      Add SHA512/224 and SHA512/256 test vectors from NIST CAVP (#4237)
      remove block size as a required part of HashAlgorithm (#4249)
      build and test libre on travis (#4256)
      deprecate pythons without hmac.compare_digest (#4261)
      parametrize a few things in test_ec (#4268)
      simplify and parametrize DSA tests (#4267)
      LibreSSL 2.7.x support (#4270)
      Add py37 (#4298)
      6 and 7 are right next to each other (#4302)
      since the generator order is 570 bits this should be 570 (#4307)
      set an OPENSSL_API_COMPAT level (#4313)
      reduce number of deprecated signer/verifier calls in test_rsa (#4314)
      document that an ECPublicNumbers object has some unexpected properties (#4319)
      fix encoding BMPString in x509 name entries (#4321)
      fix a memory leak when calling X25519PrivateKey.public_key() (#4326)
      we don't actually care about the errstack here, it's an invalid signature (#4325)
      don't install docs when we build openssls...and do it parallel (#4327)
      try compiling with asm for our custom openssl (#4328)
      raise valueerror for null x25519 derived keys (#4332)
      switch cryptography wheel builders back to pip wheel (#4334)
      document one shot AEAD length restrictions (#4322)
      add crl.get_revoked_certificate method (#4331)
      add DSA wycheproof tests (#4346)
      add aes cbc pkcs5 wycheproof tests (#4347)
      disallow implicit tag truncation with finalize_with_tag (#4342)
      add wycheproof tests for AES CMAC (#4344)
      add chacha20poly1305 wycheproof tests (#4345)
      raise ValueError on zero length GCM IV (#4348)
      also check iv length for GCM nonce in AEAD (#4350)
      min_tag_length is an int (#4351)
      add wycheproof gcm tests (#4349)
      improve skip msg when skipping an ECDH test in test_ec (#4355)
      bump version and changelog for 2.3 release (#4356)

Quinten Stokkink (1):
      Exposed OpenSSL prime methods (#4292)

Thierry Bastian (1):
      OpenSSL 1.0.2o has switched to winsock2 (#4184)

Thom Dixon (1):
      Make AuthorityKeyIdentifier docs reflect reality (#4252)

Tim D. Smith (1):
      Fix typo (#4178)

Tux (1):
      Expose OpenSSL constant time bignum arithmetic (#4200)

Ville Skyttä (1):
      Use pytest instead of py.test per upstream recommendation, #dropthedot (#4236)

Vladyslav Moisieienkov (1):
      Add clearer message when key type is not bytes (#4289)

dumol (1):
      Fixed build errors on HP-UX. (#4259)

Коренберг Марк (1):
      Add serialisation output examples (#4286)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@arealjim arealjim arealjim approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@mattsb42-aws @arealjim