Skip to content

ci: compile with openjdk11, validate against other JDKs #293

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 22 commits into from
Jul 27, 2021
Merged

ci: compile with openjdk11, validate against other JDKs #293

Show file tree
Hide file tree
Changes from 11 commits
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
507483a
ci: compile with openjdk11, validate against other JDKs
texastony Jul 23, 2021
bb6f9a6
ci: address too many builds in Q error
texastony Jul 24, 2021
97f56ec
ci: correct build images
texastony Jul 24, 2021
e15a32b
ci: try abstracting the secrets
texastony Jul 24, 2021
b53bd83
ci(mvn): do not log transfers
texastony Jul 25, 2021
133640c
ci: after busy engineer test, run all ESDK tests
texastony Jul 25, 2021
6599ec2
ci: correct typo
texastony Jul 26, 2021
217ea70
ci: suppress transfer progress
texastony Jul 26, 2021
426dfd1
ci: suppress info logs from javadoc
texastony Jul 26, 2021
7d5eda1
ci: correct typo
texastony Jul 26, 2021
859124d
ci: correct typo
texastony Jul 26, 2021
5495e7d
refactor(pom): group dependencies by scope
texastony Jul 26, 2021
b9d4a49
ci: copy and run the ESDK tests through the busy-bucket
texastony Jul 26, 2021
6731450
Update codebuild/ci/validate-ci.yml
texastony Jul 26, 2021
b95a553
revert: "Update codebuild/ci/validate-ci.yml"
texastony Jul 26, 2021
3d2dcab
revert: "ci: copy and run the ESDK tests through the busy-bucket"
texastony Jul 26, 2021
fb10165
revert: "refactor(pom): group dependencies by scope"
texastony Jul 26, 2021
0f881f3
ci: move vector tests to it's own buildspec
texastony Jul 26, 2021
d9db99f
ci: spacing typo in buildspec
texastony Jul 26, 2021
60a6982
ci: spacing typo in buildspec
texastony Jul 26, 2021
16070bf
ci: optimize build times via parallel
texastony Jul 26, 2021
1980282
ci: run unit/integ/example/vector tests against all JDKs
texastony Jul 27, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
54 changes: 39 additions & 15 deletions buildspec.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,29 +2,53 @@ version: 0.2

batch:
fast-fail: false
build-list:
- identifier: openjdk8
buildspec: codebuild/openjdk8.yml
build-graph:
- identifier: static_analysis
buildspec: codebuild/ci/static-analysis.yml
env:
env:
image: aws/codebuild/standard:3.0
- identifier: openjdk11
buildspec: codebuild/openjdk11.yml
image: aws/codebuild/amazonlinux2-x86_64-standard:3.0
- identifier: release_ci
depend-on:
- static_analysis
buildspec: codebuild/ci/release-ci.yml
env:
env:
image: aws/codebuild/standard:3.0
- identifier: corretto8
buildspec: codebuild/corretto8.yml
- identifier: validate_ci_openjdk8
depend-on:
- release_ci
buildspec: codebuild/ci/validate-ci.yml
env:
variables:
JAVA_ENV_VERSION: openjdk8
JAVA_NUMERIC_VERSION: 8
env:
image: aws/codebuild/amazonlinux2-x86_64-standard:3.0
- identifier: corretto11
buildspec: codebuild/corretto11.yml
image: aws/codebuild/standard:3.0
- identifier: validate_ci_openjdk11
depend-on:
- release_ci
buildspec: codebuild/ci/validate-ci.yml
env:
env:
variables:
JAVA_ENV_VERSION: openjdk11
JAVA_NUMERIC_VERSION: 11
image: aws/codebuild/standard:3.0
- identifier: validate_ci_corretto8
depend-on:
- release_ci
buildspec: codebuild/ci/validate-ci.yml
env:
variables:
JAVA_ENV_VERSION: corretto8
JAVA_NUMERIC_VERSION: 8
image: aws/codebuild/amazonlinux2-x86_64-standard:3.0
- identifier: static_analysis
buildspec: codebuild/static-analysis.yml
- identifier: validate_ci_corretto11
depend-on:
- release_ci
buildspec: codebuild/ci/validate-ci.yml
env:
env:
variables:
JAVA_ENV_VERSION: corretto11
JAVA_NUMERIC_VERSION: 11
image: aws/codebuild/amazonlinux2-x86_64-standard:3.0
61 changes: 61 additions & 0 deletions codebuild/ci/release-ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
version: 0.2

env:
variables:
REGION: us-east-1
DOMAIN: crypto-tools-internal
REPOSITORY: java-esdk-ci
NAMESPACE: com.amazonaws
PACKAGE: aws-encryption-sdk-java
parameter-store:
ACCOUNT: /CodeBuild/AccountId
secrets-manager:
GPG_KEY: Maven-GPG-Keys-Credentials:Keyname
GPG_PASS: Maven-GPG-Keys-Credentials:Passphrase

phases:
install:
runtime-versions:
java: openjdk11
pre_build:
commands:
- export SETTINGS_FILE=$(pwd)/codebuild/release/settings.xml
- export CODEARTIFACT_TOKEN=$(aws codeartifact get-authorization-token --domain $DOMAIN --domain-owner $ACCOUNT --query authorizationToken --output text --region ${REGION})
- export CODEARTIFACT_REPO_URL=https://${DOMAIN}-${ACCOUNT}.d.codeartifact.${REGION}.amazonaws.com/maven/${REPOSITORY}
- aws secretsmanager get-secret-value --region us-west-2 --secret-id Maven-GPG-Keys --query SecretBinary --output text | base64 -d > ~/mvn_gpg.tgz
- tar -xvf ~/mvn_gpg.tgz -C ~
build:
commands:
- VERSION_HASH="$(mvn help:evaluate -Dexpression=project.version -q -DforceStdout)-$CODEBUILD_RESOLVED_SOURCE_VERSION"
# See https://docs.aws.amazon.com/codebuild/latest/userguide/build-env-ref-env-vars.html
- echo "Setting version in POM to $VERSION_HASH"
- mvn versions:set -DnewVersion="$VERSION_HASH" --no-transfer-progress
- echo "Version is now $(grep version pom.xml | head -n 1 | sed -n 's/[ \t]*<version>\(.*\)<\/version>/\1/p')"
- |
mvn deploy \
-PpublishingCodeArtifact \
-Pfast-tests-only \
-DperformRelease \
-Dgpg.homedir="$HOME/mvn_gpg" \
-DautoReleaseAfterClose=true \
-Dgpg.keyname="$GPG_KEY" \
-Dgpg.passphrase="$GPG_PASS" \
-Dcodeartifact.token=$CODEARTIFACT_TOKEN \
-DaltDeploymentRepository=codeartifact::default::$CODEARTIFACT_REPO_URL \
--no-transfer-progress \
-s $SETTINGS_FILE
finally:
- |
if expr ${CODEBUILD_BUILD_SUCCEEDING} != 1; then
echo "An error occured while building and uploading $REPOSITORY."
echo "Did a previous build already upload $VERSION_HASH to $REPOSITORY?"
echo "Try deleting $VERSION_HASH from $REPOSITORY and restart"
echo "Delete Package Version Command \n
aws codeartifact delete-package-versions --domain $DOMAIN \
--repository $REPOSITORY \
--format maven \
--namespace $NAMESPACE \
--package $PACKAGE \
--versions $VERSION_HASH \
--region $REGION";
fi
25 changes: 25 additions & 0 deletions codebuild/ci/settings.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
<settings xmlns="http://maven.apache.org/SETTINGS/1.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.0.0
http://maven.apache.org/xsd/settings-1.0.0.xsd">
<servers>
<server>
<id>codeartifact</id>
<username>aws</username>
<password>${codeartifact.token}</password>
</server>
</servers>

<profiles>
<profile>
<id>codeartifact</id>
<repositories>
<repository>
<id>codeartifact</id>
<name>codeartifact</name>
<url>${codeartifact.url}</url> <!-- passed via command line to avoid hardcoding it here -->
</repository>
</repositories>
</profile>
</profiles>
</settings>
0 codebuild/static-analysis.yml → codebuild/ci/static-analysis.yml
View file
Open in desktop
File renamed without changes.
56 changes: 56 additions & 0 deletions codebuild/ci/validate-ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
version: 0.2

env:
variables:
REGION: us-east-1
DOMAIN: crypto-tools-internal
REPOSITORY: java-esdk-ci
parameter-store:
ACCOUNT: /CodeBuild/AccountId

phases:
install:
commands:
- pip install awscli
runtime-versions:
java: $JAVA_ENV_VERSION
pre_build:
commands:
- export SRC_DIR=$(pwd)
- VERSION_HASH="$(mvn help:evaluate -Dexpression=project.version -q -DforceStdout)-$CODEBUILD_RESOLVED_SOURCE_VERSION"
- export SETTINGS_FILE=$(pwd)/codebuild/ci/settings.xml
- git clone https://github.com/aws-samples/busy-engineers-document-bucket.git
- cd busy-engineers-document-bucket/exercises/java/encryption-context-complete
- export CODEARTIFACT_TOKEN=$(aws codeartifact get-authorization-token --domain $DOMAIN --domain-owner $ACCOUNT --query authorizationToken --output text --region ${REGION})
- export CODEARTIFACT_REPO_URL=https://${DOMAIN}-${ACCOUNT}.d.codeartifact.${REGION}.amazonaws.com/maven/${REPOSITORY}
build:
commands:
- |
mvn verify \
-Pcodeartifact \
-Dcheckstyle.skip \
-Desdk.version=$VERSION_HASH \
-Dmaven.compiler.target=$JAVA_NUMERIC_VERSION \
-Dmaven.compiler.source=$JAVA_NUMERIC_VERSION \
-Dcodeartifact.token=$CODEARTIFACT_TOKEN \
-Dcodeartifact.url=$CODEARTIFACT_REPO_URL \
--no-transfer-progress \
-s $SETTINGS_FILE
post_build:
commands:
- cd $$CODEBUILD_SRC_DIR
- |
mvn verify \
-Pcodeartifact \
-Dcheckstyle.skip \
-Desdk.version=$VERSION_HASH \
-Dmaven.compiler.target=$JAVA_NUMERIC_VERSION \
-Dmaven.compiler.source=$JAVA_NUMERIC_VERSION \
-Dcodeartifact.token=$CODEARTIFACT_TOKEN \
-Dcodeartifact.url=$CODEARTIFACT_REPO_URL \
--no-transfer-progress \
-Dgpg.skip=true \
"-DtestVectorZip=file://$CODEBUILD_SRC_DIR/src/test/resources/aws-encryption-sdk-test-vectors/vectors/awses-decrypt/python-2.3.0-mrks.zip" \
-s $SETTINGS_FILE
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will not work. It's not just the $$ typo, but the tests will run from the src code that will compile with the local JDK, so not what we want.

I am looking into getting mvn to run all the tests for the ESDK with a -D testVectorZip via the busy-engineer-bucket-test-pom.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No... I am not sure what to do here. @robin-aws or @lavaleri . I tried passing the test directory from the ESDK, hoping the sure-fire test runner in the busy-engineer-bucket would run those tests. I had no luck.
There are some other ideas on stack-overflow about bringing tests from another jar.
But our ESDK jar does not have the tests in it :(, so I would have to figure out how to edit the POM.xml :

mvn \
          -Pcodeartifact \
          -Dcheckstyle.skip \
          -Desdk.version=$VERSION_HASH \
          -Dmaven.compiler.target=$JAVA_NUMERIC_VERSION \
          -Dmaven.compiler.source=$JAVA_NUMERIC_VERSION \
          -Dcodeartifact.token=$CODEARTIFACT_TOKEN \
          -Dcodeartifact.url=$CODEARTIFACT_REPO_URL \
          --no-transfer-progress \
          -Dgpg.skip=true \
          "-DtestVectorZip=file://$CODEBUILD_SRC_DIR/src/test/resources/aws-encryption-sdk-test-vectors/vectors/awses-decrypt/python-2.3.0-mrks.zip" \
          "-DtestSourceDirectory=$CODEBUILD_SRC_DIR/src/test/java" \
          -s $SETTINGS_FILE \
          clean test

What actually got farther along, and maybe a decent idea, (y'all let me know), is copying the test directory for the ESDK into the test path here: cp -r $CODEBUILD_SRC_DIR/src/test/ src/test.

The two projects have different dependencies, which causes conflicts. What we could do is have a custom pom.xml that we also shunt in here. Then the consumed ESDK from CodeArtifact will be tested by busy-engineer AND all of its tests...

There are more challenges here: We may have to figure out how to not run the example tests here, since the examples are not in the jar either. Thoughts?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's the intention here? To verify the jar on JDK11 by running through the document-bucket example? Or to verify the jar on JDK11 by running the ESDK test vectors? (it sounds like the latter but I want to double check that that is actually what we want to do right now, or if just running the document-bucket should be enough for now)

If the former, then we just mvn verify the document-bucket repo.

If the latter, you're correct that I'm not sure how to do this with our current jar/project structure. You can install the ESDK jar you want to test directly, but then I'm not sure how to execure mvn test such that it uses that jar instead of re-compiling the source (assuming you're in the ESDK repo). This SO suggests that you may be able to skip compile with a particular option, but I'm not sure if it will do the right thing here.

If that doesn't work, I would look into how to include the tests in our ESDK jar. I don't have much experience with that, but that sounds much less brittle than copying the test source into a different project.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As an alternative, building the test vector runner as a separate project isn't a bad idea either. The JS ESDK essentially does this, and the DDBEC for Java does this with the examples. We could consider doing something similar here.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well... b9d4a49 is the implementation of what may be a bad idea.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I did not see lavaleri comments... but I can answer them now:

What's the intention here?

Take the openjdk11 compiled jar and test it with other JAVA runtimes.

To verify the jar on JDK11 by running through the document-bucket example?

To verify the JDK11 jar by running through the document-bucket in different java environments.

Or to verify the jar on JDK11 by running the ESDK test vectors?

Well, I took the goal to be: compile with openjdk11, and then test with the other JDKs. So, I am trying to run both the Busy-Bucket-Tests AND the Vectors. To ensure that the ESDK jar used was the one from CodeArtifact, I tried to do it through the busy-bucket.

This idea does not work yet. I had it working on my local, but the vector tests failed to execute in the latest CI run.

Making a custom test-vector runner is an interesting idea...



9 changes: 0 additions & 9 deletions codebuild/corretto11.yml
View file
Open in desktop

This file was deleted.

9 changes: 0 additions & 9 deletions codebuild/corretto8.yml
View file
Open in desktop

This file was deleted.

9 changes: 0 additions & 9 deletions codebuild/openjdk11.yml
View file
Open in desktop

This file was deleted.

9 changes: 0 additions & 9 deletions codebuild/openjdk8.yml
View file
Open in desktop

This file was deleted.

1 change: 1 addition & 0 deletions codebuild/release/release-prod.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,7 @@ phases:
-Dgpg.passphrase="$GPG_PASS" \
-Dsonatype.username="$SONA_USERNAME" \
-Dsonatype.password="$SONA_PASSWORD" \
--no-transfer-progress \
-s $SETTINGS_FILE


Expand Down
1 change: 1 addition & 0 deletions codebuild/release/release-staging.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,7 @@ phases:
-Dgpg.passphrase="$GPG_PASS" \
-Dcodeartifact.token=$CODEARTIFACT_TOKEN \
-DaltDeploymentRepository=codeartifact::default::$CODEARTIFACT_REPO_URL \
--no-transfer-progress \
-s $SETTINGS_FILE

batch:
Expand Down
3 changes: 2 additions & 1 deletion codebuild/release/validate-prod.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,4 +15,5 @@ phases:
-Dcheckstyle.skip \
-Desdk.version=$VERSION \
-Dmaven.compiler.target=$JAVA_NUMERIC_VERSION \
-Dmaven.compiler.source=$JAVA_NUMERIC_VERSION
-Dmaven.compiler.source=$JAVA_NUMERIC_VERSION \
--no-transfer-progress
1 change: 1 addition & 0 deletions codebuild/release/validate-staging.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,5 +32,6 @@ phases:
-Dmaven.compiler.source=$JAVA_NUMERIC_VERSION \
-Dcodeartifact.token=$CODEARTIFACT_TOKEN \
-Dcodeartifact.url=$CODEARTIFACT_REPO_URL \
--no-transfer-progress \
-s $SETTINGS_FILE

1 change: 1 addition & 0 deletions pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -129,6 +129,7 @@
<maxmemory>1024m</maxmemory>
<doclint>none</doclint>
<source>8</source>
<quiet>true</quiet>
</configuration>
</plugin>

Expand Down