ci: compile with openjdk11, validate against other JDKs

[texastony]

Issue #, if available:

Description of changes:

Use CodeBuild's (CB) Batch Build in conjunction with CodeArtifact (CA) to Compile the ESDK-Java with openjdk11, publish to internal maven repository hosted on CA, and have subsequent CB builds test artifact with other JDKs.

This PR is in draft until I finish drafting, testing, and PR-ing the Infrastructure changes with these changes.

No feature has been added, nor are there any performance improvements/bug fixes. Just CI work.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Check any applicable:

• Were any files moved? Moving files changes their URL, which breaks all hyperlinks to the files.
  Yes. They have not been moved yet, but this will end up replacing all of the yaml files in codebuild with those in codebuild/ci.

[texastony]

@robin-aws
I was not able to completely stand up the same CFN CodeBuild stack as we have in a regular account in my test account.
I could not figure out how to configure a GPG key to be consumed by Secrets Manager correctly.
But I was able to recreate every thing else, to the extent that if I ran the release-ci build on my local (laptop), writing the artifacts to CodeArtifacts, all the other builds were successful.

I will talk with you or the team about recording the steps to set up the GPG Key in Secrets Manager.

The CodeBuild Test is failing because there is no esdk-java-ci Repository in the account. I am seeking a +1 before authoring changes to our AWS Infrastructure.

[texastony]

This will not work. It's not just the $$ typo, but the tests will run from the src code that will compile with the local JDK, so not what we want.

I am looking into getting mvn to run all the tests for the ESDK with a -D testVectorZip via the busy-engineer-bucket-test-pom.

[texastony]

No... I am not sure what to do here. @robin-aws or @lavaleri . I tried passing the test directory from the ESDK, hoping the sure-fire test runner in the busy-engineer-bucket would run those tests. I had no luck.
There are some other ideas on stack-overflow about bringing tests from another jar.
But our ESDK jar does not have the tests in it :(, so I would have to figure out how to edit the POM.xml :

mvn \
          -Pcodeartifact \
          -Dcheckstyle.skip \
          -Desdk.version=$VERSION_HASH \
          -Dmaven.compiler.target=$JAVA_NUMERIC_VERSION \
          -Dmaven.compiler.source=$JAVA_NUMERIC_VERSION \
          -Dcodeartifact.token=$CODEARTIFACT_TOKEN \
          -Dcodeartifact.url=$CODEARTIFACT_REPO_URL \
          --no-transfer-progress \
          -Dgpg.skip=true \
          "-DtestVectorZip=file://$CODEBUILD_SRC_DIR/src/test/resources/aws-encryption-sdk-test-vectors/vectors/awses-decrypt/python-2.3.0-mrks.zip" \
          "-DtestSourceDirectory=$CODEBUILD_SRC_DIR/src/test/java" \
          -s $SETTINGS_FILE \
          clean test

What actually got farther along, and maybe a decent idea, (y'all let me know), is copying the test directory for the ESDK into the test path here: cp -r $CODEBUILD_SRC_DIR/src/test/ src/test.

The two projects have different dependencies, which causes conflicts. What we could do is have a custom pom.xml that we also shunt in here. Then the consumed ESDK from CodeArtifact will be tested by busy-engineer AND all of its tests...

There are more challenges here: We may have to figure out how to not run the example tests here, since the examples are not in the jar either. Thoughts?

[lavaleri]

What's the intention here? To verify the jar on JDK11 by running through the document-bucket example? Or to verify the jar on JDK11 by running the ESDK test vectors? (it sounds like the latter but I want to double check that that is actually what we want to do right now, or if just running the document-bucket should be enough for now)

If the former, then we just mvn verify the document-bucket repo.

If the latter, you're correct that I'm not sure how to do this with our current jar/project structure. You can install the ESDK jar you want to test directly, but then I'm not sure how to execure mvn test such that it uses that jar instead of re-compiling the source (assuming you're in the ESDK repo). This SO suggests that you may be able to skip compile with a particular option, but I'm not sure if it will do the right thing here.

If that doesn't work, I would look into how to include the tests in our ESDK jar. I don't have much experience with that, but that sounds much less brittle than copying the test source into a different project.

[lavaleri]

As an alternative, building the test vector runner as a separate project isn't a bad idea either. The JS ESDK essentially does this, and the DDBEC for Java does this with the examples. We could consider doing something similar here.

[texastony]

Well... b9d4a49 is the implementation of what may be a bad idea.

[texastony]

I did not see lavaleri comments... but I can answer them now:

What's the intention here?

Take the openjdk11 compiled jar and test it with other JAVA runtimes.

To verify the jar on JDK11 by running through the document-bucket example?

To verify the JDK11 jar by running through the document-bucket in different java environments.

Or to verify the jar on JDK11 by running the ESDK test vectors?

Well, I took the goal to be: compile with openjdk11, and then test with the other JDKs. So, I am trying to run both the Busy-Bucket-Tests AND the Vectors. To ensure that the ESDK jar used was the one from CodeArtifact, I tried to do it through the busy-bucket.

This idea does not work yet. I had it working on my local, but the vector tests failed to execute in the latest CI run.

Making a custom test-vector runner is an interesting idea...

[texastony]

This works.

[lavaleri]

Unfortunately, I think we'll still want to attempt to build on JDK8 because the tests run on the document bucket only exercise a subset of the codepaths. For example, I tested here whether it would catch me putting something non-JDK8 compatible in a MKP that's not used in the document bucket example. This change should cause the CI to fail.

While it's inelegant, I think we want to keep building and running the full ESDK test suite in each JDK under test AND also perform the smoke test for the actual jar in each JDK under test using the document bucket. We keep all the coverage we currently have, and we additionally cover some of the gap for not having any testing of the actual jar in JDK8.

[texastony]

Unfortunately, I think we'll still want to attempt to build on JDK8 because the tests run on the document bucket only exercise a subset of the codepaths...

Addressed in 1980282