From 9bc87edd9a9102ac9458591bdb3c9a8224b8cdee Mon Sep 17 00:00:00 2001 From: Bond_009 Date: Tue, 19 Nov 2024 15:36:06 +0100 Subject: [PATCH 1/8] Add docs on how to create exercises for the Docker judge --- en/references/judges/docker-judge/index.md | 67 ++++++++++++++++++++++ 1 file changed, 67 insertions(+) create mode 100644 en/references/judges/docker-judge/index.md diff --git a/en/references/judges/docker-judge/index.md b/en/references/judges/docker-judge/index.md new file mode 100644 index 000000000..4a893b084 --- /dev/null +++ b/en/references/judges/docker-judge/index.md @@ -0,0 +1,67 @@ +--- +title: "Docker judge" +description: "Docker judge" +order: 7 +--- +# Docker judge + +The Docker judge has 4 main functions. +Firstly it uses dodona-containerfile-evaluator, a custom utility, to check the usage of certain Docker instruction e.g. USER and WORKDIR. +Secondly it uses Hadolint to annotate the code and optionally fail the solution if it errors out. +Thirdly it uses kaniko to build the image from the Dockerfile. +Lastly the judge checks the existence of files and directories in the created image. + +## Configuring Hadolint + +The behavior of Hadolint can be modified by creating a [Hadolint configuration file](https://github.com/hadolint/hadolint#configure) with one of the following names inside of the `evaluation` directory: `.hadolint.yml`, `.hadolint.yaml`, `hadolint.yml` or `hadolint.yaml`. +Using this it's possible to change the severity of annotations, fail when a certain severity is hit, check if labels exist and more. + +An example configuration file looks as follows: +```yml +no-fail: false +failure-threshold: error +label-schema: + org.opencontainers.image.title: text + org.opencontainers.image.description: text + org.opencontainers.image.source: text + org.opencontainers.image.revision: text + org.opencontainers.image.licenses: text + org.opencontainers.image.vendor: text +override: + error: + - DL3049 +ignored: + - DL3004 +``` + +`no-fail` specifies if the judge should fail the solution if the linter fails. +`failure-threshold` sets the threshold for which rules will cause a fail, valid values are `error`, `warning`, `info` and `style`. +`label-schema` is a dictionary where the key is the label name and the value of a label can be either of `text`, `url`, `semver`, `hash`, `rfc3339`, `spdx` or `email`. +`override` is used to override the severity of rules, valid severities are the same as for `failure-threshold`. +`ignored` is a list or rules that should be ignored. +For more configuration options can be found in the [Hadolint README](https://github.com/hadolint/hadolint#configure) + + +## Judge configuration + +The judge is configured by specifying a `judge.json` configuration file inside of the `evaluation` directory. +Using this file it's possible to verify if the `USER` or `WORKDIR` instructions are used with the desired arguments. +This file also contains a `files` property that is a JSON array of JSON objects. +Each object contains a `path` property with the desired location in the resulting image. +The object represents either a file or a directory, this is specified by the `type` property. +Additionally object representing files can also contain a `compare` or `regex` property. +`compare` should be the name of a file inside of the `workdir` with which the file at `path` should be compared with. +`regex` contains a ("extended") "regular expression that should match on the content of the file at `path`. + +```json +{ + "user": "runner", + "workdir": "/course", + "files": [ + { "type": "directory", "path": "/course" }, + { "type": "file", "path": "/environment.yml", "compare": "environment.yml" }, + { "type": "directory", "path": "/usr/miniconda3/envs/pipeline-tools-1.0.0" }, + { "type": "file", "path": "/etc/os-release", "regex": "^ID=\"ubuntu\"$" } + ] +} +``` From f459dbda18a96dc6d983273cf9826ad74a5ec7bf Mon Sep 17 00:00:00 2001 From: Bond_009 Date: Tue, 19 Nov 2024 16:20:36 +0100 Subject: [PATCH 2/8] Add links to used projects --- en/references/judges/docker-judge/index.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/en/references/judges/docker-judge/index.md b/en/references/judges/docker-judge/index.md index 4a893b084..cdf55c5da 100644 --- a/en/references/judges/docker-judge/index.md +++ b/en/references/judges/docker-judge/index.md @@ -6,9 +6,9 @@ order: 7 # Docker judge The Docker judge has 4 main functions. -Firstly it uses dodona-containerfile-evaluator, a custom utility, to check the usage of certain Docker instruction e.g. USER and WORKDIR. -Secondly it uses Hadolint to annotate the code and optionally fail the solution if it errors out. -Thirdly it uses kaniko to build the image from the Dockerfile. +Firstly it uses [dodona-containerfile-evaluator](https://github.com/Bond-009/dodona-containerfile-evaluator), a custom utility, to check the usage of certain Docker instruction e.g. USER and WORKDIR. +Secondly it uses [Hadolint](https://github.com/hadolint/hadolint) to annotate the code and optionally fail the solution if it errors out. +Thirdly it uses [kaniko](https://github.com/GoogleContainerTools/kaniko) to build the image from the Dockerfile. Lastly the judge checks the existence of files and directories in the created image. ## Configuring Hadolint From 4246719b29d0e0439f0f455b8fe32fd5fb73406b Mon Sep 17 00:00:00 2001 From: Bond_009 Date: Mon, 2 Dec 2024 15:51:21 +0100 Subject: [PATCH 3/8] Add documentation for checking the FROM instruction --- en/references/judges/docker-judge/index.md | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/en/references/judges/docker-judge/index.md b/en/references/judges/docker-judge/index.md index cdf55c5da..978e7949f 100644 --- a/en/references/judges/docker-judge/index.md +++ b/en/references/judges/docker-judge/index.md @@ -45,7 +45,8 @@ For more configuration options can be found in the [Hadolint README](https://git ## Judge configuration The judge is configured by specifying a `judge.json` configuration file inside of the `evaluation` directory. -Using this file it's possible to verify if the `USER` or `WORKDIR` instructions are used with the desired arguments. +Using this file it's possible to verify the base image, the `from` object requires the `image` to contain the image name, optionally a `tag` or `hash` property can be added to check the used tag or digest. +It's also possible to check if the `USER` or `WORKDIR` instructions are used with the desired arguments. This file also contains a `files` property that is a JSON array of JSON objects. Each object contains a `path` property with the desired location in the resulting image. The object represents either a file or a directory, this is specified by the `type` property. @@ -55,6 +56,10 @@ Additionally object representing files can also contain a `compare` or `regex` p ```json { + "from": { + "image": "alpine", + "tag": "3:20" + }, "user": "runner", "workdir": "/course", "files": [ From 3808deb1ca77d9afae816d42a675245b41d61280 Mon Sep 17 00:00:00 2001 From: Bond_009 Date: Thu, 5 Dec 2024 14:34:35 +0100 Subject: [PATCH 4/8] Fix typo --- en/references/judges/docker-judge/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/en/references/judges/docker-judge/index.md b/en/references/judges/docker-judge/index.md index 978e7949f..8f801b284 100644 --- a/en/references/judges/docker-judge/index.md +++ b/en/references/judges/docker-judge/index.md @@ -50,7 +50,7 @@ It's also possible to check if the `USER` or `WORKDIR` instructions are used wit This file also contains a `files` property that is a JSON array of JSON objects. Each object contains a `path` property with the desired location in the resulting image. The object represents either a file or a directory, this is specified by the `type` property. -Additionally object representing files can also contain a `compare` or `regex` property. +Additionally objects representing files can also contain a `compare` or `regex` property. `compare` should be the name of a file inside of the `workdir` with which the file at `path` should be compared with. `regex` contains a ("extended") "regular expression that should match on the content of the file at `path`. From 40a7a88f795aa7dd16871b82ebe1cd86f14f4bd5 Mon Sep 17 00:00:00 2001 From: Bond_009 Date: Mon, 16 Dec 2024 10:17:04 +0100 Subject: [PATCH 5/8] Add info about limitations and the checking of comments --- en/references/judges/docker-judge/index.md | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/en/references/judges/docker-judge/index.md b/en/references/judges/docker-judge/index.md index 8f801b284..14e64203a 100644 --- a/en/references/judges/docker-judge/index.md +++ b/en/references/judges/docker-judge/index.md @@ -47,6 +47,7 @@ For more configuration options can be found in the [Hadolint README](https://git The judge is configured by specifying a `judge.json` configuration file inside of the `evaluation` directory. Using this file it's possible to verify the base image, the `from` object requires the `image` to contain the image name, optionally a `tag` or `hash` property can be added to check the used tag or digest. It's also possible to check if the `USER` or `WORKDIR` instructions are used with the desired arguments. +Another feature is the ability to check if the comments contain certain strings using the `comments` array. This file also contains a `files` property that is a JSON array of JSON objects. Each object contains a `path` property with the desired location in the resulting image. The object represents either a file or a directory, this is specified by the `type` property. @@ -62,6 +63,7 @@ Additionally objects representing files can also contain a `compare` or `regex` }, "user": "runner", "workdir": "/course", + "comments": [ "docker run" ], "files": [ { "type": "directory", "path": "/course" }, { "type": "file", "path": "/environment.yml", "compare": "environment.yml" }, @@ -70,3 +72,17 @@ Additionally objects representing files can also contain a `compare` or `regex` ] } ``` + +## Limitations +Due to the design of the judge it isn't possible to test the existence of files if no `COPY`, `RUN` or `ADD` instructions are present in the final stage. +For example, if the following solution is submitted, a test for the existence of the file `/etc/os-release` which is present in the alpine base image will fail. +```docker +FROM alpine:3.21 + +USER runner + +WORKDIR /home/runner + +LABEL org.opencontainers.image.title="test" +``` +But once you add one of the instructions that can modify the filesystem this same test will succeed. From 71f5c8511e12da1c8b9bd9ecf9f0e0e0ce9f1d4d Mon Sep 17 00:00:00 2001 From: James Collier Date: Mon, 16 Dec 2024 10:41:30 +0100 Subject: [PATCH 6/8] Change formatting and ordering of the documentation I believe this reads more clearly if trying to reference what an instructor needs. --- en/references/judges/docker-judge/index.md | 76 ++++++++++++---------- 1 file changed, 40 insertions(+), 36 deletions(-) diff --git a/en/references/judges/docker-judge/index.md b/en/references/judges/docker-judge/index.md index 14e64203a..20beb16e2 100644 --- a/en/references/judges/docker-judge/index.md +++ b/en/references/judges/docker-judge/index.md @@ -5,46 +5,18 @@ order: 7 --- # Docker judge -The Docker judge has 4 main functions. -Firstly it uses [dodona-containerfile-evaluator](https://github.com/Bond-009/dodona-containerfile-evaluator), a custom utility, to check the usage of certain Docker instruction e.g. USER and WORKDIR. -Secondly it uses [Hadolint](https://github.com/hadolint/hadolint) to annotate the code and optionally fail the solution if it errors out. -Thirdly it uses [kaniko](https://github.com/GoogleContainerTools/kaniko) to build the image from the Dockerfile. -Lastly the judge checks the existence of files and directories in the created image. +The Docker judge has 4 main functions: +1. Check usage of Docker instructions like `USER` or `WORKDIR` using [dodona-containerfile-evaluator](https://github.com/Bond-009/dodona-containerfile-evaluator). +2. Static analysis (linting) using [Hadolint](https://github.com/hadolint/hadolint) to annotate the code and optionally fail the solution if it contains errors. +3. Build the submitted Dockerfile using [kaniko](https://github.com/GoogleContainerTools/kaniko). +4. Check the existence and/or contents of files and directories in the image that is build from the submitted Dockerfile. -## Configuring Hadolint - -The behavior of Hadolint can be modified by creating a [Hadolint configuration file](https://github.com/hadolint/hadolint#configure) with one of the following names inside of the `evaluation` directory: `.hadolint.yml`, `.hadolint.yaml`, `hadolint.yml` or `hadolint.yaml`. -Using this it's possible to change the severity of annotations, fail when a certain severity is hit, check if labels exist and more. - -An example configuration file looks as follows: -```yml -no-fail: false -failure-threshold: error -label-schema: - org.opencontainers.image.title: text - org.opencontainers.image.description: text - org.opencontainers.image.source: text - org.opencontainers.image.revision: text - org.opencontainers.image.licenses: text - org.opencontainers.image.vendor: text -override: - error: - - DL3049 -ignored: - - DL3004 -``` - -`no-fail` specifies if the judge should fail the solution if the linter fails. -`failure-threshold` sets the threshold for which rules will cause a fail, valid values are `error`, `warning`, `info` and `style`. -`label-schema` is a dictionary where the key is the label name and the value of a label can be either of `text`, `url`, `semver`, `hash`, `rfc3339`, `spdx` or `email`. -`override` is used to override the severity of rules, valid severities are the same as for `failure-threshold`. -`ignored` is a list or rules that should be ignored. -For more configuration options can be found in the [Hadolint README](https://github.com/hadolint/hadolint#configure) +Items 1 and 2 above are configurable. -## Judge configuration +## Configuring Docker instructions -The judge is configured by specifying a `judge.json` configuration file inside of the `evaluation` directory. +Create a file called `judge.json` inside of the `evaluation` directory. Using this file it's possible to verify the base image, the `from` object requires the `image` to contain the image name, optionally a `tag` or `hash` property can be added to check the used tag or digest. It's also possible to check if the `USER` or `WORKDIR` instructions are used with the desired arguments. Another feature is the ability to check if the comments contain certain strings using the `comments` array. @@ -55,6 +27,7 @@ Additionally objects representing files can also contain a `compare` or `regex` `compare` should be the name of a file inside of the `workdir` with which the file at `path` should be compared with. `regex` contains a ("extended") "regular expression that should match on the content of the file at `path`. +An example `judge.json`: ```json { "from": { @@ -73,6 +46,37 @@ Additionally objects representing files can also contain a `compare` or `regex` } ``` +## Configuring Hadolint + +The behavior of Hadolint can be modified by creating a [Hadolint configuration file](https://github.com/hadolint/hadolint#configure) with one of the following names inside of the `evaluation` directory: `.hadolint.yml`, `.hadolint.yaml`, `hadolint.yml` or `hadolint.yaml`. +Using this it's possible to change the severity of annotations, fail when a certain severity is hit, check if labels exist and more. + +An example configuration file looks as follows: +```yml +no-fail: false +failure-threshold: error +label-schema: + org.opencontainers.image.title: text + org.opencontainers.image.description: text + org.opencontainers.image.source: text + org.opencontainers.image.revision: text + org.opencontainers.image.licenses: text + org.opencontainers.image.vendor: text +override: + error: + - DL3049 +ignored: + - DL3004 +``` + +`no-fail` specifies if the judge should fail the solution if the linter fails. +`failure-threshold` sets the threshold for which rules will cause a fail, valid values are `error`, `warning`, `info` and `style`. +`label-schema` is a dictionary where the key is the label name and the value of a label can be either of `text`, `url`, `semver`, `hash`, `rfc3339`, `spdx` or `email`. +`override` is used to override the severity of rules, valid severities are the same as for `failure-threshold`. +`ignored` is a list or rules that should be ignored. +For more configuration options can be found in the [Hadolint README](https://github.com/hadolint/hadolint#configure) + + ## Limitations Due to the design of the judge it isn't possible to test the existence of files if no `COPY`, `RUN` or `ADD` instructions are present in the final stage. For example, if the following solution is submitted, a test for the existence of the file `/etc/os-release` which is present in the alpine base image will fail. From 4e5b495e5a34b779510bffc7b1325f992e1e36c9 Mon Sep 17 00:00:00 2001 From: James Collier Date: Mon, 16 Dec 2024 11:46:51 +0100 Subject: [PATCH 7/8] Add Docker to the list of judges --- en/references/judges/index.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/en/references/judges/index.md b/en/references/judges/index.md index 301668b0d..6a3aed36a 100644 --- a/en/references/judges/index.md +++ b/en/references/judges/index.md @@ -20,6 +20,12 @@ C is a judge that uses the GTester framework to run tests on C exercises.\ **Get started** [Documentation](https://github.com/mvdcamme/C-Judge), [examples](https://github.com/mvdcamme/C-Judge/tree/master/example_exercises) \ **Created by:** [Maarten Vandercammen](mailto:mvdcamme@vub.ac.be) +## Docker +Docker is a judge that uses dodona-containerfile-evaluator, Hadolint, and Kaniko to evaluate Dockerfile exercises. +**Programming languages**: Dockerfile +**Get started** [Git repo](https://github.com/dodona-edu/judge-docker), [examples](https://github.com/Bond-009/dodona-docker-exercises) \ +**Created by:** [Lode Willems](mailto:bits@vib.be) + ## Haskell Haskell is a judge that uses HUnit to test Haskell exercises. \ **Programming languages:** Haskell\ From c6e99a35d2625d7e5b54fd0b616075de33e68e89 Mon Sep 17 00:00:00 2001 From: Bond_009 Date: Mon, 16 Dec 2024 13:21:35 +0100 Subject: [PATCH 8/8] Fix typo --- en/references/judges/docker-judge/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/en/references/judges/docker-judge/index.md b/en/references/judges/docker-judge/index.md index 20beb16e2..a043787db 100644 --- a/en/references/judges/docker-judge/index.md +++ b/en/references/judges/docker-judge/index.md @@ -74,7 +74,7 @@ ignored: `label-schema` is a dictionary where the key is the label name and the value of a label can be either of `text`, `url`, `semver`, `hash`, `rfc3339`, `spdx` or `email`. `override` is used to override the severity of rules, valid severities are the same as for `failure-threshold`. `ignored` is a list or rules that should be ignored. -For more configuration options can be found in the [Hadolint README](https://github.com/hadolint/hadolint#configure) +More configuration options can be found in the [Hadolint README](https://github.com/hadolint/hadolint#configure) ## Limitations