Add annotation tests from @hyperjump/json-schema

.github/workflows/annotation-tests.yml

@@ -0,0 +1,21 @@
+name: Validate annotation tests
+
+on:
+  pull_request:
+    paths:
+      - "annotations/**"
+
+jobs:
+  annotate:
+    runs-on: ubuntu-latest
+
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Set up Deno
+        uses: denoland/setup-deno@v2
+        with:
+          deno-version: "2.x"
+
+      - name: Validate annotation tests
+        run: deno --node-modules-dir=auto --allow-read --no-prompt bin/annotation-tests.ts

annotations/README.md

@@ -0,0 +1,116 @@
+# Annotations Tests Suite
+
+The Annotations Test Suite tests which annotations should appear (or not appear)
+on which values of an instance. These tests are agnostic of any output format.
+
+## Supported Dialects
+
+Although the annotation terminology of didn't appear in the spec until 2019-09,
+the concept is compatible with every version of JSON Schema. Test Cases in this
+Test Suite are designed to be compatible with as many releases of JSON Schema as
+possible. They do not include `$schema` or `$id`/`id` keywords so
+implementations can run the same Test Suite for each dialect they support.
+
+Since this Test Suite can be used for a variety of dialects, there are a couple
+of options that can be used by Test Runners to filter out Test Cases that don't
+apply to the dialect under test.
+
+## Test Case Components
+
+### description
+
+A short description of what behavior the Test Case is covering.
+
+### compatibility
+
+The `compatibility` option allows you to set which dialects the Test Case is
+compatible with. Test Runners can use this value to filter out Test Cases that
+don't apply the to dialect currently under test. The terminology for annotations
+didn't appear in the spec until 2019-09, but the concept is compatible with
+older releases as well. When setting `compatibility`, test authors should take
+into account dialects before 2019-09 for implementations that chose to support
+annotations for older dialects.
+
+Dialects are indicated by the number corresponding to their release. Date-based
+releases use just the year. If this option isn't present, it means the Test Case
+is compatible with any dialect.
+
+If this option is present with a number, the number indicates the minimum
+release the Test Case is compatible with. This example indicates that the Test
+Case is compatible with draft-07 and up.
+
+**Example**: `"compatibility": "7"`
+
+You can use a `<=` operator to indicate that the Test Case is compatible with
+releases less then or equal to the given release. This example indicates that
+the Test Case is compatible with 2019-09 and under.
+
+**Example**: `"compatibility": "<=2019"`
+
+You can use comma-separated values to indicate multiple constraints if needed.
+This example indicates that the Test Case is compatible with releases between
+draft-06 and 2019-09.
+
+**Example**: `"compatibility": "6,<=2019"`
+
+For convenience, you can use the `=` operator to indicate a Test Case is only
+compatible with a single release. This example indicates that the Test Case is
+compatible only with 2020-12.
+
+**Example**: `"compatibility": "=2020"`
+
+### schema
+
+The schema that will serve as the subject for the tests. Whenever possible, this
+schema shouldn't include `$schema` or `id`/`$id` because Test Cases should be
+designed to work with as many releases as possible.
+
+### externalSchemas
+
+This allows you to define additional schemas that `schema` makes references to.
+The value is an object where the keys are retrieval URIs and values are schemas.
+Most external schemas aren't self identifying (using `id`/`$id`) and rely on the
+retrieval URI for identification. This is done to increase the number of
+dialects that the test is compatible with.  Because `id` changed to `$id` in
+draft-06, if you use `$id`, the test becomes incompatible with draft-03/4 and in
+most cases, that's not necessary.
+
+### tests
+
+A collection of Tests to run to verify the Test Case.
+
+## Test Components
+
+### instance
+
+The JSON instance to be annotated.
+
+### assertions
+
+A collection of assertions that must be true for the test to pass.
+
+## Assertions Components
+
+### location
+
+The instance location.
+
+### keyword
+
+The annotating keyword.
+
+### expected
+
+A collection of `keyword` annotations expected on the instance at `location`.
+`expected` is an object where the keys are schema locations and the values are
+the annotation that schema location contributed for the given `keyword`.
+
+There can be more than one expected annotation because multiple schema locations
+could contribute annotations for a single keyword.
+
+An empty object is an assertion that the annotation must not appear at the
+`location` for the `keyword`.
+
+As a convention for this Test Suite, the `expected` array should be sorted such
+that the most recently encountered value for an annotation given top-down
+evaluation of the schema comes before previously encountered values.

annotations/assertion.schema.json

@@ -0,0 +1,24 @@
+{
+  "$schema": "http://json-schema.org/draft-07/schema#",
+
+  "type": "object",
+  "properties": {
+    "location": {
+      "markdownDescription": "The instance location.",
+      "type": "string",
+      "format": "json-pointer"
+    },
+    "keyword": {
+      "markdownDescription": "The annotation keyword.",
+      "type": "string"
+    },
+    "expected": {
+      "markdownDescription": "An object of schemaLocation/annotations pairs for `keyword` annotations expected on the instance at `location`.",
+      "type": "object",
+      "propertyNames": {
+        "format": "uri"
+      }
+    }
+  },
+  "required": ["location", "keyword", "expected"]
+}

annotations/test-case.schema.json

@@ -0,0 +1,38 @@
+{
+  "$schema": "http://json-schema.org/draft-07/schema#",
+
+  "type": "object",
+  "properties": {
+    "description": {
+      "markdownDescription": "A short description of what behavior the Test Case is covering.",
+      "type": "string"
+    },
+    "compatibility": {
+      "markdownDescription": "Set which dialects the Test Case is compatible with. Examples:\n- `\"7\"` -- draft-07 and above\n- `\"<=2019\"` -- 2019-09 and previous\n- `\"6,<=2019\"` -- Between draft-06 and 2019-09\n- `\"=2020\"` -- 2020-12 only",
+      "type": "string",
+      "pattern": "^(<=|=)?([123467]|2019|2020)(,(<=|=)?([123467]|2019|2020))*$"
+    },
+    "schema": {
+      "markdownDescription": "This schema shouldn't include `$schema` or `id`/`$id` unless necesary for the test because Test Cases should be designed to work with as many releases as possible.",
+      "type": ["boolean", "object"]
+    },
+    "externalSchemas": {
+      "markdownDescription": "The keys are retrieval URIs and values are schemas.",
+      "type": "object",
+      "patternProperties": {
+        "": {
+          "type": ["boolean", "object"]
+        }
+      },
+      "propertyNames": {
+        "format": "uri"
+      }
+    },
+    "tests": {
+      "markdownDescription": "A collection of Tests to run to verify the Test Case.",
+      "type": "array",
+      "items": { "$ref": "./test.schema.json" }
+    }
+  },
+  "required": ["description", "schema", "tests"]
+}

annotations/test-suite.schema.json

@@ -0,0 +1,15 @@
+{
+  "$schema": "http://json-schema.org/draft-07/schema#",
+
+  "type": "object",
+  "properties": {
+    "description": {
+      "type": "string"
+    },
+    "suite": {
+      "type": "array",
+      "items": { "$ref": "./test-case.schema.json" }
+    }
+  },
+  "required": ["description", "suite"]
+}

annotations/test.schema.json

@@ -0,0 +1,16 @@
+{
+  "$schema": "http://json-schema.org/draft-07/schema#",
+
+  "type": "object",
+  "properties": {
+    "instance": {
+      "markdownDescription": "The JSON instance to be annotated."
+    },
+    "assertions": {
+      "markdownDescription": "A collection of assertions that must be true for the test to pass.",
+      "type": "array",
+      "items": { "$ref": "./assertion.schema.json" }
+    }
+  },
+  "required": ["instance", "assertions"]
+}