Skip to content

Commit 0aef5f8

Browse files
jdesrosiersjdesrosiers
committed
Add annotation tests
1 parent 83e866b commit 0aef5f8

13 files changed

+1860
-0
lines changed

‎annotations/README.md

+104
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,104 @@
1+
# Annotation Tests
2+
3+
The annotations Test Suite tests which annotations should appear (or not appear)
4+
on which values of an instance. These tests are agnostic of any output format.
5+
6+
## Supported Dialects
7+
8+
Although the concept of annotations didn't appear in the spec until 2019-09, the
9+
concept is compatible with every version of JSON Schema. Test Cases in this Test
10+
Suite are designed to be compatible with as many releases of JSON Schema as
11+
possible. They do not include `$schema` or `$id`/`id` keywords so that
12+
implementations can run the same Test Suite for each dialect they support.
13+
14+
Since this Test Suite can be used for a variety of dialects, there are a couple
15+
of options that can be used by Test Runners to filter out Test Cases that don't
16+
apply to the dialect under test.
17+
18+
## Test Case Components
19+
20+
### description
21+
22+
A short description of what behavior the Test Case is covering.
23+
24+
### compatibility
25+
26+
The `compatibility` option allows you to set which dialects the Test Case is
27+
compatible with. Test Runners can use this value to filter out Test Cases that
28+
don't apply the to dialect currently under test. Dialects are indicated by the
29+
number corresponding to their release. Date-based releases use just the year.
30+
31+
If this option isn't present, it means the Test Case is compatible with any
32+
dialect.
33+
34+
If this option is present with a number, the number indicates the minimum
35+
release the Test Case is compatible with. This example indicates that the Test
36+
Case is compatible with draft-07 and up.
37+
38+
**Example**: `"compatibility": "7"`
39+
40+
You can use a `<=` operator to indicate that the Test Case is compatible with
41+
releases less then or equal to the given release. This example indicates that
42+
the Test Case is compatible with 2019-09 and under.
43+
44+
**Example**: `"compatibility": "<=2019"`
45+
46+
You can use comma-separated values to indicate multiple constraints if needed.
47+
This example indicates that the Test Case is compatible with releases between
48+
draft-06 and 2019-09.
49+
50+
**Example**: `"compatibility": "6,<=2019"`
51+
52+
For convenience, you can use the `=` operator to indicate a Test Case is only
53+
compatible with a single release. This example indicates that the Test Case is
54+
compatible only with 2020-12.
55+
56+
**Example**: `"compatibility": "=2020"`
57+
58+
### schema
59+
60+
The schema that will serve as the subject for the tests. Whenever possible, this
61+
schema shouldn't include `$schema` or `id`/`$id` because Test Cases should be
62+
designed to work with as many releases as possible.
63+
64+
### externalSchemas
65+
66+
`externalSchemas` allows you to define additional schemas that `schema` makes
67+
references to. The value is an object where the keys are retrieval URIs and
68+
values are schemas. Most external schemas aren't self identifying (using
69+
`id`/`$id`) and rely on the retrieval URI for identification. This is done to
70+
increase the number of dialects that the test is compatible with. Because `id`
71+
changed to `$id` in draft-06, if you use `$id`, the test becomes incompatible
72+
with draft-03/4 and in most cases, that's not necessary.
73+
74+
### tests
75+
76+
A collection of Tests to run to verify the Test Case.
77+
78+
## Test Components
79+
80+
### instance
81+
82+
The JSON instance to be annotated.
83+
84+
### assertions
85+
86+
`assertions` are a collection of assertions that must be true for the test to pass.
87+
88+
## Assertions Components
89+
90+
### location
91+
92+
The instance location.
93+
94+
### keyword
95+
96+
The annotating keyword.
97+
98+
### expected
99+
100+
An array of annotations on the `keyword` - instance `location` pair. `expected`
101+
is an array because there's always a chance that an annotation is applied
102+
multiple times to any given instance location. The `expected` array should be
103+
sorted such that the most recently encountered value for an annotation during
104+
evaluation comes before previously encountered values.

‎annotations/assertion.schema.json

+21
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
{
2+
"$schema": "http://json-schema.org/draft-07/schema#",
3+
4+
"type": "object",
5+
"properties": {
6+
"location": {
7+
"markdownDescription": "The instance location.",
8+
"type": "string",
9+
"format": "json-pointer"
10+
},
11+
"keyword": {
12+
"markdownDescription": "The annotation keyword.",
13+
"type": "string"
14+
},
15+
"expected": {
16+
"markdownDescription": "An array of annotations on the `keyword` - instance `location` pair.",
17+
"type": "array"
18+
}
19+
},
20+
"required": ["location", "keyword", "expected"]
21+
}

‎annotations/test-case.schema.json

+38
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
{
2+
"$schema": "http://json-schema.org/draft-07/schema#",
3+
4+
"type": "object",
5+
"properties": {
6+
"description": {
7+
"markdownDescription": "A short description of what behavior the Test Case is covering.",
8+
"type": "string"
9+
},
10+
"compatibility": {
11+
"markdownDescription": "Set which dialects the Test Case is compatible with. Examples:\n- `\"7\"` -- draft-07 and above\n- `\"<=2019\"` -- 2019-09 and previous\n- `\"6,<=2019\"` -- Between draft-06 and 2019-09\n- `\"=2020\"` -- 2020-12 only",
12+
"type": "string",
13+
"pattern": "^(<=|=)?([123467]|2019|2020)(,(<=|=)?([123467]|2019|2020))*$"
14+
},
15+
"schema": {
16+
"markdownDescription": "This schema shouldn't include `$schema` or `id`/`$id` unless necesary for the test because Test Cases should be designed to work with as many releases as possible.",
17+
"type": ["boolean", "object"]
18+
},
19+
"externalSchemas": {
20+
"markdownDescription": "The keys are retrieval URIs and values are schemas.",
21+
"type": "object",
22+
"patternProperties": {
23+
"": {
24+
"type": ["boolean", "object"]
25+
}
26+
},
27+
"propertyNames": {
28+
"format": "uri"
29+
}
30+
},
31+
"tests": {
32+
"markdownDescription": "A collection of Tests to run to verify the Test Case.",
33+
"type": "array",
34+
"items": { "$ref": "./test.schema.json" }
35+
}
36+
},
37+
"required": ["description", "schema", "tests"]
38+
}

‎annotations/test-suite.schema.json

+15
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
{
2+
"$schema": "http://json-schema.org/draft-07/schema#",
3+
4+
"type": "object",
5+
"properties": {
6+
"description": {
7+
"type": "string"
8+
},
9+
"suite": {
10+
"type": "array",
11+
"items": { "$ref": "./test-case.schema.json" }
12+
}
13+
},
14+
"required": ["description", "suite"]
15+
}

‎annotations/test.schema.json

+16
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
{
2+
"$schema": "http://json-schema.org/draft-07/schema#",
3+
4+
"type": "object",
5+
"properties": {
6+
"instance": {
7+
"markdownDescription": "The JSON instance to be annotated."
8+
},
9+
"assertions": {
10+
"markdownDescription": "An array of annotations on the `keyword` - instance `location` pair.",
11+
"type": "array",
12+
"items": { "$ref": "./assertion.schema.json" }
13+
}
14+
},
15+
"required": ["instance", "assertions"]
16+
}

0 commit comments

Comments
 (0)