Skip to content

Update the test schema to Draft 2020, and fix a bug in it. #565

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Jun 30, 2022
Merged

Update the test schema to Draft 2020, and fix a bug in it. #565

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
69 changes: 19 additions & 50 deletions test-schema.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,88 +1,57 @@
{
"$schema": "http://json-schema.org/draft-06/schema#",
"description": "Schema for tests",
"$schema": "https://json-schema.org/draft/2020-12/schema",
"description": "A schema for files contained within this suite",

"type": "array",
"minItems": 1,
"items": {
"description": "An individual test case, containing multiple tests of a single schema's behavior",

"type": "object",
"required": [ "description", "schema", "tests" ],
"properties": {
"id": {
"description": "Uniquely identifies a set of tests",
"type": "string",
"format": "uri"
},
"description": {
"description": "The test set description",
"description": "The test case description",
"type": "string"
},
"comment": {
"description": "Any additional comments about the test set",
"description": "Any additional comments about the test case",
"type": "string"
},
"schema": {
"description": "This should be a valid schema. This should be a ref to a meta-schema if schema keywords need testing."
"description": "A valid JSON Schema (one written for the corresponding version directory that the file sits within)."
Copy link
Member

@gregsdennis gregsdennis Jun 30, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we add an enum property to the test files, we could probably key off of that and enforce a specific metaschema here. (Idea for separate PR)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As in provide some other mechanism for indicating a metaschema test? We could, but each of these changes is delicate, since someone's existing test runner still may not be expecting that. Obviously we're going to need to do something like breaking our informal backwards compatibility anyhow to resolve the optional/ folder situation, so we could in theory do so for that too, but lemme know if I misunderstood the idea entirely.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, I mean actually validate the schemas in the test files. If the test file specifies what version schemas its for (because JSON Schema does see the file path) then we can use that to select a metaschema to validate the test schema against.

I was thinking something like the below, but then I realized the test file is an array, and you'd have to specify the version on every test... Might be more headache than it's worth now that I see it.

Leaving for posterity

In the test file

{
  "version": "2020-12",
   ... // the rest of the test file
}

In the above schema:

{
  ...
  "oneOf": [
    {
      "properties": {
        "version": { "const": "2020-12" },
        "schema": { "$ref": "<2020-12 metaschema>" }
    },
    ... // more for other versions
  ]
  ...
}

},
"tests": {
"description": "A set of related tests all using the same schema",
"type": "array",
"items": { "$ref": "#/definitions/test" },
"items": { "$ref": "#/$defs/test" },
"minItems": 1
}
},
"additionalProperties": false,
"minItems": 1
"additionalProperties": false
},
"definitions": {
"outputItem": {
"type": "object",
"properties": {
"valid": { "type": "boolean" },
"keywordLocation": { "type": "string" },
"absoluteKeywordLocation": {
"type": "string",
"format": "uri"
},
"instanceLocation": { "type": "string" },
"annotations": {
"type": "array",
"items": { "$ref": "#/definitions/outputItem" }
},
"errors": {
"type": "array",
"items": { "$ref": "#/definitions/outputItem" }
}
}
},

"$defs": {
"test": {
"description": "A single test",

"type": "object",
"required": [ "description", "data", "valid" ],
"properties": {
"id": {
"description": "Uniquely identifies a single test",
"type": "string",
"format": "uri"
},
"description": {
"description": "The test description",
"description": "The test description, briefly explaining which behavior it exercises",
"type": "string"
},
"comment": {
"description": "Any additional comments about the test",
"type": "string"
},
"data": {
"description": "This is the instance to be validated against the schema in \"schema\"."
"description": "The instance which should be validated against the schema in \"schema\"."
},
"valid": { "type": "boolean" },
"output": {
"type": "object",
"required": [ "basic", "detailed", "verbose" ],
"properties": {
"basic": { "$ref": "#/definitions/outputItem" },
"detailed": { "$ref": "#/definitions/outputItem" },
"verbose": { "$ref": "#/definitions/outputItem" }
}
"valid": {
"description": "Whether the validation process of this instance should consider the instance valid or not",
"type": "boolean"
}
},
"additionalProperties": false
Expand Down