Squashed 'json/' changes from cf78d97d0..f0f619d19

f0f619d19 Merge pull request #560 from json-schema-org/ether/fix-383
fa791ada9 remove tests that are not valid for draft7
0f341dca0 Merge pull request #429 from notEthan/decimal_minItems
32ec6305c test validation of keywords which expect integers when their value has a decimal
547330dd6 Tweak the markdown table syntax in the README.
28ed3022c Restructure the README further.
45d6e5e7f README: Add new content and update the structure
4de0966a7 Merge pull request #558 from yakimun/replace-definitions-with-defs
aad0f350a Replace definitions with  for 2019 and later drafts
0b777ffa2 Merge branch 'ether/more-anchor-and-id-tests'
d593591d7 Forward port the just-added tests to 2020 and next.
e157bc0b7 squash: use a unique $id to prevent namespace collisions across tests
1d5583de6 some tests of the interactions between $id, $anchor and $ref
07c45c0d7 Restore the dependencies tests as optional for 2019, 2020, and next

git-subtree-dir: json
git-subtree-split: f0f619d19696a81f7683116f707a51c34150d28f

Author: Julian

README.md

@@ -1,85 +1,186 @@
-# JSON Schema Test Suite 
+# JSON Schema Test Suite
+
 [![Contributor Covenant](https://img.shields.io/badge/Contributor%20Covenant-2.1-4baaaa.svg)](https://github.com/json-schema-org/.github/blob/main/CODE_OF_CONDUCT.md)
 [![Project Status: Active – The project has reached a stable, usable state and is being actively developed.](https://www.repostatus.org/badges/latest/active.svg)](https://www.repostatus.org/#active)
 [![Financial Contributors on Open Collective](https://opencollective.com/json-schema/all/badge.svg?label=financial+contributors)](https://opencollective.com/json-schema)
 
 [![Build Status](https://github.com/json-schema-org/JSON-Schema-Test-Suite/workflows/Test%20Suite%20Sanity%20Checking/badge.svg)](https://github.com/json-schema-org/JSON-Schema-Test-Suite/actions?query=workflow%3A%22Test+Suite+Sanity+Checking%22)
 
-This repository contains a set of JSON objects that implementors of JSON Schema
-validation libraries can use to test their validators.
+This repository contains a set of JSON objects that implementers of JSON Schema validation libraries can use to test their validators.
 
 It is meant to be language agnostic and should require only a JSON parser.
+The conversion of the JSON objects into tests within a specific language and test framework of choice is left to be done by the validator implementer.
+
+## Coverage
+
+All JSON Schema specification releases should be well covered by this suite, including drafts 2020-12, 2019-09, 07, 06, 04 and 03.
+Drafts 04 and 03 are considered "frozen" in that less effort is put in to backport new tests to these versions.
+
+Additional coverage is always welcome, particularly for bugs encountered in real-world implementations.
+If you see anything missing or incorrect, please feel free to [file an issue](https://github.com/json-schema-org/JSON-Schema-Test-Suite/issues) or [submit a PR](https://github.com/json-schema-org/JSON-Schema-Test-Suite).
+
+## Introduction to the Test Suite Structure
+
+The tests in this suite are contained in the `tests` directory at the root of this repository.
+Inside that directory is a subdirectory for each released version of the specification.
+
+The structure and contents of each file in these directories is described below.
+
+In addition to the version-specific subdirectories, two additional directories are present:
 
-The conversion of the JSON objects into tests within your test framework of
-choice is still the job of the validator implementor.
+1. `draft-next/`: containing tests for the next version of the specification whilst it is in development
+2. `latest/`: a symbolic link which points to the directory which is the most recent release (which may be useful for implementations providing specific entry points for validating against the latest version of the specification)
 
-## Structure of a Test
+Inside each version directory there are a number of `.json` files each containing a collection of related tests.
+Often the grouping is by property under test, but not always.
+In addition to the `.json` files, each version directory contains one or more special subdirectories whose purpose is [described below](#subdirectories-within-each-draft), and which contain additional `.json` files.
 
-The tests in this suite are contained in the `tests` directory at the root of
-this repository. Inside that directory is a subdirectory for each draft or
-version of the specification.
+Each `.json` file consists of a single JSON array of test cases.
 
-Inside each draft directory, there are a number of `.json` files and one or more
-special subdirectories. The subdirectories contain `.json` files meant for a
-specific testing purpose, and each `.json` file logically groups a set of test
-cases together. Often the grouping is by property under test, but not always.
+### Terminology
 
-The subdirectories are described in the next section.
+For clarity, we first define this document's usage of some testing terminology:
 
-Inside each `.json` file is a single array containing objects. It's easiest to
-illustrate the structure of these with an example:
+| term            | definition                                                                                                                                                        |
+|-----------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| **test suite**  | the entirety of the contents of this repository, containing tests for multiple different releases of the JSON Schema specification                                |
+| **test case**   | a single schema, along with a description and an array of *test*s                                                                                                 |
+| **test**        | within a *test case*, a single test example, containing a description, instance and a boolean indicating whether the instance is valid under the test case schema |
+| **test runner** | a program, external to this repository and authored by a user of this suite, which is executing each of the tests in the suite                                    |
+
+An example illustrating this structure is immediately below, and a JSON Schema containing a formal definition of the contents of test cases can be found [alongside this README](./test-schema.json).
+
+### Sample Test Case
+
+Here is a single *test case*, containing one or more tests:
 
 ```json
 {
-    "description": "The description of the test case",
-    "schema": {
-        "description": "The schema against which the data in each test is validated",
-        "type": "string"
-    },
+    "description": "The test case description",
+    "schema": { "type": "string" },
     "tests": [
         {
-            "description": "Test for a valid instance",
-            "data": "the instance to validate",
+            "description": "a test with a valid instance",
+            "data": "a string",
             "valid": true
         },
         {
-            "description": "Test for an invalid instance",
+            "description": "a test with an invalid instance",
             "data": 15,
             "valid": false
         }
     ]
 }
 ```
 
-In short: a description, a schema under test, and some tests, where each test
-in the `tests` array is an objects with a description of the case itself, the
-instance under test, and a boolean indicating whether it should be valid
-or invalid.
+### Subdirectories Within Each Draft
 
-## Test Subdirectories
+There is currently only one additional subdirectory that may exist within each draft test directory.
 
-There is currently only one subdirectory that may exist within each draft
-directory. This is:
+This is:
 
 1. `optional/`: Contains tests that are considered optional.
 
-## Coverage
+Note, the `optional/` subdirectory today conflates many reasons why a test may be optional -- it may be because tests within a particular file are indeed not required by the specification but still potentially useful to an implementer, or it may be because tests within it only apply to programming languages with particular functionality (in
+which case they are not truly optional in such a language).
+In the future this directory structure will be made richer to reflect these differences more clearly.
+
+## Using the Suite to Test a Validator Implementation
+
+The test suite structure was described [above](#introduction-to-the-test-suite-structure).
+
+If you are authoring a new validator implementation, or adding support for an additional version of the specification, this section describes:
+
+1. How to implement a test runner which passes tests to your validator
+2. Assumptions the suite makes about how the test runner will configure your validator
+3. Invariants the test suite claims to hold for its tests
 
-All JSON Schema specification releases should be well covered by this suite,
-including drafts 2020-12, 2019-09, 07, 06, 04 and 03. Additional coverage is
-always welcome, particularly for bugs encountered in real-world
-implementations.
+### How to Implement a Test Runner
 
-Drafts 04 and 03 are considered "frozen" in that less effort is put in to
-backport new tests to these versions.
+Presented here is a possible implementation of a test runner.
+The precise steps described do not need to be followed exactly, but the results of your own procedure should produce the same effects.
 
-Contributions are very welcome, especially from implementers as they add support
-to their own implementations.
+To test a specific version:
 
-If you see anything missing from the current supported drafts, or incorrect on
-any draft still accepting bug fixes, please
-[file an issue](https://github.com/json-schema-org/JSON-Schema-Test-Suite/issues)
-or [submit a PR](https://github.com/json-schema-org/JSON-Schema-Test-Suite).
+* Load any remote references [described below](additional-assumptions) and configure your implementation to retrieve them via their URIs
+* Walk the filesystem tree for that version's subdirectory and for each `.json` file found:
+
+    * if the file is located in the root of the version directory:
+
+        * for each test case present in the file:
+
+            * load the schema from the `"schema"` property
+            * load (or log) the test case description from the `"description"` property for debugging or outputting
+            * for each test in the `"tests"` property:
+
+                * load the instance to be tested from the `"data"` property
+                * load (or log) the individual test description from the `"description"` property for debugging or outputting
+
+                * use the schema loaded above to validate whether the instance is considered valid under your implementation
+
+                * if the result from your implementation matches the value found in the `"valid"` property, your implementation correctly implements the specific example
+                * if the result does not match, or your implementation errors or crashes, your implementation does not correctly implement the specific example
+
+    * otherwise it is located in a special subdirectory as described above.
+      Follow the additional assumptions and restrictions for the containing subdirectory, then run the test case as above.
+
+If your implementation supports multiple versions, run the above procedure for each version supported, configuring your implementation as appropriate to call each version individually.
+
+### Additional Assumptions
+
+1. The suite, notably in its `refRemote.json` file in each draft, expects a number of remote references to be configured.
+   These are JSON documents, identified by URI, which are used by the suite to test the behavior of the `$ref` keyword (and related keywords).
+   Depending on your implementation, you may configure how to "register" these either by retrieving them from the `remotes/` directory at the root of the repository, *or* you may execute `bin/jsonschema_suite remotes` using the executable in the `bin/` directory, which will output a JSON object containing all of the remotes combined.
+
+2. Test cases found within [special subdirectories](#subdirectories-within-each-draft) may require additional configuration to run.
+   In particular, tests within the `optional/format` subdirectory may require implementations to change the way they treat the `"format"`keyword (particularly on older drafts which did not have a notion of vocabularies).
+
+### Invariants & Guarantees
+
+The test suite guarantees a number of things about tests it defines.
+Any deviation from the below is generally considered a bug.
+If you suspect one, please [file an issue](https://github.com/json-schema-org/JSON-Schema-Test-Suite/issues/new):
+
+1. All files containing test cases are valid JSON.
+2. The contents of the `"schema"` property in a test case are always valid
+   JSON Schemas under the corresponding specification.
+
+   The rationale behind this is that we are testing instances in a test's `"data"` element, and not the schema itself.
+   A number of tests *do* test the validity of a schema itself, but do so by representing the schema as an instance inside a test, with the associated meta-schema in the `"schema"` property (via the `"$ref"` keyword):
+
+   ```json
+   {
+       "description": "Test the \"type\" schema keyword",
+       "schema": {
+           "$ref": "https://json-schema.org/draft/2019-09/schema"
+        },
+       "tests": [
+           {
+               "description": "Valid: string",
+               "data": {
+                   "type": "string"
+               },
+               "valid": true
+           },
+           {
+               "description": "Invalid: null",
+               "data": {
+                   "type": null
+               },
+               "valid": false
+           }
+       ]
+   }
+   ```
+   See below for some [known limitations](#known-limitations).
+
+## Known Limitations
+
+This suite expresses its assertions about the behavior of an implementation *within* JSON Schema itself.
+Each test is the application of a schema to a particular instance.
+This means that the suite of tests can test against any behavior a schema can describe, and conversely cannot test against any behavior which a schema is incapable of representing, even if the behavior is mandated by the specification.
+
+For example, a schema can require that a string is a _URI-reference_ and even that it matches a certain pattern, but even though the specification contains [recommendations about URIs being normalized](https://json-schema.org/draft/2020-12/json-schema-core.html#name-the-id-keyword), a JSON schema cannot today represent this assertion within the core vocabularies of the specifications, so no test covers this behavior.
 
 ## Who Uses the Test Suite
 
@@ -153,12 +254,9 @@ This suite is being used by:
 
 ### Node.js
 
-For node.js developers, the suite is also available as an
-[npm](https://www.npmjs.com/package/@json-schema-org/tests) package.
+For node.js developers, the suite is also available as an [npm](https://www.npmjs.com/package/@json-schema-org/tests) package.
 
-Node-specific support is maintained in a [separate
-repository](https://github.com/json-schema-org/json-schema-test-suite-npm)
-which also welcomes your contributions!
+Node-specific support is maintained in a [separate repository](https://github.com/json-schema-org/json-schema-test-suite-npm) which also welcomes your contributions!
 
 ### .NET
 

remotes/locationIndependentIdentifier.json

@@ -1,5 +1,5 @@
 {
-    "definitions": {
+    "$defs": {
         "refToInteger": {
             "$ref": "#foo"
         },

tests/draft-next/anchor.json

@@ -169,5 +169,36 @@
                 "valid": false
             }
         ]
+    },
+    {
+        "description": "non-schema object containing an $anchor property",
+        "schema": {
+            "$defs": {
+                "const_not_anchor": {
+                    "const": {
+                        "$anchor": "not_a_real_anchor"
+                    }
+                }
+            },
+            "if": {
+                "const": "skip not_a_real_anchor"
+            },
+            "then": true,
+            "else" : {
+                "$ref": "#/$defs/const_not_anchor"
+            }
+        },
+        "tests": [
+            {
+                "description": "skip traversing definition for a valid result",
+                "data": "skip not_a_real_anchor",
+                "valid": true
+            },
+            {
+                "description": "const at const_not_anchor does not match",
+                "data": 1,
+                "valid": false
+            }
+        ]
     }
 ]

tests/draft-next/id.json

@@ -254,5 +254,36 @@
                 "valid": false
             }
         ]
+    },
+    {
+        "description": "non-schema object containing an $id property",
+        "schema": {
+            "$defs": {
+                "const_not_id": {
+                    "const": {
+                        "$id": "not_a_real_id"
+                    }
+                }
+            },
+            "if": {
+                "const": "skip not_a_real_id"
+            },
+            "then": true,
+            "else" : {
+                "$ref": "#/$defs/const_not_id"
+            }
+        },
+        "tests": [
+            {
+                "description": "skip traversing definition for a valid result",
+                "data": "skip not_a_real_id",
+                "valid": true
+            },
+            {
+                "description": "const at const_not_id does not match",
+                "data": 1,
+                "valid": false
+            }
+        ]
     }
 ]

tests/draft-next/maxContains.json

@@ -86,6 +86,25 @@
             }
         ]
     },
+    {
+        "description": "maxContains with contains, value with a decimal",
+        "schema": {
+            "contains": {"const": 1},
+            "maxContains": 1.0
+        },
+        "tests": [
+            {
+                "description": "one element matches, valid maxContains",
+                "data": [ 1 ],
+                "valid": true
+            },
+            {
+                "description": "too many elements match, invalid maxContains",
+                "data": [ 1, 1 ],
+                "valid": false
+            }
+        ]
+    },
     {
         "description": "minContains < maxContains",
         "schema": {

tests/draft-next/maxItems.json

@@ -24,5 +24,21 @@
                 "valid": true
             }
         ]
+    },
+    {
+        "description": "maxItems validation with a decimal",
+        "schema": {"maxItems": 2.0},
+        "tests": [
+            {
+                "description": "shorter is valid",
+                "data": [1],
+                "valid": true
+            },
+            {
+                "description": "too long is invalid",
+                "data": [1, 2, 3],
+                "valid": false
+            }
+        ]
     }
 ]

tests/draft-next/maxLength.json

@@ -29,5 +29,21 @@
                 "valid": true
             }
         ]
+    },
+    {
+        "description": "maxLength validation with a decimal",
+        "schema": {"maxLength": 2.0},
+        "tests": [
+            {
+                "description": "shorter is valid",
+                "data": "f",
+                "valid": true
+            },
+            {
+                "description": "too long is invalid",
+                "data": "foo",
+                "valid": false
+            }
+        ]
     }
 ]