Rework ref deferencing

This is an attempt to simplify dereferencing refs and address issues
exposed by new json-schema-test-suite tests. The main thing here is
giving schema objects a `base_uri` that's used when resolving ids (and
as the initial base URI for instances). Child schemas use the `ref_uri`
that was used to resolve them as their `base_uri` in order to get nested
refs and ids to resolve properly.

More details about specific issues:

- `$id` is ignored when `$ref` is present (`instance.base_uri` is
  updated after `validate_ref`), because `$ref` isn't supposed to take
  sibling `$id` values into account. `@base_uri` handles nested refs.
  Exposed by: json-schema-org/JSON-Schema-Test-Suite#493

- JSON pointers are evaluated relative to the ref URI. Previously,
  pointers were always evaluated using the root schema. Now they're
  evaluated relative to the schema with a matching `$id` (usually
  nearest parent with an `$id`; or specific id (see below); default is
  root).
  Exposed by: json-schema-org/JSON-Schema-Test-Suite#457

- JSON pointers are evaluated for id refs. This allows a ref to look up
  a schema by `$id` and then apply a JSON pointer to use a subschema.
  This uses the same logic as above. The important part is removing the
  fragment from `ref_uri` if it's a JSON pointer so that the lookup in
  `ids` works properly. The fragment is kept if it's not a JSON pointer
  to support location-independent ids.
  Exposed by: json-schema-org/JSON-Schema-Test-Suite#578

- JSON pointer refs are always joined with the base URI. I [started
  handling them][0] separately because of an [issue][1] with invalid
  URIs. But now I think that was incorrect and that fragment pointers
  need to be encoded properly for URIs. The [specification says][2]:

  > In all cases, dereferencing a "$ref" reference involves first
  > resolving its value as a URI reference against the current base URI.

- Empty fragments are removed in `join_uri` to have consistent URIs to
  lookup in `ids`. Meta schemas, for example, have empty fragments in
  their top-level ids (eg, `http://json-schema.org/draft-07/schema#`)
  and removing the JSON pointer fragments causes them not to be found.

[0]: b91115e
[1]: #54
[2]: https://datatracker.ietf.org/doc/html/draft-handrews-json-schema-01#section-8.3.2

Author: davishmcclurg

lib/json_schemer.rb

@@ -57,28 +57,25 @@ def schema(schema, default_schema_class: Schema::Draft7, **options)
       when String
         schema = JSON.parse(schema)
       when Pathname
-        uri = URI.parse(File.join('file:', URI::DEFAULT_PARSER.escape(schema.realpath.to_s)))
-        if options.key?(:ref_resolver)
-          schema = FILE_URI_REF_RESOLVER.call(uri)
+        base_uri = URI.parse(File.join('file:', URI::DEFAULT_PARSER.escape(schema.realpath.to_s)))
+        options[:base_uri] = base_uri
+        schema = if options.key?(:ref_resolver)
+          FILE_URI_REF_RESOLVER.call(base_uri)
         else
           ref_resolver = CachedResolver.new(&FILE_URI_REF_RESOLVER)
-          schema = ref_resolver.call(uri)
           options[:ref_resolver] = ref_resolver
+          ref_resolver.call(base_uri)
         end
-        schema[draft_class(schema, default_schema_class)::ID_KEYWORD] ||= uri.to_s
       end
-      draft_class(schema, default_schema_class).new(schema, **options)
-    end
-
-  private
 
-    def draft_class(schema, default_schema_class)
-      if schema.is_a?(Hash) && schema.key?('$schema')
+      schema_class = if schema.is_a?(Hash) && schema.key?('$schema')
         meta_schema = schema.fetch('$schema')
         SCHEMA_CLASS_BY_META_SCHEMA[meta_schema] || raise(UnsupportedMetaSchema, meta_schema)
       else
         default_schema_class
       end
+
+      schema_class.new(schema, **options)
     end
   end
 end

lib/json_schemer/schema/base.rb

@@ -4,17 +4,17 @@ module Schema
     class Base
       include Format
 
-      Instance = Struct.new(:data, :data_pointer, :schema, :schema_pointer, :parent_uri, :before_property_validation, :after_property_validation) do
+      Instance = Struct.new(:data, :data_pointer, :schema, :schema_pointer, :base_uri, :before_property_validation, :after_property_validation) do
         def merge(
           data: self.data,
           data_pointer: self.data_pointer,
           schema: self.schema,
           schema_pointer: self.schema_pointer,
-          parent_uri: self.parent_uri,
+          base_uri: self.base_uri,
           before_property_validation: self.before_property_validation,
           after_property_validation: self.after_property_validation
         )
-          self.class.new(data, data_pointer, schema, schema_pointer, parent_uri, before_property_validation, after_property_validation)
+          self.class.new(data, data_pointer, schema, schema_pointer, base_uri, before_property_validation, after_property_validation)
         end
       end
 
@@ -53,6 +53,7 @@ def merge(
 
       def initialize(
         schema,
+        base_uri: nil,
         format: true,
         insert_property_defaults: false,
         before_property_validation: nil,
@@ -64,6 +65,7 @@ def initialize(
       )
         raise InvalidSymbolKey, 'schemas must use string keys' if schema.is_a?(Hash) && !schema.empty? && !schema.first.first.is_a?(String)
         @root = schema
+        @base_uri = base_uri
         @format = format
         @before_property_validation = [*before_property_validation]
         @before_property_validation.unshift(INSERT_DEFAULT_PROPERTY) if insert_property_defaults
@@ -82,15 +84,17 @@ def initialize(
       end
 
       def valid?(data)
-        valid_instance?(Instance.new(data, '', root, '', nil, @before_property_validation, @after_property_validation))
+        valid_instance?(Instance.new(data, '', root, '', @base_uri, @before_property_validation, @after_property_validation))
       end
 
       def validate(data)
-        validate_instance(Instance.new(data, '', root, '', nil, @before_property_validation, @after_property_validation))
+        validate_instance(Instance.new(data, '', root, '', @base_uri, @before_property_validation, @after_property_validation))
       end
 
     protected
 
+      attr_reader :root
+
       def valid_instance?(instance)
         validate_instance(instance).none?
       end
@@ -120,13 +124,13 @@ def validate_instance(instance, &block)
         ref = schema['$ref']
         id = schema[id_keyword]
 
-        instance.parent_uri = join_uri(instance.parent_uri, id)
-
         if ref
           validate_ref(instance, ref, &block)
           return
         end
 
+        instance.base_uri = join_uri(instance.base_uri, id)
+
         if format? && custom_format?(format)
           validate_custom_format(instance, formats.fetch(format), &block)
         end
@@ -228,7 +232,7 @@ def ids
 
     private
 
-      attr_reader :root, :formats, :keywords, :ref_resolver, :regexp_resolver
+      attr_reader :formats, :keywords, :ref_resolver, :regexp_resolver
 
       def id_keyword
         ID_KEYWORD
@@ -246,10 +250,11 @@ def spec_format?(format)
         !custom_format?(format) && supported_format?(format)
       end
 
-      def child(schema)
+      def child(schema, base_uri:)
         JSONSchemer.schema(
           schema,
           default_schema_class: self.class,
+          base_uri: base_uri,
           format: format?,
           formats: formats,
           keywords: keywords,
@@ -306,50 +311,38 @@ def validate_type(instance, type, &block)
       end
 
       def validate_ref(instance, ref, &block)
-        if ref.start_with?('#')
-          schema_pointer = ref.slice(1..-1)
-          if valid_json_pointer?(schema_pointer)
-            ref_pointer = Hana::Pointer.new(URI.decode_www_form_component(schema_pointer))
-            subinstance = instance.merge(
-              schema: ref_pointer.eval(root),
-              schema_pointer: schema_pointer,
-              parent_uri: (pointer_uri(root, ref_pointer) || instance.parent_uri)
-            )
-            validate_instance(subinstance, &block)
-            return
-          end
-        end
-
-        ref_uri = join_uri(instance.parent_uri, ref)
+        ref_uri = join_uri(instance.base_uri, ref)
 
+        ref_uri_pointer = ''
         if valid_json_pointer?(ref_uri.fragment)
-          ref_pointer = Hana::Pointer.new(URI.decode_www_form_component(ref_uri.fragment))
-          ref_root = resolve_ref(ref_uri)
-          ref_object = child(ref_root)
-          subinstance = instance.merge(
-            schema: ref_pointer.eval(ref_root),
-            schema_pointer: ref_uri.fragment,
-            parent_uri: (pointer_uri(ref_root, ref_pointer) || ref_uri)
-          )
-          ref_object.validate_instance(subinstance, &block)
-        elsif id = ids[ref_uri.to_s]
-          subinstance = instance.merge(
-            schema: id.fetch(:schema),
-            schema_pointer: id.fetch(:pointer),
-            parent_uri: ref_uri
-          )
-          validate_instance(subinstance, &block)
+          ref_uri_pointer = ref_uri.fragment
+          ref_uri.fragment = nil
+        end
+
+        ref_object = if ids.key?(ref_uri) || ref_uri.to_s == @base_uri.to_s
+          self
         else
-          ref_root = resolve_ref(ref_uri)
-          ref_object = child(ref_root)
-          id = ref_object.ids[ref_uri.to_s] || { schema: ref_root, pointer: '' }
-          subinstance = instance.merge(
-            schema: id.fetch(:schema),
-            schema_pointer: id.fetch(:pointer),
-            parent_uri: ref_uri
-          )
-          ref_object.validate_instance(subinstance, &block)
+          child(resolve_ref(ref_uri), base_uri: ref_uri)
+        end
+
+        ref_schema, ref_schema_pointer = ref_object.ids[ref_uri] || [ref_object.root, '']
+
+        ref_uri_pointer_parts = Hana::Pointer.parse(URI.decode_www_form_component(ref_uri_pointer))
+        schema, base_uri = ref_uri_pointer_parts.reduce([ref_schema, ref_uri]) do |(obj, uri), token|
+          if obj.is_a?(Array)
+            [obj.fetch(token.to_i), uri]
+          else
+            [obj.fetch(token), join_uri(uri, obj[id_keyword])]
+          end
         end
+
+        subinstance = instance.merge(
+          schema: schema,
+          schema_pointer: "#{ref_schema_pointer}#{ref_uri_pointer}",
+          base_uri: base_uri
+        )
+
+        ref_object.validate_instance(subinstance, &block)
       end
 
       def validate_custom_format(instance, custom_format)
@@ -631,7 +624,7 @@ def escape_json_pointer_token(token)
 
       def join_uri(a, b)
         b = URI.parse(b) if b
-        if a && b && a.relative? && b.relative?
+        uri = if a && b && a.relative? && b.relative?
           b
         elsif a && b
           URI.join(a, b)
@@ -640,35 +633,19 @@ def join_uri(a, b)
         else
           a
         end
+        uri.fragment = nil if uri.is_a?(URI) && uri.fragment == ''
+        uri
       end
 
-      def pointer_uri(schema, pointer)
-        uri_parts = nil
-        pointer.reduce(schema) do |obj, token|
-          next obj.fetch(token.to_i) if obj.is_a?(Array)
-          if obj_id = obj[id_keyword]
-            uri_parts ||= []
-            uri_parts << obj_id
-          end
-          obj.fetch(token)
-        end
-        uri_parts ? URI.join(*uri_parts) : nil
-      end
-
-      def resolve_ids(schema, ids = {}, parent_uri = nil, pointer = '')
+      def resolve_ids(schema, ids = {}, base_uri = @base_uri, pointer = '')
         if schema.is_a?(Array)
-          schema.each_with_index { |subschema, index| resolve_ids(subschema, ids, parent_uri, "#{pointer}/#{index}") }
+          schema.each_with_index { |subschema, index| resolve_ids(subschema, ids, base_uri, "#{pointer}/#{index}") }
         elsif schema.is_a?(Hash)
-          uri = join_uri(parent_uri, schema[id_keyword])
+          uri = join_uri(base_uri, schema[id_keyword])
           schema.each do |key, value|
             case key
             when id_keyword
-              unless uri == parent_uri
-                ids[uri.to_s] = {
-                  schema: schema,
-                  pointer: pointer
-                }
-              end
+              ids[uri] ||= [schema, pointer]
             when 'items', 'allOf', 'anyOf', 'oneOf', 'additionalItems', 'contains', 'additionalProperties', 'propertyNames', 'if', 'then', 'else', 'not'
               resolve_ids(value, ids, uri, "#{pointer}/#{key}")
             when 'properties', 'patternProperties', 'definitions', 'dependencies'

test/ref_test.rb

@@ -215,7 +215,7 @@ def test_it_handles_nested_refs
   def test_it_handles_json_pointer_refs_with_special_characters
     schema = JSONSchemer.schema({
       'type' => 'object',
-      'properties' => { 'foo' => { '$ref' => '#/definitions/~1some~1{id}'} },
+      'properties' => { 'foo' => { '$ref' => '#/definitions/~1some~1%7Bid%7D'} },
       'definitions' => { '/some/{id}' => { 'type' => 'string' } }
     })
     assert(schema.valid?({ 'foo' => 'bar' }))