misleading erros when failing anyof tests

[ssbarnea]

I discovered that anyOf related errors can be very mistleading.

- jobs:  # must fail as correct is 'job'

Validating snippet above with https://github.com/pycontribs/zuul-lint/blob/master/zuul_lint/zuul-schema.json would fail with a misleading error:

'nodeset' is a required property

Failed validating 'required' in schema[0]:
    {'required': ['nodeset']}

On instance:
    {'jobs': {'name': 'foo'}}

That schema allows multiple value and nodeset happens to just one of several. As you can see what user gets is an erros that makes no sense for him.

[Julian]

I assume you mean when you're using jsonschema.validate -- which is trying to be as beginner friendly as can be -- as you say, it can be misleading at times, but more often than not descending into errors ends up producing clearer errors for most cases, which is why that API does so.

If you don't though want that behavior, you can use jsonschema.DraftXValidator.iter_errors or .validate directly, and not call best_match on the result.

[ssbarnea]

You are right I use validate at https://github.com/pycontribs/zuul-lint/blob/master/zuul_lint/__main__.py#L22 but I am going to test more advanced API.

Thanks a lot fo the quick reply. That new linter I am writing is very new stuff, is not even 10 hours since I started it. I am also new to jsonschema in general and I do not rule out that my schema may not be well defined.

I raised the bug at pycontribs/zuul-lint#2

[Julian]

No worries at all! You're spot on on this one looks like, so if there's a way to improve the error message for just these kinds of cases sounds great, but yeah it can get a bit tricky unfortunately... Needs some thought.

…

On Sun, Jan 12, 2020, 20:32 Sorin Sbarnea ***@***.***> wrote: You are right I use validate at https://github.com/pycontribs/zuul-lint/blob/master/zuul_lint/__main__.py#L22 but I am going to test more advanced API. Thanks a lot fo the quick reply. That new linter I am writing is very new stuff, is not even 10 hours since I started it. I am also new to jsonschema in general and I do not rule out that my schema may not be well defined. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#646>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACQQXXUNPGZ3BEJ6K77EFLQ5N47HANCNFSM4KFZN2QQ> .

[willson-chen]

Same as #698, since validator of oneOf and anyOf, the error instance with longest ValidationError.path will be considered as the best match in best_match. And if the length of all ValidationError.path are the same, the first error instance will be returned default.

[Julian]

I believe the commit I just pushed should make these kinds of cases clearer (by not recursing into them). Feel free to try it out and provide feedback, otherwise a release with it should come out in a day or two.

[ssbarnea]

@Julian I tested 4.8.0 and I can see the benefits in improved messaging. It broke two tests on my suite but that only helped me upgrade it and fix the tests. Thanks for working on this.

[Julian]

Nice (other than the breakage :)! Thanks for the feedback and the report.