Cannot take allOf a non-object #674

ahuang11 · 2022-09-23T23:54:53Z

Describe the bug

Thanks for creating this; I love how explicit it is on errors and warnings!

I am currently encountering this warning:

WARNING parsing POST /2.1/jobs/create within default. Endpoint will not be generated.

Cannot take allOf a non-object

Raised from here. However, I don't fully understand this error, specifically what is a non-object.

Here is the contents that it warns on:

  /2.1/jobs/create:
  ...
      requestBody:
        required: true
        content:
          application/json:
            schema:
              allOf:
                - $ref: '#/components/schemas/JobSettings'
                - $ref: '#/components/schemas/AccessControlList'

I double checked to make sure these reference schemas exist:

    JobSettings:
      properties:
        name:
          type: string
          example: A multitask job
...

    AccessControlList:
      type: object
      properties:
        access_control_list:
          type: array
          description: List of permissions to set on the job.
          items:
            $ref: '#/components/schemas/AccessControlRequest'

Any thoughts and suggestions appreciated!

To Reproduce
Steps to reproduce the behavior:

wget https://docs.databricks.com/_static/api-refs/jobs-2.1-aws.yaml
Try running openapi-python-client generate --path jobs-2.1-aws.yaml which results in multiple errors similar to:

components -> schemas -> GitSource -> properties -> git_url -> required
  value is not a valid list (type=type_error.list)

Manually update the offenders' required from bool to list, e.g.

required: false

to

required:
  - false

Rerun openapi-python-client generate --path jobs-2.1-aws.yaml
Results in multiple warnings:


WARNING parsing POST /2.1/jobs/create within default. Endpoint will not be generated.

Cannot take allOf a non-object


WARNING parsing POST /2.1/jobs/run-now within default. Endpoint will not be generated.

Cannot take allOf a non-object

Expected behavior
A clear and concise description of what you expected to happen.

No warnings; succesful generation. I am happy to help contribute a PR if this is indeed a bug and if we can figure out the issue.

OpenAPI Spec File
A link to your openapi.json which produces this issue.
https://docs.databricks.com/_static/api-refs/jobs-2.1-aws.yaml (needs manual modification to the yaml to make it more compliant)

Desktop (please complete the following information):

OS: Mac M1
Python Version: 3.9.13
openapi-python-client version 0.11.6

Additional context
Add any other context about the problem here.

The text was updated successfully, but these errors were encountered:

dbanty · 2022-09-24T18:09:25Z

A schema is only considered an object if it has type: object. It seems like JobSettings does not have that set. Maybe we should assume type: object if properties is set? I'm not sure it makes sense to have properties if not an object 🤔.

dbanty · 2022-09-24T18:16:24Z

Side note, that required: list should be a list of the properties that are required. (e.g., required: ["name", "git_source"]) required should also be defined on the schema itself, not on each property therein.

I think there was an older version of the spec (maybe Swagger?) which had required: set on each property? Sounds like it was not updated properly. What you probably want is to process each model, loop through the properties, find out which ones have required: true, and move those into a required array up a level.

ahuang11 · 2022-09-25T06:08:26Z

Thanks for the quick reply!

A schema is only considered an object if it has type: object. It seems like JobSettings does not have that set. Maybe we should assume type: object if properties is set? I'm not sure it makes sense to have properties if not an object 🤔.

That makes sense to me. Would you like me to create a PR for that? If so, can you point me to the relevant code section?

And also thanks for the pointer on required; I will try these out tomorrow and see if I can generate the output fully.

dbanty · 2022-09-25T06:21:20Z

Sure thing, I think adding an or to line 649 of openapi_python_client/parser/properties/__init__.py (the if statement looking for type == object should do it.

The easiest way to test will be via an end-to-end test, which should be described in CONTRIBUTING.md (let me know if it's not clear so I can fix it!)

ahuang11 · 2022-09-25T18:42:24Z

Thanks for all your pointers; adding type: object to JobSettings fixed those warnings and I was able to generate the code without any warnings (after correcting below).

When I do the same to RunParameters, I get this:

Unable to parse schema /components/schemas/RunParameters

Unable to parse this part of your OpenAPI document: : type array must have items defined

Schema(title=None, multipleOf=None, maximum=None, exclusiveMaximum=None, minimum=None, exclusiveMinimum=None, maxLength=None, minLength=None, pattern=None, maxItems=None, minItems=None, uniqueItems=None, maxProperties=None, minProperties=None, required=None, enum=None, type=<DataType.ARRAY: 'array'>, allOf=[], oneOf=[], anyOf=[], schema_not=None, items=None, properties=None, additionalProperties=None, description='An array of commands to execute for jobs with the dbt task, for example `"dbt_commands": ["dbt deps", "dbt seed", "dbt run"]`', schema_format=None, default=None, nullable=False, discriminator=None, readOnly=None, writeOnly=None, xml=None, externalDocs=None, example=['dbt deps', 'dbt seed', 'dbt run'], deprecated=None)

So I updated:

dbt_commands:
  ...
  items:
    type: string

Wondering if we should also default to string if no type is defined for an array, and warn accordingly?

ahuang11 · 2022-09-25T18:56:00Z

Also, when I tried adding an or statement:

    if data.type == oai.DataType.OBJECT or data.allOf or (data.type is None and data.properties):

I get a cascading error effect:

Updating jobs_api_2_1_client
Warning(s) encountered while generating. Client was generated, but some pieces may be missing

WARNING parsing POST /2.1/jobs/create within default. Endpoint will not be generated.

Reference #/components/schemas/JobSettings not found


WARNING parsing GET /2.1/jobs/list within default.

Cannot parse response for status code 200 (Could not find reference in parsed models or enums), response will be ommitted from generated client

Reference(ref='#/components/schemas/Job')

WARNING parsing GET /2.1/jobs/get within default.

Cannot parse response for status code 200 (Could not find reference in parsed models or enums), response will be ommitted from generated client

Reference(ref='#/components/schemas/JobSettings', description='Settings for this job and all of its runs. These settings can be updated using the [Reset](https://docs.databricks.com/dev-tools/api/latest/jobs.html#operation/JobsReset) or [Update](https://docs.databricks.com/dev-tools/api/latest/jobs.html#operation/JobsUpdate) endpoints.')

WARNING parsing POST /2.1/jobs/reset within default. Endpoint will not be generated.

Could not find reference in parsed models or enums

Reference(ref='#/components/schemas/JobSettings', description='The new settings of the job. These settings completely replace the old settings.\n\nChanges to the field `JobSettings.timeout_seconds` are applied to active runs. Changes to other fields are applied to future runs only.')

WARNING parsing POST /2.1/jobs/update within default. Endpoint will not be generated.

Could not find reference in parsed models or enums

Reference(ref='#/components/schemas/JobSettings', description='The new settings for the job. Any top-level fields specified in `new_settings` are completely replaced. Partially updating nested fields is not supported.\n\nChanges to the field `JobSettings.timeout_seconds` are applied to active runs. Changes to other fields are applied to future runs only.')

WARNING parsing GET /2.1/jobs/runs/list within default.

Cannot parse response for status code 200 (Could not find reference in parsed models or enums), response will be ommitted from generated client

Reference(ref='#/components/schemas/Run')

WARNING parsing GET /2.1/jobs/runs/get within default.

Cannot parse response for status code 200 (Reference #/components/schemas/Run not found), response will be ommitted from generated client


WARNING parsing GET /2.1/jobs/runs/get-output within default.

Cannot parse response for status code 200 (Could not find reference in parsed models or enums), response will be ommitted from generated client

Reference(ref='#/components/schemas/Run', description='All details of the run except for its output.')

Unable to parse schema /components/schemas/Job

Unable to parse this part of your OpenAPI document: : Could not find reference in parsed models or enums

Reference(ref='#/components/schemas/JobSettings', description='Settings for this job and all of its runs. These settings can be updated using the `resetJob` method.')

Unable to parse schema /components/schemas/JobSettings

invalid data in items of array named "tasks": Could not find reference in parsed models or enums

Reference(ref='#/components/schemas/JobTaskSettings')

Unable to parse schema /components/schemas/JobTaskSettings

Unable to parse this part of your OpenAPI document: : Could not find reference in parsed models or enums

Reference(ref='#/components/schemas/DbtTask', description='If dbt_task, indicates that this must execute a dbt task. It requires both Databricks SQL and the ability to use a Serverless SQL warehouse.')

Unable to parse schema /components/schemas/RunTask

Unable to parse this part of your OpenAPI document: : Could not find reference in parsed models or enums

Reference(ref='#/components/schemas/DbtTask', description='If dbt_task, indicates that this must execute a dbt task. It requires both Databricks SQL and the ability to use a Serverless SQL warehouse.')

Unable to parse schema /components/schemas/Run

invalid data in items of array named "tasks": Could not find reference in parsed models or enums

Reference(ref='#/components/schemas/RunTask')

Unable to parse schema /components/schemas/DbtTask

Unable to parse this part of your OpenAPI document: : type array must have items defined

Schema(title=None, multipleOf=None, maximum=None, exclusiveMaximum=None, minimum=None, exclusiveMinimum=None, maxLength=None, minLength=None, pattern=None, maxItems=None, minItems=None, uniqueItems=None, maxProperties=None, minProperties=None, required=None, enum=None, type=<DataType.ARRAY: 'array'>, allOf=[], oneOf=[], anyOf=[], schema_not=None, items=None, properties=None, additionalProperties=None, description='A list of dbt commands to execute. All commands must start with `dbt`. This parameter must not be empty. A maximum of up to 10 commands can be provided.', schema_format=None, default=None, nullable=False, discriminator=None, readOnly=None, writeOnly=None, xml=None, externalDocs=None, example=['dbt deps', 'dbt seed', 'dbt run --models 123'], deprecated=None)

ahuang11 · 2022-10-21T00:29:32Z

Interestingly, all I had to do was fix one, and then the cascading warnings all disappeared.

Specifically:

Unable to parse schema /components/schemas/DbtTask

Unable to parse this part of your OpenAPI document: : type array must have items defined

I added:

          items:
            type: string

Should array default to type string if it's not specified?

@ahuang11

…ect [#691, #674]. Thanks @ahuang11! Co-authored-by: Dylan Anthony <[email protected]>

ahuang11 added the 🐞bug Something isn't working label Sep 23, 2022

ahuang11 mentioned this issue Oct 21, 2022

fix: If data.type is None but has data.properties, assume type is object #691

Merged

dbanty closed this as completed in #691 Nov 12, 2022

dbanty added a commit that referenced this issue Nov 12, 2022

fix: If data.type is None but has data.properties, assume type is obj…

d4e14f9

…ect [#691, #674]. Thanks @ahuang11! Co-authored-by: Dylan Anthony <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

Cannot take allOf a non-object #674

Cannot take allOf a non-object #674

ahuang11 commented Sep 23, 2022 •

edited

Loading

dbanty commented Sep 24, 2022

Uh oh!

dbanty commented Sep 24, 2022

Uh oh!

ahuang11 commented Sep 25, 2022

Uh oh!

dbanty commented Sep 25, 2022

Uh oh!

ahuang11 commented Sep 25, 2022 •

edited

Loading

Uh oh!

ahuang11 commented Sep 25, 2022 •

edited

Loading

Uh oh!

ahuang11 commented Oct 21, 2022

Uh oh!

Uh oh!

Cannot take allOf a non-object #674

Cannot take allOf a non-object #674

Comments

ahuang11 commented Sep 23, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

dbanty commented Sep 24, 2022

Uh oh!

dbanty commented Sep 24, 2022

Uh oh!

ahuang11 commented Sep 25, 2022

Uh oh!

dbanty commented Sep 25, 2022

Uh oh!

ahuang11 commented Sep 25, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ahuang11 commented Sep 25, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ahuang11 commented Oct 21, 2022

Uh oh!

ahuang11 commented Sep 23, 2022 •

edited

Loading

ahuang11 commented Sep 25, 2022 •

edited

Loading

ahuang11 commented Sep 25, 2022 •

edited

Loading