Skip to content

Connector SyncJob API spec #2503

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 10 commits into from
Apr 22, 2024
Merged
Show file tree
Hide file tree
Changes from 5 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
390 changes: 390 additions & 0 deletions output/openapi/elasticsearch-serverless-openapi.json

Large diffs are not rendered by default.

908 changes: 908 additions & 0 deletions output/schema/schema.json

Large diffs are not rendered by default.

6 changes: 6 additions & 0 deletions output/schema/validation-errors.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

80 changes: 80 additions & 0 deletions output/typescript/types.ts

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

5 changes: 5 additions & 0 deletions specification/_doc_ids/table.csv
Original file line number Diff line number Diff line change
Expand Up @@ -78,6 +78,11 @@ cluster-stats,https://www.elastic.co/guide/en/elasticsearch/reference/{branch}/c
cluster-update-settings,https://www.elastic.co/guide/en/elasticsearch/reference/{branch}/cluster-update-settings.html
cluster,https://www.elastic.co/guide/en/elasticsearch/reference/{branch}/cluster.html
common-options,https://www.elastic.co/guide/en/elasticsearch/reference/{branch}/common-options.html
connector-sync-job-cancel,https://www.elastic.co/guide/en/elasticsearch/reference/{branch}/cancel-connector-sync-job-api.html
connector-sync-job-delete,https://www.elastic.co/guide/en/elasticsearch/reference/{branch}/delete-connector-sync-job-api.html
connector-sync-job-get,https://www.elastic.co/guide/en/elasticsearch/reference/{branch}/get-connector-sync-job-api.html
connector-sync-job-post,https://www.elastic.co/guide/en/elasticsearch/reference/{branch}/create-connector-sync-job-api.html
connector-sync-job-list,https://www.elastic.co/guide/en/elasticsearch/reference/{list}/list-connector-sync-jobs-api.html
connector-checkin,https://www.elastic.co/guide/en/elasticsearch/reference/{branch}/check-in-connector-api.html
connector-delete,https://www.elastic.co/guide/en/elasticsearch/reference/{branch}/delete-connector-api.html
connector-get,https://www.elastic.co/guide/en/elasticsearch/reference/{branch}/get-connector-api.html
Expand Down
27 changes: 27 additions & 0 deletions specification/_json_spec/connector_sync_job.cancel.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
{
"connector_sync_job.cancel": {
"documentation": {
"url": "https://www.elastic.co/guide/en/elasticsearch/reference/master/cancel-connector-sync-job-api.html",
"description": "Cancels a connector sync job."
},
"stability": "experimental",
"visibility": "public",
"headers": {
"accept": ["application/json"]
},
"url": {
"paths": [
{
"path": "/_connector/_sync_job/{connector_sync_job_id}/_cancel",
"methods": ["PUT"],
"parts": {
"connector_sync_job_id": {
"type": "string",
"description": "The unique identifier of the connector sync job to be canceled"
}
}
}
]
}
}
}
27 changes: 27 additions & 0 deletions specification/_json_spec/connector_sync_job.delete.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
{
"connector_sync_job.delete": {
"documentation": {
"url": "https://www.elastic.co/guide/en/elasticsearch/reference/master/delete-connector-sync-job-api.html",
"description": "Deletes a connector sync job."
},
"stability": "experimental",
"visibility": "public",
"headers": {
"accept": ["application/json"]
},
"url": {
"paths": [
{
"path": "/_connector/_sync_job/{connector_sync_job_id}",
"methods": ["DELETE"],
"parts": {
"connector_sync_job_id": {
"type": "string",
"description": "The unique identifier of the connector sync job to be deleted."
}
}
}
]
}
}
}
27 changes: 27 additions & 0 deletions specification/_json_spec/connector_sync_job.get.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
{
"connector_sync_job.get": {
"documentation": {
"url": "https://www.elastic.co/guide/en/elasticsearch/reference/master/get-connector-sync-job-api.html",
"description": "Returns the details about a connector sync job."
},
"stability": "experimental",
"visibility": "public",
"headers": {
"accept": ["application/json"]
},
"url": {
"paths": [
{
"path": "/_connector/_sync_job/{connector_sync_job_id}",
"methods": ["GET"],
"parts": {
"connector_sync_job_id": {
"type": "string",
"description": "The unique identifier of the connector sync job to be returned."
}
}
}
]
}
}
}
45 changes: 45 additions & 0 deletions specification/_json_spec/connector_sync_job.list.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
{
"connector_sync_job.list": {
"documentation": {
"url": "https://www.elastic.co/guide/en/elasticsearch/reference/master/list-connector-sync-jobs-api.html",
"description": "Lists all connector sync jobs."
},
"stability": "experimental",
"visibility": "public",
"headers": {
"accept": ["application/json"]
},
"url": {
"paths": [
{
"path": "/_connector/_sync_job",
"methods": ["GET"]
}
]
},
"params": {
"from": {
"type": "int",
"default": 0,
"description": "Starting offset (default: 0)"
},
"size": {
"type": "int",
"default": 100,
"description": "specifies a max number of results to get (default: 100)"
},
"status": {
"type": "string",
"description": "Sync job status, which sync jobs are fetched for"
},
"connector_id": {
"type": "string",
"description": "Id of the connector to fetch the sync jobs for"
},
"job_type": {
"type": "list",
"description": "A comma-separated list of job types"
}
}
}
}
26 changes: 26 additions & 0 deletions specification/_json_spec/connector_sync_job.post.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
{
"connector_sync_job.post": {
"documentation": {
"url": "https://www.elastic.co/guide/en/elasticsearch/reference/master/create-connector-sync-job-api.html",
"description": "Creates a connector sync job."
},
"stability": "experimental",
"visibility": "public",
"headers": {
"accept": ["application/json"],
"content_type": ["application/json"]
},
"url": {
"paths": [
{
"path": "/_connector/_sync_job",
"methods": ["POST"]
}
]
},
"body": {
"description": "The connector sync job data.",
"required": true
}
}
}
70 changes: 70 additions & 0 deletions specification/connector_sync_job/_types/SyncJob.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,70 @@
/*
* Licensed to Elasticsearch B.V. under one or more contributor
* license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright
* ownership. Elasticsearch B.V. licenses this file to you under
* the Apache License, Version 2.0 (the "License"); you may
* not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing,
* software distributed under the License is distributed on an
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
* KIND, either express or implied. See the License for the
* specific language governing permissions and limitations
* under the License.
*/
import { Id } from '@_types/common'
import { long } from '@_types/Numeric'
import { Dictionary } from '@spec_utils/Dictionary'
import { UserDefinedValue } from '@spec_utils/UserDefinedValue'
import {
ConnectorConfiguration,
FilteringConfig,
IngestPipelineParams,
SyncStatus
} from '../../connector/_types/Connector'

interface SyncJobConnectorReference {
configuration: ConnectorConfiguration
filtering: FilteringConfig
id: Id
index_name: string
language?: string
pipeline: IngestPipelineParams
service_type: string
}

enum SyncJobType {
full,
incremental,
access_control
}

enum TriggerMethod {
on_demand,
scheduled
}

export interface ConnectorSyncJob {
cancelation_requested_at?: string
canceled_at?: string
completed_at?: string
connector: SyncJobConnectorReference
created_at: string
deleted_document_count: long
error?: string
id: Id
indexed_document_count: long
indexed_document_volume: long
job_type: SyncJobType
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can job_type be a SyncJobType or SyncJobType[] enum consistently?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same for trigger_method

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In a ConnectorSyncJob representation they are not an array, there is singe value associated with trigger_method and job_type

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What I mean is, in this class job_type and trigger_method are enums, while in the classes SyncJobPostRequest.ts and SyncJobListRequest.ts they are Name/Names, which are just strings, can't they be the same enum everywhere?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ahh I see what you mean! For job_type this is passed as a comma-separated list in the URL param of SyncJobListRequest. I was not sure if defining this as SyncJobType[] would work with URL param list, and I saw Names in other examples of requests that pass comma separated lists in url params.

If the usage of SyncJobType[] and SyncJobTriggerMethod is correct, I can adapt this in the PR.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks :) I think it's more coherent this way from the clients perspective, to always provide enums where there's fixed values

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I addressed that feedback in the updated PR :)

last_seen?: string
metadata: Dictionary<string, UserDefinedValue>
started_at?: string
status: SyncStatus
total_document_count?: long
trigger_method: TriggerMethod
worker_hostname?: string
}
36 changes: 36 additions & 0 deletions specification/connector_sync_job/cancel/SyncJobCancelRequest.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
/*
* Licensed to Elasticsearch B.V. under one or more contributor
* license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright
* ownership. Elasticsearch B.V. licenses this file to you under
* the Apache License, Version 2.0 (the "License"); you may
* not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing,
* software distributed under the License is distributed on an
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
* KIND, either express or implied. See the License for the
* specific language governing permissions and limitations
* under the License.
*/
import { RequestBase } from '@_types/Base'
import { Id } from '@_types/common'

/**
* Cancels a connector sync job.
* @rest_spec_name connector_sync_job.cancel
* @availability stack since=8.12.0 stability=experimental
* @availability serverless stability=experimental visibility=public
* @doc_id connector-sync-job-cancel
*/
export interface Request extends RequestBase {
path_parts: {
/**
* The unique identifier of the connector sync job
*/
connector_sync_job_id: Id
}
}
Loading