[OpenAPI] Merge multiple paths in a single operation

[swallez]

Fixes #4277

Adds an option in the OpenAPI export to group the many operations resulting from endpoints with multiple paths in a single operation. This works around the current issue where there are many duplicated operations in the nav bar and in the documentation.

A "main path" is chosen to become the path of the single OpenAPI operation that is produced. This is the endpoint's longest path, which will show all path parameters, be them required or optional. All paths are then listed at the top of the description (including the main one), see rendering below.

WIP: do not merge this PR now. This option must be configurable, as it is destructive from an OpenAPI schema perspective since it removes some operations.

---

Search endpoint rendered by Bump with this PR:

[lcawl]

Looking at the current output, for example:

  "description": "**All methods and paths for this operation:**\n\n<div><operation-summary>\n 
<span class=\"operation-verb get\">GET</span>\n 
<span class=\"operation-path\">/_analyze</span>\n
</operation-summary></div>\n
...

IMO if it make it more usable in general to just accomplish this list via simple markdown, that's okay too. For example:

"Valid methods and paths for this operation include:\n
- `GET /_analyze`\n
- `POST /_analyze`\n
- `GET /{index}/_analyze`\n
- `POST /{index}/_analyze`\n
...

[lcawl]

WIP: do not merge this PR now. This option must be configurable, as it is destructive from an OpenAPI schema perspective since it removes some operations.

I see 179 operations where this new text appears in the OpenAPI document, so it's a pretty impactful change. It's not something we ideally do manually, but I think it will be necessary to confirm that:

1. none of the operations that have been removed are ones that are affected by overlays (or if they are, that the overlay is moved to the appropriate consolidated operation), and
2. none of the operations that have been removed are target URLs (e.g. from https://github.com/elastic/elasticsearch-specification/blob/main/specification/_doc_ids/table.csv, https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-doc-links/src/get_doc_links.ts, or I guess even from docs in general... which would imply that in addition to cleaning up their usage we'll likely need to create redirects just in case we don't catch all usages).

For example, if we need to create redirects, it will require information like this:

Old URL	New URL
https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-async-search-submit-1	https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-async-search-submit
https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-bulk-1	https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-bulk
https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-bulk-3	https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-bulk
https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-bulk-2	https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-bulk
https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-watcher-execute-watch-1	https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-watcher-execute-watch
https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-watcher-execute-watch-2	https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-watcher-execute-watch
https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-watcher-execute-watch-3	https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-watcher-execute-watch

[swallez]

IMO if it make it more usable in general to just accomplish this list via simple markdown

The generated markup uses the CSS classes used by Bump in the main path description to provide a consistent rendering.

I've added a commit that removes the <operation-summary> that doesn't have any benefit in this context. The remaining <div><span> still make it valid Markdown and will be rendered correctly even outside of Bump. Now can also produce plain Markdown if you prefer.

I see 179 operations where this new text appears in the OpenAPI document, so it's a pretty impactful change

Yes. As we already discussed, it cannot replace the current OpenAPI file. It has to be a new one, dedicated to the docs pipeline.

none of the operations that have been removed are target URLs

So I understand we're good on that front?

Some background: when we generate one operation for each method+path, the first one (as found in schema.json) has a "plain" (no suffix) identifier, and the others have a counter suffix (-1, -2, etc). When merging multiple path, all these suffixed operations are removed, and we only have the unsuffixed operation.

[flobernd]

Good to know there will be a dedicated OpenAPI file for the docs. This should also make it trivial to convert GHF markdown admonitions to the bump.sh syntax.

[lcawl]

Some background: when we generate one operation for each method+path, the first one (as found in schema.json) has a "plain" (no suffix) identifier, and the others have a counter suffix (-1, -2, etc). When merging multiple path, all these suffixed operations are removed, and we only have the unsuffixed operation.

That's great and should make it easier to check we don't have any of those suffixed URLs used elsewhere.
Looping in @georgewallace and @szabosteve so they can weigh in on the outstanding questions around whether it's preferrable to use plain markdown for the list of alternatives and when/if we want to do redirects and check for impact to overlays, link services, etc per #4415 (comment)

[swallez]

I rebased on main where openapi generation now accepts command-line arguments (see PR #4437).

We now have an additional make transform-to-openapi-for-docs that has a different set of parameters, in particular the one added by this PR to merge operations.

This new make target stores is result in output/openapi/elasticsearch-openapi-docs.json, which has not been added to the repository. So we must run it manually (for now) to feed Bump and enjoy a streamlined navigation.

@szabosteve, @georgewallace can you give your opinion on outputting html with classes that will keep a consistent rendering in Bump vs just plain markdown? My opinion is that html with classes is fine as this is feature that specifically targets document rending with Bump for now, and we can always revisit it later if needed.

[georgewallace]

@swallez I think for now HTML is fine. I am unsure on if bump can render markdown into html in their process. For cleaniness and ease of reading we should probably look at markdown in the future but no need to change at the moment.

[lcawl]

We now have an additional make transform-to-openapi-for-docs that has a different set of parameters, in particular the one added by this PR to merge operations.
This new make target stores is result in output/openapi/elasticsearch-openapi-docs.json, which has not been added to the repository. So we must run it manually (for now) to feed Bump and enjoy a streamlined navigation.

I presume we can update the make transform-to-openapi-for-docs ought to generate an elasticsearch-openapi-serverless-docs.json too, right? And then update the make overlay-docs and make lint-docs to target those new files as well. If you want me to add those changes to this PR via a commit lmk!