Skip to content

[Failure store] Add data stream options to the es specs #4414

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 8 commits into
base: main
Choose a base branch
from
Open

[Failure store] Add data stream options to the es specs #4414

wants to merge 8 commits into from

Conversation

gmarouli
Copy link
Contributor

As part of the failure store feature, we introduce the data stream options. Data stream options have the following format:

{
  "failure_store": {
    "enabled": true, 
    "lifecycle": {
      "enabled": true,
      "data_retention": "7d" 
    }
  }
}

Currently they only configure the failure store and they can be set in their dedicated APIs and via the templates.

@gmarouli gmarouli requested a review from jbaiera May 26, 2025 15:50
@gmarouli gmarouli changed the title Add data stream options to the es spex [Failure store] Add data stream options to the es spex May 26, 2025
gmarouli
gmarouli commented May 26, 2025
View reviewed changes
output/schema/validation-errors.json
Comment on lines +60 to +65
"indices.get_data_stream_options": {
"request": [
"Request: missing json spec query parameter 'include_defaults'"
],
"response": []
},
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

About this param. Currently, this flag has no effect. It is already in the API because initially, we considered to expose the default rollover via this flag, but we did not do this. @jbaiera can we clean it up from the API, or should I add it here and explain that it doesn't have an effect yet?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the flag has no effect and we aren't planning on adding anything for it, we should probably just remove it from the API. Considering the api is so recently new and it hasn't been fully documented yet, I think it's a pretty low impact change to pursue.

gmarouli
gmarouli commented May 26, 2025
View reviewed changes
...on/indices/get_data_lifecycle/examples/response/IndicesGetDataLifecycleResponseExample1.yaml
@@ -1,5 +1,5 @@
# summary:
description: A successful response from `GET _lifecycle/stats?human&pretty`.
description: A successful response from `GET /_data_stream/{name}/_lifecycle?human&pretty`.
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Drive-by fix, this is not the lifecycle stats API.

@gmarouli gmarouli mentioned this pull request May 26, 2025
Open
@github-actions GitHub Actions
Copy link
Contributor

Following you can find the validation results for the APIs you have changed.

API Status Request Response
cluster.allocation_explain 🟢 4/4 4/4
cluster.delete_component_template 🟢 2/2 2/2
cluster.delete_voting_config_exclusions 🟢 1/1 1/1
cluster.exists_component_template ⚪ Missing test Missing test
cluster.get_component_template 🟢 12/12 12/12
cluster.get_settings 🟢 7/7 7/7
cluster.health 🟢 157/157 157/157
cluster.info 🟢 4/4 4/4
cluster.pending_tasks 🟢 3/3 3/3
cluster.post_voting_config_exclusions 🟢 5/5 5/5
cluster.put_component_template 🟢 29/29 29/29
cluster.put_settings 🟢 59/59 58/58
cluster.remote_info 🟢 3/3 3/3
cluster.reroute 🟢 8/8 7/7
cluster.state 🟢 79/79 78/78
cluster.stats 🔴 27/27 0/27
indices.add_block 🟢 2/2 2/2
indices.analyze 🟢 246/246 246/246
indices.cancel_migrate_reindex 🟢 2/2 2/2
indices.clear_cache 🟢 4/4 4/4
indices.clone 🟢 5/5 5/5
indices.close 🟢 49/49 49/49
indices.create_data_stream 🟢 31/31 31/31
indices.create_from 🟢 4/4 4/4
indices.create 🔴 1179/1225 1225/1225
indices.data_streams_stats 🟢 4/4 4/4
indices.delete_alias 🔴 12/12 5/12
indices.delete_data_lifecycle ⚪ Missing test Missing test
indices.delete_data_stream_options 🟠 Missing type Missing type
indices.delete_data_stream 🟢 28/28 28/28
indices.delete_index_template 🟢 1/1 1/1
indices.delete_template 🟢 9/9 9/9
indices.delete 🟢 105/105 105/105
indices.disk_usage 🟢 6/6 6/6
indices.downsample 🟢 4/4 4/4
indices.exists_alias 🟢 37/37 37/37
indices.exists_index_template ⚪ Missing test Missing test
indices.exists_template 🟢 15/15 15/15
indices.exists 🟢 39/39 39/39
indices.explain_data_lifecycle ⚪ Missing test Missing test
indices.field_usage_stats 🟢 5/5 5/5
indices.flush 🟢 41/41 41/41
indices.forcemerge 🟢 12/12 12/12
indices.get_alias 🟢 68/68 68/68
indices.get_data_lifecycle_stats ⚪ Missing test Missing test
indices.get_data_lifecycle ⚪ Missing test Missing test
indices.get_data_stream_options ⚪ Missing test Missing test
indices.get_data_stream_settings 🟠 Missing type Missing type
indices.get_data_stream 🔴 11/11 3/11
indices.get_field_mapping 🟢 15/15 15/15
indices.get_index_template 🟢 21/21 21/21
indices.get_mapping 🟢 142/142 142/142
indices.get_migrate_reindex_status 🟢 1/1 1/1
indices.get_settings 🔴 76/76 47/76
indices.get_template 🟢 32/32 32/32
indices.get 🟢 55/55 55/55
indices.migrate_reindex 🟢 2/2 2/2
indices.migrate_to_data_stream ⚪ Missing test Missing test
indices.modify_data_stream ⚪ Missing test Missing test
indices.open 🟢 20/20 20/20
indices.promote_data_stream ⚪ Missing test Missing test
indices.put_alias 🔴 58/58 20/58
indices.put_data_lifecycle ⚪ Missing test Missing test
indices.put_data_stream_options ⚪ Missing test Missing test
indices.put_data_stream_settings 🟠 Missing type Missing type
indices.put_index_template 🟢 68/68 68/68
indices.put_mapping 🔴 124/125 128/128
indices.put_settings 🔴 56/58 58/58
indices.put_template 🔴 45/46 46/46
indices.recovery 🟢 9/9 9/9
indices.refresh 🟢 283/283 283/283
indices.reload_search_analyzers 🟢 2/2 2/2
indices.resolve_cluster 🟢 13/13 13/13
indices.resolve_index 🟢 6/6 6/6
indices.rollover 🔴 27/27 2/27
indices.segments 🔴 6/6 5/6
indices.shard_stores 🔴 5/5 4/5
indices.shrink 🟢 5/5 5/5
indices.simulate_index_template 🔴 6/7 5/7
indices.simulate_template 🔴 5/5 4/5
indices.split 🟢 11/11 11/11
indices.stats 🔴 123/123 31/122
indices.update_aliases 🔴 26/26 7/26
indices.validate_query 🟢 12/12 12/12

You can validate these APIs yourself by using the make validate target.

@github-actions GitHub Actions
Copy link
Contributor

Following you can find the validation results for the APIs you have changed.

API Status Request Response
cluster.allocation_explain 🟢 4/4 4/4
cluster.delete_component_template 🟢 2/2 2/2
cluster.delete_voting_config_exclusions 🟢 1/1 1/1
cluster.exists_component_template ⚪ Missing test Missing test
cluster.get_component_template 🟢 12/12 12/12
cluster.get_settings 🟢 7/7 7/7
cluster.health 🟢 157/157 157/157
cluster.info 🟢 4/4 4/4
cluster.pending_tasks 🟢 3/3 3/3
cluster.post_voting_config_exclusions 🟢 5/5 5/5
cluster.put_component_template 🟢 29/29 29/29
cluster.put_settings 🟢 59/59 58/58
cluster.remote_info 🟢 3/3 3/3
cluster.reroute 🟢 8/8 7/7
cluster.state 🟢 79/79 78/78
cluster.stats 🔴 27/27 0/27
indices.add_block 🟢 2/2 2/2
indices.analyze 🟢 246/246 246/246
indices.cancel_migrate_reindex 🟢 2/2 2/2
indices.clear_cache 🟢 4/4 4/4
indices.clone 🟢 5/5 5/5
indices.close 🟢 49/49 49/49
indices.create_data_stream 🟢 31/31 31/31
indices.create_from 🟢 4/4 4/4
indices.create 🔴 1179/1225 1225/1225
indices.data_streams_stats 🟢 4/4 4/4
indices.delete_alias 🔴 12/12 5/12
indices.delete_data_lifecycle ⚪ Missing test Missing test
indices.delete_data_stream_options ⚪ Missing test Missing test
indices.delete_data_stream 🟢 28/28 28/28
indices.delete_index_template 🟢 1/1 1/1
indices.delete_template 🟢 9/9 9/9
indices.delete 🟢 105/105 105/105
indices.disk_usage 🟢 6/6 6/6
indices.downsample 🟢 4/4 4/4
indices.exists_alias 🟢 37/37 37/37
indices.exists_index_template ⚪ Missing test Missing test
indices.exists_template 🟢 15/15 15/15
indices.exists 🟢 39/39 39/39
indices.explain_data_lifecycle ⚪ Missing test Missing test
indices.field_usage_stats 🟢 5/5 5/5
indices.flush 🟢 41/41 41/41
indices.forcemerge 🟢 12/12 12/12
indices.get_alias 🟢 68/68 68/68
indices.get_data_lifecycle_stats ⚪ Missing test Missing test
indices.get_data_lifecycle ⚪ Missing test Missing test
indices.get_data_stream_options ⚪ Missing test Missing test
indices.get_data_stream_settings 🟠 Missing type Missing type
indices.get_data_stream 🔴 11/11 3/11
indices.get_field_mapping 🟢 15/15 15/15
indices.get_index_template 🟢 21/21 21/21
indices.get_mapping 🟢 142/142 142/142
indices.get_migrate_reindex_status 🟢 1/1 1/1
indices.get_settings 🔴 76/76 47/76
indices.get_template 🟢 32/32 32/32
indices.get 🟢 55/55 55/55
indices.migrate_reindex 🟢 2/2 2/2
indices.migrate_to_data_stream ⚪ Missing test Missing test
indices.modify_data_stream ⚪ Missing test Missing test
indices.open 🟢 20/20 20/20
indices.promote_data_stream ⚪ Missing test Missing test
indices.put_alias 🔴 58/58 20/58
indices.put_data_lifecycle ⚪ Missing test Missing test
indices.put_data_stream_options ⚪ Missing test Missing test
indices.put_data_stream_settings 🟠 Missing type Missing type
indices.put_index_template 🟢 68/68 68/68
indices.put_mapping 🔴 124/125 128/128
indices.put_settings 🔴 56/58 58/58
indices.put_template 🔴 45/46 46/46
indices.recovery 🟢 9/9 9/9
indices.refresh 🟢 283/283 283/283
indices.reload_search_analyzers 🟢 2/2 2/2
indices.resolve_cluster 🟢 13/13 13/13
indices.resolve_index 🟢 6/6 6/6
indices.rollover 🔴 27/27 2/27
indices.segments 🔴 6/6 5/6
indices.shard_stores 🔴 5/5 4/5
indices.shrink 🟢 5/5 5/5
indices.simulate_index_template 🔴 6/7 5/7
indices.simulate_template 🔴 5/5 4/5
indices.split 🟢 11/11 11/11
indices.stats 🔴 123/123 31/122
indices.update_aliases 🔴 26/26 7/26
indices.validate_query 🟢 12/12 12/12

You can validate these APIs yourself by using the make validate target.

@github-actions GitHub Actions
Copy link
Contributor

Following you can find the validation results for the APIs you have changed.

API Status Request Response
cluster.allocation_explain 🟢 4/4 4/4
cluster.delete_component_template 🟢 2/2 2/2
cluster.delete_voting_config_exclusions 🟢 1/1 1/1
cluster.exists_component_template ⚪ Missing test Missing test
cluster.get_component_template 🟢 12/12 12/12
cluster.get_settings 🟢 7/7 7/7
cluster.health 🟢 157/157 157/157
cluster.info 🟢 4/4 4/4
cluster.pending_tasks 🟢 3/3 3/3
cluster.post_voting_config_exclusions 🟢 5/5 5/5
cluster.put_component_template 🟢 29/29 29/29
cluster.put_settings 🟢 59/59 58/58
cluster.remote_info 🟢 3/3 3/3
cluster.reroute 🟢 8/8 7/7
cluster.state 🟢 79/79 78/78
cluster.stats 🔴 27/27 0/27
indices.add_block 🟢 2/2 2/2
indices.analyze 🟢 246/246 246/246
indices.cancel_migrate_reindex 🟢 2/2 2/2
indices.clear_cache 🟢 4/4 4/4
indices.clone 🟢 5/5 5/5
indices.close 🟢 49/49 49/49
indices.create_data_stream 🟢 31/31 31/31
indices.create_from 🟢 4/4 4/4
indices.create 🔴 1179/1225 1225/1225
indices.data_streams_stats 🟢 4/4 4/4
indices.delete_alias 🔴 12/12 5/12
indices.delete_data_lifecycle ⚪ Missing test Missing test
indices.delete_data_stream_options ⚪ Missing test Missing test
indices.delete_data_stream 🟢 28/28 28/28
indices.delete_index_template 🟢 1/1 1/1
indices.delete_template 🟢 9/9 9/9
indices.delete 🟢 105/105 105/105
indices.disk_usage 🟢 6/6 6/6
indices.downsample 🟢 4/4 4/4
indices.exists_alias 🟢 37/37 37/37
indices.exists_index_template ⚪ Missing test Missing test
indices.exists_template 🟢 15/15 15/15
indices.exists 🟢 39/39 39/39
indices.explain_data_lifecycle ⚪ Missing test Missing test
indices.field_usage_stats 🟢 5/5 5/5
indices.flush 🟢 41/41 41/41
indices.forcemerge 🟢 12/12 12/12
indices.get_alias 🟢 68/68 68/68
indices.get_data_lifecycle_stats ⚪ Missing test Missing test
indices.get_data_lifecycle ⚪ Missing test Missing test
indices.get_data_stream_options ⚪ Missing test Missing test
indices.get_data_stream_settings 🟠 Missing type Missing type
indices.get_data_stream 🔴 11/11 3/11
indices.get_field_mapping 🟢 15/15 15/15
indices.get_index_template 🟢 21/21 21/21
indices.get_mapping 🟢 142/142 142/142
indices.get_migrate_reindex_status 🟢 1/1 1/1
indices.get_settings 🔴 76/76 47/76
indices.get_template 🟢 32/32 32/32
indices.get 🟢 55/55 55/55
indices.migrate_reindex 🟢 2/2 2/2
indices.migrate_to_data_stream ⚪ Missing test Missing test
indices.modify_data_stream ⚪ Missing test Missing test
indices.open 🟢 20/20 20/20
indices.promote_data_stream ⚪ Missing test Missing test
indices.put_alias 🔴 58/58 20/58
indices.put_data_lifecycle ⚪ Missing test Missing test
indices.put_data_stream_options ⚪ Missing test Missing test
indices.put_data_stream_settings 🟠 Missing type Missing type
indices.put_index_template 🟢 68/68 68/68
indices.put_mapping 🔴 124/125 128/128
indices.put_settings 🔴 56/58 58/58
indices.put_template 🔴 45/46 46/46
indices.recovery 🟢 9/9 9/9
indices.refresh 🟢 283/283 283/283
indices.reload_search_analyzers 🟢 2/2 2/2
indices.resolve_cluster 🟢 13/13 13/13
indices.resolve_index 🟢 6/6 6/6
indices.rollover 🔴 27/27 2/27
indices.segments 🔴 6/6 5/6
indices.shard_stores 🔴 5/5 4/5
indices.shrink 🟢 5/5 5/5
indices.simulate_index_template 🔴 6/7 5/7
indices.simulate_template 🔴 5/5 4/5
indices.split 🟢 11/11 11/11
indices.stats 🔴 123/123 31/122
indices.update_aliases 🔴 26/26 7/26
indices.validate_query 🟢 12/12 12/12

You can validate these APIs yourself by using the make validate target.

@gmarouli gmarouli changed the title [Failure store] Add data stream options to the es spex [Failure store] Add data stream options to the es specs May 28, 2025
@github-actions GitHub Actions
Copy link
Contributor

Following you can find the validation results for the APIs you have changed.

API Status Request Response
cluster.allocation_explain 🟢 4/4 4/4
cluster.delete_component_template 🟢 2/2 2/2
cluster.delete_voting_config_exclusions 🟢 1/1 1/1
cluster.exists_component_template ⚪ Missing test Missing test
cluster.get_component_template 🟢 12/12 12/12
cluster.get_settings 🟢 7/7 7/7
cluster.health 🟢 157/157 157/157
cluster.info 🟢 4/4 4/4
cluster.pending_tasks 🟢 3/3 3/3
cluster.post_voting_config_exclusions 🟢 5/5 5/5
cluster.put_component_template 🟢 29/29 29/29
cluster.put_settings 🟢 59/59 58/58
cluster.remote_info 🟢 3/3 3/3
cluster.reroute 🟢 8/8 7/7
cluster.state 🟢 79/79 78/78
cluster.stats 🔴 27/27 0/27
indices.add_block 🟢 2/2 2/2
indices.analyze 🟢 246/246 246/246
indices.cancel_migrate_reindex 🟢 2/2 2/2
indices.clear_cache 🟢 4/4 4/4
indices.clone 🟢 5/5 5/5
indices.close 🟢 49/49 49/49
indices.create_data_stream 🟢 31/31 31/31
indices.create_from 🟢 4/4 4/4
indices.create 🔴 1179/1225 1225/1225
indices.data_streams_stats 🟢 4/4 4/4
indices.delete_alias 🔴 12/12 5/12
indices.delete_data_lifecycle ⚪ Missing test Missing test
indices.delete_data_stream_options ⚪ Missing test Missing test
indices.delete_data_stream 🟢 28/28 28/28
indices.delete_index_template 🟢 1/1 1/1
indices.delete_template 🟢 9/9 9/9
indices.delete 🟢 105/105 105/105
indices.disk_usage 🟢 6/6 6/6
indices.downsample 🟢 4/4 4/4
indices.exists_alias 🟢 37/37 37/37
indices.exists_index_template ⚪ Missing test Missing test
indices.exists_template 🟢 15/15 15/15
indices.exists 🟢 39/39 39/39
indices.explain_data_lifecycle ⚪ Missing test Missing test
indices.field_usage_stats 🟢 5/5 5/5
indices.flush 🟢 41/41 41/41
indices.forcemerge 🟢 12/12 12/12
indices.get_alias 🟢 68/68 68/68
indices.get_data_lifecycle_stats ⚪ Missing test Missing test
indices.get_data_lifecycle ⚪ Missing test Missing test
indices.get_data_stream_options ⚪ Missing test Missing test
indices.get_data_stream_settings 🟠 Missing type Missing type
indices.get_data_stream 🔴 11/11 3/11
indices.get_field_mapping 🟢 15/15 15/15
indices.get_index_template 🟢 21/21 21/21
indices.get_mapping 🟢 142/142 142/142
indices.get_migrate_reindex_status 🟢 1/1 1/1
indices.get_settings 🔴 76/76 47/76
indices.get_template 🟢 32/32 32/32
indices.get 🟢 55/55 55/55
indices.migrate_reindex 🟢 2/2 2/2
indices.migrate_to_data_stream ⚪ Missing test Missing test
indices.modify_data_stream ⚪ Missing test Missing test
indices.open 🟢 20/20 20/20
indices.promote_data_stream ⚪ Missing test Missing test
indices.put_alias 🔴 58/58 20/58
indices.put_data_lifecycle ⚪ Missing test Missing test
indices.put_data_stream_options ⚪ Missing test Missing test
indices.put_data_stream_settings 🟠 Missing type Missing type
indices.put_index_template 🟢 68/68 68/68
indices.put_mapping 🔴 124/125 128/128
indices.put_settings 🔴 56/58 58/58
indices.put_template 🔴 45/46 46/46
indices.recovery 🟢 9/9 9/9
indices.refresh 🟢 283/283 283/283
indices.reload_search_analyzers 🟢 2/2 2/2
indices.resolve_cluster 🟢 13/13 13/13
indices.resolve_index 🟢 6/6 6/6
indices.rollover 🔴 27/27 2/27
indices.segments 🔴 6/6 5/6
indices.shard_stores 🔴 5/5 4/5
indices.shrink 🟢 5/5 5/5
indices.simulate_index_template 🔴 6/7 5/7
indices.simulate_template 🔴 5/5 4/5
indices.split 🟢 11/11 11/11
indices.stats 🔴 123/123 31/122
indices.update_aliases 🔴 26/26 7/26
indices.validate_query 🟢 12/12 12/12

You can validate these APIs yourself by using the make validate target.

jbaiera
jbaiera approved these changes May 30, 2025
View reviewed changes
Copy link
Member

@jbaiera jbaiera left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changes LGTM!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jbaiera jbaiera jbaiera approved these changes

Assignees
No one assigned
Labels
backport 8.19 specification
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@gmarouli @jbaiera