Skip to content

Commit b134508

Browse files
CristigeoCristigeo
authored
Support for transforms (#284)
* initial commit * support for Update * adding missing properties for transform * misc corrections * some documentation * support for transform start/stop * addiong options on the client calls for transform start/stop * initial commit * support for Update * adding missing properties for transform * misc corrections * some documentation * support for transform start/stop * addiong options on the client calls for transform start/stop * code updates based on feedback * updated documentation (added md template) * timeout query param is only available from v7.17.0 * fixed the messed up md file * settings promoted to individual arguments; drift detection for updatable props * check versions before using features * updated doc * support for terraform import * minor corrections/updates * corrected min version for align_checkpoints * updates on acc tests * removed some defaults; updated docs; updated acc test
1 parent a7d9905 commit b134508

File tree

12 files changed

+1956
-0
lines changed

12 files changed

+1956
-0
lines changed

‎CHANGELOG.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,7 @@
1717
}
1818
```
1919
- Add support for managing Kibana spaces ([#272](https://github.com/elastic/terraform-provider-elasticstack/pull/272))
20+
- Add support for managing Elasticsearch transforms ([#284](https://github.com/elastic/terraform-provider-elasticstack/pull/284))
2021

2122
### Fixed
2223
- Respect `ignore_unavailable` and `include_global_state` values when configuring SLM policies ([#224](https://github.com/elastic/terraform-provider-elasticstack/pull/224))

‎docs/resources/elasticsearch_transform.md

Lines changed: 170 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,170 @@
1+
---
2+
subcategory: "Transform"
3+
layout: ""
4+
page_title: "Elasticstack: elasticstack_elasticsearch_transform Resource"
5+
description: |-
6+
Manages transforms. Transforms enable you to convert existing Elasticsearch indices into summarized indices.
7+
---
8+
9+
# Resource: elasticstack_elasticsearch_transform
10+
11+
Creates, updates, starts and stops a transform. See: https://www.elastic.co/guide/en/elasticsearch/reference/current/transforms.html
12+
13+
**NOTE:** Some transform settings require a minimum Elasticsearch version. Such settings will be ignored when applied to versions below the required one (a warning will be issued in the logs).
14+
15+
## Example Usage
16+
17+
```terraform
18+
resource "elasticstack_elasticsearch_transform" "transform_with_pivot" {
19+
name = "transform-pivot"
20+
description = "A meaningful description"
21+
22+
source {
23+
indices = ["names_or_patterns_for_input_index"]
24+
}
25+
26+
destination {
27+
index = "destination_index_for_transform"
28+
}
29+
30+
pivot = jsonencode({
31+
"group_by" : {
32+
"customer_id" : {
33+
"terms" : {
34+
"field" : "customer_id",
35+
"missing_bucket" : true
36+
}
37+
}
38+
},
39+
"aggregations" : {
40+
"max_price" : {
41+
"max" : {
42+
"field" : "taxful_total_price"
43+
}
44+
}
45+
}
46+
})
47+
48+
frequency = "5m"
49+
50+
retention_policy {
51+
time {
52+
field = "order_date"
53+
max_age = "30d"
54+
}
55+
}
56+
57+
sync {
58+
time {
59+
field = "order_date"
60+
delay = "10s"
61+
}
62+
}
63+
64+
max_page_search_size = 2000
65+
66+
enabled = false
67+
defer_validation = false
68+
}
69+
```
70+
71+
<!-- schema generated by tfplugindocs -->
72+
## Schema
73+
74+
### Required
75+
76+
- `destination` (Block List, Min: 1, Max: 1) The destination for the transform. (see [below for nested schema](#nestedblock--destination))
77+
- `name` (String) Name of the transform you wish to create.
78+
- `source` (Block List, Min: 1, Max: 1) The source of the data for the transform. (see [below for nested schema](#nestedblock--source))
79+
80+
### Optional
81+
82+
- `align_checkpoints` (Boolean) Specifies whether the transform checkpoint ranges should be optimized for performance.
83+
- `dates_as_epoch_millis` (Boolean) Defines if dates in the output should be written as ISO formatted string (default) or as millis since epoch.
84+
- `deduce_mappings` (Boolean) Specifies whether the transform should deduce the destination index mappings from the transform config.
85+
- `defer_validation` (Boolean) When true, deferrable validations are not run upon creation, but rather when the transform is started. This behavior may be desired if the source index does not exist until after the transform is created. Default is `false`
86+
- `description` (String) Free text description of the transform.
87+
- `docs_per_second` (Number) Specifies a limit on the number of input documents per second. Default (unset) value disables throttling.
88+
- `enabled` (Boolean) Controls wether the transform should be started or stopped. Default is `false` (stopped).
89+
- `frequency` (String) The interval between checks for changes in the source indices when the transform is running continuously. Defaults to `1m`.
90+
- `latest` (String) The latest method transforms the data by finding the latest document for each unique key. JSON definition expected. Either 'pivot' or 'latest' must be present.
91+
- `max_page_search_size` (Number) Defines the initial page size to use for the composite aggregation for each checkpoint. Default is 500.
92+
- `metadata` (String) Defines optional transform metadata.
93+
- `num_failure_retries` (Number) Defines the number of retries on a recoverable failure before the transform task is marked as failed. The default value is the cluster-level setting num_transform_failure_retries.
94+
- `pivot` (String) The pivot method transforms the data by aggregating and grouping it. JSON definition expected. Either 'pivot' or 'latest' must be present.
95+
- `retention_policy` (Block List, Max: 1) Defines a retention policy for the transform. (see [below for nested schema](#nestedblock--retention_policy))
96+
- `sync` (Block List, Max: 1) Defines the properties transforms require to run continuously. (see [below for nested schema](#nestedblock--sync))
97+
- `timeout` (String) Period to wait for a response from Elastisearch when performing any management operation. If no response is received before the timeout expires, the operation fails and returns an error. Defaults to `30s`.
98+
- `unattended` (Boolean) In unattended mode, the transform retries indefinitely in case of an error which means the transform never fails.
99+
100+
### Read-Only
101+
102+
- `id` (String) Internal identifier of the resource
103+
104+
<a id="nestedblock--destination"></a>
105+
### Nested Schema for `destination`
106+
107+
Required:
108+
109+
- `index` (String) The destination index for the transform.
110+
111+
Optional:
112+
113+
- `pipeline` (String) The unique identifier for an ingest pipeline.
114+
115+
116+
<a id="nestedblock--source"></a>
117+
### Nested Schema for `source`
118+
119+
Required:
120+
121+
- `indices` (List of String) The source indices for the transform.
122+
123+
Optional:
124+
125+
- `query` (String) A query clause that retrieves a subset of data from the source index.
126+
- `runtime_mappings` (String) Definitions of search-time runtime fields that can be used by the transform.
127+
128+
129+
<a id="nestedblock--retention_policy"></a>
130+
### Nested Schema for `retention_policy`
131+
132+
Required:
133+
134+
- `time` (Block List, Min: 1, Max: 1) Specifies that the transform uses a time field to set the retention policy. This is currently the only supported option. (see [below for nested schema](#nestedblock--retention_policy--time))
135+
136+
<a id="nestedblock--retention_policy--time"></a>
137+
### Nested Schema for `retention_policy.time`
138+
139+
Required:
140+
141+
- `field` (String) The date field that is used to calculate the age of the document.
142+
- `max_age` (String) Specifies the maximum age of a document in the destination index.
143+
144+
145+
146+
<a id="nestedblock--sync"></a>
147+
### Nested Schema for `sync`
148+
149+
Required:
150+
151+
- `time` (Block List, Min: 1, Max: 1) Specifies that the transform uses a time field to synchronize the source and destination indices. This is currently the only supported option. (see [below for nested schema](#nestedblock--sync--time))
152+
153+
<a id="nestedblock--sync--time"></a>
154+
### Nested Schema for `sync.time`
155+
156+
Required:
157+
158+
- `field` (String) The date field that is used to identify new documents in the source.
159+
160+
Optional:
161+
162+
- `delay` (String) The time delay between the current time and the latest input data time. The default value is 60s.
163+
164+
## Import
165+
166+
Import is supported using the following syntax:
167+
168+
```shell
169+
terraform import elasticstack_elasticsearch_tranform.my_new_transform <cluster_uuid>/<transform_name>
170+
```

‎examples/resources/elasticstack_elasticsearch_transform/import.sh

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
terraform import elasticstack_elasticsearch_tranform.my_new_transform <cluster_uuid>/<transform_name>

‎examples/resources/elasticstack_elasticsearch_transform/resource.tf

Lines changed: 51 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,51 @@
1+
resource "elasticstack_elasticsearch_transform" "transform_with_pivot" {
2+
name = "transform-pivot"
3+
description = "A meaningful description"
4+
5+
source {
6+
indices = ["names_or_patterns_for_input_index"]
7+
}
8+
9+
destination {
10+
index = "destination_index_for_transform"
11+
}
12+
13+
pivot = jsonencode({
14+
"group_by" : {
15+
"customer_id" : {
16+
"terms" : {
17+
"field" : "customer_id",
18+
"missing_bucket" : true
19+
}
20+
}
21+
},
22+
"aggregations" : {
23+
"max_price" : {
24+
"max" : {
25+
"field" : "taxful_total_price"
26+
}
27+
}
28+
}
29+
})
30+
31+
frequency = "5m"
32+
33+
retention_policy {
34+
time {
35+
field = "order_date"
36+
max_age = "30d"
37+
}
38+
}
39+
40+
sync {
41+
time {
42+
field = "order_date"
43+
delay = "10s"
44+
}
45+
}
46+
47+
max_page_search_size = 2000
48+
49+
enabled = false
50+
defer_validation = false
51+
}

0 commit comments

Comments
 (0)