Transform.GetAsync() not deserializing a percentiles aggregation with a script

[karoberts]

NEST/Elasticsearch.Net version: 7.12.1

Elasticsearch version: 7.11.2

Description of the problem including expected versus actual behavior:
I have a transform with an aggregation like this

            "my_median": {
                "percentiles": {
                    "script": "doc['field'].value * 2",
                    "percents": [
                        50
                    ]
                }
            }

But calling Transform.GetAsync() on that transform returns an object with the Script field as null

Steps to reproduce:

1. Create transform with a median aggregation based on a script
2. call Transform.GetAsync()
3. inspect returned object

Expected behavior
The Script field should be non-null and match the transform definition

[stevejgordon]

Hi @karoberts. I have just tried to reproduce this and I'm unable to. The following code writes the original script to the console.

var putResponse = client.Transform.Put<MyType>("testing", t => t
	.Source(s => s.Index("my-index-01"))
	.Destination(d => d.Index("my-index-20"))
	.Frequency("5m")
	.Pivot(p => p
		.Aggregations(a =>
			a.Percentiles("my_median", pc => pc.Script("doc['counter'].value * 2").Percents(50)))
		.GroupBy(g => g.Terms("a-term", term => term.Field("location.keyword")))));

var getResponse = client.Transform.Get(r => r.TransformId("testing"));

var script =
	getResponse.Transforms.First().Pivot.Aggregations["my_median"].Percentiles.Script as InlineScript;

Console.WriteLine(script?.Source ?? "Not found");

It's possible your use case is more complex. Are you able to provide the code necessary to reproduce the issue and I'll dig into it? Thanks!

[karoberts]

Here is the a template definition json (I added this via postman)

    "source": {
        "index": [
            "my-index"
        ],
        "query": {
            "bool": {
                "must": [
                    {
                        "exists": {
                            "field": "zip"
                        }
                    }
                ]
            }
        }
    },
    "pivot": {
        "group_by": {
            "key": {
                "terms": {
                    "field": "zip"
                }
            }
        },
        "aggregations": {
            "with_filter": {
                "filter": {
                    "range": {
                        "recording_date": {
                            "gte": "now-6M/M",
                            "lt": "now/M"
                        }
                    }
                },
                "aggs": {
                    "percentiles": {
                        "percentiles": {
                            "script": "doc['sales_price'].value * 2",
                            "percents": [
                                50
                            ]
                        }
                    }
                }
            },
            "no_filter": {
                "percentiles": {
                    "script": "doc['sales_price'].value * 2",
                    "percents": [
                        50
                    ]
                }
            }
        }
    },
    "dest": {
        "index": "test"
    },
    "settings": {
        "max_page_search_size": 5000,
        "docs_per_second": null
    }
}

My use case uses the one with the filter, but I tried both ways

var templateInfo = await _elasticClient.Transform.GetAsync(t => t.TransformId("test").ExcludeGenerated());
log.Warn((templateInfo.Transforms.First().Pivot.Aggregations["with_filter"].Aggregations.First().Value.Percentiles.Script as InlineScript)?.Source);
log.Warn((templateInfo.Transforms.First().Pivot.Aggregations["no_filter"].Percentiles.Script as InlineScript)?.Source);

both of those log null

[stevejgordon]

Thanks, @karoberts. I see the issue. Your template defines the script using just the string containing the source. This is valid on the server and it is returned as such. The client though expects to deserialise into an IScript so this is currently not supported. Due to the way these types are used, it might be difficult to address this without introducing a breaking change in a future major release. I'll review it and see what options we have for the 7.x time frame. In your case, would it be possible to define your transform using the object representation for your script?

[karoberts]

@stevejgordon I can confirm that changing to the object representation then allows Transform.GetAsync to deserialize it correctly.

no good

"script": "someScript"

works

"script": {
    "source": "someScript"
}

I can see how supporting both might be challenging. For now I will either use the low level client, or use the object representation. Thanks!

[stevejgordon]

Thanks for confirming @karoberts. I'll keep this open to track whether this can be safely implemented in 7.x.

[stevejgordon]

@karoberts Apologies for the long delay on this one. I have added a serialisation fix which adds support for the string-based scripts that will go out in the next release.