Aggregation result deserialization error

[phoeniks-sk]

NEST/Elasticsearch.Net version: 7.4.0

Elasticsearch version: 7.4.0

Description of the problem including expected versus actual behavior:
Upgrading from 6.8 to 7.4 our aggregations stopped working in some cases. Getting an Elasticsearch.Net.UnexpectedElasticsearchClientException with message expected:'String Begin Token', actual:']'.

This problem seems to be related to #3673 .

Steps to reproduce:

1. Index a document with a date field:

client.Index(new Doc { ModificationDate = DateTime.Parse("2019-10-09T10:43:07.8633456+02:00") }, i => i.Index("pptest-collections"));

2. Run search with aggregations:

        client.Search<Doc>(s => s
            .Index("pptest-collections")
            .Aggregations(
                 a => a.Filters("Modification date",
                     f => f
                         .AnonymousFilters(
                             q => q.DateRange(dr => dr.Field(d => d.ModificationDate)
                                   .GreaterThan(DateMath.Now.Subtract(TimeSpan.FromDays(120))))
                         )))
        );

Provide DebugInformation (if relevant):

 - [1] BadResponse: Node: http://localhost:9200/ Exception: JsonParsingException Took: 00:00:00.2654590
# Audit exception in step 1 BadResponse:
Elasticsearch.Net.Utf8Json.JsonParsingException: expected:'String Begin Token', actual:']', at offset:241
   at Elasticsearch.Net.Utf8Json.JsonReader.ReadStringSegmentCore(Byte[]& resultBytes, Int32& resultOffset, Int32& resultLength)
   at Elasticsearch.Net.Utf8Json.JsonReader.ReadString()
   at Elasticsearch.Net.Utf8Json.JsonReader.ReadPropertyName()
   at Nest.AggregateFormatter.GetFiltersBucket(JsonReader& reader, IJsonFormatterResolver formatterResolver)
   at Nest.AggregateFormatter.ReadBucket(JsonReader& reader, IJsonFormatterResolver formatterResolver)
   at Nest.AggregateFormatter.GetMultiBucketAggregate(JsonReader& reader, IJsonFormatterResolver formatterResolver, ArraySegment`1& propertyName, IReadOnlyDictionary`2 meta)
   at Nest.AggregateFormatter.ReadAggregate(JsonReader& reader, IJsonFormatterResolver formatterResolver)
   at Nest.AggregateFormatter.Deserialize(JsonReader& reader, IJsonFormatterResolver formatterResolver)
   at Nest.AggregateDictionaryFormatter.ParseAggregate(JsonReader& reader, IJsonFormatterResolver formatterResolver, String name, Dictionary`2 dictionary)
   at Nest.AggregateDictionaryFormatter.ReadAggregate(JsonReader& reader, IJsonFormatterResolver formatterResolver, String[] tokens, Dictionary`2 dictionary)
   at Nest.AggregateDictionaryFormatter.Deserialize(JsonReader& reader, IJsonFormatterResolver formatterResolver)
   at Deserialize(Object[] , JsonReader& , IJsonFormatterResolver )
   at Elasticsearch.Net.Utf8Json.Resolvers.DynamicMethodAnonymousFormatter`1.Deserialize(JsonReader& reader, IJsonFormatterResolver formatterResolver)
   at Elasticsearch.Net.Utf8Json.JsonSerializer.Deserialize[T](Byte[] bytes, Int32 offset, IJsonFormatterResolver resolver)
   at Elasticsearch.Net.Utf8Json.JsonSerializer.Deserialize[T](Byte[] bytes, IJsonFormatterResolver resolver)
   at Elasticsearch.Net.Utf8Json.JsonSerializer.Deserialize[T](Stream stream, IJsonFormatterResolver resolver)
   at Nest.DefaultHighLevelSerializer.Deserialize[T](Stream stream)
   at Elasticsearch.Net.DiagnosticsSerializerProxy.Deserialize[T](Stream stream)
   at Elasticsearch.Net.ResponseBuilder.SetBody[TResponse](ApiCallDetails details, RequestData requestData, Stream responseStream, String mimeType)
   at Elasticsearch.Net.ResponseBuilder.ToResponse[TResponse](RequestData requestData, Exception ex, Nullable`1 statusCode, IEnumerable`1 warnings, Stream responseStream, String mimeType)
   at Elasticsearch.Net.HttpConnection.Request[TResponse](RequestData requestData)
   at Elasticsearch.Net.RequestPipeline.CallElasticsearch[TResponse](RequestData requestData)
   at Elasticsearch.Net.Transport`1.Request[TResponse](HttpMethod method, String path, PostData data, IRequestParameters requestParameters)
# Inner Exception: expected:'String Begin Token', actual:']', at offset:241
Elasticsearch.Net.Utf8Json.JsonParsingException: expected:'String Begin Token', actual:']', at offset:241
   at Elasticsearch.Net.Utf8Json.JsonReader.ReadStringSegmentCore(Byte[]& resultBytes, Int32& resultOffset, Int32& resultLength)
   at Elasticsearch.Net.Utf8Json.JsonReader.ReadString()
   at Elasticsearch.Net.Utf8Json.JsonReader.ReadPropertyName()
   at Nest.AggregateFormatter.GetFiltersBucket(JsonReader& reader, IJsonFormatterResolver formatterResolver)
   at Nest.AggregateFormatter.ReadBucket(JsonReader& reader, IJsonFormatterResolver formatterResolver)
   at Nest.AggregateFormatter.GetMultiBucketAggregate(JsonReader& reader, IJsonFormatterResolver formatterResolver, ArraySegment`1& propertyName, IReadOnlyDictionary`2 meta)
   at Nest.AggregateFormatter.ReadAggregate(JsonReader& reader, IJsonFormatterResolver formatterResolver)
   at Nest.AggregateFormatter.Deserialize(JsonReader& reader, IJsonFormatterResolver formatterResolver)
   at Nest.AggregateDictionaryFormatter.ParseAggregate(JsonReader& reader, IJsonFormatterResolver formatterResolver, String name, Dictionary`2 dictionary)
   at Nest.AggregateDictionaryFormatter.ReadAggregate(JsonReader& reader, IJsonFormatterResolver formatterResolver, String[] tokens, Dictionary`2 dictionary)
   at Nest.AggregateDictionaryFormatter.Deserialize(JsonReader& reader, IJsonFormatterResolver formatterResolver)
   at Deserialize(Object[] , JsonReader& , IJsonFormatterResolver )
   at Elasticsearch.Net.Utf8Json.Resolvers.DynamicMethodAnonymousFormatter`1.Deserialize(JsonReader& reader, IJsonFormatterResolver formatterResolver)
   at Elasticsearch.Net.Utf8Json.JsonSerializer.Deserialize[T](Byte[] bytes, Int32 offset, IJsonFormatterResolver resolver)
   at Elasticsearch.Net.Utf8Json.JsonSerializer.Deserialize[T](Byte[] bytes, IJsonFormatterResolver resolver)
   at Elasticsearch.Net.Utf8Json.JsonSerializer.Deserialize[T](Stream stream, IJsonFormatterResolver resolver)
   at Nest.DefaultHighLevelSerializer.Deserialize[T](Stream stream)
   at Elasticsearch.Net.DiagnosticsSerializerProxy.Deserialize[T](Stream stream)
   at Elasticsearch.Net.ResponseBuilder.SetBody[TResponse](ApiCallDetails details, RequestData requestData, Stream responseStream, String mimeType)
   at Elasticsearch.Net.ResponseBuilder.ToResponse[TResponse](RequestData requestData, Exception ex, Nullable`1 statusCode, IEnumerable`1 warnings, Stream responseStream, String mimeType)
   at Elasticsearch.Net.HttpConnection.Request[TResponse](RequestData requestData)
   at Elasticsearch.Net.RequestPipeline.CallElasticsearch[TResponse](RequestData requestData)
   at Elasticsearch.Net.Transport`1.Request[TResponse](HttpMethod method, String path, PostData data, IRequestParameters requestParameters)

[russcam]

Thanks for reporting @phoeniks-sk, this looks like a bug. Do you have an example JSON response that I can work with, to fix this?

[russcam]

I've opened #4214

[phoeniks-sk]

I don't know if still needed, just in case, this is the result for the above aggregation:

{
  "took" : 1,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 1,
      "relation" : "eq"
    },
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "pptest-collections",
        "_type" : "_doc",
        "_id" : "Ed_cXm4BBc-fPZmR7xGC",
        "_score" : 1.0,
        "_source" : {
          "modificationDate" : "2019-10-09T10:43:07.8633456+02:00"
        }
      }
    ]
  },
  "aggregations" : {
    "filters#Modification date" : {
      "meta" : { },
      "buckets" : [
        {
          "doc_count" : 1
        }
      ]
    }
  }
}