BoxPlotAggregate - JsonParsingException: expected:'Number Token', actual:'"NaN"', at offset:318

[smissingham]

NEST/Elasticsearch.Net version: 7.15

Elasticsearch version: 8.0.0-alpha2

.NET runtime version: .NETFramework v4.8

Operating system version: Windows 11

Description of the problem including expected versus actual behavior:
Json deserializer throws expected:'Number Token', actual:'"NaN"', at offset:318 when BoxPlotAggregate results returns "NaN".
Same for other potential non-numeric values "Infinity", "-Infinity"

Steps to reproduce:
Run the below BoxPlotAggregate query with NEST, it will fail to deserialize the "NaN" for "lower" prop

POST _bulk
{ "index" : { "_index" : "australian-states" } }
{ "state" : "NSW", "population": 8176368	 }
{ "index" : { "_index" : "australian-states" } }
{ "state" : "QLD", "population": 5206400	 }
{ "index" : { "_index" : "australian-states" } }
{ "state" : "SA", "population": 1771703	 }
{ "index" : { "_index" : "australian-states" } }
{ "state" : "TAS", "population": 541965	 }
{ "index" : { "_index" : "australian-states" } }
{ "state" : "VIC", "population": 6648564	 }
{ "index" : { "_index" : "australian-states" } }
{ "state" : "WA", "population": 2675797	 }

GET australian-states/_search
{
  "size": 0,
  "aggs": {
    "summary_boxplot": {
      "boxplot": {
        "field": "population"
      }
    }
  },
  "query": {
    "range": {
      "population": {
        "gte": "1",
        "lte": "1000000"
      }
    }
  }
}

Expected behavior
I think these non-numeric results should deserialize to null (but I'm sure there's some debate to be had here).

Stack Trace

[JsonParsingException: expected:'Number Token', actual:'"NaN"', at offset:318]
   Elasticsearch.Net.Utf8Json.JsonReader.ReadDouble() in C:\Users\SteveGordon\Code\elasticsearch-net\src\Elasticsearch.Net\Utf8Json\JsonReader.cs:1096
   Nest.AggregateFormatter.GetBoxplotAggregate(JsonReader& reader, IJsonFormatterResolver formatterResolver, IReadOnlyDictionary`2 meta) in C:\Users\SteveGordon\Code\elasticsearch-net\src\Nest\Aggregations\AggregateFormatter.cs:282
   Nest.AggregateFormatter.ReadAggregate(JsonReader& reader, IJsonFormatterResolver formatterResolver) in C:\Users\SteveGordon\Code\elasticsearch-net\src\Nest\Aggregations\AggregateFormatter.cs:180
   Nest.AggregateDictionaryFormatter.Deserialize(JsonReader& reader, IJsonFormatterResolver formatterResolver) in C:\Users\SteveGordon\Code\elasticsearch-net\src\Nest\Aggregations\AggregateDictionaryFormatter.cs:37
   Deserialize(Object[] , JsonReader& , IJsonFormatterResolver ) +1619
   Elasticsearch.Net.Utf8Json.JsonSerializer.Deserialize(Byte[] bytes, Int32 offset, IJsonFormatterResolver resolver) in C:\Users\SteveGordon\Code\elasticsearch-net\src\Elasticsearch.Net\Utf8Json\JsonSerializer.cs:206
   Elasticsearch.Net.Utf8Json.JsonSerializer.Deserialize(Stream stream, IJsonFormatterResolver resolver) in C:\Users\SteveGordon\Code\elasticsearch-net\src\Elasticsearch.Net\Utf8Json\JsonSerializer.cs:261
   Elasticsearch.Net.DiagnosticsSerializerProxy.Deserialize(Stream stream) in C:\Users\SteveGordon\Code\elasticsearch-net\src\Elasticsearch.Net\Serialization\DiagnosticsSerializerProxy.cs:82
   Elasticsearch.Net.ResponseBuilder.SetBody(ApiCallDetails details, RequestData requestData, Stream responseStream, String mimeType) in C:\Users\SteveGordon\Code\elasticsearch-net\src\Elasticsearch.Net\Transport\Pipeline\ResponseBuilder.cs:181
   Elasticsearch.Net.ResponseBuilder.ToResponse(RequestData requestData, Exception ex, Nullable`1 statusCode, IEnumerable`1 warnings, Stream responseStream, String productName, String mimeType) in C:\Users\SteveGordon\Code\elasticsearch-net\src\Elasticsearch.Net\Transport\Pipeline\ResponseBuilder.cs:55
   Elasticsearch.Net.HttpWebRequestConnection.Request(RequestData requestData) in C:\Users\SteveGordon\Code\elasticsearch-net\src\Elasticsearch.Net\Connection\HttpWebRequestConnection.cs:92
   Elasticsearch.Net.RequestPipeline.CallElasticsearch(RequestData requestData) in C:\Users\SteveGordon\Code\elasticsearch-net\src\Elasticsearch.Net\Transport\Pipeline\RequestPipeline.cs:185
   Elasticsearch.Net.Transport`1.Request(HttpMethod method, String path, PostData data, IRequestParameters requestParameters) in C:\Users\SteveGordon\Code\elasticsearch-net\src\Elasticsearch.Net\Transport\Transport.cs:87

[stevejgordon]

Thanks for raising this @smissingham and my apologies for the very slow response from my side. We have a potential solution for the NaN response that we've used elsewhere. As you suggest, we treat that as null. Infinity and -Infinity are more complicated and need some thought.

[stevejgordon]

@smissingham I've decided on the cleanest solution which is to use the double constants for the infinity cases. I've pushed up a PR which will go into the next release that ensures boxplot properties use a custom formatter that can handle these less common string values.

[smissingham]

@smissingham I've decided on the cleanest solution which is to use the double constants for the infinity cases. I've pushed up a PR which will go into the next release that ensures boxplot properties use a custom formatter that can handle these less common string values.

Good point @stevejgordon using double constants. Maybe double.NaN is a better fit as well for the NaN values, than null?

[stevejgordon]

https://github.com/elastic/elasticsearch-net/security/dependabot/tests/Tests/Tests.csproj/SharpZipLib/open

Yep, that should be in place with the change.