Error response is masked as a Json parsing exception

[LBoraz]

Java API client version

8.6.2

Java version

19

Elasticsearch Version

8.4.1

Problem description

When the server responds with an error, for example a http 400 Bad Request, the client reports a JsonParsingException which masks the true cause of the error.

One way to reproduce is:

Configure the restClient to use a HTTPS only host, but do not set the scheme:

new HttpHost("my.https-only.host", 443)

Any api call now will respond with a 400, which is masked by the following exception.
At this stage it's even possible to omit the CredentialsProvider, and we would still get the same exception

Exception in thread "main" jakarta.json.stream.JsonParsingException: Jackson exception: Unexpected character ('<' (code 60)): expected a valid value (JSON String, Number, Array, Object or token 'null', 'true' or 'false')
at [Source: (ByteArrayInputStream); line: 1, column: 2]
    at co.elastic.clients.json.jackson.JacksonJsonpParser.convertException(JacksonJsonpParser.java:89)
    at co.elastic.clients.json.jackson.JacksonJsonpParser.fetchNextToken(JacksonJsonpParser.java:96)
    at co.elastic.clients.json.jackson.JacksonJsonpParser.next(JacksonJsonpParser.java:123)
    at co.elastic.clients.json.JsonpDeserializer.deserialize(JsonpDeserializer.java:71)
    at co.elastic.clients.json.ObjectBuilderDeserializer.deserialize(ObjectBuilderDeserializer.java:79)
    at co.elastic.clients.json.DelegatingDeserializer$SameType.deserialize(DelegatingDeserializer.java:43)
    at co.elastic.clients.transport.rest_client.RestClientTransport.getHighLevelResponse(RestClientTransport.java:280)
    at co.elastic.clients.transport.rest_client.RestClientTransport.performRequest(RestClientTransport.java:148)
    at co.elastic.clients.elasticsearch.tasks.ElasticsearchTasksClient.list(ElasticsearchTasksClient.java:150)
    at co.elastic.clients.elasticsearch.tasks.ElasticsearchTasksClient.list(ElasticsearchTasksClient.java:166)

The client should provide a way to access to the true error.
What happens instead in RestClientTransport.getHighLevelResponse is that the TransportException is skipped (deserializer != null), so the method continues until eventually an api object is returned (for example ListResponse).
The invalid api object then is deserialized and throws the JsonParsingException above.

if (endpoint.isError(statusCode)) {
                JsonpDeserializer<ErrorT> errorDeserializer = endpoint.errorDeserializer(statusCode);
                if (errorDeserializer == null) {
                   /* It should probably throw this - but the deserializer exists  */
                    throw new TransportException(
                        "Request failed with status code '" + statusCode + "'",
                        endpoint.id(), new ResponseException(clientResp)
                    );
                }
         [...]
}

[Ayush1gupta]

Same case with 401 response. #563

[swallez]

The error indicates that the response most likely comes from a proxy server that returns XML. This is fixed in #579