Skip to content

Commit d110a99

Browse files
authored
[WIP] Pre-flight product check (#5728)
* Fix comments in IConnectionConfigurationValues * Fix test method name typo * Add audit event types * Support product name of ApiCallDetails * Add types for response deserialisation * Add product check status to connection pool * Read product name from HTTP responses * Update InMemoryConnection to support product checks * Add product checks * Support product checking on virtual cluster * Add product check doc and tests * Remove ping integration tests These no longer work using a custom path as the product check fails with a 404 and is assumed invalid. * Add async version of concurrent first usage test * Add product check info to audit trail doc * Add more product check tests * Further test cleanup and fixup * Regnerate documentation * Fix encoding for new file * Remove async test * Update link in documentation * Fix documentation * Optimise validation for 7.14 versions * Update messages and documentation * Generate asciidoc * Fix BOM
1 parent 862b14b commit d110a99

File tree

52 files changed

+1381
-407
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

52 files changed

+1381
-407
lines changed

docs/client-concepts/connection-pooling/exceptions/unexpected-exceptions.asciidoc

+1-1
Original file line numberDiff line numberDiff line change
@@ -105,7 +105,7 @@ An unexpected hard exception on ping and sniff is something we *do* try to recov
105105

106106
Here, pinging nodes on first use is enabled and the node on port 9200 throws an exception on ping; when this happens,
107107
we still fallover to retry the ping on node on port 9201, where it succeeds.
108-
Following this, the client call on 9201 throws a hard exception that we are not able to recover from
108+
Following this, the client call on 9201 throws a hard exception that we are not able to recover from.
109109

110110
[source,csharp]
111111
----

docs/client-concepts/connection-pooling/failover/falling-over.asciidoc

+5-5
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,7 @@ please modify the original csharp file found at the link and submit the PR with
1616
=== Fail over
1717

1818
When using a connection pool with more than one node, a request will be retried if
19-
the call to a node throws an exception or returns a 502, 503 or 504 response
19+
the call to a node throws an exception or returns a 502, 503 or 504 response.
2020

2121
[source,csharp]
2222
----
@@ -39,7 +39,7 @@ audit = await audit.TraceCall(
3939
[[bad-gateway]]
4040
==== 502 Bad Gateway
4141

42-
Will be treated as an error that requires retrying
42+
Will be treated as an error that requires retrying.
4343

4444
[source,csharp]
4545
----
@@ -62,7 +62,7 @@ audit = await audit.TraceCall(
6262
[[service-unavailable]]
6363
==== 503 Service Unavailable
6464

65-
Will be treated as an error that requires retrying
65+
Will be treated as an error that requires retrying.
6666

6767
[source,csharp]
6868
----
@@ -85,7 +85,7 @@ audit = await audit.TraceCall(
8585
[[gateway-timeout]]
8686
==== 504 Gateway Timeout
8787

88-
Will be treated as an error that requires retrying
88+
Will be treated as an error that requires retrying.
8989

9090
[source,csharp]
9191
----
@@ -108,7 +108,7 @@ audit = await audit.TraceCall(
108108
If a call returns a __valid__ HTTP status code other than 502 or 503, the request won't be retried.
109109

110110
IMPORTANT: Different requests may have different status codes that are deemed __valid__. For example,
111-
a *404 Not Found* response is a __valid__ status code for an index exists request
111+
a *404 Not Found* response is a __valid__ status code for an index exists request.
112112

113113
[source,csharp]
114114
----
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,114 @@
1+
:ref_current: https://www.elastic.co/guide/en/elasticsearch/reference/7.x
2+
3+
:github: https://github.com/elastic/elasticsearch-net
4+
5+
:nuget: https://www.nuget.org/packages
6+
7+
////
8+
IMPORTANT NOTE
9+
==============
10+
This file has been generated from https://github.com/elastic/elasticsearch-net/tree/7.x/src/Tests/Tests/ClientConcepts/ConnectionPooling/ProductChecking/ProductCheckAtStartup.doc.cs.
11+
If you wish to submit a PR for any spelling mistakes, typos or grammatical errors for this file,
12+
please modify the original csharp file found at the link and submit the PR with that change. Thanks!
13+
////
14+
15+
[[product-check-at-startup]]
16+
== Product check at startup
17+
18+
Since v7.14.0, the client performs a required product check before the first call.
19+
This pre-flight product check allows the client to establish the version of Elasticsearch that it is communicating with.
20+
21+
The product check requires one additional HTTP request to be sent to the server as part of the request pipeline before
22+
the main API call is sent. In most cases, this will succeed during the very first API call that the client sends.
23+
Once the product check succeeds, no further product check HTTP requests are sent for subsequent API calls.
24+
25+
[source,csharp]
26+
----
27+
var audit = new Auditor(() => VirtualClusterWith
28+
.Nodes(1)
29+
.ClientCalls(r => r.SucceedAlways())
30+
.StaticConnectionPool()
31+
.Settings(s => s.DisablePing())
32+
);
33+
34+
audit = await audit.TraceCalls(skipProductCheck: false,
35+
new ClientCall() {
36+
{ ProductCheckOnStartup },
37+
{ ProductCheckSuccess, 9200 }, <1>
38+
{ HealthyResponse, 9200 } <2>
39+
},
40+
new ClientCall() {
41+
{ HealthyResponse, 9200 } <3>
42+
}
43+
);
44+
----
45+
<1> as this is the first call, the product check is executed
46+
<2> following the product check, the actual request is sent
47+
<3> subsequent calls no longer perform product check
48+
49+
Here's an example with a single node cluster which fails for some reason during the first product check attempt.
50+
51+
[source,csharp]
52+
----
53+
var audit = new Auditor(() => VirtualClusterWith
54+
.Nodes(1, productCheckAlwaysSucceeds: false)
55+
.ProductCheck(r => r.Fails(TimesHelper.Once))
56+
.ProductCheck(r => r.SucceedAlways())
57+
.ClientCalls(r => r.SucceedAlways())
58+
.StaticConnectionPool()
59+
.Settings(s => s.DisablePing())
60+
);
61+
62+
audit = await audit.TraceCalls(skipProductCheck: false,
63+
new ClientCall() {
64+
{ ProductCheckOnStartup },
65+
{ ProductCheckFailure, 9200 }, <1>
66+
{ HealthyResponse, 9200 } <2>
67+
},
68+
new ClientCall() {
69+
{ ProductCheckOnStartup },
70+
{ ProductCheckSuccess, 9200 }, <3>
71+
{ HealthyResponse, 9200 }
72+
},
73+
new ClientCall() {
74+
{ HealthyResponse, 9200 } <4>
75+
}
76+
);
77+
----
78+
<1> as this is the first call, the product check is executed, but fails
79+
<2> the actual request is still sent and succeeds
80+
<3> as the previous product check failed, it runs again on the second call
81+
<4> subsequent calls no longer perform product check
82+
83+
Here's an example with a three node cluster which fails for some reason during the first and second product check attempts.
84+
85+
[source,csharp]
86+
----
87+
var audit = new Auditor(() => VirtualClusterWith
88+
.Nodes(3, productCheckAlwaysSucceeds: false)
89+
.ProductCheck(r => r.FailAlways())
90+
.ProductCheck(r => r.OnPort(9202).SucceedAlways())
91+
.ClientCalls(r => r.SucceedAlways())
92+
.StaticConnectionPool()
93+
.Settings(s => s.DisablePing())
94+
);
95+
96+
audit = await audit.TraceCalls(skipProductCheck: false,
97+
new ClientCall() {
98+
{ ProductCheckOnStartup },
99+
{ ProductCheckFailure, 9200 }, <1>
100+
{ ProductCheckFailure, 9201 }, <2>
101+
{ ProductCheckSuccess, 9202 }, <3>
102+
{ HealthyResponse, 9200 } <4>
103+
},
104+
new ClientCall() {
105+
{ HealthyResponse, 9201 } <5>
106+
}
107+
);
108+
----
109+
<1> this is the first call, the product check is executed, but fails on this node
110+
<2> the next node is also tried and fails
111+
<3> the third node is tried, successfully responds and the product check succeeds
112+
<4> the actual request is sent and succeeds
113+
<5> subsequent calls no longer perform product check
114+

docs/client-concepts/connection-pooling/request-overrides/disable-sniff-ping-per-request.asciidoc

+1-1
Original file line numberDiff line numberDiff line change
@@ -36,7 +36,7 @@ var audit = new Auditor(() => VirtualClusterWith
3636
----
3737
<1> sniff on startup
3838

39-
Now We disable sniffing on the request so even though it's our first call,
39+
Now we disable sniffing on the request so even though it's our first call,
4040
we do not want to sniff on startup.
4141

4242
Instead, the sniff on startup is deferred to the second call into the cluster that

docs/client-concepts/connection-pooling/request-overrides/request-timeouts-overrides.asciidoc

+3-3
Original file line numberDiff line numberDiff line change
@@ -15,9 +15,9 @@ please modify the original csharp file found at the link and submit the PR with
1515
[[request-timeout]]
1616
=== Request timeouts
1717

18-
While you can specify Request time out globally you can override this per request too
18+
While you can specify Request time out globally, you can override this per request too.
1919

20-
we set up a 10 node cluster with a global time out of 20 seconds.
20+
We set up a 10 node cluster with a global time out of 20 seconds.
2121
Each call on a node takes 10 seconds. So we can only try this call on 2 nodes
2222
before the max request time out kills the client call.
2323

@@ -38,7 +38,7 @@ audit = await audit.TraceCalls(
3838
{ MaxTimeoutReached }
3939
},
4040
/**
41-
* On the second request we specify a request timeout override to 80 seconds
41+
* On the second request we specify a request timeout override to 80 seconds.
4242
* We should now see more nodes being tried.
4343
*/
4444
new ClientCall(r => r.RequestTimeout(TimeSpan.FromSeconds(80)))

docs/client-concepts/troubleshooting/audit-trail.asciidoc

+16-12
Original file line numberDiff line numberDiff line change
@@ -20,7 +20,7 @@ occur when a request is made. This audit trail is available on the response as d
2020
following example.
2121

2222
We'll use a Sniffing connection pool here since it sniffs on startup and pings before
23-
first usage, so we can get an audit trail with a few events out
23+
first usage, so we can get an audit trail with a few events out.
2424

2525
[source,csharp]
2626
----
@@ -33,7 +33,7 @@ var connectionSettings = new ConnectionSettings(pool)
3333
var client = new ElasticClient(connectionSettings);
3434
----
3535

36-
After issuing the following request
36+
After issuing the following request:
3737

3838
[source,csharp]
3939
----
@@ -48,10 +48,12 @@ readable fashion, similar to
4848
....
4949
Valid NEST response built from a successful low level call on POST: /project/doc/_search
5050
# Audit trail of this API call:
51-
- [1] SniffOnStartup: Took: 00:00:00.0360264
52-
- [2] SniffSuccess: Node: http://localhost:9200/ Took: 00:00:00.0310228
53-
- [3] PingSuccess: Node: http://127.0.0.1:9200/ Took: 00:00:00.0115074
54-
- [4] HealthyResponse: Node: http://127.0.0.1:9200/ Took: 00:00:00.1477640
51+
- [1] ProductCheckOnStartup: Took: 00:00:00.1014934
52+
- [2] ProductCheckSuccess: Node: http://ipv4.fiddler:9200/ Took: 00:00:00.1002862
53+
- [3] SniffOnStartup: Took: 00:00:00.0440780
54+
- [4] SniffSuccess: Node: http://localhost:9200/ Took: 00:00:00.0257506
55+
- [5] PingSuccess: Node: http://127.0.0.1:9200/ Took: 00:00:00.0115074
56+
- [6] HealthyResponse: Node: http://127.0.0.1:9200/ Took: 00:00:00.1477640
5557
# Request:
5658
<Request stream not captured or already read to completion by serializer. Set DisableDirectStreaming() on ConnectionSettings to force it to be set on the response.>
5759
# Response:
@@ -69,15 +71,17 @@ But can also be accessed manually:
6971

7072
[source,csharp]
7173
----
72-
response.ApiCall.AuditTrail.Count.Should().Be(4, "{0}", debug);
73-
response.ApiCall.AuditTrail[0].Event.Should().Be(AuditEvent.SniffOnStartup, "{0}", debug);
74-
response.ApiCall.AuditTrail[1].Event.Should().Be(AuditEvent.SniffSuccess, "{0}", debug);
75-
response.ApiCall.AuditTrail[2].Event.Should().Be(AuditEvent.PingSuccess, "{0}", debug);
76-
response.ApiCall.AuditTrail[3].Event.Should().Be(AuditEvent.HealthyResponse, "{0}", debug);
74+
response.ApiCall.AuditTrail.Count.Should().Be(6, "{0}", debug);
75+
response.ApiCall.AuditTrail[0].Event.Should().Be(AuditEvent.ProductCheckOnStartup, "{0}", debug);
76+
response.ApiCall.AuditTrail[1].Event.Should().Be(AuditEvent.ProductCheckSuccess, "{0}", debug);
77+
response.ApiCall.AuditTrail[2].Event.Should().Be(AuditEvent.SniffOnStartup, "{0}", debug);
78+
response.ApiCall.AuditTrail[3].Event.Should().Be(AuditEvent.SniffSuccess, "{0}", debug);
79+
response.ApiCall.AuditTrail[4].Event.Should().Be(AuditEvent.PingSuccess, "{0}", debug);
80+
response.ApiCall.AuditTrail[5].Event.Should().Be(AuditEvent.HealthyResponse, "{0}", debug);
7781
----
7882

7983
Each audit has a started and ended `DateTime` on it that will provide
80-
some understanding of how long it took
84+
some understanding of how long it took.
8185

8286
[source,csharp]
8387
----

docs/client-concepts/troubleshooting/logging-with-on-request-completed.asciidoc

+1-1
Original file line numberDiff line numberDiff line change
@@ -51,7 +51,7 @@ var client = FixedResponseClient.Create( <1>
5151
500,
5252
connectionSettings => connectionSettings
5353
.ThrowExceptions() <2>
54-
.OnRequestCompleted(r => counter++)
54+
.OnRequestCompleted(r => counter++), productCheckSucceeds: false
5555
);
5656
5757
Assert.Throws<ElasticsearchClientException>(() => client.RootNodeInfo()); <3>

docs/code-standards/requests.asciidoc

+2-2
Original file line numberDiff line numberDiff line change
@@ -34,14 +34,14 @@ var searchResponse = client.Search<Project>(s => s.AllIndices());
3434
searchResponse.ApiCall.Uri.ToString().Should().Be("http://localhost:9200/_all/_search?typed_keys=true");
3535
3636
var pool = new SingleNodeConnectionPool(new Uri("http://localhost:9200/elasticsearch"));
37-
var settings = new ConnectionSettings(pool, new InMemoryConnection());
37+
var settings = new ConnectionSettings(pool, new InMemoryConnection("elasticsearch"));
3838
var client = new ElasticClient(settings);
3939
var searchResponse = client.Search<Project>(s => s.AllIndices());
4040
4141
searchResponse.ApiCall.Uri.ToString().Should().Be("http://localhost:9200/elasticsearch/_all/_search?typed_keys=true");
4242
4343
var pool = new SingleNodeConnectionPool(new Uri("http://localhost:9200/elasticsearch/"));
44-
var settings = new ConnectionSettings(pool, new InMemoryConnection());
44+
var settings = new ConnectionSettings(pool, new InMemoryConnection("elasticsearch/"));
4545
var client = new ElasticClient(settings);
4646
var searchResponse = client.Search<Project>(s => s.AllIndices());
4747

docs/conventions.asciidoc

+4
Original file line numberDiff line numberDiff line change
@@ -35,6 +35,8 @@ Conventions that apply to both Elasticsearch.Net and NEST
3535

3636
* <<transports-and-pipelines, Transports and pipelines>>
3737

38+
* <<product-check-at-startup, Pre-flight product check at startup>>
39+
3840
* <<sniffing-behaviour, Sniffing behaviour>>
3941

4042
* <<pinging-behaviour, Pinging behaviour>>
@@ -97,6 +99,8 @@ include::{building-blocks}/date-time-providers.asciidoc[]
9799

98100
include::{building-blocks}/keeping-track-of-nodes.asciidoc[]
99101

102+
include::client-concepts/connection-pooling/product-checking/product-check-at-startup.asciidoc[]
103+
100104
[[sniffing-behaviour]]
101105
== Sniffing behaviour
102106

0 commit comments

Comments
 (0)