Skip to content

ExpandWildcards option is not correctly serialized on Delete indices request #7823

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
gribunin opened this issue Jul 12, 2023 · 4 comments · Fixed by #7840
Closed

ExpandWildcards option is not correctly serialized on Delete indices request #7823

gribunin opened this issue Jul 12, 2023 · 4 comments · Fixed by #7840
Labels
8.x Relates to a 8.x client version

Comments

@gribunin
Copy link

Elastic.Clients.Elasticsearch version: 8.1.2

Elasticsearch version: 8.8.2

.NET runtime version: .NET 6

Operating system version: Windows 10 Enterprise

Description of the problem including expected versus actual behavior:
When trying to delete indices with the wildcard with the following code:

            var response = _EsClient.Indices.Delete("rules*", configureRequest =>
            {
                configureRequest.ExpandWildcards(new ExpandWildcard[] { ExpandWildcard.All });
            });

The server returns unsuccesful response:

Invalid Elasticsearch response built from a unsuccessful (400) low level call on DELETE: /rules%2A?pretty=true&error_trace=true&expand_wildcards=Elastic.Clients.Elasticsearch.ExpandWildcard%5B%5D

obviously because the library generates incorrect query string parameter "expand_wildcards", which has to be a comma-separated string of any of "all,open,close,hidden,none" values, but instead the "Elastic.Clients.Elasticsearch.ExpandWildcard" string is added to this parameter.

Steps to reproduce:

  1. Write the code above.
  2. Run.

Expected behavior
Succesful return code (and the query string with the correct "expand_wildcards" parameter).

Provide DebugInformation (if relevant):

Invalid Elasticsearch response built from a unsuccessful (400) low level call on DELETE: /rules%2A?pretty=true&error_trace=true&expand_wildcards=Elastic.Clients.Elasticsearch.ExpandWildcard%5B%5D
 Exception: Request failed to execute. Call: Status code 400 from: DELETE /rules%2A?pretty=true&error_trace=true&expand_wildcards=Elastic.Clients.Elasticsearch.ExpandWildcard%5B%5D. ServerError: Type: illegal_argument_exception Reason: "No valid expand wildcard value [Elastic.Clients.Elasticsearch.ExpandWildcard[]]"

# Audit trail of this API call:
 - [1] PingSuccess: Node: http://elkdev02:9200/ Took: 00:00:00.2743429
 - [2] BadResponse: Node: http://elkdev02:9200/ Took: 00:00:00.0503984
# OriginalException: Elastic.Transport.TransportException: Request failed to execute. Call: Status code 400 from: DELETE /rules%2A?pretty=true&error_trace=true&expand_wildcards=Elastic.Clients.Elasticsearch.ExpandWildcard%5B%5D. ServerError: Type: illegal_argument_exception Reason: "No valid expand wildcard value [Elastic.Clients.Elasticsearch.ExpandWildcard[]]"
# Request:
<Request stream not captured or already read to completion by serializer. Set DisableDirectStreaming() on TransportConfiguration to force it to be set on the response.>
# Response:
{
  "error" : {
    "root_cause" : [
      {
        "type" : "illegal_argument_exception",
        "reason" : "No valid expand wildcard value [Elastic.Clients.Elasticsearch.ExpandWildcard[]]",
        "stack_trace" : "org.elasticsearch.ElasticsearchException$1: No valid expand wildcard value [Elastic.Clients.Elasticsearch.ExpandWildcard[]]\n\tat [email protected]/org.elasticsearch.ElasticsearchException.guessRootCauses(ElasticsearchException.java:669)\n\tat [email protected]/org.elasticsearch.ElasticsearchException.generateFailureXContent(ElasticsearchException.java:597)\n\tat [email protected]/org.elasticsearch.rest.RestResponse.build(RestResponse.java:176)\n\tat [email protected]/org.elasticsearch.rest.RestResponse.<init>(RestResponse.java:124)\n\tat [email protected]/org.elasticsearch.rest.RestResponse.<init>(RestResponse.java:103)\n\tat [email protected]/org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:412)\n\tat [email protected]/org.elasticsearch.rest.RestController.tryAllHandlers(RestController.java:539)\n\tat [email protected]/org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:311)\n\tat [email protected]/org.elasticsearch.http.AbstractHttpServerTransport.dispatchRequest(AbstractHttpServerTransport.java:381)\n\tat [email protected]/org.elasticsearch.http.AbstractHttpServerTransport.handleIncomingRequest(AbstractHttpServerTransport.java:477)\n\tat [email protected]/org.elasticsearch.http.AbstractHttpServerTransport.incomingRequest(AbstractHttpServerTransport.java:354)\n\tat [email protected]/org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.handlePipelinedRequest(Netty4HttpPipeliningHandler.java:128)\n\tat [email protected]/org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.channelRead(Netty4HttpPipeliningHandler.java:118)\n\tat [email protected]/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)\n\tat [email protected]/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)\n\tat [email protected]/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)\n\tat [email protected]/io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)\n\tat [email protected]/io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111)\n\tat [email protected]/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)\n\tat [email protected]/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)\n\tat [email protected]/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)\n\tat [email protected]/io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)\n\tat [email protected]/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)\n\tat [email protected]/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)\n\tat [email protected]/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)\n\tat [email protected]/io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)\n\tat [email protected]/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)\n\tat [email protected]/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)\n\tat [email protected]/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)\n\tat [email protected]/io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)\n\tat [email protected]/io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)\n\tat [email protected]/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)\n\tat [email protected]/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)\n\tat [email protected]/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)\n\tat [email protected]/io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)\n\tat [email protected]/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)\n\tat [email protected]/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)\n\tat [email protected]/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)\n\tat [email protected]/io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)\n\tat [email protected]/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)\n\tat [email protected]/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)\n\tat [email protected]/io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)\n\tat [email protected]/io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)\n\tat [email protected]/io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)\n\tat [email protected]/io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689)\n\tat [email protected]/io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652)\n\tat [email protected]/io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)\n\tat [email protected]/io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)\n\tat [email protected]/io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)\n\tat java.base/java.lang.Thread.run(Thread.java:1623)\nCaused by: java.lang.IllegalArgumentException: No valid expand wildcard value [Elastic.Clients.Elasticsearch.ExpandWildcard[]]\n\tat [email protected]/org.elasticsearch.action.support.IndicesOptions$WildcardStates.updateSetForValue(IndicesOptions.java:83)\n\tat [email protected]/org.elasticsearch.action.support.IndicesOptions$WildcardStates.parseParameter(IndicesOptions.java:56)\n\tat [email protected]/org.elasticsearch.action.support.IndicesOptions.fromParameters(IndicesOptions.java:420)\n\tat [email protected]/org.elasticsearch.action.support.IndicesOptions.fromRequest(IndicesOptions.java:375)\n\tat [email protected]/org.elasticsearch.rest.action.admin.indices.RestDeleteIndexAction.prepareRequest(RestDeleteIndexAction.java:44)\n\tat [email protected]/org.elasticsearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:80)\n\tat [email protected]/org.elasticsearch.xpack.security.rest.SecurityRestFilter.doHandleRequest(SecurityRestFilter.java:98)\n\tat [email protected]/org.elasticsearch.xpack.security.rest.SecurityRestFilter.handleRequest(SecurityRestFilter.java:74)\n\tat [email protected]/org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:410)\n\t... 44 more\n"
      }
    ],
    "type" : "illegal_argument_exception",
    "reason" : "No valid expand wildcard value [Elastic.Clients.Elasticsearch.ExpandWildcard[]]",
    "stack_trace" : "java.lang.IllegalArgumentException: No valid expand wildcard value [Elastic.Clients.Elasticsearch.ExpandWildcard[]]\n\tat [email protected]/org.elasticsearch.action.support.IndicesOptions$WildcardStates.updateSetForValue(IndicesOptions.java:83)\n\tat [email protected]/org.elasticsearch.action.support.IndicesOptions$WildcardStates.parseParameter(IndicesOptions.java:56)\n\tat [email protected]/org.elasticsearch.action.support.IndicesOptions.fromParameters(IndicesOptions.java:420)\n\tat [email protected]/org.elasticsearch.action.support.IndicesOptions.fromRequest(IndicesOptions.java:375)\n\tat [email protected]/org.elasticsearch.rest.action.admin.indices.RestDeleteIndexAction.prepareRequest(RestDeleteIndexAction.java:44)\n\tat [email protected]/org.elasticsearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:80)\n\tat [email protected]/org.elasticsearch.xpack.security.rest.SecurityRestFilter.doHandleRequest(SecurityRestFilter.java:98)\n\tat [email protected]/org.elasticsearch.xpack.security.rest.SecurityRestFilter.handleRequest(SecurityRestFilter.java:74)\n\tat [email protected]/org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:410)\n\tat [email protected]/org.elasticsearch.rest.RestController.tryAllHandlers(RestController.java:539)\n\tat [email protected]/org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:311)\n\tat [email protected]/org.elasticsearch.http.AbstractHttpServerTransport.dispatchRequest(AbstractHttpServerTransport.java:381)\n\tat [email protected]/org.elasticsearch.http.AbstractHttpServerTransport.handleIncomingRequest(AbstractHttpServerTransport.java:477)\n\tat [email protected]/org.elasticsearch.http.AbstractHttpServerTransport.incomingRequest(AbstractHttpServerTransport.java:354)\n\tat [email protected]/org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.handlePipelinedRequest(Netty4HttpPipeliningHandler.java:128)\n\tat [email protected]/org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.channelRead(Netty4HttpPipeliningHandler.java:118)\n\tat [email protected]/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)\n\tat [email protected]/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)\n\tat [email protected]/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)\n\tat [email protected]/io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)\n\tat [email protected]/io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111)\n\tat [email protected]/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)\n\tat [email protected]/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)\n\tat [email protected]/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)\n\tat [email protected]/io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)\n\tat [email protected]/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)\n\tat [email protected]/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)\n\tat [email protected]/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)\n\tat [email protected]/io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)\n\tat [email protected]/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)\n\tat [email protected]/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)\n\tat [email protected]/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)\n\tat [email protected]/io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)\n\tat [email protected]/io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)\n\tat [email protected]/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)\n\tat [email protected]/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)\n\tat [email protected]/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)\n\tat [email protected]/io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)\n\tat [email protected]/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)\n\tat [email protected]/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)\n\tat [email protected]/io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)\n\tat [email protected]/io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)\n\tat [email protected]/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)\n\tat [email protected]/io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)\n\tat [email protected]/io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)\n\tat [email protected]/io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)\n\tat [email protected]/io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)\n\tat [email protected]/io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689)\n\tat [email protected]/io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652)\n\tat [email protected]/io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)\n\tat [email protected]/io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)\n\tat [email protected]/io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)\n\tat java.base/java.lang.Thread.run(Thread.java:1623)\n"
  },
  "status" : 400
}

# TCP states:
  TimeWait: 37
  Established: 97
  CloseWait: 2

# ThreadPool statistics:
  Worker: 
    Busy: 0
    Free: 32767
    Min: 4
    Max: 32767
  IOCP: 
    Busy: 0
    Free: 1000
    Min: 4
    Max: 1000
@gribunin
Copy link
Author

The root of the problem is in the library elastic-transport-net, the corresponding pull request is here:
elastic/elastic-transport-net#89

There is also a different problem. Even if this serialization problem is fixed in elastic-transport-net, the other problem will arise. The ExpandWildcard enum values have a capital first letter in its enum values, but elastic search requires values for the parameter "expand_wildcards" to be all lowercase. For example, for expand_wildcards=All the elasticsearch also will return 400 bad request.

@gribunin
Copy link
Author

The current workaround is to use the more low level code:

            var parameters = new DefaultRequestParameters();
            parameters.QueryString.Add("expand_wildcards", "all");

            var response = _EsClient.Transport.Delete<DeleteIndexResponse>("/rules%2A", null, parameters);

@flobernd flobernd added the bug label Jul 14, 2023
@flobernd
Copy link
Member

There is also a different problem. Even if this serialization problem is fixed in elastic-transport-net, the other problem will arise. The ExpandWildcard enum values have a capital first letter in its enum values, but elastic search requires values for the parameter "expand_wildcards" to be all lowercase. For example, for expand_wildcards=All the elasticsearch also will return 400 bad request.

Capitalization good to me:

[JsonConverter(typeof(ExpandWildcardConverter))]
public enum ExpandWildcard
{
/// <summary>
/// <para>Match open, non-hidden indices. Also matches any non-hidden data stream.</para>
/// </summary>
[EnumMember(Value = "open")]
Open,
/// <summary>
/// <para>Wildcard expressions are not accepted.</para>
/// </summary>
[EnumMember(Value = "none")]
None,
/// <summary>
/// <para>Match hidden data streams and hidden indices. Must be combined with open, closed, or both.</para>
/// </summary>
[EnumMember(Value = "hidden")]
Hidden,
/// <summary>
/// <para>Match closed, non-hidden indices. Also matches any non-hidden data stream. Data streams cannot be closed.</para>
/// </summary>
[EnumMember(Value = "closed")]
Closed,
/// <summary>
/// <para>Match any data stream or index, including hidden ones.</para>
/// </summary>
[EnumMember(Value = "all")]
All
}

@gribunin
Copy link
Author

gribunin commented Jul 14, 2023

There is also a different problem. Even if this serialization problem is fixed in elastic-transport-net, the other problem will arise. The ExpandWildcard enum values have a capital first letter in its enum values, but elastic search requires values for the parameter "expand_wildcards" to be all lowercase. For example, for expand_wildcards=All the elasticsearch also will return 400 bad request.

Capitalization good to me:

[JsonConverter(typeof(ExpandWildcardConverter))]
public enum ExpandWildcard
{
/// <summary>
/// <para>Match open, non-hidden indices. Also matches any non-hidden data stream.</para>
/// </summary>
[EnumMember(Value = "open")]
Open,
/// <summary>
/// <para>Wildcard expressions are not accepted.</para>
/// </summary>
[EnumMember(Value = "none")]
None,
/// <summary>
/// <para>Match hidden data streams and hidden indices. Must be combined with open, closed, or both.</para>
/// </summary>
[EnumMember(Value = "hidden")]
Hidden,
/// <summary>
/// <para>Match closed, non-hidden indices. Also matches any non-hidden data stream. Data streams cannot be closed.</para>
/// </summary>
[EnumMember(Value = "closed")]
Closed,
/// <summary>
/// <para>Match any data stream or index, including hidden ones.</para>
/// </summary>
[EnumMember(Value = "all")]
All
}

Did you test it? When converting to JSON the code indeed uses this EnumMemberAttribute, but when converting to an URL query string parameter -- then a simple .ToString() method is called on the enum value (first the IEnumerable branch will be called after my patch and then for each member of the collection ResolveUrlParameterOrDefault will be called. The latter just will call .ToString()). And .ToString() doesn't respect the EnumMemberAttribute.

public static string CreateString(object value, ITransportConfiguration settings)
	{
		switch (value)
		{
			case null: return null;
			case string s: return s;
			case string[] ss: return string.Join(",", ss);
			case Enum e: return e.GetStringValue();
			case bool b: return b ? "true" : "false";
			case DateTimeOffset offset: return offset.ToString("o");
			case IEnumerable<object> pns:
				return string.Join(",", pns.Select(o => ResolveUrlParameterOrDefault(o, settings)));
			case TimeSpan timeSpan: return timeSpan.ToTimeUnit();
			default:
				return ResolveUrlParameterOrDefault(value, settings);
		}
	}

	private static string ResolveUrlParameterOrDefault(object value, ITransportConfiguration settings) =>
		value is IUrlParameter urlParam ? urlParam.GetString(settings) : value.ToString();

https://github.com/elastic/elastic-transport-net/blob/04320b1bd6eb771f8e57db0602de730b766bd339/src/Elastic.Transport/Requests/UrlFormatter.cs#L62C2-L63C85

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
8.x Relates to a 8.x client version
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants