InstantAsStringAttributeConverter does not appear to have lexicographically correct sorting as claimed

[andrewparmet]

Describe the bug

InstantAsStringAttributeConverter claims that its provided serialization is lexicographically sortable given that the year of the Instant is non-negative. It seems to also not be lexicographically sortable if any value falls directly on a second, since InstantAsStringAttributeConverter simply calls toString(), which uses an adapting ISO 8601 formatter that will truncate 3-groups of zero values:

From Instant's toString():

The format used is the same as DateTimeFormatter.ISO_INSTANT.

From DateTimeFormatter's ISO_INSTANT:

The nano-of-second outputs zero, three, six or nine digits as necessary.

The manifestation of this problem was a query on a sort key that, when re-sorted in application code, changed its order.

Expected Behavior

When using an Instant as a sort key on a DynamoDbBean, I expect sort orderings like this to be returned by a query:

2020-12-23T07:05:28.991Z
2020-12-23T07:05:29Z
2020-12-23T07:05:29.017Z

Current Behavior

Instead I'm getting orderings like this:

2020-12-23T07:05:28.991Z
2020-12-23T07:05:29.017Z
2020-12-23T07:05:29Z

since Z comes after . lexicographically. This disagrees with the natural sort order of the corresponding Instants.

Steps to Reproduce

Create a Dynamo schema with a string sort key, write two instants that share a second, but ensure one of them falls directly on the second while the other has some fractional component.

Possible Solution

Use new DateTimeFormatterBuilder().appendInstant(9).toFormatter() to control the number of zeros. For any values taken from System.currentTimeMillis() there will be a lot of unnecessary zeros, but I don't think there's any other correct default behavior.

Context

It's nice to combine the enhanced client's ability to use native Java types with querying by time. In this case the workaround is pretty straightforward for my case.

Your Environment

• AWS Java SDK version used: 2.15.50
• JDK version used: 11
• Operating System and version: Mac OS X 10.15.7

[debora-ito]

@andrewparmet I'm trying to understand what's the issue exactly. In the current behavior the results are listed in lexicographically order "since Z comes after . lexicographically" as you stated, so this behavior matches the documentation, it's not a bug.

Would this be a feature request for the result be sorted like in the expected behavior?

[andrewparmet]

@debora-ito For ISO 8601-formatted instants, which is what the converter specifies as its destination format, you can get lexicographical ordering that matches natural ordering. One requirement to get that guarantee is that the precision after the decimal is consistent among elements.

In particular the sentence "This serialization is lexicographically orderable when the year is not negative," by indicating the other necessary condition on the elements - namely, that the year is not negative - implies that it's meant to be used in this way. (Actually, you can still get natural lexicographical ordering if all years are negative and you reverse the sort.) This guarantee makes it useful for mapping Instants used as sort keys, since a time-based query will translate into a sort key query on a string attribute.

All UTF-8 text is trivially lexicographically orderable. I think if the current behavior is desired there should be a disclaimer saying that Instants mapped this way should not be used as sort keys with the intent to be used with time-based queries.

[brandondahler]

+1 - this also has implications for people who wish to use Instant values in ConditionExpressions.

It is easy to fall into the trap of assuming that the Instant value will be strictly increasing lexicographically, but it leaves corner cases where, e.g. 2021-02-09T00:00:00Z happens after 2021-02-09T00:00:00.999999999Z according to dynamodb's expressions because of how the Instant value is converted to a string.

As a developer my general assumption is that all "normal" instants (those between 0000-01-01T00:00:00Z and 9999-12-31T23:59:59.999999999Z) that are saved to DynamoDb via the mapper behave the same way they would outside of the database.

As for whether this is a bug or a feature request - it feels like it is in the grey area between the two. At very least a warning should be added to the documentation.