TASTy format versioning scheme

[anatoliykmetyuk]

Currently, we are only using major and minor version numbers to identify the TASTy format version. We need to rethink this scheme before the release.

After the release, in versions 3.0.x, we will maintain both backward and forward compatibility of the format. However, there will be a transition period between 3.0.x and 3.1.x. During this period, there will be commits that break forward compatibility of the format in preparation for 3.1.x while still on 3.0.x. For this situation, we need to modify the TASTy versioning scheme to specify whether the version is the transition version. In general, we need to think of the logistics of TASTy evolution between 3.0.x and 3.1.x.

This issue is the place for this discussion. We need to come to a decision about it before the 3.0.0-RC1 release since we'd like to try out the new versioning format of TASTy in that release.

/cc @odersky @sjrd @smarter @bishabosha

[bishabosha]

@sjrd with the way Tasty unpickling is currently implemented it seems that nothing needs to be done to support the above proposed process:

• we keep same major version between two backwards compatible Tasty releases
• we increase the minor version in Tasty for forward incompatible (but still backwards compatible) releases, and so the compiler will reject Tasty with a higher minor version than it knows, but continues to support Tasty with a smaller minor version and same major version.

[sjrd]

Yes, that is true. But the limitation of the current scheme is that, when we read an invalid TASTy version, all we can tell the user is "version 7.45 is invalid, expected 7.40 or an earlier 7.x" or something like that. But that doesn't help the user know what version of Scala does 7.45 correspond to.

The idea here would be to align the TASTy versions better with the language version. So we could say "version 3.1 is invalid, expected 3.0; please upgrade to Scala to 3.1.0 or later".

But with that scheme comes the issue of "transient" TASTy versions: the ones in between 3.0.n and 3.1.0, when we're in snapshots/nighly builds and we've already broken forward compat, but we will do it again before 3.1.0. In that case we cannot use 3.0 nor 3.1. We would need something else that is "in the middle", and which is backward compatible with 3.0 but not compatible in any way with 3.1.

In Scala.js, we use 3.1-SNAPSHOT for that purpose, or 3.1-M1, etc. That requires a third component which is a string, though.

[smarter]

That requires a third component which is a string, though.

Since that's a bit complicated in a binary format, my suggestion is to just use a number for the third field, this field would be 0 for final versions and non-zero for preview versions.

[bishabosha]

It should be fine to put after the uuid (or minor version) the string length as a nat, then simply treat the next n bytes as UTF-8

[smarter]

Yes, but it's some extra complexity that might not be worth it.

[bishabosha]

I propose that an additional two Longs worth of header can be used to fully encode the same information as is currently displayed to the user when displaying the compiler version using -version flag, e.g. 3.0.0-M4-NIGHTLY-git-c2f24d2ee7938e59, we could call this a vendor string payload.

but then this does not account for some tool different to the scala compiler producing a tasty file

[bishabosha]

is the purpose of this extra field to simply display a better error message?, which would imply it needs to be able to decode a future version string that represents a scala compiler version for them to upgrade to.

[sjrd]

Yes, I don't think there's any other reason.

[bishabosha]

At the Dotty meeting we discussed that it would be desirable for the "data format" version to still be independent of the source language/library version.

• but it is still important to track the addition of primitive types in the compiler in a forward incompatible tasty version.

A point has also been made that we do not want to bump the minor version in a snapshot/nightly as that version is not stable/for-release, so we shouldn't be growing the minor version out of control.

So a potential plan - one extra field encodes that a tasty version is experimental, and observing that flag should cause a different error mode to if the version is not experimental.

There can then potentially still be an extra field for encoding a string of the compiler version, but the format has not been discussed.

[sjrd]

After more discussion, we will use the following. We use three fields:

• The first two match the major and minor version of the last compiler version that broken forward compatibility
• The third one is 0 for the TASTy format released by the final version of the above compiler
• It is a positive number if the compiler is in-between minor versions and has broken forward compatibility since the previous stable version.

[bishabosha]

This scheme does not address how to identify something that is not backwards compatible, for example: a lower major version is now considered to be backwards compatible, but before, any change to major version signals "not backwards compatible".

Or is Scala 4.0.0 the first non-TASTY-backwards compatible release?

[bishabosha]

And also, do we need a way to quickly reject the old scheme, or should we not worry about someone trying to read code compiled by Dotty 0.6.0-RC1 (Major version 3, minor version 0)

[sjrd]

This scheme does not address how to identify something that is not backwards compatible, for example: a lower major version is now considered to be backwards compatible, but before, any change to major version signals "not backwards compatible".

Or is Scala 4.0.0 the first non-TASTY-backwards compatible release?

Ah! That is a good point. I believe that kills the idea of tying up the "public" version number and the TASTy version number. Unless we're ready to commit to 3.T being in fact 4.0, but that we know we're not ready to.

[bishabosha]

I believe that kills the idea of tying up the "public" version number and the TASTy version number. Unless we're ready to commit to 3.T being in fact 4.0, but that we know we're not ready to.

so perhaps the first two fields remain the major and minor versions as before, and continuing from their current values.
Then we just add the "experimental" field, and then two fields for the Scala compiler major/minor versions. Three extra Nats

[smarter]

Ah! That is a good point. I believe that kills the idea of tying up the "public" version number and the TASTy version number. Unless we're ready to commit to 3.T being in fact 4.0, but that we know we're not ready to.

We don't have to commit to it, we can go with the scheme where the tasty and scala version number match now, then when we get to the 3.T/4.0 release we can decide whether to decouple them if we really want to go with 3.T (even if we decouple them, we could keep a mapping that tasty version 4.x is Scala version 3.T+x, but it's not something I'd worry about now).

[bishabosha]

In todays meeting we decided to go with four fields - MajorVersion, MinorVersion, ExperimentalVersion all Natural numbers, and CompilerVersion, and arbitrary string for printing error messages.

@sjrd what should the comparison semantics be for when MajorVersion and MinorVersion in the tasty file match what the compiler expects?

For example:

• release Scala 3.0.5 - 28.0 @ 0
  • say this is our final forward compatible Scala 3.0.x release
• release Scala 3.1.0-M1 - 28.1 @ 1
  • say we make a forward incompatible change to TASTy, so we bump the minor version and experimental version
• release Scala 3.1.0-M2 - 28.1 @ 2
  • say we try an alternative and incompatible implementation to 28.1 @ 1, so we bump the experimental version
• release Scala 3.1.0 - 28.1 @ 0
  • we have decided a final implementation of the forward incompatible change

none of these experimental version can read each other, and the final version does not know which experimental version it was based from, so it must reject those too. So it seems to me that the experimental version must just always match.

[sjrd]

Let's call myVersion the version of the current compiler, and fileVersion the version it's reading from the file. The algorithm is:

• If fileVersion.major != myVersion.major, then return incompatible
• If myVersion.experimental == 0, then
  • If fileVersion.experimental != 0, then return incompatible
  • If fileVersion.minor > myVersion.minor, then return incompatible
  • Otherwise, return compatible
• Otherwise (myVersion.experimental != 0),
  • If fileVersion.experimental == myVersion.experimental, then
    • If fileVersion.minor == myVersion.minor, then return compatible (all fields equal)
    • Otherwise, return incompatible
  • Otherwise, if fileVersion.experimental == 0,
    • If fileVersion.minor < myVersion.minor, then return compatible (an experimental version can read a previous released version)
    • Otherwise, return incompatible (an experimental version cannot read its own minor version or any later version)
  • Otherwise (fileVersion.experimental is non-0 and different than myVersion.experimental), return incompatible

There might be a way to "optimize" that, but that's the spec.