You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: _overviews/tutorials/binary-compatibility-for-library-authors.md
+59-53Lines changed: 59 additions & 53 deletions
Original file line number
Diff line number
Diff line change
@@ -23,10 +23,13 @@ Before we start, let's understand how code is compiled and executed on the Java
23
23
24
24
## The JVM execution model
25
25
26
-
Scala is compiled to a platform-independent format called **JVM bytecode** and stored in `.class` files. These class files are collated in JAR files for distribution.
26
+
Scala is compiled to a platform-independent format called **JVM bytecode** and stored in `.class` files.
27
+
These class files are collated in JAR files for distribution.
27
28
28
-
When some code depends on a library, its compiled bytecode references the library's bytecode. The library's bytecode is referenced by its class/method signatures and loaded lazily
29
-
by the JVM classloader during runtime. If a class or method matching the signature is not found, an exception is thrown.
29
+
When some code depends on a library, its compiled bytecode references the library's bytecode.
30
+
The library's bytecode is referenced by its class/method signatures and loaded lazily
31
+
by the JVM classloader during runtime. If a class or method matching the signature is not found,
32
+
an exception is thrown.
30
33
31
34
As a result of this execution model:
32
35
@@ -42,7 +45,14 @@ Consider an application `App` that depends on `A` which itself depends on librar
42
45
for all of `App`, `A` and `C` (something like `java -cp App.jar:A.jar:C.jar:. MainClass`). If we did not provide `C.jar` or if we provided a `C.jar` that does not contain some classes/methods
43
46
which `A` calls, we will get classloading exceptions when our code attempts to invoke the missing classes/methods.
44
47
45
-
These are what we call **Binary Incompatibility Errors** -- errors that happen when the compiled bytecode references a name that cannot be resolved during runtime.
48
+
These are what we call **Linkage Errors** -- errors that happen when the compiled bytecode references a name that cannot be resolved during runtime.
49
+
50
+
### What about Scala.js and Scala Native?
51
+
Similarly to the JVM, Scala.js and Scala Native have their respective equivalents of .class files, namely .sjsir files and .nir files. Similarly to .class files, they are distributed in .jars and linked together at the end.
52
+
53
+
However, contrary to the JVM, Scala.js and Scala Native link their respective IR files at link time, so eagerly, instead of lazily at run-time. Failure to correctly link the entire program results in linking errors reported while trying to invoke `fastOptJS`/`fullOptJS` or `nativeLink`.
54
+
55
+
Besides that difference in the timing of linkage errors, the models are extremely similar. **Unless otherwise noted, the contents of this guide apply equally to the JVM, Scala.js and Scala Native.**
46
56
47
57
Before we look at how to avoid binary incompatibility errors, let us first
48
58
establish some key terminologies we will be using for the rest of the guide.
@@ -59,15 +69,14 @@ Because of this, having multiple versions of the same library in the classpath i
59
69
Therefore, when resolving JARs to use for compilation and packaging, most build tools will pick only one version of each library and **evict** the rest.
60
70
61
71
### Source Compatibility
62
-
Two library versions are **Source Compatible** if switching one for the other does not incur any compile errors or unintended behavioral changes at runtime.
63
-
For example, If we can upgrade `v1.0.0` of a dependency to `v1.1.0` and recompile our code without any compilation errors, `v1.1.0` is source compatible with `v1.0.0`.
72
+
Two library versions are **Source Compatible**with each other if switching one for the other does not incur any compile errors or unintended behavioral changes (semantic errors).
73
+
For example, If we can upgrade `v1.0.0` of a dependency to `v1.1.0` and recompile our code without any compilation errors or semantic errors, `v1.1.0` is source compatible with `v1.0.0`.
64
74
65
75
### Binary Compatibility
66
-
Two library versions are **Binary Compatible** if the compiled bytecode of these versions can be interchanged without causing binary compatibility errors.
67
-
For example, if we can replace the class files of a library's `v1.0.0` with the class files of `v1.1.0` without any binary compatibility errors during runtime,
68
-
`v1.1.0` is binary compatible with `v1.0.0`.
76
+
Two library versions are **Binary Compatible** with each other if the compiled bytecode of these versions can be interchanged without causing Linkage Errors.
69
77
70
-
**NOTE:** While breaking source compatibility normally results in binary compatibility breakages as well, they are actually orthogonal -- breaking one does not imply breaking the other.
78
+
### Relationship between source and binary compatibility
79
+
While breaking source compatibility often results in binary incompatibilities as well, they are actually orthogonal -- breaking one does not imply breaking the other.
71
80
72
81
#### Forwards and Backwards Compatibility
73
82
@@ -76,20 +85,20 @@ There are two "directions" when we describe compatibility of a library release:
76
85
**Backwards Compatible** means that a newer library version can be used in an environment where an older version is expected. When talking about binary and source compatibility,
77
86
this is the common and implied direction.
78
87
79
-
**Forwards Compatible** means that an older library can be used in an environment where a newer version is expected.
80
-
Forward compatibility is generally not upheld for userland libraries. It is only important in situations where an older version of a library is commonly used at runtime against code that is compiled with newer versions. (for example, [Scala's standard library](http://docs.scala-lang.org/overviews/core/binary-compatibility-of-scala-releases.html))
88
+
**Forwards Compatible** means that an older library can be used in an environment where a newer version is expected.
89
+
Forward compatibility is generally not upheld for libraries.
81
90
82
91
Let's look at an example where library `A v1.0.0` is compiled with library `C v1.1.0`.
`C v1.1.0 ` is **Forwards Binary Compatible** with `v1.0.0` if we can use `v1.0.0`'s JAR at runtime instead of `v1.1.0`'s JAR without any binary compatibility errors.
95
+
`C v1.1.0 ` is **Forwards Binary Compatible** with `v1.0.0` if we can use `v1.0.0`'s JAR at runtime instead of `v1.1.0`'s JAR without any linkage errors.
87
96
88
-
`C v1.2.0 ` is **Backwards Binary Compatible** with `v1.1.0` if we can use `v1.2.0`'s JAR at runtime instead of `v1.1.0`'s JAR without any binary compatibility errors.
97
+
`C v1.2.0 ` is **Backwards Binary Compatible** with `v1.1.0` if we can use `v1.2.0`'s JAR at runtime instead of `v1.1.0`'s JAR without any linkage errors.
89
98
90
99
## Why binary compatibility matters
91
100
92
-
Binary Compatibility matters because breaking binary compatibility have bad consequences on the ecosystem around the software.
101
+
Binary Compatibility matters because breaking binary compatibility has bad consequences on the ecosystem around the software.
93
102
94
103
* End users have to update versions transitively in all their dependency tree such that they are binary compatible. This process is time-consuming and error-prone, and it can change the semantics of end program.
95
104
* Library authors need to update their library dependencies to avoid "falling behind" and causing dependency hell for their users. Frequent binary breakages increase the effort required to maintain libraries.
@@ -134,30 +143,32 @@ How can we, as library authors, spare our users of runtime errors and dependency
134
143
## MiMa - Checking binary compatibility against previous library versions
135
144
136
145
The [Migration Manager for Scala](https://github.com/typesafehub/migration-manager) (MiMa) is a tool for diagnosing binary incompatibilities between different library versions.
137
-
It works by comparing the class files of two provided JARs and report any binary incompatibilities found.
146
+
It works by comparing the class files of two provided JARs and report any binary incompatibilities found.
147
+
Both backwards and forwards binary incompatibility can be detected by swapping input order of the JARs.
138
148
139
149
By incorporating [MiMa SBT plugin](https://github.com/typesafehub/migration-manager/wiki/Sbt-plugin) into your SBT build, you can easily check whether
140
150
you have accidentally introduced binary incompatible changes. Detailed instruction on how to use the SBT plugin can be found in the link.
141
151
142
-
We strongly encourage every library author to incorporate MiMa into their library release workflow.
152
+
We strongly encourage every library author to incorporate MiMa into their continuous integration and release workflow.
153
+
154
+
Detecting backwards source compatibility is difficult with Scala due to language features like implicit
155
+
and named parameters. The best approximation to checking backwards source compatibility is running
156
+
both forwards and backwards binary compatibility check, as this can detect most cases
157
+
of source-incompatible changes. For example, adding/removing public class members is a source
158
+
incompatible change, and will be caught through forward + backward binary compatibility check.
143
159
144
160
## Evolving code without breaking binary compatibility
145
161
146
-
Binary compatibility breakages can often be avoided through careful use of certain Scala features as well as some techniques you can apply when modifying code.
162
+
Binary compatibility breakages can often be avoided through the careful use of certain Scala features
163
+
as well as some techniques you can apply when modifying code.
147
164
148
-
Some language features may break binary compatibility:
165
+
For example, the use of these language features are a common source of binary compatibility breakages
166
+
in library releases:
149
167
150
168
* Default parameter values for methods or classes
151
169
* Case classes
152
-
* Default methods on traits (doesn't cause breakages since 2.12)
153
-
154
-
Techniques you can use to avoid breaking binary compatibility:
155
-
156
-
* Annotate public method's return type explicitly
157
-
* Mark methods as package private when you want to remove a method or modify its signature
158
-
* Don't turn on inlining for methods from dependencies
159
170
160
-
For brevity of this guide, detailed explanation and runnable code examples can be found in [Binary Compatibility Code Examples & Explanation](https://github.com/jatcwang/binary-compatibility-guide).
171
+
You can find detailed explanations, runnable examples and tips to maintain binary compatibility in [Binary Compatibility Code Examples & Explanation](https://github.com/jatcwang/binary-compatibility-guide).
161
172
162
173
Again, we recommend using MiMa to double check that you have not broken binary compatibility after making changes.
163
174
@@ -171,45 +182,40 @@ Semantic Versioning v2.0.0.
171
182
172
183
### Recommended Versioning Scheme
173
184
174
-
* If backward **binary compatibility** is broken, **major version number** must be increased
175
-
* If backward **source compatibility** is broken, **minor version number** must be increased
176
-
* A change in **patch version number** signals **no binary nor source incompatibility**. According to SemVer, patch versions should contain only bug fixes that fix incorrect behavior so major behavioral
185
+
* If backward **binary compatibility** is broken, **major version number** must be increased.
186
+
* If backward **source compatibility** is broken, **minor version number** must be increased.
187
+
* A change in **patch version number** signals **no binary nor source incompatibility**.
188
+
According to SemVer, patch versions should contain only bug fixes that fix incorrect behavior so major behavioral
177
189
change in method/classes should result in a minor version bump.
178
-
* When major version is `0`, a minor version bump **may contain both source and binary breakages**
179
-
* Some libraries may take a harder stance on maintaining source compatibility, bumping the major version number for ANY source incompatibility even if they are binary compatible
190
+
* When major version is `0`, a minor version bump **may contain both source and binary breakages**.
180
191
181
192
Some examples:
182
193
183
-
*`v1.0.0 -> v2.0.0` is <spanstyle="color: red">binary incompatible</span>. Cares needs to be taken to make sure no evicted versions are still in the `v1.x.x` range to avoid runtime errors
184
-
*`v1.0.0 -> v1.1.0` is <spanstyle="color: blue">binary compatible</span> and maybe source incompatible
185
-
*`v1.0.0 -> v1.0.1` is <spanstyle="color: blue">binary compatible</span> and source compatible
186
-
*`v0.4.0 -> v0.5.0` is <spanstyle="color: red">binary incompatible</span> and maybe source incompatible
187
-
*`v0.4.0 -> v0.4.1` is <spanstyle="color: blue">binary compatible</span> and source compatible
194
+
*`v1.0.0 -> v2.0.0` is <spanstyle="color: #c10000">binary incompatible</span>.
195
+
End users and library maintainers need to update all their dependency graph to remove all dependency on `v1.0.0`.
196
+
*`v1.0.0 -> v1.1.0` is <spanstyle="color: #2b2bd4">binary compatible</span>. Classpath can safely contain both `v1.0.0` and `v1.1.0`. End user may need to fix minor source breaking changes introduced
197
+
*`v1.0.0 -> v1.0.1` is <spanstyle="color: #2b2bd4">binary compatible</span>. This is a safe upgrade that does not introduce binary or source incompatibilities.
198
+
*`v0.4.0 -> v0.5.0` is <spanstyle="color: #c10000">binary incompatible</span>.
199
+
End users and library maintainers need to update all their dependency graph to remove all dependency on `v0.4.0`.
200
+
*`v0.4.0 -> v0.4.1` is <spanstyle="color: #2b2bd4">binary compatible</span>. Classpath can safely contain both `v1.0.0` and `v1.1.0`. End user may need to fix minor source breaking changes introduced
188
201
189
202
Many libraries in the Scala ecosystem has adopted this versioning scheme. A few examples are [Akka](http://doc.akka.io/docs/akka/2.5/scala/common/binary-compatibility-rules.html),
190
203
[Cats](https://github.com/typelevel/cats#binary-compatibility-and-versioning) and [Scala.js](https://www.scala-js.org/).
191
204
192
-
If this version scheme is followed, reasoning about binary compatibility is now very simple:
193
-
194
-
* Ensure major versions of all versions of a library in the dependency tree are the same
195
-
* Pick the latest version and evict the rest (This is the default behavior of SBT).
196
-
197
-
### Explanation
205
+
## Conclusion
198
206
199
-
Why do we use the major version number to signal binary incompatible releases?
207
+
Why is binary compatibility so important such that we recommend using the major version number to track it?
200
208
201
209
From our [example](#why-binary-compatibility-matters) above, we have learned two important lessons:
202
210
203
-
* Binary incompatibility releases often leads to dependency hell, rendering your users unable to update any of their libraries without breaking their application
204
-
* If a new library version is binary compatible but source incompatible, the user can simply fix the compile errors in their application and everything will work
211
+
* Binary incompatibility releases often lead to dependency hell, rendering your users unable to update any of their libraries without breaking their application.
212
+
* If a new library version is binary compatible but source incompatible, the user can fix the compile errors and their application should work.
205
213
206
-
Therefore, **binary incompatible releases should be avoided if possible** and be more noticeable when they happen, warranting the use of the major version number. While source compatibility
207
-
is also important, if they are minor breakages that do not require effort to fix, then it is best to let the major number signal just binary compatibility.
208
-
209
-
## Conclusion
214
+
Therefore, **binary incompatible releases should be avoided if possible** and unambiguously documented
215
+
when they happen, warranting the use of the major version number. Users of your library can then enjoy
216
+
simple version upgrades and have clear warnings when they need to align library versions in their dependency tree
217
+
due to a binary incompatible release.
210
218
211
-
In this guide, we covered the importance of binary compatibility and showed you a few tricks to avoid breaking binary compatibility. Finally, we laid out a versioning scheme to communicate
212
-
binary compatibility breakages clearly to your users.
219
+
If we follow all recommendations laid out in this guide, we as a community can spend less time untangling dependency hell and more time building cool things!
213
220
214
-
If we follow these guidelines, we as a community can spend less time untangling dependency hell and more time building cool things!
0 commit comments