diff --git a/tck/README.md b/tck/README.md index b401417e..3ba21fff 100644 --- a/tck/README.md +++ b/tck/README.md @@ -3,21 +3,23 @@ The purpose of the *Reactive Streams Technology Compatibility Kit* (from here on referred to as: *the TCK*) is to guide and help Reactive Streams library implementers to validate their implementations against the rules defined in [the Specification](https://github.com/reactive-streams/reactive-streams-jvm). -The TCK is implemented using **plain Java (1.6)** and **TestNG** tests, and should be possible to use from other languages and testing libraries (such as Scala, Groovy, JRuby or others). +The TCK is implemented using **plain Java (1.6)** and **TestNG** tests, and should be possible to use from other JVM-based languages and testing libraries. ## Structure of the TCK The TCK aims to cover all rules defined in the Specification, however for some rules outlined in the Specification it is -not possible (or viable) to construct automated tests, thus the TCK does not claim to completely verify an implementation, however it is very helpful and is able to validate the most important rules. +not possible (or viable) to construct automated tests, thus the TCK can not claim to fully verify an implementation, however it is very helpful and is able to validate the most important rules. -The TCK is split up into 4 TestNG test classes which should be extended by implementers, providing their `Publisher` / `Subscriber` implementations for the test harness to validate them. The tests are split in the following way: +The TCK is split up into 4 TestNG test classes which are to be extended by implementers, providing their `Publisher` / `Subscriber` / `Processor` implementations for the test harness to validate. + +The tests are split in the following way: * `PublisherVerification` * `SubscriberWhiteboxVerification` * `SubscriberBlackboxVerification` * `IdentityProcessorVerification` -The next sections include examples on how these can be used and describe the various configuration options. +The sections below include examples on how these can be used and describe the various configuration options. The TCK is provided as binary artifact on [Maven Central](http://search.maven.org/#search|ga|1|reactive-streams-tck): @@ -25,23 +27,22 @@ The TCK is provided as binary artifact on [Maven Central](http://search.maven.or org.reactivestreams reactive-streams-tck - ... + 1.0.0.RC3 test ``` -Please refer to the [Reactive Streams Specification](https://github.com/reactive-streams/reactive-streams-jvm) for the current latest version number. Make sure that the API and TCK dependency versions are equal. +Please refer to the [Reactive Streams Specification](https://github.com/reactive-streams/reactive-streams-jvm) for the current latest version number. Make sure that your Reactive Streams API and TCK dependency versions match. ### Test method naming convention -Since the TCK is aimed at Reactive Stream implementers, looking into the sources of the TCK is well expected, -and should help during a libraries implementation cycle. +Since the TCK is aimed at Reactive Stream implementers, looking into the sources of the TCK is well expected and encouraged as it should help during the implementation cycle. In order to make mapping between test cases and Specification rules easier, each test case covering a specific Specification rule abides the following naming convention: `TYPE_spec###_DESC` where: * `TYPE` is one of: [#type-required](required), [#type-optional](optional), [#type-stochastic](stochastic) or [#type-untested](untested) which describe if this test is covering a Rule that MUST or SHOULD be implemented. The specific words are explained in detail below. -* `###` is the Rule number (`1.xx` Rules are about Publishers, `2.xx` Rules are about Subscribers etc.) +* `###` is the Rule number (`1.xx` Rules are about `Publisher`s, `2.xx` Rules are about Subscribers etc.) * `DESC` is a short explanation of what exactly is being tested in this test case, as sometimes one Rule may have multiple test cases in order to cover the entire Rule. Here is an example test method signature: @@ -68,7 +69,7 @@ The `required_` means that this test case is a hard requirement, it covers a *MU ``` -The `optional_` means that this test case is optional, it covers a *MAY* or *SHOULD* Rule of the Specification. +The `optional_` means that this test case is an optional requirement, it covers a *MAY* or *SHOULD* Rule of the Specification. This prefix is also used if more configuration is needed in order to run it, e.g. `@Additional(implement = "createFailedPublisher") @Test` signals the implementer that in order to run this test one has to implement the `Publisher createFailedPublisher()` method. @@ -78,8 +79,8 @@ one has to implement the `Publisher createFailedPublisher()` method. ``` -The `stochastic_` means that the Rule is either racy, and/or inherently hard to verify without heavy modification of the tested implementation. -Usually this means that this test case can yield false positives ("be green") even if for some case, the given implementation may violate the tested behaviour. +The `stochastic_` means that the Rule is impossible or infeasible to deterministically verify— +usually this means that this test case can yield false positives ("be green") even if for some case, the given implementation may violate the tested behaviour. ```java @Test public void untested_spec106_mustConsiderSubscriptionCancelledAfterOnErrorOrOnCompleteHasBeenCalled() throws Throwable @@ -87,8 +88,8 @@ Usually this means that this test case can yield false positives ("be green") ev The `untested_` means that the test case is not implemented, either because it is inherently hard to verify (e.g. Rules which use -the wording "*SHOULD consider X as Y*"). Such tests will show up in your test runs as `SKIPPED`, with a message pointing out that the TCK is unable to validate this Rule. If you figure out a way to deterministically test Rules which have been -marked with this prefix – pull requests are encouraged! +the wording "*SHOULD consider X as Y*"). Such tests will show up in your test runs as `SKIPPED`, with a message pointing out that the TCK is unable to validate this Rule. Solutions to deterministically test Rules which have been +marked with this prefix are most welcome – pull requests are encouraged! ### Test isolation @@ -96,15 +97,13 @@ All test assertions are isolated within the required `TestEnvironment`, so it is ### Testing Publishers with restricted capabilities -Some `Publisher`s will not be able to pass through all TCK tests due to some internal or fundamental decissions in their design. -For example, a `FuturePublisher` can be implemented such that it can only ever `onNext` **exactly once** - this means that it is not possible -to run all TCK tests against it, since the tests sometimes require multiple elements to be emitted. +Some `Publisher`s will not be able to pass through all TCK tests due to some internal or fundamental decisions in their design. +For example, a `FuturePublisher` can be implemented such that it can only ever `onNext` **exactly once**—this means that it's not possible to run all TCK tests against it since some of them require multiple elements to be emitted. -In order to allow such restricted capabilities to be tested against the spec's rules, the TCK provides the `maxElementsFromPublisher()` method -as means of communicating to the TCK the limited capabilities of the Publisher. For example, if a publisher can only ever emit up to `2` elements, -tests in the TCK which require more than 2 elements to verify a rule can be skipped. +In order to allow such `Publisher`s to be tested against the spec's rules, the TCK provides the `maxElementsFromPublisher()` method as means of communicating the limited capabilities of the Publisher. For example, if a `Publisher` can only ever emit up to `2` elements, +tests in the TCK which require more than 2 elements to verify a rule will be skipped. -In order to inform the TCK your Publisher is only able to signal up to `2` elements, override the `maxElementsFromPublisher` method like this: +In order to inform the TCK that the `Publisher` is only able to signal up to `2` elements, override the `maxElementsFromPublisher` method like this: ```java @Override public long maxElementsFromPublisher() { @@ -112,12 +111,12 @@ In order to inform the TCK your Publisher is only able to signal up to `2` eleme } ``` -The TCK also supports Publishers which are not able to signal completion. For example you might have a Publisher being -backed by a timer. Such Publisher does not have a natural way to "complete" after some number of ticks. It would be -possible to implement a Processor which would "take n elements from the TickPublisher and then signal completion to the -downstream", but this adds a layer of indirection between the TCK and the Publisher one initially wanted to test. -It is suggested to test such unbouded Publishers either way - using a "TakeNElementsProcessor" or by informing the TCK -that the publisher is not able to signal completion. The TCK will then skip all tests which require `onComplete` signals to be emitted. +The TCK also supports `Publisher`s which are not able to signal completion. Imagine a `Publisher` being +backed by a timer—such a `Publisher` does not have a natural way to "complete" after some number of ticks. It would be +possible to implement a `Processor` which would "take n elements from the TickPublisher and then signal completion to the +downstream", but this adds a layer of indirection between the TCK and the `Publisher` one initially wanted to test. +It is suggested to test such unbouded `Publisher`s either way—using a "TakeNElementsProcessor" or by informing the TCK +that the `Publisher` is not able to signal completion. The TCK will then skip all tests which require `onComplete` signals to be emitted. In order to inform the TCK that your Publiher is not able to signal completion, override the `maxElementsFromPublisher` method like this: @@ -128,13 +127,13 @@ In order to inform the TCK that your Publiher is not able to signal completion, ``` ### Testing a "failed" Publisher -The Reactive Streams spec mandates certain behaviours for Publishers which are "failed", +The Reactive Streams Specification mandates certain behaviours for `Publisher`s which are "failed", e.g. it was unable to initialize a connection it needs to emit elements. -It may be useful to specifically such known to be failed Publisher using the TCK. +It may be useful to specifically such known to be failed `Publisher` using the TCK. -In order to run additional tests on your failed publisher you can implement the `createFailedPublisher` method. -The expected behaviour from the returned implementation is to follow Rule 1.4 and Rule 1.9 - which are concerned -with the order of emiting the Subscription and signaling the failure. +In order to run additional tests on a failed `Publisher` implement the `createFailedPublisher` method. +The expected behaviour from the returned implementation is to follow Rule 1.4 and Rule 1.9—which are concerned +with the order of emiting the `Subscription` and signaling the failure. ```java @Override public Publisher createFailedPublisher() { @@ -143,14 +142,13 @@ with the order of emiting the Subscription and signaling the failure. } ``` -In case you don't really have a known up-front error state you can put your Publisher into, -you can easily ignore these tests by returning `null` from the `createFailedPublisher` method. -It is important to remember that it is **illegal** to signal `onNext / onComplete / onError` before -you signal the `Subscription`, for details on this rule refer to the Reactive Streams specification. +In case there isn't a known up-front error state to put the `Publisher` into, +ignore these tests by returning `null` from the `createFailedPublisher` method. +It is important to remember that it is **illegal** to signal `onNext / onComplete / onError` before signalling the `Subscription` through `onSubscribe`, for details on this rule refer to the Reactive Streams specification. ## Publisher Verification -`PublisherVerification` tests verify Publisher as well as some Subscription Rules of the Specification. +`PublisherVerification` tests verify `Publisher` as well as some `Subscription` Rules of the Specification. In order to include it's tests in your test suite simply extend it, like this: @@ -187,7 +185,7 @@ public class RangePublisherTest extends PublisherVerification { @Override public long maxElementsFromPublisher() { - return Long.MAX_VALUE - 1; + return Long.MAX_VALUE—1; } @Override @@ -199,16 +197,16 @@ public class RangePublisherTest extends PublisherVerification { Notable configuration options include: -* `maxElementsFromPublisher` – must be overridden in case the Publisher being tested is of bounded length, e.g. it's wrapping a `Future` and thus can only publish up to 1 element, in which case you +* `maxElementsFromPublisher` – must be overridden in case the `Publisher` being tested is of bounded length, e.g. it's wrapping a `Future` and thus can only publish up to 1 element, in which case you would return `1` from this method. It will cause all tests which require more elements in order to validate a certain Rule to be skipped, -* `boundedDepthOfOnNextAndRequestRecursion` – which must be overridden when verifying synchronous Publishers. +* `boundedDepthOfOnNextAndRequestRecursion` – which must be overridden when verifying synchronous `Publisher`s. This number returned by this method will be used to validate if a `Subscription` adheres to Rule 3.3 and avoids "unbounded recursion". ### Timeout configuration Publisher tests make use of two kinds of timeouts, one is the `defaultTimeoutMillis` which corresponds to all methods used within the TCK which await for something to happen. The other timeout is `publisherReferenceGCTimeoutMillis` which is only used in order to verify -[Rule 3.13](https://github.com/reactive-streams/reactive-streams-jvm#3.13) which defines that subscriber references MUST be dropped +[Rule 3.13](https://github.com/reactive-streams/reactive-streams-jvm#3.13) which defines that `Subscriber` references MUST be dropped by the Publisher. In order to configure these timeouts (for example when running on a slow continious integtation machine), you can either: @@ -240,30 +238,25 @@ Note that explicitly passed in values take precedence over values provided by th ## Subscriber Verification -Subscriber rules Verification is split up into two files (styles) of tests. - -The Blackbox Verification tests do not require the implementation under test to be modified at all, yet they are *not* able -to verify most rules. In Whitebox Verification, more control over `request()` calls etc. is required in order to validate rules more precisely. +`Subscriber` Verification is split up into two files (styles) of tests. -It is highly recommended to implement the `SubscriberWhiteboxVerification` instead of the blackbox one even if it is -more work to do so, as it can test far more corner cases in implementations that would otherwise be left untested -(if only blackbox tests would be used). +It is highly recommended to implement the `SubscriberWhiteboxVerification` instead of the `SubscriberBlackboxVerification` even if it is more work to do so, as it can test far more rules and corner cases in implementations that would otherwise be left untested—which is the case when using the Blackbox Verification. ### createElement and Helper Publisher implementations -Since testing a `Subscriber` is not possible without a corresponding `Publisher` the TCK Subscriber Verifications -both provide a default "*helper publisher*" to drive its test and also alow to replace this Publisher with a custom implementation. -The helper publisher is an asynchronous publisher by default - meaning that your subscriber can not blindly assume single threaded execution. - -When extending subscriber verification classes a type parameter representing the element type passed through the stream must be given. -Implementations usually should not be sensitive to the type of element being signalled, but sometimes a Subscriber may be limited to only be able to work within a known set of types - -like a `FileSubscriber extends Subscriber` for example, that writes each element it receives as new line into a file. -For element type agnostic Subscribers it is the simplest to parameterize the tests using `Integer` and in the `createElement(int idx)` method (explained below in futher detail), return the incoming `int`. -In case an implementation needs to work on a specific type, the verification class should be parameterized using that type (e.g. `class StringSubTest extends SubscriberWhiteboxVerification`) and a `String` must be returned from the `createElement` method. - -While the `Publisher` implementation is provided, creating the signal elements is not – this is because a given Subscriber -may for example only work with `HashedMessage` or some other specific kind of signal. The TCK is unable to generate such +Since testing a `Subscriber` is not possible without a corresponding `Publisher` the TCK `Subscriber` Verifications +both provide a default "*helper publisher*" to drive its tests and also allow to replace this `Publisher` with a custom implementation. +The helper `Publisher` is an asynchronous `Publisher` by default—meaning that a `Subscriber` can not blindly assume single threaded execution. + +When extending `Subscriber` Verification classes a type parameter representing the element type passed through the stream must be given. +Implementations are typically not sensitive to the type of element being signalled, but sometimes a `Subscriber` may be limited to only be able to work within a known set of types - +like a `FileSubscriber extends Subscriber` for example, that writes each element (ByteBuffer) it receives into a file. +For element type agnostic Subscribers the simplest way is to parameterize the tests using `Integer` and in the `createElement(int idx)` method (explained below in futher detail), return the incoming `int`. +In case an implementation needs to work on a specific type, the verification class should be parameterized using that type (e.g. `class StringSubTest extends SubscriberWhiteboxVerification`) and the `createElement` method must be overriden to return a `String`. + +While the Helper `Publisher` implementation is provided, creating its elements is not – this is because a given `Subscriber` +may for example only work with `HashedMessage` or some other specific kind of element. The TCK is unable to generate such special messages automatically, so the TCK provides the `T createElement(Integer id)` method to be implemented as part of -Subscriber Verifications which should take the given ID and return an element of type `T` (where `T` is the type of +`Subscriber` Verifications which should take the given `id` and return an element of type `T` (where `T` is the type of elements flowing into the `Subscriber`, as known thanks to `... extends SubscriberWhiteboxVerification`) representing an element of the stream that will be passed on to the Subscriber. @@ -281,58 +274,23 @@ public class MySubscriberTest extends SubscriberBlackboxVerification { ``` -The `createElement` method MAY be called concurrently from multiple threads, -so in case of more complicated implementations, please be aware of this fact. +NOTE: The `createElement` method *MAY* be called *concurrently from multiple threads*. -**Very Advanced**: While it is not expected for many implementations having to do so, it is possible to take full -which will be driving the TCKs test. This can be achieved by implementing the `createHelperPublisher` method in which one can implement the -`createHelperPublisher` method by returning a custom Publisher implementation which will then be used by the TCK to drive your Subscriber tests: +**Very advanced**: While it is not expected for many implementations having to do so, it is possible to take full control of the `Publisher` which will be driving the TCKs test. This can be achieved by implementing the `createHelperPublisher` method in which one can implement the `createHelperPublisher` method by returning a custom `Publisher` implementation which will then be used by the TCK to drive your `Subscriber` tests: ```java @Override public Publisher createHelperPublisher(long elements) { - return new Publisher() { /* CUSTOM IMPL HERE */ }; -} -``` - -### Subscriber Blackbox Verification - -Blackbox Verification does not require any additional work except from providing a `Subscriber` and `Publisher` instances to the TCK: - -```java -package com.example.streams; - -import org.reactivestreams.Publisher; -import org.reactivestreams.Subscriber; -import org.reactivestreams.Subscription; -import org.reactivestreams.tck.SubscriberBlackboxVerification; -import org.reactivestreams.tck.TestEnvironment; - -public class MySubscriberBlackboxVerificationTest extends SubscriberBlackboxVerification { - - public MySubscriberBlackboxVerificationTest() { - super(new TestEnvironment()); - } - - @Override - public Subscriber createSubscriber() { - return new MySubscriber(); - } - - @Override - public Integer createElement(int element) { - return element; - } + return new Publisher() { /* CUSTOM IMPL HERE WHICH OF COURSE ALSO SHOULD PASS THE TCK */ }; } ``` ### Subscriber Whitebox Verification -The Whitebox Verification tests are able to verify most of the Specification, at the additional cost that control over demand generation and cancellation must be handed over to the TCK via the `SubscriberPuppet`. +The Whitebox Verification is able to verify most of the `Subscriber` Specification, at the additional cost that control over demand generation and cancellation must be handed over to the TCK via the `SubscriberPuppet`. -Based on experiences so far implementing the `SubscriberPuppet` is non-trivial and can be hard for some implementations. -We keep the whitebox verification, as it is tremendously useful in the `ProcessorVerification`, where the Puppet is implemented within the TCK and injected to the tests. -We do not expect all implementations to make use of the plain `SubscriberWhiteboxVerification`, using the `SubscriberBlackboxVerification` instead. +Based on experience implementing the `SubscriberPuppet`—it can be tricky or even impossible for some implementations, +as such, not all implementations are expected to make use of the plain `SubscriberWhiteboxVerification`, instead having to fall back to using the `SubscriberBlackboxVerification`. For the simplest possible (and most common) `Subscriber` implementation using the whitebox verification boils down to exteding (or delegating to) your implementation with additionally signalling and registering the test probe, as shown in the below example: @@ -364,7 +322,7 @@ public class MySubscriberWhiteboxVerificationTest extends SubscriberWhiteboxVeri public void onSubscribe(final Subscription s) { super.onSubscribe(s); - // register a successful subscription, and create a Puppet, + // register a successful Subscription, and create a Puppet, // for the WhiteboxVerification to be able to drive its tests: probe.registerOnSubscribe(new SubscriberPuppet() { @@ -411,9 +369,40 @@ public class MySubscriberWhiteboxVerificationTest extends SubscriberWhiteboxVeri } ``` +### Subscriber Blackbox Verification + +Blackbox Verification does not require anything besides providing a `Subscriber` and `Publisher` instances to the TCK, +at the expense of not being able to verify as much as the `SubscriberWhiteboxVerification`: + +```java +package com.example.streams; + +import org.reactivestreams.Publisher; +import org.reactivestreams.Subscriber; +import org.reactivestreams.Subscription; +import org.reactivestreams.tck.SubscriberBlackboxVerification; +import org.reactivestreams.tck.TestEnvironment; + +public class MySubscriberBlackboxVerificationTest extends SubscriberBlackboxVerification { + + public MySubscriberBlackboxVerificationTest() { + super(new TestEnvironment()); + } + + @Override + public Subscriber createSubscriber() { + return new MySubscriber(); + } + + @Override + public Integer createElement(int element) { + return element; + } +} +``` + ### Timeout configuration -Similarily to `PublisherVerification`, it is possible to set the timeouts used by the TCK to validate subscriber behaviour. -This can be set either by using env variables or hardcoded explicitly. +Similarily to `PublisherVerification`, it is possible to set the timeouts used by the TCK to validate `Subscriber` behaviour either hard-coded or by using environment variables. **Use env variables** to set the timeout value to be used by the TCK: @@ -424,7 +413,7 @@ export DEFAULT_TIMEOUT_MILLIS=300 Or **define the timeout explicitly in code**: ```java -public class MySubscriberTest extends BlackboxSubscriberVerification { +public class MySubscriberTest extends SubscriberBlackboxVerification { public static final long DEFAULT_TIMEOUT_MILLIS = 300L; @@ -436,19 +425,19 @@ public class MySubscriberTest extends BlackboxSubscriberVerification { } ``` -Note that hard-coded values *take precedence* over environment set values (!). +NOTE: hard-coded values *take precedence* over environment set values (!). ## Subscription Verification Please note that while `Subscription` does **not** have it's own test class, it's rules are validated inside of the `Publisher` and `Subscriber` tests – depending if the Rule demands specific action to be taken by the publishing, or -subscribing side of the subscription contract. +subscribing side of the `Subscription` contract. ## Identity Processor Verification An `IdentityProcessorVerification` tests the given `Processor` for all `Subscriber`, `Publisher` as well as -`Subscription` rules. Internally the `WhiteboxSubscriberVerification` is used for `Subscriber` rules. +`Subscription` rules (internally the `WhiteboxSubscriberVerification` is used for that). ```java package com.example.streams; @@ -491,7 +480,7 @@ public class MyIdentityProcessorVerificationTest extends IdentityProcessorVerifi } // OPTIONAL CONFIGURATION OVERRIDES - // override these only if you understand why you'd need to do so for your impl. + // only override these if understanding the implications of doing so. @Override public long maxElementsFromPublisher() { @@ -505,16 +494,16 @@ public class MyIdentityProcessorVerificationTest extends IdentityProcessorVerifi } ``` -The additional configuration options reflect the options available in the Subscriber and Publisher Verifications. +The additional configuration options reflect the options available in the `Subscriber` and `Publisher` Verifications. -The `IdentityProcessorVerification` also runs additional sanity verifications, which are not directly mapped to -Specification rules, but help to verify that a Processor won't "get stuck" or face similar problems. Please refer to the +The `IdentityProcessorVerification` also runs additional "sanity" verifications, which are not directly mapped to +Specification rules, but help to verify that a `Processor` won't "get stuck" or face similar problems. Please refer to the sources for details on the tests included. ## Ignoring tests -Since you inherit these tests instead of defining them yourself it's not possible to use the usual `@Ignore` annotations -to skip certain tests (which may be perfectly reasonable if your implementation has some know constraints on what it -cannot implement). We recommend the below pattern to skip tests inherited from the TCK's base classes: +Since the tests are inherited instead of user defined it's not possible to use the usual `@Ignore` annotations +to skip certain tests (which may be perfectly reasonable if the implementation has some know constraints on what it +cannot implement). Below is a recommended pattern to skip tests inherited from the TCK's base classes: ```java package com.example.streams; @@ -557,7 +546,7 @@ public class MyIdentityProcessorTest extends IdentityProcessorVerification createIdentityProcessor(int bufferSize) { - return new MyProcessor(buffer Size); // return your implementation + return new MyProcessor(bufferSize); // return implementation to be tested } @Override @@ -570,32 +559,29 @@ public class MyIdentityProcessorTest extends IdentityProcessorVerification