Fixed #188 by making 2.12 an untested rule, as it cannot be deterministically verified.

Introduces prefix naming for TCK rules using "required_", "optional_", "untested_" and "stochastic_".

Updates tck/README.md to reflect these changes

Deletes the Annotations.java file as the annotations are no longer used.

Introduces interfaces for the TCK test methods so it is possible to keep track of implementations,
perform deprecations etc.

Author: viktorklang

tck/README.md

@@ -45,38 +45,40 @@ Specification rule abides the following naming convention: `spec###_DESC` where:
 
 ```java
   // Verifies rule: https://github.com/reactive-streams/reactive-streams#1.1
-  @Required @Test
-  public void spec101_subscriptionRequestMustResultInTheCorrectNumberOfProducedElements() throws Throwable {
+  @Test public void required_spec101_subscriptionRequestMustResultInTheCorrectNumberOfProducedElements() throws Throwable
     // ...
   }
 ```
 
-The annotations on the test methods are used in order to signify the character of the test. For example, these are the kinds of annotations you may find:
+The prefixes of the names of the test methods are used in order to signify the character of the test. For example, these are the kinds of prefixes you may find:
+"required_", "optional_", "stochastic_", "untested_".
+
+Explanations:
 
 ```java
-@Required @Test
+@Test public void required_spec101_subscriptionRequestMustResultInTheCorrectNumberOfProducedElements() throws Throwable
 ```
 
 ... means that this test case is a hard requirement, it covers a *MUST* or *MUST NOT* Rule of the Specification.
 
 
 ```java
-@Additional @Test
+@Test public void optional_spec104_mustSignalOnErrorWhenFails() throws Throwable
 ```
 
-... means that this test case is optional, it covers a *MAY* or *SHOULD* Rule of the Specification. This annotation is also used if more configuration is needed in order to run it, e.g. `@Additional(implement = "createErrorStatePublisher") @Test` signals the implementer that in order to include this test case in his test runs, (s)he must implement the `Publisher<T> createErrorStatePublisher()` method.
+... means that this test case is optional, it covers a *MAY* or *SHOULD* Rule of the Specification. This prefix is also used if more configuration is needed in order to run it, e.g. `@Additional(implement = "createErrorStatePublisher") @Test` signals the implementer that in order to include this test case in his test runs, (s)he must implement the `Publisher<T> createErrorStatePublisher()` method.
 
 ```java
-@Stochastic @Test
+@Test public void stochastic_spec103_mustSignalOnMethodsSequentially() throws Throwable
 ```
 
 ... means that the Rule is either racy, and/or inherently hard to verify without heavy modification of the tested implementation. Usually this means that this test case can yield false positives ("be green") even if for some case, the given implementation may violate the tested behaviour.
 
 ```java
-@NotVerified @Test
+@Test public void untested_spec106_mustConsiderSubscriptionCancelledAfterOnErrorOrOnCompleteHasBeenCalled() throws Throwable
 ```
 
-... means that the test case is not implemented, either because it is inherently hard to verify (e.g. Rules which use the wording "*SHOULD consider X as Y*") or have not been implemented yet (though we hope we have implemented all we could!). Such tests will show up in your test runs as `SKIPPED`, with a message pointing out that the TCK is unable to validate this Rule. We would be delighted if you can figure out a way to deterministically test Rules, which have been marked with this annotation – pull requests are very welcome!
+... means that the test case is not implemented, either because it is inherently hard to verify (e.g. Rules which use the wording "*SHOULD consider X as Y*") or have not been implemented yet (though we hope we have implemented all we could!). Such tests will show up in your test runs as `SKIPPED`, with a message pointing out that the TCK is unable to validate this Rule. We would be delighted if you can figure out a way to deterministically test Rules, which have been marked with this prefix – pull requests are very welcome!
 
 ### Test isolation
 

tck/src/main/java/org/reactivestreams/tck/IdentityProcessorVerification.java

@@ -67,7 +67,7 @@ public void setUp() throws Exception {
 
   // Verifies rule: https://github.com/reactive-streams/reactive-streams#2.1
   @Override @Test
-  public void required___spec201_blackbox_mustSignalDemandViaSubscriptionRequest() throws Throwable {
+  public void required_spec201_blackbox_mustSignalDemandViaSubscriptionRequest() throws Throwable {
     blackboxSubscriberTest(new BlackboxTestStageTestRun() {
       @Override
       public void run(BlackboxTestStage stage) throws InterruptedException {
@@ -82,13 +82,13 @@ public void run(BlackboxTestStage stage) throws InterruptedException {
 
   // Verifies rule: https://github.com/reactive-streams/reactive-streams#2.2
   @Override @Test
-  public void unverified___spec202_blackbox_shouldAsynchronouslyDispatch() throws Exception {
+  public void untested_spec202_blackbox_shouldAsynchronouslyDispatch() throws Exception {
     notVerified(); // cannot be meaningfully tested, or can it?
   }
 
   // Verifies rule: https://github.com/reactive-streams/reactive-streams#2.3
   @Override @Test
-  public void required___spec203_blackbox_mustNotCallMethodsOnSubscriptionOrPublisherInOnComplete() throws Throwable {
+  public void required_spec203_blackbox_mustNotCallMethodsOnSubscriptionOrPublisherInOnComplete() throws Throwable {
     blackboxSubscriberWithoutSetupTest(new BlackboxTestStageTestRun() {
       @Override
       public void run(BlackboxTestStage stage) throws Throwable {
@@ -125,7 +125,7 @@ public void cancel() {
 
   // Verifies rule: https://github.com/reactive-streams/reactive-streams#2.3
   @Override @Test
-  public void required___spec203_blackbox_mustNotCallMethodsOnSubscriptionOrPublisherInOnError() throws Throwable {
+  public void required_spec203_blackbox_mustNotCallMethodsOnSubscriptionOrPublisherInOnError() throws Throwable {
     blackboxSubscriberWithoutSetupTest(new BlackboxTestStageTestRun() {
       @Override
       public void run(BlackboxTestStage stage) throws Throwable {
@@ -164,13 +164,13 @@ public void cancel() {
 
   // Verifies rule: https://github.com/reactive-streams/reactive-streams#2.4
   @Override @Test
-  public void unverified___spec204_blackbox_mustConsiderTheSubscriptionAsCancelledInAfterRecievingOnCompleteOrOnError() throws Exception {
+  public void untested_spec204_blackbox_mustConsiderTheSubscriptionAsCancelledInAfterRecievingOnCompleteOrOnError() throws Exception {
     notVerified(); // cannot be meaningfully tested, or can it?
   }
 
   // Verifies rule: https://github.com/reactive-streams/reactive-streams#2.5
   @Override @Test
-  public void required___spec205_blackbox_mustCallSubscriptionCancelIfItAlreadyHasAnSubscriptionAndReceivesAnotherOnSubscribeSignal() throws Exception {
+  public void required_spec205_blackbox_mustCallSubscriptionCancelIfItAlreadyHasAnSubscriptionAndReceivesAnotherOnSubscribeSignal() throws Exception {
     new BlackboxTestStage(env) {{
       // try to subscribe another time, if the subscriber calls `probe.registerOnSubscribe` the test will fail
       final TestEnvironment.Latch secondSubscriptionCancelled = new TestEnvironment.Latch(env);
@@ -199,26 +199,26 @@ public String toString() {
 
   // Verifies rule: https://github.com/reactive-streams/reactive-streams#2.6
   @Override @Test
-  public void unverified___spec206_blackbox_mustCallSubscriptionCancelIfItIsNoLongerValid() throws Exception {
+  public void untested_spec206_blackbox_mustCallSubscriptionCancelIfItIsNoLongerValid() throws Exception {
     notVerified(); // cannot be meaningfully tested, or can it?
   }
 
   // Verifies rule: https://github.com/reactive-streams/reactive-streams#2.7
   @Override @Test
-  public void unverified___spec207_blackbox_mustEnsureAllCallsOnItsSubscriptionTakePlaceFromTheSameThreadOrTakeCareOfSynchronization() throws Exception {
+  public void untested_spec207_blackbox_mustEnsureAllCallsOnItsSubscriptionTakePlaceFromTheSameThreadOrTakeCareOfSynchronization() throws Exception {
     notVerified(); // cannot be meaningfully tested, or can it?
     // the same thread part of the clause can be verified but that is not very useful, or is it?
   }
 
   // Verifies rule: https://github.com/reactive-streams/reactive-streams#2.8
   @Override @Test
-  public void unverified___spec208_blackbox_mustBePreparedToReceiveOnNextSignalsAfterHavingCalledSubscriptionCancel() throws Throwable {
+  public void untested_spec208_blackbox_mustBePreparedToReceiveOnNextSignalsAfterHavingCalledSubscriptionCancel() throws Throwable {
     notVerified(); // cannot be meaningfully tested as black box, or can it?
   }
 
   // Verifies rule: https://github.com/reactive-streams/reactive-streams#2.9
   @Override @Test
-  public void required___spec209_blackbox_mustBePreparedToReceiveAnOnCompleteSignalWithPrecedingRequestCall() throws Throwable {
+  public void required_spec209_blackbox_mustBePreparedToReceiveAnOnCompleteSignalWithPrecedingRequestCall() throws Throwable {
     blackboxSubscriberWithoutSetupTest(new BlackboxTestStageTestRun() {
       @Override
       public void run(BlackboxTestStage stage) throws Throwable {
@@ -238,7 +238,7 @@ public void run(BlackboxTestStage stage) throws Throwable {
 
   // Verifies rule: https://github.com/reactive-streams/reactive-streams#2.9
   @Override @Test
-  public void required___spec209_blackbox_mustBePreparedToReceiveAnOnCompleteSignalWithoutPrecedingRequestCall() throws Throwable {
+  public void required_spec209_blackbox_mustBePreparedToReceiveAnOnCompleteSignalWithoutPrecedingRequestCall() throws Throwable {
     blackboxSubscriberWithoutSetupTest(new BlackboxTestStageTestRun() {
       @Override
       public void run(BlackboxTestStage stage) throws Throwable {
@@ -262,7 +262,7 @@ public void subscribe(Subscriber<? super T> s) {
 
   // Verifies rule: https://github.com/reactive-streams/reactive-streams#2.10
   @Override @Test
-  public void required___spec210_blackbox_mustBePreparedToReceiveAnOnErrorSignalWithPrecedingRequestCall() throws Throwable {
+  public void required_spec210_blackbox_mustBePreparedToReceiveAnOnErrorSignalWithPrecedingRequestCall() throws Throwable {
     blackboxSubscriberTest(new BlackboxTestStageTestRun() {
       @Override
       @SuppressWarnings("ThrowableResultOfMethodCallIgnored")
@@ -275,63 +275,63 @@ public void run(BlackboxTestStage stage) throws Throwable {
 
   // Verifies rule: https://github.com/reactive-streams/reactive-streams#2.11
   @Override @Test
-  public void unverified___spec211_blackbox_mustMakeSureThatAllCallsOnItsMethodsHappenBeforeTheProcessingOfTheRespectiveEvents() throws Exception {
+  public void untested_spec211_blackbox_mustMakeSureThatAllCallsOnItsMethodsHappenBeforeTheProcessingOfTheRespectiveEvents() throws Exception {
     notVerified(); // cannot be meaningfully tested, or can it?
   }
 
   // Verifies rule: https://github.com/reactive-streams/reactive-streams#2.12
   @Override @Test
-  public void required___spec212_blackbox_mustNotCallOnSubscribeMoreThanOnceBasedOnObjectEquality() throws Throwable {
+  public void untested_spec212_blackbox_mustNotCallOnSubscribeMoreThanOnceBasedOnObjectEquality() throws Throwable {
     notVerified(); // cannot be meaningfully tested as black box, or can it?
   }
 
   // Verifies rule: https://github.com/reactive-streams/reactive-streams#2.13
   @Override @Test
-  public void unverified___spec213_blackbox_failingOnSignalInvocation() throws Exception {
+  public void untested_spec213_blackbox_failingOnSignalInvocation() throws Exception {
     notVerified(); // cannot be meaningfully tested, or can it?
   }
 
   ////////////////////// SUBSCRIPTION SPEC RULE VERIFICATION //////////////////
 
   // Verifies rule: https://github.com/reactive-streams/reactive-streams#3.1
   @Override @Test
-  public void unverified___spec301_blackbox_mustNotBeCalledOutsideSubscriberContext() throws Exception {
+  public void untested_spec301_blackbox_mustNotBeCalledOutsideSubscriberContext() throws Exception {
     notVerified(); // cannot be meaningfully tested, or can it?
   }
 
   // Verifies rule: https://github.com/reactive-streams/reactive-streams#3.8
   @Override @Test
-  public void required___spec308_blackbox_requestMustRegisterGivenNumberElementsToBeProduced() throws Throwable {
+  public void required_spec308_blackbox_requestMustRegisterGivenNumberElementsToBeProduced() throws Throwable {
     notVerified(); // cannot be meaningfully tested as black box, or can it?
   }
 
   // Verifies rule: https://github.com/reactive-streams/reactive-streams#3.10
   @Override @Test
-  public void unverified___spec310_blackbox_requestMaySynchronouslyCallOnNextOnSubscriber() throws Exception {
+  public void untested_spec310_blackbox_requestMaySynchronouslyCallOnNextOnSubscriber() throws Exception {
     notVerified(); // cannot be meaningfully tested, or can it?
   }
 
   // Verifies rule: https://github.com/reactive-streams/reactive-streams#3.11
   @Override @Test
-  public void unverified___spec311_blackbox_requestMaySynchronouslyCallOnCompleteOrOnError() throws Exception {
+  public void untested_spec311_blackbox_requestMaySynchronouslyCallOnCompleteOrOnError() throws Exception {
     notVerified(); // cannot be meaningfully tested, or can it?
   }
 
   // Verifies rule: https://github.com/reactive-streams/reactive-streams#3.14
   @Override @Test
-  public void unverified___spec314_blackbox_cancelMayCauseThePublisherToShutdownIfNoOtherSubscriptionExists() throws Exception {
+  public void untested_spec314_blackbox_cancelMayCauseThePublisherToShutdownIfNoOtherSubscriptionExists() throws Exception {
     notVerified(); // cannot be meaningfully tested, or can it?
   }
 
   // Verifies rule: https://github.com/reactive-streams/reactive-streams#3.15
   @Override @Test
-  public void unverified___spec315_blackbox_cancelMustNotThrowExceptionAndMustSignalOnError() throws Exception {
+  public void untested_spec315_blackbox_cancelMustNotThrowExceptionAndMustSignalOnError() throws Exception {
     notVerified(); // cannot be meaningfully tested, or can it?
   }
 
   // Verifies rule: https://github.com/reactive-streams/reactive-streams#3.16
   @Override @Test
-  public void unverified___spec316_blackbox_requestMustNotThrowExceptionAndMustOnErrorTheSubscriber() throws Exception {
+  public void untested_spec316_blackbox_requestMustNotThrowExceptionAndMustOnErrorTheSubscriber() throws Exception {
     notVerified(); // cannot be meaningfully tested, or can it?
   }