Clarification on which TCK tests are mandatory

[purplefox]

Hello again,

I wonder if someone could clarify what tests must be run in order to say "X implementation has passed the TCK?"

There seems some ambiguity in the README:

https://github.com/reactive-streams/reactive-streams-jvm/blob/master/tck/README.md

"The TCK is split up into 4 files JUnit 4 test classes which should be extended by implementers"

Taking the RFC meaning of "should" I assume this means the expectation is they are implemented unless there is a very good reason not to.

But later on in the README it states with respect to subscriber whitebox verification:

"We do not expect all implementations to make use of the plain SubscriberWhiteboxVerification, using the SubscriberBlackboxVerification instead."

Which appears to contradict the earlier statement that all tests should be implemented.

Can someone clarify exact which of the tests are mandatory and which are optional to implement in order to claim that an implementation passes the TCK?

Cheers!

[ktoso]

Hi @purplefox!
Awesome that you're starting to have a stab at implementing RS / using the TCK :-)

You're right to assume SHOULD in RFC style wording here.
The reason we have whitebox and blackbox tests is that sometimes you may not be able to implement a SubscriberWhiteboxVerification - notice how many "hooks" it requires you to provide within the subject-under-test.

SubscriberWhiteboxVerification tests by far more rules so it is able to catch more problems an implementation may have, it comes at the cost of having to instrument your Subscriber. SubscriberBlackboxVerification tests very little things, because it's not able to verify many rules without the "puppet" which Whitebox has, on the other hand it is simple to test with it - just drop in any Subscriber and you're done.

My general recommendation is:

• try to bite the bullet and implement whitebox,
• if unable to go with blackbox (maybe you're still able to use whitebox only for some "in development version?")

Also, the specs themselves contain optional and required tests. You know which one is which by looking at their method name prefix: https://github.com/reactive-streams/reactive-streams-jvm/blob/master/tck/src/main/java/org/reactivestreams/tck/support/SubscriberBlackboxVerificationRules.java, so for example void optional_spec111_... (that's a publisher test) means that the spec rule contains SHOULD for example, and if you don't pass this test it will become skipped, instead of failing).

Hope this helps!

@reactive-streams-jvm Shall we improve the wording of the readme to explain what we "require to implement in order to be TCK approved"? // I want @ ping-ing powers! :<

[purplefox]

Hi Konrad,

Thanks for getting back to me.

From your reply I can see that you recommend that whitebox tests are implemented, but it's not really answering the question of what is required to be implemented in order for someone to claim they pass the TCK.

That's the question I would like to clarify (and I agree this should be stated unambiguously in the README). :)

Cheers

[ktoso]

I don't think we can requite the whitebox specs to be implemented. They're cool if you can, but not a MUST I think.

Required ones would be (IMHO): PublisherVerification, SubscriberBlackboxVerification and I would include IdentityProcessorVerification in the required ones with a "if you have a Processor you MUST, otherwise not" clause.

Hope this helps!
Would be interesting to hear opinions from other implementers (the above is based on what we do in akka-streams).

[smaldini]

+1 @ktoso we mostly did the same in Reactor.

[purplefox]

According to the README IdentityProcessorVerification uses whitebox tests: "An IdentityProcessorVerification tests the given Processor for all Subscriber, Publisher as well as Subscription rules. Internally the WhiteboxSubscriberVerification...".

This implies it can't be mandatory as the whitebox tests aren't required...

[ktoso]

IdentityProcessorVerification is a funky beast:

Since it has a "publisher side" and a "subscriber side", we use the WhiteboxSubscriberVerification on the end where we can attach an TCK provided Subscriber which implements the hooks. So the burden of implementing the whitebox instrumentation is taken away from the implementer.

See here for details: https://github.com/reactive-streams/reactive-streams-jvm/blob/master/tck/src/main/java/org/reactivestreams/tck/IdentityProcessorVerification.java#L418

So not being able to implement SubscriberWhiteboxVerification directly does not mean you can not implement IdentityProcessorVerification :)

[ktoso]

// Btw, thanks - good questions, seems the readme needs more love in explaining these things! I'll do this soon.

[viktorklang]

@ktoso any status here?

[ktoso]

Pushed a PR now: #245