Improve error handling for JavaFX initialization

Now, the JavaFX initialization may fail with an exception in case
something went wrong.

The driver for this change was that the initialization started
hanging in headless environments with transition to JDK 11.

Before, the initialization logic had a flaw. If a call to one API
failed, another API would be attempted. However, this approach is
problematic: if the first call failed with an exception for some
reason, it would leave JavaFX in a broken state where a flag would
imply that the system is being initialized. Subsequent calls would
then proceed to wait forever for the initialization to complete.

Now, exceptions are checked more carefully, ensuring that we only
fall back to the internal API in case the public one is unavailable
and not failed for some valid reason. This differentiation also
allows to more boldly rethrow exceptions upwards, being more or
less confident that these are relevant to the user.

Author: dkhalanskyjb

ui/kotlinx-coroutines-javafx/src/JavaFxDispatcher.kt

@@ -11,6 +11,7 @@ import javafx.util.*
 import kotlinx.coroutines.*
 import kotlinx.coroutines.internal.*
 import kotlinx.coroutines.javafx.JavaFx.delay
+import java.lang.UnsupportedOperationException
 import java.lang.reflect.*
 import java.util.concurrent.*
 import kotlin.coroutines.*
@@ -115,41 +116,42 @@ private class PulseTimer : AnimationTimer() {
     }
 }
 
+/** @return [true] if initialized successfully, and [false] if no display is detected */
 internal fun initPlatform(): Boolean = PlatformInitializer.success
 
 // Lazily try to initialize JavaFx platform just once
 private object PlatformInitializer {
     val success = run {
         /*
          * Try to instantiate JavaFx platform in a way which works
-         * both on Java 8 and Java 11 and does not produce "illegal reflective access":
-         *
-         * 1) Try to invoke javafx.application.Platform.startup if this class is
-         *    present in a classpath.
-         * 2) If it is not successful and does not because it is already started,
-         *    fallback to PlatformImpl.
-         *
-         * Ignore exception anyway in case of unexpected changes in API, in that case
-         * user will have to instantiate it manually.
+         * both on Java 8 and Java 11 and does not produce "illegal reflective access".
          */
-        val runnable = Runnable {}
-        runCatching {
-            // Invoke public API if it is present
-            Class.forName("javafx.application.Platform")
-                .getMethod("startup", java.lang.Runnable::class.java)
-                .invoke(null, runnable)
-        }.recoverCatching { exception ->
-            // Recover -> check re-initialization
-            val cause = exception.cause
-            if (exception is InvocationTargetException && cause is IllegalStateException
-                && "Toolkit already initialized" == cause.message) {
-                // Toolkit is already initialized -> success, return
-                Unit
-            } else { // Fallback to Java 8 API
-                Class.forName("com.sun.javafx.application.PlatformImpl")
+        try {
+            val runnable = Runnable {}
+            // Invoke the public API if it is present.
+            runCatching {
+                Class.forName("javafx.application.Platform")
+                        .getMethod("startup", java.lang.Runnable::class.java)
+            }.map { method ->
+                method.invoke(null, runnable)
+                return@run true
+            }
+            // If we are here, it means the public API is not present. Try the private API.
+            Class.forName("com.sun.javafx.application.PlatformImpl")
                     .getMethod("startup", java.lang.Runnable::class.java)
                     .invoke(null, runnable)
+            true
+        } catch (exception: InvocationTargetException) {
+            // Can only happen as a result of [Method.invoke].
+            val cause = exception.cause!!
+            when {
+                // Maybe the problem is that JavaFX is already initialized? Everything is good then.
+                cause is IllegalStateException && "Toolkit already initialized" == cause.message -> true
+                // If the problem is the headless environment, it is okay.
+                cause is UnsupportedOperationException && "Unable to open DISPLAY" == cause.message -> false
+                // Otherwise, the exception demonstrates an anomaly.
+                else -> throw cause
             }
-        }.isSuccess
+        }
     }
 }