Stacktrace recovery improvements and documentation

README.md

@@ -7,9 +7,6 @@
 Library support for Kotlin coroutines with [multiplatform](#multiplatform) support.
 This is a companion version for Kotlin `1.3.20` release.
 
-**NOTE**: `0.30.2` was the last release with Kotlin 1.2 and experimental coroutines.
-See [COMPATIBILITY.md](COMPATIBILITY.md) for details of migration onto the stable Kotlin 1.3 coroutines.
-
 ```kotlin
 GlobalScope.launch {
     delay(1000)
@@ -57,6 +54,8 @@ GlobalScope.launch {
   * [Guide to kotlinx.coroutines by example](docs/coroutines-guide.md) (**read it first**)
   * [Guide to UI programming with coroutines](ui/coroutines-guide-ui.md)
   * [Guide to reactive streams with coroutines](reactive/coroutines-guide-reactive.md)
+  * [Debugging capabilities in kotlinx.coroutines](docs/debugging.md)
+* [Compatibility policy and experimental annotations](docs/compatibility.md)
 * [Change log for kotlinx.coroutines](CHANGES.md)
 * [Coroutines design document (KEEP)](https://github.com/Kotlin/KEEP/blob/master/proposals/coroutines.md)
 * [Full kotlinx.coroutines API reference](http://kotlin.github.io/kotlinx.coroutines)

binary-compatibility-validator/reference-public-api/kotlinx-coroutines-core.txt

@@ -121,6 +121,10 @@ public final class kotlinx/coroutines/CompletionHandlerException : java/lang/Run
 	public fun <init> (Ljava/lang/String;Ljava/lang/Throwable;)V
 }
 
+public abstract interface class kotlinx/coroutines/CopyableThrowable {
+	public abstract fun createCopy ()Ljava/lang/Throwable;
+}
+
 public final class kotlinx/coroutines/CoroutineContextKt {
 	public static final fun newCoroutineContext (Lkotlinx/coroutines/CoroutineScope;Lkotlin/coroutines/CoroutineContext;)Lkotlin/coroutines/CoroutineContext;
 }

docs/compatibility.md

@@ -0,0 +1,115 @@
+<!--- 
+/*
+ * Copyright 2016-2019 JetBrains s.r.o. Use of this source code is governed by the Apache 2.0 license.
+ */
+---> 
+
+<!--- TOC -->
+
+* [Compatibility](#compatibility)
+* [Public API types](#public-api-types)
+  * [Experimental API](#experimental-api)
+  * [Obsolete API](#obsolete-api)
+  * [Internal API](#internal-api)
+  * [Stable API](#stable-api)
+  * [Deprecation cycle](#deprecation-cycle)
+* [Using annotated API](#using-annotated-api)
+  * [Programmatically](#programmatically)
+  * [Gradle](#gradle)
+  * [Maven](#maven)
+
+<!--- END_TOC -->
+
+## Compatibility
+This document describes the compatibility policy of `kotlinx.coroutines` library since version 1.0.0 and semantics of compatibility-specific annotations.
+
+
+## Public API types
+`kotlinx.coroutines` public API comes in five flavours: stable, experimental, obsolete, internal and deprecated. 
+All public API except stable is marked with the corresponding annotation.
+
+### Experimental API
+Experimental API is marked with [@ExperimentalCoroutinesApi][ExperimentalCoroutinesApi] annotation.
+API is marked experimental when its design has potential open questions which may eventually lead to 
+either semantics changes of the API or its deprecation.
+
+By default, most of the new API is marked as experimental and becomes stable in one of the next major releases if no new issues arise.
+Otherwise, either semantics is fixed without changes in ABI or API goes through deprecation cycle. 
+
+When using experimental API may be dangerous:
+* You are writing a library which depends on `kotlinx.coroutines` and want to use experimental coroutines API in a stable library API.
+It may lead to undesired consequences when end users of your library update their `kotlinx.coroutines` version where experimental API
+has slightly different semantics.
+* You want to build core infrastructure of the application around experimental API. 
+
+### Obsolete API
+Obsolete API is marked with [@ObsoleteCoroutinesApi][ObsoleteCoroutinesApi] annotation.
+Obsolete API is similar to experimental, but already known to have serious design flaws and its potential replacement, 
+but replacement is not yet implemented.
+
+The semantics of this API won't be changed, but it will go through a deprecation cycle as soon as the replacement is ready.
+
+### Internal API
+Internal API is marked with [@InternalCoroutinesApi][InternalCoroutinesApi] or is part of `kotlinx.coroutines.internal` package.
+This API has no guarantees on its stability, can and will be changed and/or removed in the future releases. 
+If you can't avoid using internal API, please report it to [issue tracker](https://github.com/Kotlin/kotlinx.coroutines/issues/new).
+
+### Stable API
+Stable API is guaranteed to preserve its ABI and documented semantics. If at some point unfixable design flaws will be discovered, 
+this API will go through a deprecation cycle and remain binary compatible as long as possible.
+
+### Deprecation cycle
+When some API is deprecated, it goes through multiple stages and there is at least one major release between stages.
+* Feature is deprecated with compilation warning. Most of the time, proper replacement 
+(and corresponding `replaceWith` declaration) is provided to automatically migrate deprecated usages with a help of IntelliJ IDEA.
+* Deprecation level is increased to `error` or `hidden`. It is no longer possible to compile new code against deprecated API, 
+  though it is still present in the ABI.
+* API is completely removed. While we give our best efforts not to do so and have no plans of removing any API, we still are leaving 
+this option in case of unforeseen problems such as security holes.
+
+## Using annotated API
+All API annotations are [kotlin.Experimental](https://kotlinlang.org/api/latest/jvm/stdlib/kotlin/-experimental/index.html).
+It is done in order to produce compilation warning about using experimental or obsolete API.
+Warnings can be disabled either programmatically for a specific call site or globally for the whole module.
+
+### Programmatically
+For a specific call-site, warning can be disabled by using [UseExperimental](https://kotlinlang.org/api/latest/jvm/stdlib/kotlin/-use-experimental/index.html) annotation:
+```kotlin
+@UseExperimental(ExperimentalCoroutinesApi::class) // Disables warning about experimental coroutines API 
+fun experimentalApiUsage() {
+    someKotlinxCoroutinesExperimentalMethod()
+}
+``` 
+
+### Gradle
+For the Gradle project, a warning can be disabled by passing a compiler flag in your `build.gradle` file:
+
+```groovy
+tasks.withType(org.jetbrains.kotlin.gradle.tasks.AbstractKotlinCompile).all {
+    kotlinOptions.freeCompilerArgs += ["-Xuse-experimental=kotlinx.coroutines.ExperimentalCoroutinesApi"]
+}
+
+```
+
+### Maven
+For the Maven project, a warning can be disabled by passing a compiler flag in your `pom.xml` file:
+```xml
+<plugin>
+    <artifactId>kotlin-maven-plugin</artifactId>
+    <groupId>org.jetbrains.kotlin</groupId>
+    ... your configuration ...
+    <configuration>
+        <args>
+            <arg>-Xuse-experimental=kotlinx.coroutines.ExperimentalCoroutinesApi</arg>
+        </args>
+    </configuration>
+</plugin>
+```
+
+
+<!--- MODULE kotlinx-coroutines-core -->
+<!--- INDEX kotlinx.coroutines -->
+[ExperimentalCoroutinesApi]: https://kotlin.github.io/kotlinx.coroutines/kotlinx-coroutines-core/kotlinx.coroutines/-experimental-coroutines-api/index.html
+[ObsoleteCoroutinesApi]: https://kotlin.github.io/kotlinx.coroutines/kotlinx-coroutines-core/kotlinx.coroutines/-obsolete-coroutines-api/index.html
+[InternalCoroutinesApi]: https://kotlin.github.io/kotlinx.coroutines/kotlinx-coroutines-core/kotlinx.coroutines/-internal-coroutines-api/index.html
+<!--- END -->

docs/debugging.md

@@ -0,0 +1,94 @@
+**Table of contents**
+
+<!--- TOC -->
+
+* [Debugging coroutines](#debugging-coroutines)
+* [Debug mode](#debug-mode)
+* [Stacktrace recovery](#stacktrace-recovery)
+  * [Stacktrace recovery machinery](#stacktrace-recovery-machinery)
+* [Debug agent](#debug-agent)
+  * [Debug agent and Android](#debug-agent-and-android)
+
+<!--- END_TOC -->
+
+
+## Debugging coroutines
+Asynchronous programming is hard and debugging asynchronous programs is even harder.
+To improve user experience, `kotlinx.coroutines` comes with additional features for debugging: debug mode, stacktrace recovery 
+and debug agent.
+
+## Debug mode
+
+The first debugging feature of `kotlinx.coroutines` is debug mode.
+It can be enabled either by setting system property [DEBUG_PROPERTY_NAME] or by running Java with enabled assertions (`-ea` flag).
+The latter is helpful to have debug mode enabled by default in unit tests.
+
+Debug mode attaches a unique [name][CoroutineName] to every launched coroutine, which then can be seen in a regular Java debugger, 
+a string representation of coroutine and thread name executing named coroutine. 
+Overhead of this feature is negligible and it can be safely turned on by default to simplify logging and diagnostic.
+
+## Stacktrace recovery
+
+Stacktrace recovery is another useful feature of debug mode. It is enabled by default in the debug mode, 
+but can be separately disabled by setting `kotlinx.coroutines.stacktrace.recovery` system property to `false`.
+
+Stacktrace recovery tries to knit asynchronous exception stacktrace with a stacktrace of the receiver by copying it, providing
+not only information where an exception was thrown, but also where it was asynchronously rethrown or caught.
+
+It is easy to demonstrate with actual stacktraces of the same program that awaits asynchronous operation in `main` function:
+
+| Without recovery | With recovery |
+| - | - |
+| ![before](images/before.png "before") | ![after](images/after.png "after") |
+
+The only downside of this approach is losing referential transparency of the exception. 
+
+### Stacktrace recovery machinery
+
+This section explains the inner mechanism of stacktrace recovery and can be skipped.
+
+When an exception is rethrown between coroutines (e.g. through `withContext` or `Deferred.await` boundary), stacktrace recovery
+machinery tries to create a copy of the original exception (with the original exception as the cause), then rewrite stacktrace
+of the copy with coroutine-related stack frames (using [Throwable.setStackTrace](https://docs.oracle.com/javase/9/docs/api/java/lang/Throwable.html#setStackTrace-java.lang.StackTraceElement:A-)) 
+and then throws resulting exception instead of the original one.
+
+Exception copy logic is straightforward:
+  1) If exception class implements [CopyableThrowable], [CopyableThrowable.createCopy] is used.
+  2) If exception class has class-specific fields not inherited from Throwable, the exception is not copied.
+  3) Otherwise, one of the public exception's constructor is invoked reflectively with optional an `initCause` call.  
+
+## Debug agent
+
+[kotlinx-coroutines-debug](../kotlinx-coroutines-debug) module provides one of the most powerful debug capabilities in `kotlinx.coroutines`.
+
+This is a separate module with a JVM agent that keeps track of all alive coroutines, introspect and dump them similar to thread dump command,
+additionally enhancing stacktraces with information where coroutine was created.
+
+The full tutorial of how to use debug agent can be found in a corresponding [readme](../kotlinx-coroutines-debug/README.md).
+
+### Debug agent and Android
+
+Unfortunately, Android runtime does not support Instrument API necessary for `kotlinx-coroutines-debug` to function, triggering `java.lang.NoClassDefFoundError: Failed resolution of: Ljava/lang/management/ManagementFactory;`.
+
+Nevertheless, it will be possible to support debug agent on Android as soon as [GradleAspectJ-Android](https://github.com/Archinamon/android-gradle-aspectj)  will support androin-gradle 3.3 
+
+<!---
+Make an exception googlable
+java.lang.NoClassDefFoundError: Failed resolution of: Ljava/lang/management/ManagementFactory;
+        at kotlinx.coroutines.repackaged.net.bytebuddy.agent.ByteBuddyAgent$ProcessProvider$ForCurrentVm$ForLegacyVm.resolve(ByteBuddyAgent.java:1055)
+        at kotlinx.coroutines.repackaged.net.bytebuddy.agent.ByteBuddyAgent$ProcessProvider$ForCurrentVm.resolve(ByteBuddyAgent.java:1038)
+        at kotlinx.coroutines.repackaged.net.bytebuddy.agent.ByteBuddyAgent.install(ByteBuddyAgent.java:374)
+        at kotlinx.coroutines.repackaged.net.bytebuddy.agent.ByteBuddyAgent.install(ByteBuddyAgent.java:342)
+        at kotlinx.coroutines.repackaged.net.bytebuddy.agent.ByteBuddyAgent.install(ByteBuddyAgent.java:328)
+        at kotlinx.coroutines.debug.internal.DebugProbesImpl.install(DebugProbesImpl.kt:39)
+        at kotlinx.coroutines.debug.DebugProbes.install(DebugProbes.kt:49)
+-->
+
+<!--- MODULE kotlinx-coroutines-core -->
+<!--- INDEX kotlinx.coroutines -->
+[DEBUG_PROPERTY_NAME]: https://kotlin.github.io/kotlinx.coroutines/kotlinx-coroutines-core/kotlinx.coroutines/-d-e-b-u-g_-p-r-o-p-e-r-t-y_-n-a-m-e.html
+[CoroutineName]: https://kotlin.github.io/kotlinx.coroutines/kotlinx-coroutines-core/kotlinx.coroutines/-coroutine-name/index.html
+[CopyableThrowable]: https://kotlin.github.io/kotlinx.coroutines/kotlinx-coroutines-core/kotlinx.coroutines/-copyable-throwable/index.html
+[CopyableThrowable.createCopy]: https://kotlin.github.io/kotlinx.coroutines/kotlinx-coroutines-core/kotlinx.coroutines/-copyable-throwable/create-copy.html
+<!--- MODULE kotlinx-coroutines-debug -->
+<!--- END -->

kotlinx-coroutines-core/common/src/Annotations.kt

@@ -30,8 +30,6 @@ public annotation class ObsoleteCoroutinesApi
  * Marks declarations that are **internal** in coroutines API, which means that should not be used outside of
  * `kotlinx.coroutines`, because their signatures and semantics will be changing between release without any
  * warnings and without providing any migration aids.
- *
- * @suppress **This an internal API and should not be used from general code.**
  */
 @Retention(value = AnnotationRetention.BINARY)
 @Experimental(level = Experimental.Level.ERROR)

kotlinx-coroutines-core/jvm/src/Debug.kt

@@ -4,8 +4,8 @@
 
 package kotlinx.coroutines
 
-import kotlin.coroutines.Continuation
 import kotlinx.coroutines.internal.*
+import kotlin.coroutines.*
 
 /**
  * Name of the property that controls coroutine debugging. See [newCoroutineContext][CoroutineScope.newCoroutineContext].
@@ -26,6 +26,32 @@ public const val DEBUG_PROPERTY_NAME = "kotlinx.coroutines.debug"
  */
 internal const val STACKTRACE_RECOVERY_PROPERTY_NAME = "kotlinx.coroutines.stacktrace.recovery"
 
+/**
+ * Throwable which can be cloned during stacktrace recovery in a class-specific way.
+ * For additional information about stacktrace recovery see [STACKTRACE_RECOVERY_PROPERTY_NAME]
+ *
+ * Example of usage:
+ * ```
+ * class BadResponseCodeException(val responseCode: Int) : Exception(), CopyableThrowable<BadResponseCodeException> {
+ *
+ *  override fun createCopy(): BadResponseCodeException {
+ *    val result = BadResponseCodeException(responseCode)
+ *    result.initCause(this)
+ *    return result
+ *  }
+ * ```
+ */
+@ExperimentalCoroutinesApi
+public interface CopyableThrowable<T> where T : Throwable, T : CopyableThrowable<T> {
+
+    /**
+     * Creates a copy of the current instance.
+     * For better debuggability, it is recommended to use original exception as [cause][Throwable.cause] of the resulting one.
+     * Stacktrace of copied exception will be overwritten by stacktrace recovery machinery by [Throwable.setStackTrace] call.
+     */
+    public fun createCopy(): T
+}
+
 /**
  * Automatic debug configuration value for [DEBUG_PROPERTY_NAME]. See [newCoroutineContext][CoroutineScope.newCoroutineContext].
  */

kotlinx-coroutines-core/jvm/src/internal/ExceptionsConstuctor.kt

@@ -4,21 +4,35 @@
 
 package kotlinx.coroutines.internal
 
+import kotlinx.coroutines.*
+import java.lang.reflect.*
 import java.util.*
 import java.util.concurrent.locks.*
 import kotlin.concurrent.*
 
+private val throwableFields = Throwable::class.java.fieldsCountOrDefault(-1)
 private val cacheLock = ReentrantReadWriteLock()
 // Replace it with ClassValue when Java 6 support is over
 private val exceptionConstructors: WeakHashMap<Class<out Throwable>, (Throwable) -> Throwable?> = WeakHashMap()
 
 @Suppress("UNCHECKED_CAST")
 internal fun <E : Throwable> tryCopyException(exception: E): E? {
+    if (exception is CopyableThrowable<*>) {
+        return runCatching { exception.createCopy() as E }.getOrNull()
+    }
+
     val cachedCtor = cacheLock.read {
         exceptionConstructors[exception.javaClass]
     }
 
     if (cachedCtor != null) return cachedCtor(exception) as E?
+    /*
+     * Skip reflective copy if an exception has additional fields (that are usually populated in user-defined constructors)
+     */
+    if (throwableFields != exception.javaClass.fieldsCountOrDefault(0)) {
+        cacheLock.write { exceptionConstructors[exception.javaClass] = { null } }
+        return null
+    }
 
     /*
      * Try to reflectively find constructor(), constructor(message, cause) or constructor(cause).
@@ -43,3 +57,12 @@ internal fun <E : Throwable> tryCopyException(exception: E): E? {
     cacheLock.write { exceptionConstructors[exception.javaClass] = (ctor ?: { null }) }
     return ctor?.invoke(exception) as E?
 }
+
+private fun Class<*>.fieldsCountOrDefault(defaultValue: Int) = kotlin.runCatching { fieldsCount() }.getOrDefault(defaultValue)
+
+private tailrec fun Class<*>.fieldsCount(accumulator: Int = 0): Int {
+    val fieldsCount = declaredFields.count { !Modifier.isStatic(it.modifiers) }
+    val totalFields = accumulator + fieldsCount
+    val superClass = superclass ?: return totalFields
+    return superClass.fieldsCount(totalFields)
+}