Eliminate unneeded LimitedDispatcher instances on Dispatchers.Default and Dispatchers.IO

[dovchinnikov]

Before the change LimitedDispatcher(parallelism) was returned. While it does work as expected, any submitted task
goes through its queue and parallelism number of workers. The change allows to eliminate the LimitedDispatcher
instance and its queue in case the requested parallelism is greater or equal to the effective parallelism of Dispatchers.Default and/or Dispatchers.IO

[qwwdfsad]

No conceptual objections from me. @dkhalanskyjb mind taking a look at the general approach?

[dovchinnikov]

Another approach was to expose open val effectiveParallelism: Int get() = Int.MAX_VALUE to CoroutineDispatcher, move the check if (parallelism >= effectiveParallelism) return this, then replace limitedParallelism implementations with override val effectiveParallelism. I've decided not to go that way since it requires exposing additional member (@InternalCoroutinesApi but still), and it can still be done later.

[dkhalanskyjb]

effectiveParallelism seems to me like a leaky abstraction because Dispatchers.IO has an effective parallelism of 64, but Dispatchers.IO.limitedParallelism(n) == UnlimitedIoDispatcher.limitedParallelism(n) has an effective parallelism of two million threads. So, I like what you propose now more.

[dovchinnikov]

I'd argue that Dispatchers.IO.limitedParallelism() providing views of unbounded scheduler is leaky :)
I still think the unbounded scheduler should be exposed somehow differently, e.g. via fun Dispatchers.giveMeNewIoDispatcher(parallelism: Int); then Dispatchers.IO.limitedParallelism() would provide a dispatcher view on top of 64 threads as expected, and Dispatchers.IO.effectiveParallelism would return 64 as expected. Consistency FTW

(There is still time since limitedParallelism is experimental)

[dkhalanskyjb]

Practicality outweighed consistency in this case. I'm sure nobody will be sad if they try to limit Dispatchers.IO to 100 threads and, in the worst case, observe those 100 threads and not 64. However, people will be sad if they observe starvation, which may well happen. The following is an excerpt from our design document:

A naive and problematic attempt at migrating some backend code to limitedParallelism from dedicated thread pools:

val dbConnections = Dispatchers.IO.limitedParallelism(100)
val loggers = Dispatchers.IO.limitedParallelism(2)
val cachers = Dispatchers.IO.limitedParallelism(5)

The number of max connections for PostgreSQL and MySQL is about 100, and the need to further increase this limit is common.
The number of threads in Dispatchers.IO is 64.

• The code will run just fine until Christmas Eve when it will become evident that the number of connections is half of what was possible before the migration. Given how rare the condition is, no tutorials will cover this.
• With fairness = 16, even assuming an equal length of all tasks, about 4 threads on average will be dedicated to fair execution; the rest may be busy serving the DB connections. This is fairly small, which may affect the system's responsiveness.

[qwwdfsad]

(Changed the base twice for GH to catch the most recent develop state).

Thanks!
Additional kudos for detailed commit message