Expose net::driver::Watcher mechanism

[dvc94ch]

Crates implementing Evented should be able to register a reactor.

Follow up from discussion #65

[yoshuawuyts]

This API still requires some work, but just to set expectations: it's unlikely we'll expose mio::Evented in the API bounds. Instead we're more likely to use AsRawFd for Unix and AsRawSocket for Windows for compat with std.

On top of that we should also provide a way to access the epoll / kqueue / iocp instance directly so people can perform syscalls on it outside of the read_with / write_with APIs.

[dvc94ch]

I think this is really important. Just got blocked trying to set the ecn bits on a udp datagram. Most async system programs need this in some form.

@dvc94ch What if we had an API along these lines? Would that work for you?

enum Interest {
    Read,
    Write,
    Both,
}

// Registers an I/O handle so that the current task gets woken up when it becomes ready.
fn register(cx: &mut Context<'_>, fd: impl AsRawFd, interest: Interest);

// Unregisters an I/O handle.
fn unregister(fd: impl AsRawFd);

[dvc94ch]

So how would this be used exactly? Are these operations idempotent? But it seems reasonable.

fn poll_send(&self, cx: &mut Context, data: &[u8]) -> Poll<Result<()>> {
    match self.socket.try_send(data) {
        Ok(res) => {
            unregister(cx);
            Poll::Ready(Ok(res))
        }
        Err(ref e) if e.kind() == ErrorKind::WouldBlock => {
            register(cx, self.socket.as_raw_fd(), Interest::Write);
            Poll::Pending
        }
        Err(e) => {
            unregister(cx);
            Poll::Ready(Err(e))
        }
    }
}

@dvc94ch Almost -- this is closer to what I had in mind:

impl MyIoHandle {
    fn poll_send(&self, cx: &mut Context, data: &[u8]) -> Poll<Result<()>> {
        let mut registered = false;
        loop {
            match self.socket.try_send(data) {
                Ok(res) => return Poll::Ready(Ok(res)),
                Err(ref e) if e.kind() == ErrorKind::WouldBlock => {
                    if registered {
                        return Poll::Pending;
                    } else {
                        register(cx, &self.socket, Interest::Write);
                        registered = true;
                    }
                }
                Err(e) => return Poll::Ready(Err(e)),
            }
        }
    }
}

impl Drop for MyIoHandle {
    fn drop(&mut self) {
        unregister(&self.socket);
    }
}

[Demi-Marie]

This blocks quinn-rs/quinn#502, which needs ECN support.

[kornelski]

I need to call send_sas/recv_sas on a UdpSocket. AsRawFd allows doing it only in a blocking fashion, because I have no way of handling WouldBlock properly.

What I'm missing is a public API equivalent of self.watcher.poll_read_with() / poll_write_with().