Prevent bytes from lingering in the serial buffer

This fixes a problem with the Serial UTF-8 decoder. This decoding moves
data from char[] buf, into a ByteBuffer inFromSerial, then decodes them
into a CharBuffer outToMessage and converts to a char[] to pass on.

When the buf read contained just over a full buffer worth of bytes and
contained some multi-byte characters, a situation could arise where two
decodes were needed to fill up outToMessage, leaving some data in
inFromSerial. If in this case no data would be left in buf, decoding
would stop until more data came in from serial.

This commit fixes this problem by:
 - Changing the outer loop to continue running when buf is empty, but
   inFromSerial is not.
 - Changing the inner loop to run at least once (so it runs when buf is
   empty, but inFromSerial is no).
 - Breaking out of the outer loop when no characters were produced (this
   handles the case where only an incomplete UTF-8 character remains in
   inFromSerial, which would otherwise prevent the loop from
   terminating.
 - Removes a `if (outToMessage.hasRemaining()` check that is now
   necessarily true if the break was not done.

This fixes arduino#9808.

Author: matthijskooijman

Diff for: arduino-core/src/processing/app/Serial.java

@@ -189,21 +189,41 @@ public synchronized void serialEvent(SerialPortEvent serialEvent) {
 
   public void processSerialEvent(byte[] buf) {
     int next = 0;
-    while(next < buf.length) {
-      while(next < buf.length && outToMessage.hasRemaining()) {
+    // This uses a CharsetDecoder to convert from bytes to UTF-8 in
+    // a streaming fashion (i.e. where characters might be split
+    // over multiple reads). This needs the data to be in a
+    // ByteBuffer (inFromSerial, which we also use to store leftover
+    // incomplete characters for the nexst run) and produces a
+    // CharBuffer (outToMessage), which we then convert to char[] to
+    // pass onwards.
+    // Note that these buffers switch from input to output mode
+    // using flip/compact/clear
+    while (next < buf.length || inFromSerial.position() > 0) {
+      do {
+        // This might be 0 when all data was already read from buf
+        // (but then there will be data in inFromSerial left to
+        // decode).
         int copyNow = Math.min(buf.length - next, inFromSerial.remaining());
         inFromSerial.put(buf, next, copyNow);
         next += copyNow;
+
         inFromSerial.flip();
         bytesToStrings.decode(inFromSerial, outToMessage, false);
         inFromSerial.compact();
-      }
+
+        // When there are multi-byte characters, outToMessage might
+        // still have room, so add more bytes if we have any.
+      } while (next < buf.length && outToMessage.hasRemaining());
+
+      // If no output was produced, the input only contained
+      // incomplete characters, so we're done processing
+      if (outToMessage.position() == 0)
+        break;
+
       outToMessage.flip();
-      if(outToMessage.hasRemaining()) {
-        char[] chars = new char[outToMessage.remaining()];
-        outToMessage.get(chars);
-        message(chars, chars.length);
-      }
+      char[] chars = new char[outToMessage.remaining()];
+      outToMessage.get(chars);
+      message(chars, chars.length);
       outToMessage.clear();
     }
   }