5 min with serial monitor results in OutOfMemoryException

[willie68]

I have to debug some longer running tasks. But after some minutes i get an OutOfMEmoryException than the IDE is not resonding. I think it's the buffer for the serial monitor text area.
As a workaround: Is there a possibility to clear the text area? Maybe an autoclear on low memory would help.
Setting the memory params of the IDE doesn't help, because this would only postpone the problem. And without the autosave feature this bug is more serious. I have again lost some code because of this.
Arduino IDE 1.5.7, SerialMOnitor with 57600 Baud.

[ffissore]

Wow! How much RAM does your computer have? If it's the serial monitor, you should have printed an amount of text almost equal to the amount of free ram
+1 with limiting the number of rows displayed in the serial monitor

[willie68]

8GB. But i have try only 1GB for the JavaVM. Because than you get the next issue. If your vm is blown up to 1GB the garbage collector needs huge time to work. So the IDE freezes sometimes.
As i see in the source the serial monitor is an JTextArea. And so it uses the Document model from swing which uses an double buffer algo. for the presentation. So every char in the textarea will use more than 1 bytes (i think, i have readed, that it will use up to 6 Bytes (2 Buffers and the displaybuffer with 2 Bytes))

[ffissore]

It seems you know java swing quite well: mind to provide some code that limits the number of rows to a given value? :)

[willie68]

You have two mails from me with attached source...

[ffissore]

Read. I'll try and report back asap

[PaulStoffregen]

This problem happens with Teensy, if a sketch uses Serial.print() in loop() without any delay. At best speed, Teensy can send about 1 Mbyte/sec. In just a few minutes, the JVM reports an out of memory exception and pretty much nothing works until Arduino is started. I use 64 bit Linux and "top" reports the process has allocated 10 GB of virtual memory.

If this were only the text stored by the window, it should take at least 2 hours to send that much data over 12 Mbit/sec USB. The out of memory condition happens in only a matter of minutes.

[PaulStoffregen]

If anyone tries testing this, I'd recommend using Arduino Due with the native port, running this sketch:

void setup() {
SerialUSB.begin(9600);
while (!SerialUSB) ; // wait
delay(250);
}
unsigned long count=1;
void loop() {
SerialUSB.print("Printing simple text, count # ");
SerialUSB.println(count++);
}

[PaulStoffregen]

Here's a quick screenshot showing Arduino 1.5.7 has allocated 5.9 gigabytes of RAM after 7805055 lines have printed. Each line is 39 bytes, which should be approx 0.29 Gbyte, not 5.9 Gbyte.

[ffissore]

I've tried implementing a limit to the amount of output shown in the serial monitor. Unfortunately, after minute or so spent looking at the output of an endless Asciitable example, it starts slowing down until it blocks. I still don't know the cause and will investigate further.
Anyone wishing to give a hand is welcome: branch is this, commit is 3ae7aa7

[willie68]

Hi,

i have tried this with a LimitLinesDocumentFilter and it worked perfectly.
Maybe the lines will be stored anywhere else?

    plainDocument.setDocumentFilter(new LimitLinesDocumentFilter(maxLineCount));

import javax.swing.text.AttributeSet;
import javax.swing.text.BadLocationException;
import javax.swing.text.Document;
import javax.swing.text.DocumentFilter;
import javax.swing.text.DocumentFilter.FilterBypass;
import javax.swing.text.Element;

/**
 * 
 */

/**
 * @author w.klaas
 *
 */
public class LimitLinesDocumentFilter extends DocumentFilter {
    private int maxLineCount;

    public LimitLinesDocumentFilter(int maxLineCount) {
        this.maxLineCount = maxLineCount;
    }

    @Override
    public void insertString(FilterBypass fb, int offset, String string,
            AttributeSet attr) throws BadLocationException {
        super.insertString(fb, offset, string, attr);

        removeFromStart(fb);
    }

    @Override
    public void replace(FilterBypass fb, int offset, int length, String text,
            AttributeSet attrs) throws BadLocationException {
        super.replace(fb, offset, length, text, attrs);

        removeFromStart(fb);
    }

    private void removeFromStart(FilterBypass fb) {
        Document doc = fb.getDocument();
        Element root = doc.getDefaultRootElement();
        while (root.getElementCount() > maxLineCount) {
            removeLineFromStart(doc, root);
        }
    }

    private void removeLineFromStart(Document document, Element root) {
        Element line = root.getElement(0);
        int end = line.getEndOffset();

        try {
            document.remove(0, end);
        } catch (BadLocationException ble) {
            ble.printStackTrace();
        }
    }
}

[ffissore]

Which number did you set for maxLineCount ?

[willie68]

1000 lines.

[WingTangWong]

Question: what is the memory allocated to the jvm at start?

W

On Thursday, October 2, 2014, Wilfried Klaas [email protected]
wrote:

1000 lines.

—
Reply to this email directly or view it on GitHub
#2233 (comment).

Wing Wong
[email protected]
http://about.me/wingtangwong
https://www.facebook.com/wingedpower

[ffissore]

Tried and nope: starts slowing down after about a minute until it blocks. See this video https://drive.google.com/file/d/0B25ol7x7AsJRb0xleUJCcHhWLXc/edit?usp=sharing

[ffissore]

For the record, I'm using this sketch to test the behaviour https://gist.github.com/ffissore/6a0b19a7efa613efcf6d

[willie68]

Hi,
sorry for my late response, i was on vacation. I have tested only the java textarea itselfs.
Here is my test eclipse project for this: http://wkla.no-ip.biz/down/TestTextArea.zip
So, i think not only the Textarea will cache the lines, bt somewhere in the code there will be an issue, too. I'll take a look on it on my next lunch break.
@WingTangWong
As i can see on my task manager the vm will be created with -Xms128m and Xmx128m, so as default the IDE will consume 128MB at max. There are 2 entries in the AppData/Roaming/Arduino15/preferences.txt, but they seems not to work.

[xxxajk]

As far as I know, this is also a regression in 1.0.6. 1.0.5 worked fine before.

[PaulStoffregen]

It's not a regression. 1.0.5 and all prior versions of Arduino have suffered this problem.

[PaulStoffregen]

Ok, I think I might finally have a solution. Maybe? This is an extremely tough problem!

First of all, the number of GUI updates has to be rate limited. I added a Swing timer to schedule serial monitor redraws at a maximum of 30 Hz.

Incoming data is piled up into a 1 megabyte StringBuffer, until the timer triggers an update. This should allow receiving data up to 30 MByte/sec, which is faster than any Arduino can transmit. But if dramatically faster hardware is made with 480 MBit/sec (USB 2.0) or 5 Gbit/sec (USB 3.0) or 10 Gbit/sec (USB 3.1) in the future, this buffer might need to grow.

Limiting the stored data by lines does not seem to be practical. I tried, but every attempt resulted in Java consuming more CPU time to parse lines than Arduino Due and Teensy 3.1 need to generate and send them. I found a few ways to make it run for a couple minutes, but ultimately the imbalance in CPU usage catches up and Java ends up at 100% or 200% CPU usage (both main and swing thread) and for all practical purposes the IDE locks up because the swing thread can't clear its backlog and respond to mouse clicks.

Instead, I switched to limiting the serial monitor memory by characters. Even that consumes a lot of CPU time. Java is pretty horrible. To work around that issue, I let the buffer grow beyond the limit and trim it only once every 5 seconds.

The final challenge is what to do when not scrolling. I don't have a good solution there. I put in some code to disable trimming the serial monitor data (otherwise the window changes while you're trying to read), so when not scrolling this infinite memory problem returns. More work is probably needed to start discarding incoming data when autoscroll is turned off.

Even with all these attempts to limit memory and CPU, Java is still running about 15% to 20% CPU usage on a very fast desktop (overclocked i7-3930K). Memory usage also seems to be very slowly creeping up, but it also drops sharply sometimes, so this might just be normal Java memory fragmentation and garbage collection?

Despite all these improvements, the total data throughput rate is very slow, compared to simply receiving and discarding the data in a native C application. With Teensy 3.1, I'm seeing about 300 kbytes/sec. With Arduino Due, approx 210 kbytes/sec. In previous testing with native code simply counting and discarding data, I measured about 1.1 Mbyte/sec on Teensy 3.1 and approx 4 Mbyte/sec with Arduino Due.

Needless to say, Java is so very disappointing, but at least this seems to solve the crashing from running out of memory, while also solving the crashing from exceeding available CPU time (at least when running on a very fast desktop PC).

This patch is against my heavily modified copy of Arduino 1.0.6. Eventually, if nobody else does, I'll port this to 1.5.x and submit a pull request. Of course, I'm hoping someone else can take it from here on 1.5.x.....

PaulStoffregen/Arduino-1.0.6-Teensyduino@03cb017

[PaulStoffregen]

BTW, I've been running a test with Teensy 3.1 sending at maximum rate for over 40 minutes. Memory usage (resident) is currently 349 Mbyte, up from about 220 when first opening the serial monitor. CPU usage has been holding steady around 16 to 21%. Sustained speed is still in the 300 kbyte/sec range.

Edit: now it's been running over 2 hours. Resident memory is 219 Mbyte, CPU still about 20%, sustained speed still 300 kbytes/sec, serial monitor window still scrolling rapidly and the IDE is still very responsive. I'm not happy the overall data rate isn't 1 Mbyte/sec, but at least the running out of memory and crash from CPU usage problems seem to be solved. Well, on a fast PC anyway. More testing on slower computers needed....

[PaulStoffregen]

Oh, I should mention, I have only tested this on a single machine, running Ubuntu 12.04 (64 bit) with very fast PC hardware. Much more testing should be done, particularly on slower computers and other operating systems, using Arduino Due or Teensy 3.1 running a sketch similar to the one I posted above on August 8th.

[cmaglie]

@PaulStoffregen thanks!

I pushed your solution here:
#2491
once merged I'll port it to 1.5.x too, it should be quite straightforward

For the record: on my test machine the CPU go to 100% and memory usage slowly increase (started from 400MB going toward 800MB). BTW after 15min of test running the IDE is still stable and responsive, I guess we can hardly get better performance from a JTextArea :-)

[PaulStoffregen]

Actually, I think there are probably more small performance improvements possible...

But the really important thing left undone here is dealing with incoming data when autoscroll is turned off.

What probably should happen is setting character counter to zero when autoscroll is detected as going from on to off. The off case should increment that counter for each new message() arrival. If it gets too large, perhaps more than double the total limit, message() should discard the data rather than stuffing it into the buffer.

From the user's perspective, when they turn off autoscroll, everything will be normal for a while. But after a lot of extra data has arrived and grown below the place they're viewing, a limit will be hit and the serial monitor will stop collecting new info. If the limit is high, relatively few people will ever hit it. But those who do will be spared the entire IDE crashing and their unsaved work lost, only because they turned off autoscroll, spent some time reading through the data, and maybe went to do other stuff (like using the editor window) for a while to think about how to fix their project based on what they saw.

[cmaglie]

Merged and ported on 1.5.x, see #2491

As I already explained here: #2491 (comment)
another big improvement on performance has been possible, now IMHO we have a much better serial monitor.

Thanks!

[PaulStoffregen]

I should mention some bugs have turned up with the serial monitor using Teensy, since I added this speedup code. At this point, I don't have any good info, other than this simple warning that unexpected bugs have surfaced. They may be related to other changes, or they may be due to this speedup, or perhaps some complex interaction.

When I know more, I'll post again. At this early stage, I just want to let you know some problems have surfaced as more users have tried this code, which I didn't in my initial testing.