Serial Port selected disconnects after upload.

[ReeceM]

When I upload to my Sparkfun pro micro after selecting the port its on, after upload the selected port becomes deselected in the drop down and I can't upload again without going there and re-selecting the com port.

I am using a Hourly build when this happens :
Arduino 1.6.6 Hourly Build 2015/07/07 9:41

But if I use ver. 1.6.0 the serial port stays connected after upload.

Is this a localised problem or am I suffering from beta.

[matthijskooijman]

The micro disconnects and reconnects USB when switching from sketch to bootloader and back, perhaps this causes the problem? I haven't tried reproducing this, though.

[ReeceM]

I think the switching is causing the problem, because it switches to the port for uploading but never returns to the normal port. The IDE never reselects the port after an upload.

I think something that works with the Leonardo/micro upload style has been changed in the code, because it used to work correctly, i.e. it remembered the com port it is on after an upload.

[facchinm]

Hi @ReeceM , are you on Linux? Your problem can be triggered by the serial monitor (which is now kept in a "zombie" state during upload) respawning too soon and thus locking the previous port, so the sketch will get another one.
Can you post here your PC specs?

[ReeceM]

Hi @facchinm, I am on Windows XP, never had this problem before.

[facchinm]

Very strange, as Windows assigns different COM numbers to bootloader and sketch. Are you uploading with the serial monitor opened?

[ReeceM]

The serial monitor is not opened. The problem is that the IDE is not re-selecting the port after upload, even though it is there, example:
_BEFORE_

_AFTER_

[facchinm]

Ok, thanks for the screenshot, we'll analyze this issue as soon as @ffissore is back!

[ReeceM]

@facchinm cool 😃

[pablosun]

Hi, we can also reproduce this issue with a ATMega32U4 (3.3V/8MHz). The same cannot be reproduced with ATMega32U4(5V/16MHz) with Arduino IDE 1.6.5r2. This issue does not happen to Arduino IDE 1.6.4.

We suspect that this is related to uploadUsingPreferences() in Arduino/arduino-core/src/cc/arduino/packages/uploaders/SerialUploader.java

In recent changes, the uploader will set the BaseNoGui.selectSerialPort(uploadPort); after finishing uploading the sketch through the Bootloader COM port. However, uploadPort is actually set to Bootloader COM port at this moment - and thus overrides the setting from users (which should set to the sketch COM port). There was a logic that "detects" COM port with a 1000ms timeout in IDE 1.6.5r2 - and this timeout may not be sufficient enough for Pro Micro(3.3V/8MHz).

[NicoHood]

Does this only happen on Windows?
How does a 16mhz version behave?
Have you tried the official Bootloader as well, since the sparkfun bootloader is a bit different?

[pablosun]

Hi @NicoHood,

1. This does not happen on OS X(Yosemite) - but we also notice that on OS X, both Bootloader COM port and sketch COM port are registered as /dev/usbmodem1411 (the number may change on different machine, but keep the same when the board switches COM ports).
2. The official Leonardo (5V/16MHz) bootloader works without this issue.
3. We have only tried 8MHz version of Pro Micro bootloader - the result of Sparkfun Pro Micro bootloader at 5V/16MHz is unknown - sorry for a bit misleading when reporting this.

[NicoHood]

have you tried the 16mhz bootloader on a 16 mhz device? I mean you cannot use a slower/faster bootloader on a device with a different crystal.

You should try the sparkfun 16mhz bootloader for sure.

You could also try to recompile HoodLoader2 (see my repos) for 8mhz and see if this fixes the issue (since the original and modified bootloaders have some major problems)

[pablosun]

@NicoHood

1. Yes, we changed to 16MHz clock with 16MHz bootloaders
2. Sparkfun 5V/16MHz bootloader works fine without this issue
3. We haven't tried HoodLoader2 yet but reducing the timeout time of the Pro Micro bootloader can workaround this issue

But still I think the IDE code should have a more "formal" way to deal with Leonardo-derivatives which have disappearing COM ports after sketch uploads.

[NicoHood]

I do not have any 8mhz device, but with a clock devider I could test this though. However I dont have the time to do it. But it seems that 8mhz causes the problem. You could try to recompile HoodLoader2 with 8mhz and see if it also has the problem. Because the sparkfun and official bootloaders are not perfect, I could imagine that this causes the problem, and not the IDE itself.

[ReeceM]

Wow, there is some activity :). First off my pro micro broke a month ago as it was one from v1.3 batch where the micro USB connector did not have the extra mount pads, thus I am at present and have been unable to test this, so thank you to those who have commented on this issue.

@pablosun what IDE version did you test the Sparkfun 5v/16MHz bootloader with?

@NicoHood I don't think it could be the bootloader, talking of Sparkfuns as I haven't used any other, because this problem only started after I ran the 1.6.6 hourly build, as mentioned in the first comment. Why would the IDE and bootloader work harmoniously and then not? That makes me think the way in which the IDE handles the 32u4 port switching is not as it was a while back.

[pablosun]

@ReeceM I tested with official releases:
Arduino IDE 1.6.4(Windows): Works fine without this issue.
Arduino IDE 1.6.5(Windows): Can reproduce this issue (COM port changes after uploading sketches).
Arduino IDE 1.6.6(Windows): Can reproduce this issue (COM port changes after uploading sketches).

[NicoHood]

USB can introduce many problems. Just because the IDE changed a few things that broke the connection, doesnt mean the real cause is not in the bootloader itself.

It just popped up then (maybe!). but it is really more likely that the IDE broke the code, I just dont want to cross out a bad bootloader because it is known to have some major issues that noone is willing to fix (not because its hard to fix, just because of compatibility and not being able to tell users how to update it properly).

So here are 4 things we have to keep in mind when dealing with USB Problems:

• BootLoader
• Arduino IDE
• Kernel (Windows or Linux)
• Hardware (Long cables, hubs, 2 vs 3 etc)

For example Ive had issues with usb 2 hubs on usb 3 ports. But it was a bootloader bug. And once ago we had a problem with a special linux kernel version.

[jsnmc]

I am having the same issue. Windows 10 + Sparkfun 16MHz Pro micro on Arduino IDE 1.6.5.
Upon upload, the Arduino disconnects and reconnects on a different port.

*Update: moved to 1.6.11 and the issue no longer occurred initially. Upon changing code and attempting the upload, I am seeing the same behavior.

[facchinm]

@jsnmc , if you are still experiencing this it means that the serial port is not becoming available in the first two seconds after the end of the upload. This could be caused by:

• drivers taking a lot of time to load properly (if you are on windows)
• a "slow" bootloader
  We could push this limit up a bit but not too much since it would interfere with workflows on non-problematic boards/OSs

[Breidenbach]

Same problem with Leonardo, but not with Genuine Arduino under Win 10.

Sketch uses 7,872 bytes (27%) of program storage space. Maximum is 28,672 bytes.
Global variables use 506 bytes (19%) of dynamic memory, leaving 2,054 bytes for local variables. Maximum is 2,560 bytes.
Forcing reset using 1200bps open/close on port COM8
PORTS {COM1, COM3, COM8, } / {COM1, COM3, COM8, } => {}
PORTS {COM1, COM3, COM8, } / {COM1, COM3, COM5, COM8, } => {COM5, }
Found upload port: COM5
C:\Program Files (x86)\Arduino\hardware\tools\avr/bin/avrdude -CC:\Program Files (x86)\Arduino\hardware\tools\avr/etc/avrdude.conf -v -patmega32u4 -cavr109 -PCOM5 -b57600 -D -Uflash:w:C:\Users\HALBRE~1\AppData\Local\Temp\builda32b237ded91972840ec973ffdc37541.tmp/Lesson_04_Changing_Colours.ino.hex:i

avrdude: Version 6.0.1, compiled on Apr 15 2015 at 19:59:58
Copyright (c) 2000-2005 Brian Dean, http://www.bdmicro.com/
Copyright (c) 2007-2009 Joerg Wunsch

     System wide configuration file is "C:\Program Files (x86)\Arduino\hardware\tools\avr/etc/avrdude.conf"

     Using Port                    : COM5
     Using Programmer              : avr109
     Overriding Baud Rate          : 57600

avrdude: ser_open(): can't set com-state for ".\COM5"
avrdude: ser_send(): write error: sorry no info avail
avrdude: ser_send(): write error: sorry no info avail

Board at COM5 is not available

On Sep 13, 2016, at 3:54 AM, Martino Facchin [email protected] wrote:

@jsnmc https://github.com/jsnmc , if you are still experiencing this it means that the serial port is not becoming available in the first two seconds after the end of the upload. This could be caused by:

drivers taking a lot of time to load properly (if you are on windows)
a "slow" bootloader We could push this limit up a bit but not too much since it would interfere with workflows on non-problematic boards/OSs
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub #3495 (comment), or mute the thread https://github.com/notifications/unsubscribe-auth/ABsoINUz1N1J092vX2V5H6ANMfNaAymrks5qplaZgaJpZM4FV6qX.

[eyalwu]

I am having a similar problem with my Arduino micro-5V, which I run with windows10, IDE 1.8.2. The micro disconnects during the upload, but then reconnects again. I am having some hardware problems which I suspect are related to this problem.
Anyone has any idea how to solve the issue?

[selligv]

I am surprised to see this com port switching or disappearing problem has continued over the years. It just keeps coming back. I see lots of quasi theories. Can't someone provide a comprehensive solution layered on the IDE and/or the USB boot driver.... SOMEDAY!!!!
I have 4 TAU boards and 2 Firecricket boards (ZERO's) waiting for a solution

[selligv]

My TAU board worked under Win 7. It doesn't under Win 10 (same PC). I note that the filesize of the sounds when the board reset is pressed are 4 or 5 times larger in Win 10 than in Win 7 (the files are located in Windows/media, "Windows Hardware Insert.wav" and "Windows Hardware Remove.wav"). When uploads are initiated, I wonder if these substantial extra delays interfere in the timing required when the port is switched from serial to programming and back again. I've attempted to change these files, but they are protected under "Trusted Installer" permissions that cannot be changed. And they can't be disabled.

[ReeceM]

@selligv I don't think it would be the audio that windows plays. The IDE does not trigger the connect and disconnect audio, but rather the windows system.

This original problem came up on an XP PC, and I still have issues with it on a mac and a linux machine too. So I don't think this is the fault of the OS, I believe a part of the serial section of the IDE was changed and it has had a knock-on effect causing this issue.

[selligv]

I really have to apologize.... the problem had to do with my poor programming skills. RabidPrototypes pointed me in a direction I finally understood. I won't explain further for fear of being expelled from this forum. For shame.

[jrowberg]

FWIW, I too am having this problem with a Zero-based board in the latest Win10 64-bit, Arduino 1.8.5. The issue doesn't occur in the Arduino 1.8.5 environment on MacOS with the same board, since (probably because of the OS) it always receives the same /dev/ttyX node regardless. But on Windows, the bootloader/upload COM port is a different number than the standard running sketch COM port. For example:

1. Plug in the device, COM14 assigned
2. Open serial monitor and verify communication (works)
3. Compile and flash, 1200bps open/close trick to get the bootloader and serial monitor closes/pauses
4. COM14 disappears and COM15 appears (detected as upload port correctly)
5. Upload completes, board resets
6. COM14 reappears after 5-10 seconds or so, but...
7. Serial monitor eventually closes and IDE has switched to COM15 for uploading, which no longer exists
8. Re-uploading OR re-opening serial monitor manually requires first selecting COM14 by hand

It's pretty annoying. :(

Does the typical Zero bootloader assign VID, PID, and a serial number? I think those are usually the things that help the OS detect whether two devices are the same or not.

[facchinm]

All boards with Native USB bootloader have this behaviour; this is a consequence of how Windows works, by design.
Since it assigns a different port number for any combination of VID/PID/SerialNumber, we have to touch the actual port (this triggers a reboot to bootloader), rediscover the bootloader port number, use it for upload, wait until the board is up again to detect the newly assigned port number.
This procedure of course has timeouts, and a lot of Windows PCs are really slow in listing the new device. If it takes more than 5 seconds, the bootloader port is retained and the "unlock serial monitor" feature fails.

[Chris-Moir]

We are a new user, have an LP 2Gb/32Gb board with Windows 10, and have the same problem. In an attempt to isolate exactly where the problem is, we built the "blink" example, added Arduino.cs into the project, but in the sub Arduino(port,baud,bool,delay) we forced "serialPortName" to "COM1:". This seemed to work, but only until we powered down. On the next power up, COM1: had disappeared from the list of available serial ports available in Arduino. However, the Win10 device manager showed it as "in use", but as "USB Serial Device".
Checking back over the blogs, this problem has been around since 2015 at least. Has anyone managed a fix? If we force all of the occurrences of "serialPortName" in Arduino.cs to "COM1:", will this cure the problem, or is there something else behind (i.e. in Firmata or the bootloader) which is causing it? Not enough documentation exists on the system at this level to allow us to safely mess with it - unless someone can provide more info?

[DaMicrok]

I've read the post and understand:

" it means that the serial port is not becoming available in the first two seconds after the end of the upload. This could be caused by:

drivers taking a lot of time to load properly (if you are on windows)"etc

I'm on Windows 10, using a Due and still having this problem in 2018! What is the fix????!!!!!!???

EDIT: I can get it to work. I have a Due. The problem was trying to programme it via the native USB port. The solution (for me) was to programme it via the programme port (Doh!) and then remove the cable and insert it to the native USB port to commence reading from that. It may be obvious but it had me stumped for a few hours. I hope this may help someone else.

[wcndave]

I'm on Windows 10, using a Due and still having this problem in 2018! What is the fix????!!!!!!???

Yes! I have the same thing, and I am using a super fast and powerful Windows PC.... Yet I cannot get a small arduino micro programmed...

EDIT: I can get it to work. I have a Due. The problem was trying to programme it via the native USB port. The solution (for me) was to programme it via the programme port (Doh!) and then remove the cable and insert it to the native USB port to commence reading from that. It may be obvious but it had me stumped for a few hours. I hope this may help someone else.

Sorry, but I didn't understand what that meant. Perhaps the Due has 2 ports, however the Micro only has the one, so this is still an issue....

[ReeceM]

@wcndave and @DaMicrok I think the Due is not directly related to this as it is a different form factor / programming method of the device. It doesn't have programming and communication over the same ports like the micros which switch between the boot loader on a different usb port and then back to the user's program space.

Also I don't believe it is a driver issue, as it won't change if the driver hasn't updated, but rather the software that is making use of said driver.

[wcndave]

@wcndave and @DaMicrok I think the Due is not directly related to this as it is a different form factor / programming method of the device. It doesn't have programming and communication over the same ports like the micros which switch between the boot loader on a different usb port and then back to the user's program space.

Also I don't believe it is a driver issue, as it won't change if the driver hasn't updated, but rather the software that is making use of said driver.

okay, well my "same issue" is the same as those that started 3 years ago, on a genuine Micro.

[facchinm]

Ok, let's recap a bit since I've just tested on both a real Windows PC and on a VM with Native USB uploads and they all work on latest IDE.
#3495 (comment) still applies.

Arduino/arduino-core/src/cc/arduino/packages/uploaders/SerialUploader.java

Lines 128 to 132 in 9818dcf

	if (doTouch) {
	try {
	// Toggle 1200 bps on selected serial port to force board reset.
	List<String> before = Serial.list();
	if (before.contains(userSelectedUploadPort)) {

This is where the upload starts; all affected boards have doTouch set as true.
The touch operation is allowed to change the upload port name silently, since the bootloader may appear on a different port.

After upload, this is the snippet that restores the serial port name

Arduino/arduino-core/src/cc/arduino/packages/uploaders/SerialUploader.java

Lines 205 to 237 in 9818dcf

	String finalUploadPort = null;
	if (uploadResult && doTouch) {
	try {
	if (waitForUploadPort) {
	// For Due/Leonardo wait until the bootloader serial port disconnects and the
	// sketch serial port reconnects (or timeout after a few seconds if the
	// sketch port never comes back). Doing this saves users from accidentally
	// opening Serial Monitor on the soon-to-be-orphaned bootloader port.
	Thread.sleep(1000);
	long started = System.currentTimeMillis();
	while (System.currentTimeMillis() - started < 2000) {
	List<String> portList = Serial.list();
	if (portList.contains(userSelectedUploadPort)) {
	finalUploadPort = userSelectedUploadPort;
	break;
	}
	Thread.sleep(250);
	}
	}
	} catch (InterruptedException ex) {
	// noop
	}
	}
	if (finalUploadPort == null) {
	finalUploadPort = actualUploadPort;
	}
	if (finalUploadPort == null) {
	finalUploadPort = userSelectedUploadPort;
	}
	BaseNoGui.selectSerialPort(finalUploadPort);
	return uploadResult;

Bothe doTouch and waitForUploadPort are true (Due native for example). uploadResult is true if the upload succeeded (the uploader program must return 0 on success) .

If all conditions are true, we wait at least 1 second and then we check if the originally selected upload port has reappeared; we continue checking for 2 more seconds; if it didn't reappear yet, we set finalUploadPort = actualUploadPort;, losing the user selected port.

What I can see here is:

• if the upload doesn't succeed but the touch does, we keep selecting the actualUploadPort so the user can press upload another time. This works for Due (since the bootloader has no timeout) but not for Micro/Leonardo
• if the upload succeeds the port must reappear in 3 seconds to avoid losing the selection.

What can we do:

• remove the first behaviour by entirely removing

    if (finalUploadPort == null) {
      finalUploadPort = actualUploadPort;
    }

This will affect any further upload while the board is still in bootloader mode. Usually if upload fails there's something deeply wrong so it may not be a very common case.

• wait for reappearance for more than 3 seconds. I don't have metrics on this but maybe 5 seconds could be a good compromise.
• we don't consider the fact that the serial name might change from A to B (bootloader) to C (due to a different serialNumber on Windows). This might be solved by waiting for port A and if it doesn't reappear select the one that did.

If we agree on an approach I can prepare a PR with the changes.

[jrowberg]

I like this solution. My recent experience with similar native-upload boards is that the post-upload port changes to something new, so I have to switch manually every time. Being able to detect this change automatically would eliminate a big annoyance with the process.

[facchinm]

PR open for testing (as soon as Jenkins produces the builds): #8218

[ReeceM]

@facchinm that is cool, this would be useful that it is fixed after so long 😄

As an aside, version 1.8 on a mac doesbt experience the behaviour that seemed to be present on the windows machine, it seems to stick to one port regardless of the switch between the CDC bootloader and user space program. I presume this fix won't affect that as it will still keep the port either way?

[facchinm]

It's a Windows specific problem, on every other OS the CDC driver takes milliseconds to load, not 5 seconds 😄

[ReeceM]

@facchinm that's funny.

Thank you for finding a way to get it working 👍

[facchinm]

@jrowberg @wcndave @selligv @ReeceM @Breidenbach @Chris-Moir @DaMicrok sorry for the spam, but could you test PR #8218 and report if it solves the problem with port being deselected after upload? Thanks!

[ReeceM]

@facchinm Will do 👍

[PilotinControl25]

Has this been fixed yet as I too am having the same issues. I would like to test it as well

[facchinm]

@PilotinControl25 can you test the IDE build at PR #8218 and report if it solveds your problem?

[ReeceM]

@jrowberg @wcndave @selligv @ReeceM @Breidenbach @Chris-Moir @DaMicrok sorry for the spam, but could you test PR #8218 and report if it solves the problem with port being deselected after upload? Thanks!

Hi @facchinm, I tested the new PR #8212 and it doesn't seem to cause the same issue as before. It was tested on windows 10, unfortunately I don't have access to XP anymore.

What I also did is to run 1.8.7 as well to see if the same issue was still there, but it appears that it isn't happening, this may be related to windows 10 only and other versions are still experiencing the issue.

I presume the others who have also experienced this issue should have a look too, and possibly state the version they were running when they had the ports not connecting.

[PaulStoffregen]

Dunno if this will help, but just want to share a bit of experience from Teensy's long history of native USB. We've seen Windows 7, Vista & XP take much longer to complete USB enumeration depending on how many extra HID interfaces are included together with CDC serial (using IAD, class/subclass/protocol = 254/2/1 in the device descriptor). Years ago I put many long hours into investigating this problem. Using a USB protocol analyzer proved it was just some strange delay in Windows, not any issue where control transfers were being delayed by the device side. The good news is those delays are avoided if you use only CDC (class/subclass/protocol = 2/0/0 in the device descriptor) without any extra HID interfaces. The bad news is the delays become noticeable starting with 2 HID interfaces and really annoying for 3+ HID interfaces, and there seems to be no solution other than upgrading to Windows 8 or 10.

[ReeceM]

Thanks @PaulStoffregen that kind of makes sense because I did originally have this issue on XP. I haven't used Arduino on windows since 2016, so I haven't actually had this issue since then.

Thanks for giving some info on the reasons 👍

[NonaSuomy]

Has this ever been tested in a linux vm on a linux host or with usbip?

I have yet to get it to upload, Attempted to use udev rules to autoconnect the device back to the host after disconnect and then a script with systemd to auto add the device back to the client machine but failed with timeout so I modified the java file and increased the delays but then got a butterfly programmer error instead of the timeout.

In the VM I can't re-attach the device fast enough (QEMU with Libvirt). Anyone have any tips for using 32u4 with Linux VM or USBIP in Linux as well for both server/client?

[ReeceM]

Hi @NonaSuomy, I used to have a version of the Arduino IDE running on Ubuntu 12.04 LTS a while back, and was using the 32u4 on that without problems if I can remember properly when I was using it, but I recall I only had this issue on windows when it was opened.

I am no quite sure if the issue you are having is related to the vm story or not, but that does sound like quite a bit of a different setup to the normal way it can be implemented and the case when this was first an issue.

I think it may be better to open a new issue as it could be related to something else and would also get better attention by the community. This is quite an old issue and discussion, so may not get a response from the actual maintainers.

[NonaSuomy]

Running Arduino IDE on Linux is not an issue, VM / Delays / Reconnects seem to be.

It's pretty much similar to the same issue as above except with the addition of extracurricular delays caused by outside influences unbeknownst to the Arduino IDE causing the issue.

I've yet to test a regular Arduino with the 2 methods here: http://nonasuomy.github.io/Arduino-Remote-Upload/

The 32u4 with it's special case of the delay way it connects to serial with the IDE seems to be the issue.

Maybe having definable delays as an option. If you know it takes X time for USB/IP and Services to reconnect. Instead of hardcoded wait timeouts.

I tried to recompile the IDE by manually changing the delays set, but then hit another error about the butterfly programmer failing as I wasn't really sure what I was doing to begin with... trial and error. It found the port again though after massively increasing the delay. Finding this post though helped me find where in the code to change the delays so that's a step forward :) thanks.

I'll paste this to another issue to see if anybody has another suggestion.

[ReeceM]

Hi @NonaSuomy, okay I am glad that this issue's details helped you start off in a direction of getting somewhere with your problem.

There is a timeout option that happens with the avrdude when running in butterfly mode, I don't know where that is set in the IDE's code though, but that may be where it stopped working when you extended the delays.