Arduino 1.5.7 - Always does a Full Re-Compile

[mpflaga]

It appears that feature from: ARDUINO 1.0.1 - 2012.05.21 is broke.

• Compilation has been speeded up by only compiling changed files. (All files are recompiled when a new board is selected.) (Paul Stoffregen) http://code.google.com/p/arduino/issues/detail?id=638

Is no longer working, as Verify appears (at least on windows 7) to always do a full (aka clean) compile rather than a re-compile of only changed files.

Not that this was/is working in 1.5.6r2, and still failing in nightly build "ARDUINO 1.5.8 BETA - not yet released" dated Aug 23 2014 5:18am

---

I would also point out that this feature always had a problem if the path to the core library directory had any white spaces. Hence the windows installer placing the install by default into the "C:\Program Files (x86)\Arduino." manifested this same symptom. Where a simple work around was to relocate the path as to not have any white spaces, such as "c:\arduino.".

---

Whereas now the problem is chronic and occurs regardless of paths directory.

[willie68]

I have the same problem
Windows 7,
Arduino Path without spaces, D:\Sprachen\arduino-1.5.7
Sketchbook path: E:\Dropbox\Elektronik\Arduino\projekte\152VO_Uhr\sketchbook
all lib's, sources, everything will be recompiled on every upload...

The 1.5.6 compile only changes.

[matthijskooijman]

Could either of you provide the full compiler output for the first and second verify runs after startup?

I'm not seeing this problem here on linux, so I suspect the problem is Windows-specific. Presumably it fails to get the timestamps of files for some reason, or the timestamp of some file is off (far into the future perhaps)?

[mpflaga]

In short they are identical. There are now timestamps in the compiler output. I have included 7zip bundle of both the first and 2nd build of a blank sketch at https://dl.dropboxusercontent.com/u/99286313/CompileFullOutputOf1stAnd2nd.7z. Where the stdout.txt shows they are identical. The 2nd is appended with the first.

[mpflaga]

a variation of this problem has always existed. In that if the path to the IDE's directory had any white spaces the same would symptom would occur. I don't recall exactly but do recall that I tracked it down to possibly the file object in the java for opening and returning the status of the files was not likely reporting back correctly with white spaces. Where I am not skilled enough fix. but believe that it was not properly quoting the directory name. This smells similiar in that it is about that same file open and status always reporting what ever it is looking for does not exist and wanting to make a clean build.

[PaulStoffregen]

The incorrect handling of spaces was entirely my fault, from 4 years ago when I first wrote this speedup. I mainly use Linux, where spares are rarely used in filenames. Then I tested on Mac and Windows. On Windows, I copy to c:\arduino, because I'm mainly using Linux and that's simpler to copy files over.

I designed the code to "fail safe". There are numerous checks. If any condition fails, the file is recompiled. Only if all pass is the compilation skipped and the old .o file reused.

The problem with spaces involved my incorrect parsing of the .d dependency files created by gcc. The official gcc documentation doesn't mention any special character handling. But when people reported no speedup on Windows, eventually I figured out the issue was the space in "Program Files". By looking at the actual .d files, it turns out gcc adds a backslash in front of the space. I didn't know this, so my original code didn't translate "\ " into " " before comparing strings. Remember, it's designed to "fail safe" if anything doesn't match up.

I fixed this years ago in Teensyduino (where this code was originally developed, years before the Arduino folks accepted my contribution). I don't know if the space parsing ever got fixed in Arduino's version.

Anyway, the point you should take away from this is the code performs MANY checks. It's designed to "fail safe". If any of the files don't appear to exist, or their modification times can't be read (by the Java JRE), or the modification times appear older on either the .o files OR the .d files, or if the .d file isn't exactly the right (expected) format, or anything at all doesn't match up, then it returns false to the compile code, indicating the source needs to be recompiled.

If you're not familiar with the code, it could seem the old spaces parsing bug and whatever has recently failed might be related. The effect is the same. Well, I am familiar with this code. I wrote it. My point is the code is extremely cautious. There are MANY possible ways it can return false, causing an unnecessary recompile. It was intentionally designed with caution. Even with such an approach, it took about 2 years and even a video demo begging people to beta test with regular (non-Teensy) Arduino boards before this was accepted as a contribution to Arduino.

I know you just want your wait time reduced. I feel your pain. I've gone to great lengths to speed things up (just try a Teensy3 someday to see how fast things can be.....) But the last thing anyone wants is incorrectly compiled code, which is why the checks are so cautious. Even though the failure will look similar, unnecessarily recompiling files, there are MANY possible reasons such a bug could happen.

[mpflaga]

Thanks for the history. I think it is key. Noting that 1.5.7 took the leap of using a later version of gcc, as the windows build employes 4.8.1 as opposed to the older 4.3.2. Your insight about the intermediate files lends me to suspect the new gcc may be altering such files in new ways.

I have found your improvement very productive as I play nightly with code that has gotten quite large. I have only once seen an issue requiring a complete rebuild. Where I was messing with changing libraries on the fly. So this was understandable. Otherwise, it has been perfect when not skipped.

[willie68]

Any news on this? I'll find the Version 1.5.7 nearly unusable, because of this issue...

[matthijskooijman]

Hmm, regarding the space escaping @PaulStoffregen Paul mentioned, it seems I fixed something related to that in a recent commit: 726f2ba

Given that the original report says it used to work without spaces in the path, but stopped working somewhere after 1.5.6, I suspect that my change has actually broken things for people (at least on Windows, I've tested this on Linux).

@mpflaga, @willie68, could you post the contents of a .d file generated on Windows?

[matthijskooijman]

Hmm, I'm wondering if \ is used as a path separator as well as or instead of an escape character on Windows...

[willie68]

OK, HardwareSerial.cpp.d First compile Timestamp 02.10.2014 15:58 (German system)

C:\Users\W8858~1.KLA\AppData\Local\Temp\build7047330185438250531.tmp\HardwareSerial.cpp.o: \
 E:\Sprachen\arduino-1.5.7\hardware\arduino\avr\cores\arduino\HardwareSerial.cpp \
 E:\Sprachen\arduino-1.5.7\hardware\arduino\avr\cores\arduino\Arduino.h \
 E:\Sprachen\arduino-1.5.7\hardware\arduino\avr\cores\arduino\binary.h \
 E:\Sprachen\arduino-1.5.7\hardware\arduino\avr\cores\arduino\WCharacter.h \
 E:\Sprachen\arduino-1.5.7\hardware\arduino\avr\cores\arduino\WString.h \
 E:\Sprachen\arduino-1.5.7\hardware\arduino\avr\cores\arduino\HardwareSerial.h \
 E:\Sprachen\arduino-1.5.7\hardware\arduino\avr\cores\arduino\Stream.h \
 E:\Sprachen\arduino-1.5.7\hardware\arduino\avr\cores\arduino\Print.h \
 E:\Sprachen\arduino-1.5.7\hardware\arduino\avr\cores\arduino\Printable.h \
 E:\Sprachen\arduino-1.5.7\hardware\arduino\avr\cores\arduino/new.h \
 E:\Sprachen\arduino-1.5.7\hardware\arduino\avr\cores\arduino\USBAPI.h \
 E:\Sprachen\arduino-1.5.7\hardware\arduino\avr\variants\standard/pins_arduino.h \
 E:\Sprachen\arduino-1.5.7\hardware\arduino\avr\cores\arduino\HardwareSerial_private.h \
 E:\Sprachen\arduino-1.5.7\hardware\arduino\avr\cores\arduino\wiring_private.h

Second compile 2.10.2014 16:01

C:\Users\W8858~1.KLA\AppData\Local\Temp\build7047330185438250531.tmp\HardwareSerial.cpp.o: \
 E:\Sprachen\arduino-1.5.7\hardware\arduino\avr\cores\arduino\HardwareSerial.cpp \
 E:\Sprachen\arduino-1.5.7\hardware\arduino\avr\cores\arduino\Arduino.h \
 E:\Sprachen\arduino-1.5.7\hardware\arduino\avr\cores\arduino\binary.h \
 E:\Sprachen\arduino-1.5.7\hardware\arduino\avr\cores\arduino\WCharacter.h \
 E:\Sprachen\arduino-1.5.7\hardware\arduino\avr\cores\arduino\WString.h \
 E:\Sprachen\arduino-1.5.7\hardware\arduino\avr\cores\arduino\HardwareSerial.h \
 E:\Sprachen\arduino-1.5.7\hardware\arduino\avr\cores\arduino\Stream.h \
 E:\Sprachen\arduino-1.5.7\hardware\arduino\avr\cores\arduino\Print.h \
 E:\Sprachen\arduino-1.5.7\hardware\arduino\avr\cores\arduino\Printable.h \
 E:\Sprachen\arduino-1.5.7\hardware\arduino\avr\cores\arduino/new.h \
 E:\Sprachen\arduino-1.5.7\hardware\arduino\avr\cores\arduino\USBAPI.h \
 E:\Sprachen\arduino-1.5.7\hardware\arduino\avr\variants\standard/pins_arduino.h \
 E:\Sprachen\arduino-1.5.7\hardware\arduino\avr\cores\arduino\HardwareSerial_private.h \
 E:\Sprachen\arduino-1.5.7\hardware\arduino\avr\cores\arduino\wiring_private.h

[mpflaga]

Above I reference a link to drop box that has a 7zip of all the temp files from a 1st and 2nd build. which includes the core.a along with the *.d files that are created. Is there new build with your above mentioned change, as to test. I see the commit, but believe I would need to build a runtime exe, as to test.

[matthijskooijman]

Right, so it uses both \ and / as a path separator (e.g. in E:\Sprachen\arduino-1.5.7\hardware\arduino\avr\variants\standard/pins_arduino.h) as well as an escape character (at the end of a line).

That certainly explains why my change broke things. I wonder how a space in a pathname is encoded (and even more interestingly, I wonder how a space at the start of a filename looks). I'm trying to fire up my virtualbox to test for myself, but it's not quite working yet. Care to do another test for me?

Could you rename your arduino folder to "E:\Test\ Test\Arduino Test" (so there's a space at the start of the second folder and a space inside of the third folder). If you can also manage to put a backspace inside a folder name, that would be even better, but I think Windows does not allow this.

[matthijskooijman]

Is there new build with your above mentioned change, as to test. I see the commit, but believe I would need to build a runtime exe, as to test.

That commit has been available in the nightlies for a while and is actually the cause of your problem, not the solution :-)

It helps to allow spaces on Linux but, as I see now, it only worsens the situation on Windows.

[willie68]

Windows doesn't allow backspace in names and spaces before the name.
Spaces are Escaped with leading \ "\152VO Uhr" -> "\152VO\ Uhr"

C:\Users\W8858~1.KLA\AppData\Local\Temp\build7929098494296034026.tmp\Dogm\Dogm.cpp.o: \
 E:\privat\Dropbox\Elektronik\Arduino\projekte\152VO\ Uhr\sketchbook\libraries\Dogm\Dogm.cpp \
 E:\privat\Dropbox\Elektronik\Arduino\projekte\152VO\ Uhr\sketchbook\libraries\Dogm/Dogm.h \
 E:\privat\Dropbox\Elektronik\Arduino\projekte\152VO\ Uhr\sketchbook\libraries\Dogm/utility/dogm128.h \
 E:\privat\Dropbox\Elektronik\Arduino\projekte\152VO\ Uhr\sketchbook\libraries\Dogm/utility/dogmpgm.h \
 E:\privat\Dropbox\Elektronik\Arduino\projekte\152VO\ Uhr\sketchbook\libraries\Dogm/utility/dogmfont.h \
 E:\Sprachen\arduino-1.5.7\hardware\arduino\avr\cores\arduino/Print.h \
 E:\Sprachen\arduino-1.5.7\hardware\arduino\avr\cores\arduino/WString.h \
 E:\Sprachen\arduino-1.5.7\hardware\arduino\avr\cores\arduino/Printable.h \
 E:\Sprachen\arduino-1.5.7\hardware\arduino\avr\cores\arduino/new.h \
 E:\Sprachen\arduino-1.5.7\hardware\arduino\avr\cores\arduino/Arduino.h \
 E:\Sprachen\arduino-1.5.7\hardware\arduino\avr\cores\arduino/binary.h \
 E:\Sprachen\arduino-1.5.7\hardware\arduino\avr\cores\arduino/WCharacter.h \
 E:\Sprachen\arduino-1.5.7\hardware\arduino\avr\cores\arduino/HardwareSerial.h \
 E:\Sprachen\arduino-1.5.7\hardware\arduino\avr\cores\arduino/Stream.h \
 E:\Sprachen\arduino-1.5.7\hardware\arduino\avr\cores\arduino/Print.h \
 E:\Sprachen\arduino-1.5.7\hardware\arduino\avr\cores\arduino/USBAPI.h \
 E:\Sprachen\arduino-1.5.7\hardware\arduino\avr\variants\standard/pins_arduino.h

[mpflaga]

Actually it will if you use the dos command mkdir " foo". The Explorer GUI shell removes leading spaces. Note that mkdir " foo\bar" creates a dir of " foo" and a sub directory of "bar". At least on Windows 7.
C:\projects\Arduino>cd " test"
C:\projects\Arduino\ test>mkdir "arduino test"
C:\projects\Arduino\ test>cd "arduino test
C:\projects\Arduino\ test\arduino test>mkdir "foo\bar"
C:\projects\Arduino\ test\arduino test>cd foo
C:\projects\Arduino\ test\arduino test\foo>cd bar
C:\projects\Arduino\ test\arduino test\foo\bar>

[mpflaga]

typical output from above directory
C:\projects\Arduino\ test\arduino test\foo\bar\arduino-1.5.7/hardware/tools/avr/bin/avr-g++ -c -g -Os -w -fno-exceptions -ffunction-sections -fdata-sections -MMD -mmcu=atmega328p -DF_CPU=16000000L -DARDUINO=157 -DARDUINO_AVR_UNO -DARDUINO_ARCH_AVR -IC:\projects\Arduino\ test\arduino test\foo\bar\arduino-1.5.7\hardware\arduino\avr\cores\arduino -IC:\projects\Arduino\ test\arduino test\foo\bar\arduino-1.5.7\hardware\arduino\avr\variants\standard C:\Users\mflaga\AppData\Local\Temp\build1786728830652454607.tmp\sketch_oct02a.cpp -o C:\Users\mflaga\AppData\Local\Temp\build1786728830652454607.tmp\sketch_oct02a.cpp.o
...
C:\projects\Arduino\ test\arduino test\foo\bar\arduino-1.5.7/hardware/tools/avr/bin/avr-ar rcs C:\Users\mflaga\AppData\Local\Temp\build1786728830652454607.tmp/core.a C:\Users\mflaga\AppData\Local\Temp\build1786728830652454607.tmp\HardwareSerial0.cpp.o

[matthijskooijman]

Right. So escaping is just messy on Windows (OTOH, Makefiles and escaping / spaces are also messy in the first place...).

I dug around in gcc a bit, it seems the escaping of stuff happens here: https://github.com/gcc-mirror/gcc/blob/master/libcpp/mkdeps.c#L55-L120

So, what hapens is:

• is escaped by prefixing a \

• $ is escaped by prefixing a $
• space and tab are escaped by prefixing a . Furthermore, if any backslashes were already present directly before the space (can happen on Linux where paths can contain a backslash), those are all doubled.

Essentially, we should undo all these changes on the Arduino side. Alternatively, we could just remove all backslashes in front of spaces, tabs and # signs and accept that a backslash or a $ in a filename messes up the the dependency detection (by always recompiling the file), which I'd also find acceptable.

[JulyJim]

I have moved my 1.5.8 from Program space File to Arduino_EXE and it still compiles ALL after changing only the main ino file.

[JulyJim]

cmaglie - how can I try this? Wait for 1.5.9??

[matthijskooijman]

Hm, I was going to recommend the nightly build (which is automatically built from git every night), but I see only one nightly build on the download page and I'm not sure if it is built from master (1.0.x) or ide-1.5.x. @cmaglie, can you comment?

[matthijskooijman]

Closing this issue btw, it was fixed by #2395.

[JulyJim]

I was just adding this in case it has anything to do with the compiler "speed".
Just note that the error is reported twice.

Arduino: 1.5.8 (Windows XP), Board: "Arduino Due (Programming Port)"

Using library Wire in folder: C:\Arduino_EXE\Arduino_158\hardware\arduino\sam\libraries\Wire 

Using library LiquidCrystal in folder: C:\Documents and Settings\Vaclav\My Documents\Arduino\libraries\LiquidCrystal (legacy)

C:\Arduino_EXE\Arduino_158/hardware/tools/gcc-arm-none-eabi-4.8.3-2014q1/bin/arm-none-eabi-g++ -c -g -Os -w -ffunction-sections -fdata-sections -nostdlib -fno-threadsafe-statics --param max-inline-insns-single=500 -fno-rtti -fno-exceptions -Dprintf=iprintf -mcpu=cortex-m3 -DF_CPU=84000000L -DARDUINO=158 -DARDUINO_SAM_DUE -DARDUINO_ARCH_SAM -D__SAM3X8E__ -mthumb -DUSB_VID=0x2341 -DUSB_PID=0x003e -DUSBCON -DUSB_MANUFACTURER="Unknown" -DUSB_PRODUCT="Arduino Due" -IC:\Arduino_EXE\Arduino_158\hardware\arduino\sam\system/libsam -IC:\Arduino_EXE\Arduino_158\hardware\arduino\sam\system/CMSIS/CMSIS/Include/ -IC:\Arduino_EXE\Arduino_158\hardware\arduino\sam\system/CMSIS/Device/ATMEL/ -IC:\Arduino_EXE\Arduino_158\hardware\arduino\sam\cores\arduino -IC:\Arduino_EXE\Arduino_158\hardware\arduino\sam\variants\arduino_due_x -IC:\Arduino_EXE\Arduino_158\hardware\arduino\sam\libraries\Wire -IC:\Documents and Settings\Vaclav\My Documents\Arduino\libraries\LiquidCrystal C:\DOCUME~1\Vaclav\LOCALS~1\Temp\build3836562875884357172.tmp\Audio_DUE_20.cpp -o C:\DOCUME~1\Vaclav\LOCALS~1\Temp\build3836562875884357172.tmp\Audio_DUE_20.cpp.o 

Using library Wire in folder: C:\Arduino_EXE\Arduino_158\hardware\arduino\sam\libraries\Wire 

Using library LiquidCrystal in folder: C:\Documents and Settings\Vaclav\My Documents\Arduino\libraries\LiquidCrystal (legacy)

C:\Arduino_EXE\Arduino_158/hardware/tools/gcc-arm-none-eabi-4.8.3-2014q1/bin/arm-none-eabi-g++ -c -g -Os -w -ffunction-sections -fdata-sections -nostdlib -fno-threadsafe-statics --param max-inline-insns-single=500 -fno-rtti -fno-exceptions -Dprintf=iprintf -mcpu=cortex-m3 -DF_CPU=84000000L -DARDUINO=158 -DARDUINO_SAM_DUE -DARDUINO_ARCH_SAM -D__SAM3X8E__ -mthumb -DUSB_VID=0x2341 -DUSB_PID=0x003e -DUSBCON -DUSB_MANUFACTURER="Unknown" -DUSB_PRODUCT="Arduino Due" -IC:\Arduino_EXE\Arduino_158\hardware\arduino\sam\system/libsam -IC:\Arduino_EXE\Arduino_158\hardware\arduino\sam\system/CMSIS/CMSIS/Include/ -IC:\Arduino_EXE\Arduino_158\hardware\arduino\sam\system/CMSIS/Device/ATMEL/ -IC:\Arduino_EXE\Arduino_158\hardware\arduino\sam\cores\arduino -IC:\Arduino_EXE\Arduino_158\hardware\arduino\sam\variants\arduino_due_x -IC:\Arduino_EXE\Arduino_158\hardware\arduino\sam\libraries\Wire -IC:\Documents and Settings\Vaclav\My Documents\Arduino\libraries\LiquidCrystal C:\DOCUME~1\Vaclav\LOCALS~1\Temp\build3836562875884357172.tmp\Audio_DUE_20.cpp -o C:\DOCUME~1\Vaclav\LOCALS~1\Temp\build3836562875884357172.tmp\Audio_DUE_20.cpp.o 

In file included from Audio_DUE_20.ino:38:0:
C:\DOCUME~1\Vaclav\LOCALS~1\Temp\build3836562875884357172.tmp\LCM1602.H:18:5: error: prototype for 'int CLCM1602::Test()' does not match any in class 'CLCM1602'
 int CLCM1602::Test(void)
     ^
C:\DOCUME~1\Vaclav\LOCALS~1\Temp\build3836562875884357172.tmp\LCM1602.H:14:9: error: candidate is: int CLCM1602::Test(int)
     int Test(int a);
         ^
Error compiling.

In file included from Audio_DUE_20.ino:38:0:
C:\DOCUME~1\Vaclav\LOCALS~1\Temp\build3836562875884357172.tmp\LCM1602.H:18:5: error: prototype for 'int CLCM1602::Test()' does not match any in class 'CLCM1602'
 int CLCM1602::Test(void)
     ^
C:\DOCUME~1\Vaclav\LOCALS~1\Temp\build3836562875884357172.tmp\LCM1602.H:14:9: error: candidate is: int CLCM1602::Test(int)
     int Test(int a);
         ^
Error compiling.

[cmaglie]

The nightly build is made from ide-1.5.x branch, this fix should be already in the nightly build now.
I've just merged #2003 that improves compile speed a bit more, it should be available in the nightly within the next 24h.