Not compiling using master HEAD on macOS

[dimitre]

Platform

• Hardware: ESP-01
• Core Version: 5ca0bde
• Development Env: [Arduino IDE]
• Operating System: [MacOS] 10.14.6

Settings in IDE

• Module: [Generic ESP8266 Module]

Problem Description

No sketches are compiling using git master HEAD in macOS
here is the output in console in red

fork/exec /Applications/Arduino.app/Contents/Java/hardware/espressif/esp8266/tools/python3/python3: no such file or directory
Error compiling for board Generic ESP8266 Module.

[earlephilhower]

Did you re-run "python get.py" to install the new toolchain required for head?

You'll also need the usual "git submodule init" and "git submodule update" to ensure proper sub-git contents.

[dimitre]

yes I've did everything.
in fact I was using a different folder (espressif) instead of esp8266com inside /Applications/Arduino.app/Contents/Java/hardware and tried to use esp8266com, but after that Arduino app didnt' open too.
Now I'm using the stable version but I have availability to do any testing. I'll try it in a clean Arduino install soon.

[earlephilhower]

The JSON should be pulling in the proper tarball with the python3 symlink in it::

Arduino/package/package_esp8266com_index.template.json

Lines 178 to 184 in 5ca0bde

	{
	"host": "x86_64-apple-darwin",
	"url": "https://github.com/earlephilhower/esp-quick-toolchain/releases/download/2.5.0-4/python3-placeholder.tar.gz",
	"archiveFileName": "python3-placeholder.tar.gz",
	"checksum": "SHA-256:d8cf9d9d66423d7b90978ebe285a73a6e8611995cd0d5e6273e929a0cf2c9850",
	"size": "191"
	},

The referenced tarball does expand to the proper spot. so it looks like our install is somehow taking the tools dir from somewhere else. Odd, not sure what in the Arduino IDE could call that.

[dimitre]

I've now installed a clean Arduino, erased the ~/Library/Application Support/Arduino15 folder and installed again, same results

here is my installation script

#/bin/bash
cd /Applications/Arduino.app/Contents/Java/hardware
mkdir esp8266com
cd esp8266com
git clone https://github.com/esp8266/Arduino.git esp8266
cd esp8266
git submodule update --init

cd tools
python get.py

and the output

Cloning into 'esp8266'...
remote: Enumerating objects: 33, done.
remote: Counting objects: 100% (33/33), done.
remote: Compressing objects: 100% (28/28), done.
remote: Total 23667 (delta 6), reused 7 (delta 3), pack-reused 23634
Receiving objects: 100% (23667/23667), 81.72 MiB | 1.22 MiB/s, done.
Resolving deltas: 100% (14413/14413), done.
Submodule 'libraries/ESP8266SdFat' (https://github.com/earlephilhower/ESP8266SdFat.git) registered for path 'libraries/ESP8266SdFat'
Submodule 'libraries/LittleFS/lib/littlefs' (https://github.com/ARMmbed/littlefs.git) registered for path 'libraries/LittleFS/lib/littlefs'
Submodule 'libraries/SoftwareSerial' (https://github.com/plerup/espsoftwareserial.git) registered for path 'libraries/SoftwareSerial'
Submodule 'tools/esptool' (https://github.com/espressif/esptool.git) registered for path 'tools/esptool'
Submodule 'tools/pyserial' (https://github.com/pyserial/pyserial.git) registered for path 'tools/pyserial'
Submodule 'lwip2' (https://github.com/d-a-v/esp82xx-nonos-linklayer.git) registered for path 'tools/sdk/lwip2/builder'
Submodule 'tools/sdk/ssl/bearssl' (https://github.com/earlephilhower/bearssl-esp8266.git) registered for path 'tools/sdk/ssl/bearssl'
Cloning into '/Applications/Arduino.app/Contents/Java/hardware/esp8266com/esp8266/libraries/ESP8266SdFat'...
Cloning into '/Applications/Arduino.app/Contents/Java/hardware/esp8266com/esp8266/libraries/SoftwareSerial'...
Cloning into '/Applications/Arduino.app/Contents/Java/hardware/esp8266com/esp8266/tools/sdk/lwip2/builder'...
Cloning into '/Applications/Arduino.app/Contents/Java/hardware/esp8266com/esp8266/libraries/LittleFS/lib/littlefs'...
Cloning into '/Applications/Arduino.app/Contents/Java/hardware/esp8266com/esp8266/tools/pyserial'...
Cloning into '/Applications/Arduino.app/Contents/Java/hardware/esp8266com/esp8266/tools/sdk/ssl/bearssl'...
Cloning into '/Applications/Arduino.app/Contents/Java/hardware/esp8266com/esp8266/tools/esptool'...
Submodule path 'libraries/ESP8266SdFat': checked out 'af4ed0c5ec3084cb3883df51ec2052791ca2bff2'
Submodule path 'libraries/LittleFS/lib/littlefs': checked out 'abd90cb84c818a663b584575b019258d01d0065e'
Submodule path 'libraries/SoftwareSerial': checked out 'dd9d2326f85a22ca96ef5922df044884a2f3b529'
Submodule path 'tools/esptool': checked out '9ad444a6e06e58833d5e6044c1d5f3eb3dd56023'
Submodule path 'tools/pyserial': checked out 'c54c81d933b847458d465cd77e96cd702ff2e7be'
Submodule path 'tools/sdk/lwip2/builder': checked out 'ffa962483cc1c5d874b11bec13080359619c4cb2'
Submodule path 'tools/sdk/ssl/bearssl': checked out '89454af34e3e61ddfc9837f3da5a0bc8ed44c3aa'
Platform: x86_64-apple-darwin
Downloading python3-placeholder.tar.gz
Done
Extracting dist/python3-placeholder.tar.gz
Downloading x86_64-apple-darwin14.xtensa-lx106-elf-b40a506.1563313032.tar.gz
Done
Extracting dist/x86_64-apple-darwin14.xtensa-lx106-elf-b40a506.1563313032.tar.gz
Downloading x86_64-apple-darwin14.mkspiffs-7fefeac.1563313032.tar.gz
Done
Extracting dist/x86_64-apple-darwin14.mkspiffs-7fefeac.1563313032.tar.gz
Downloading x86_64-apple-darwin14.mklittlefs-7f77f2b.1563313032.tar.gz
Done
Extracting dist/x86_64-apple-darwin14.mklittlefs-7f77f2b.1563313032.tar.gz

Still the same result there

My System Python is installed via homebrew
so the command

brew list python 

returns

/usr/local/Cellar/python/3.7.4/bin/2to3
/usr/local/Cellar/python/3.7.4/bin/2to3-3.7
/usr/local/Cellar/python/3.7.4/bin/easy_install-3.7
/usr/local/Cellar/python/3.7.4/bin/idle3
/usr/local/Cellar/python/3.7.4/bin/idle3.7
/usr/local/Cellar/python/3.7.4/bin/pip3
/usr/local/Cellar/python/3.7.4/bin/pip3.7
/usr/local/Cellar/python/3.7.4/bin/pydoc3
/usr/local/Cellar/python/3.7.4/bin/pydoc3.7
/usr/local/Cellar/python/3.7.4/bin/python3
/usr/local/Cellar/python/3.7.4/bin/python3-config
/usr/local/Cellar/python/3.7.4/bin/python3.7
/usr/local/Cellar/python/3.7.4/bin/python3.7-config
/usr/local/Cellar/python/3.7.4/bin/python3.7m
/usr/local/Cellar/python/3.7.4/bin/python3.7m-config
/usr/local/Cellar/python/3.7.4/bin/pyvenv
/usr/local/Cellar/python/3.7.4/bin/pyvenv-3.7
/usr/local/Cellar/python/3.7.4/bin/wheel3
/usr/local/Cellar/python/3.7.4/Frameworks/Python.framework/ (2779 files)
/usr/local/Cellar/python/3.7.4/IDLE 3.app/Contents/ (8 files)
/usr/local/Cellar/python/3.7.4/lib/pkgconfig/ (3 files)
/usr/local/Cellar/python/3.7.4/libexec/bin/ (7 files)
/usr/local/Cellar/python/3.7.4/libexec/pip/ (699 files)
/usr/local/Cellar/python/3.7.4/libexec/setuptools/ (346 files)
/usr/local/Cellar/python/3.7.4/libexec/wheel/ (38 files)
/usr/local/Cellar/python/3.7.4/Python Launcher 3.app/Contents/ (16 files)
/usr/local/Cellar/python/3.7.4/share/man/ (2 files)

system python command invoke 2.7.10

python
Python 2.7.10 (default, Feb 22 2019, 21:55:15) 
[GCC 4.2.1 Compatible Apple LLVM 10.0.1 (clang-1001.0.37.14)] on darwin

and python3 command invokes 3.7.4

Python 3.7.4 (default, Jul  9 2019, 18:13:23) 
[Clang 10.0.1 (clang-1001.0.46.4)] on darwin

[earlephilhower]

Thanks for the detailed dump. The key is the python3-placeholder tarball.

Basically the core, to support Windows and Linux, doesn't try and fork python3, what it uses is tools/python3/python3 which under Linux-like OSes is a symlink to /usr/bin/python3 but under Windows is the actual local full python3 executable standalone installation.

I have a hunch that the python3 symlink used is pointing to the wrong spot for Mac. Can you report what which python3 outputs and then ls -l tools/python3 (from wherever the install was) and see if the symlink is broken? You should see the following:

earle@server:~/Arduino/hardware/esp8266com/esp8266$ ls -l tools/python3
total 0
-rw-r--r-- 1 earle earle  0 Aug  3 11:41 placeholder_for_arduino
lrwxrwxrwx 1 earle earle 16 Aug 30 08:08 python3 -> /usr/bin/python3

Or, it could be that untarring symlinks does not work on MacOS and we need to use a wrapper instead.

[earlephilhower]

And to short-circuit things, does /usr/bin/env python3 --version run properly and give you the Python 3 version on your MacOS?

earle@server:~/Arduino/hardware/esp8266com/esp8266$ /usr/bin/env python3 --version
Python 3.6.8

If that's good, then a wrapper script could work here (and under any Linux-like OS)

[dimitre]

/usr/bin/env python3 --version
Python 3.7.4

[earlephilhower]

Great, and the other info to see what's actually been installed? #6490 (comment)

[dimitre]

ls -l tools/python3
ls: tools/python3: No such file or directory

which python3
/usr/local/bin/python3

I'm online now and following here

[earlephilhower]

What exactly is in your tools directory, then? ls -l tools from inside the Arduino git clone directory...

[dimitre]

ok sorry here it goes in the right directory

total 264
-rwxr-xr-x   1 z  staff  70784 Sep  7 12:14 boards.txt.py
-rwxr-xr-x   1 z  staff   6107 Sep  7 12:14 build.py
drwxr-xr-x   6 z  staff    192 Sep  7 12:15 dist
-rwxr-xr-x   1 z  staff   5357 Sep  7 12:14 elf2bin.py
-rwxr-xr-x   1 z  staff   9372 Sep  7 12:14 espota.py
drwxr-xr-x  20 z  staff    640 Sep  7 12:15 esptool
-rwxr-xr-x   1 z  staff   4215 Sep  7 12:14 get.py
-rwxr-xr-x   1 z  staff   2451 Sep  7 12:14 makecorever.py
drwxr-xr-x   3 z  staff     96 Jul 16 18:37 mklittlefs
drwxr-xr-x   3 z  staff     96 Jul 16 18:37 mkspiffs
-rw-r--r--   1 z  staff   9754 Sep  7 12:14 platformio-build.py
drwxr-xr-x  17 z  staff    544 Sep  7 12:15 pyserial
drwxr-xr-x   4 z  staff    128 Aug  3 15:41 python3
drwxr-xr-x  10 z  staff    320 Sep  7 12:14 sdk
-rwxr-xr-x   1 z  staff   4025 Sep  7 12:14 signing.py
-rwxr-xr-x   1 z  staff   1471 Sep  7 12:14 upload.py
drwxr-xr-x   8 z  staff    256 Jul 16 19:01 xtensa-lx106-elf

[dimitre]

autechre:esp8266 z$ ls -l tools/python3
total 0
-rw-r--r--  1 z  staff   0 Aug  3 15:41 placeholder_for_arduino
lrwxr-xr-x  1 z  staff  16 Sep  7 12:15 python3 -> /usr/bin/python3

[earlephilhower]

Ah, there's the problem. OSX doesn't have a global python3, only a /usr/localone! So the symlink is there but pointing to the wrong spot. A simple fix is to make a custom macosx python symlink tarball in the release, pointing to /usr/local/bin/python.

You can do the same, simply cd into the tools/python3 dir, rm -f python3 and ln -s /usr/local/bin/python3 python3.

[dimitre]

Worked. thank you!
it may be homebrew fault.
homebrew is the unofficial but widely used package manager for macOS.
Count on me if you plan to support macOS with brew I can do the testing and reinstall

[earlephilhower]

You can give #6492 a try. It adds a real OSX sketch compile to the CI and fixes the symlink. It's basically what you've already done.

[dimitre]

Good. It is working here
I've cleaned up Arduino and installed using the following

#/bin/bash
cd /Applications/Arduino.app/Contents/Java/hardware
mkdir esp8266com
cd esp8266com
git clone https://github.com/esp8266/Arduino.git esp8266
cd esp8266
git fetch origin +refs/pull/6492/merge
git checkout FETCH_HEAD

git submodule update --init

cd tools
python get.py

[earlephilhower]

Thanks for the fast feedback and confirmation!

[oscar2211]

I was having the same problem, the way I fixed it was by following the following instructions from adafruit:
So, I’m getting a 'no such file' error compiling for ESP8266 on my Mac If your error message looks like this:
fork/exec /Users/xxxxxxx/Library/Arduino15/packages/esp8266/tools/xtensa-lx106-elf-gcc/1.20.0-26-gb404fb9- 2/bin/xtensa-lx106-elf-g++: no such file or directory
Error compiling.
To fix this problem, do this:

1. OpentheBoardsManagerintheArduinoIDE
2. UninstalltheESP8266support
3. gotoyourLIbraryfolder(intheFinder,select"Go::Gotofolder:,andenterLibrary).FindthefolderArduino15.
4. IntheArduino15folder,gointopackages,anddeletethefolderesp8266
5. GobacktotheArduinoIDE,andinstallESP8266boardsupport.
6. NowgobacktotheFinder,andcheckthatyouhavethextensa-lx106-elf-g++fileinthe
   path Arduino15/packages/esp8266/tools/xtensa-lx106-elf-gcc/1.20.0-26-gb404fb9-2/bin/xtensa-lx106-elf-g++
7. That'sit!

In addition to this, when installing the board manager, I installed version 2.5.2, NOT the latest version. The latest version continued giving me the same error.

Hope this helps.

[dperlman]

Hate to do this, but I think this "fix" is actually a bug.
Mac OS X doesn't install "command line tools" at all by default. So most "real" software doesn't work at all until you install the command line tools using xcode. If you try running commands in the terminal and get errors referring to "xcrun" then you can force the OS to install the command line tools by running:
xcode-select --install
The process is a bit janky but it will work.
Once you have the command line tools installed, python3 will be exactly where it's supposed to be: /usr/bin/python3.

The "fix" to the symlink described here is actually creating an undocumented dependency on homebrew. That's the only reason dimitre had python3 in /usr/local/bin. While homebrew is certainly popular, I don't think it's a good idea to assume the user has it set up. In fact, much of this stuff won't work without xcode command line tools installed anyway so once that's done, there's no need to rely on custom installations anyway.

For those who are still getting the python3 error, the solution is either to go back to an earlier release of these tools (I think 2.5.2 maybe will precede the homebrew dependency) or else to fix the symlink yourself manually (the path in question is listed in the error message itself).

[bdwilson]

xcode-select --install
The process is a bit janky but it will work.
Once you have the command line tools installed, python3 will be exactly where it's supposed to be: /usr/bin/python3.

This is not true on Mojave 10.14.6. Still trying to get things working there.

[dimitre]

Just to update things here : I've recently updated everything and it claim again it is missing python3. Again solved by making a symlink for python3 installation in tools/python3 folder so it is all good now.
but I think #6492 didn't fix it.
Cheers

[earlephilhower]

Please make sure you run the get.py in tools to get the updated python3 tarball. No symlinks are needed anymore, but if you did not get the new tools after updating, you'll be stuck with the older python. tarball which did have a symlink.

[bdwilson]

Please make sure you run the get.py in tools to get the updated python3 tarball. No symlinks are needed anymore, but if you did not get the new tools after updating, you'll be stuck with the older python. tarball which did have a symlink.

get.py didn't work for me. brew install python did.

[dimitre]

@earlephilhower I did it already, here is my install script

#/bin/bash
cd /Applications/Arduino.app/Contents/Java/hardware
mkdir esp8266com
cd esp8266com
git clone https://github.com/esp8266/Arduino.git esp8266 --depth 1
cd esp8266
git submodule update --init

cd tools
python get.py

@bdwilson python3 installed via homebrew ok, and xcode-install already installed

[earlephilhower]

Thanks for the info, @dimitre . Does xcode not include python3 in the standard path?

Basically, with the latest patch tools/python3/python3 is a simple script that calls python3 using the OS's standard registry (think it's actually a POSIX thing)...

#!/usr/bin/env python3

So, if python3 is registered with the OS for general use, the OS itself (well, /usr/bin/env) will find it.

What happens from a standard terminal when you run: /usr/bin/env python3 ?

Also, can you cat the tools/python3/python3 file and make sure the tarball got expanded (no issues, maybe. overwriting old symlinks)?

[earlephilhower]

Oh, @bdwilson , you need to install the real python3 yourself. Looks like Mac's getting dumbed down and not including any Python interpreter in later releases (whilst Windows is going to support Linux...it's an odd world). With your comment, it seems that Homebrew does install python3 and register it properly for the new /usr/bin/env python3 to find it.