upgrade Ethernet library to pjrc Version 2.0.0

[wolfgangr]

There is a new Ethernet library been developed by Paul Stoffregen
https://www.pjrc.com/arduino-ethernet-library-2-0-0/
which has already found its way into arduino mainstream:
https://github.com/arduino-libraries/Ethernet/blob/master/library.properties

Here at esp8266/Arduino, current master is still at "version=1.0.4"
https://github.com/esp8266/Arduino/blob/master/libraries/Ethernet/library.properties

In his linked article, Paul reports succesful testing on a Adafruit Huzzah ESP8266 Feather , so I hoped, this lib would be copmatible or even included in esp8266/Arduino.

among other, I'd desperately use Ethernet.hardwareStatus() and Ethernet.linkStatus() to debug my network test environment.

Basic Infos

• This issue complies with the issue POLICY doc.
• I have read the documentation at readthedocs and the issue is not addressed there.
  Search for "Ethernet" does not macht any document
• I have tested that the issue is present in current master branch (aka latest git).
• I have searched the issue tracker for a similar issue.
• If there is a stack dump, I have decoded it.
  no stack dump
• I have filled out all fields below.

Platform

• Hardware: [ESP-12] + WIZNET W5100
• Core Version: [2.5.0-beta2] and [2.4.2]
• Development Env: [Arduino IDE] ARDUINO 1.8.8 2018.12.07
• Operating System: [debian 9.3]

Settings in IDE

• Module: [Generic ESP8266] and |Nodemcu 0.9]
• Flash Mode: N/A
• Flash Size: [4MB] no SPIFFS
• lwip Variant: [v2 Lower Memory]
• Reset Method: N/A
• Flash Frequency: N/A
• CPU Frequency: [80Mhz]
• Upload Using: [SERIAL]
• Upload Speed: [115200]

Problem Description

I try to include wired networking into the tasmota Project. arendst/Tasmota#4936

My test setup consists of a NODEMCU 0.9 with a generic china sourced W5100.
(Yes, I got those to work with AVR MCU's, and tried different modules....)
When I try the DHCPAdressPrinter example, I don't get any print at all.

I see the initial stage of DHCP handshake in my router log, but not the DHCPREQUEST / DHCPACK of a succesful lease assignment:

Jan 16 01:21:34 spider daemon.info dnsmasq-dhcp[1862]: DHCPDISCOVER(br-dmz) 00:aa:bb:cc:de:02 
Jan 16 01:21:34 spider daemon.info dnsmasq-dhcp[1862]: DHCPOFFER(br-dmz) 192.168.1.88 00:aa:bb:cc:de:02 

I concluded that either my net module does only transmit sends, not recieve the answer, or problems buried somewhere deep in the stack. More debug would be great at that point.

Googling the issue, I found PJRC's library 2.0.0.
Scrutinizing my Arduino (good exercise :-), anyway....) revelaed that this was already included in mainstream arduino, but not yet in the ESP8266 platform.
So I tried to add it manually.

When I override the Ethernet lib shipped with esp8266/Arduino with the new 2-0-0 Ethernet lib, I get loads of error: invalid abstract return type for member function 'EthernetClient EthernetServer::available()'.
So obvioulsy there is more to change in the ESP8266 interface then just dropping in this library.

Would be great to get it to work, however. Among else, it promises extended hardware and link state debugging. Im trying at the DNSAddressPrinter example. I see DHCPDISCOVER and DHCPOFFER in my router log, but communication seems to break down then.

I could not manage to test 2-0-0 Ethernet libs using the Arduino IDE.
So I did a forceful override:

~/.arduino15/packages/esp8266/hardware/esp8266/2.5.0-beta2/libraries$ rm -rf Ethernet/
:~/.arduino15/packages/esp8266/hardware/esp8266/2.5.0-beta2/libraries$ ln -s ~/Arduino/libraries/Ethernet-2.0.0/ ./Ethernet

Debug Messages

Full error message:

Arduino: 1.8.8 (Linux), Board: "NodeMCU 0.9 (ESP-12 Module), 80 MHz, Flash, Enabled, 4M (no SPIFFS), v2 Lower Memory, Disabled, None, Only Sketch, 115200"

In file included from /home/wrosner/Arduino/libraries/Ethernet-2.0.0/src/EthernetServer.h:3:0,
                 from /home/wrosner/test/ESP8266/Ethernet-2.0.0/examples/DhcpAddressPrinter/DhcpAddressPrinter.ino:1:
/home/wrosner/Arduino/libraries/Ethernet-2.0.0/src/Ethernet.h:258:17: error: invalid abstract return type for member function 'EthernetClient EthernetServer::available()'
  EthernetClient available();
                 ^
/home/wrosner/Arduino/libraries/Ethernet-2.0.0/src/Ethernet.h:214:7: note:   because the following virtual functions are pure within 'EthernetClient':
 class EthernetClient : public Client {
       ^
In file included from /home/wrosner/Arduino/libraries/Ethernet-2.0.0/src/Ethernet.h:52:0,
                 from /home/wrosner/Arduino/libraries/Ethernet-2.0.0/src/EthernetServer.h:3,
                 from /home/wrosner/test/ESP8266/Ethernet-2.0.0/examples/DhcpAddressPrinter/DhcpAddressPrinter.ino:1:
Multiple libraries were found for "EthernetServer.h"
 Used: /home/wrosner/Arduino/libraries/Ethernet-2.0.0
 Not used: /home/wrosner/.arduino15/packages/esp8266/hardware/esp8266/2.5.0-beta2/libraries/Ethernet
/home/wrosner/.arduino15/packages/esp8266/hardware/esp8266/2.5.0-beta2/cores/esp8266/Client.h:29:21: note: 	virtual int Client::connect(const IPAddress&, uint16_t)
 Not used: /home/wrosner/test/arduino/arduino-1.8.8/arduino-1.8.8/libraries/Ethernet
 Not used: /home/wrosner/.arduino15/packages/esp8266/hardware/esp8266/2.5.0-beta2/libraries/Ethernet
 Not used: /home/wrosner/test/arduino/arduino-1.8.8/arduino-1.8.8/libraries/Ethernet
 Not used: /home/wrosner/.arduino15/packages/esp8266/hardware/esp8266/2.5.0-beta2/libraries/Ethernet
 Not used: /home/wrosner/test/arduino/arduino-1.8.8/arduino-1.8.8/libraries/Ethernet
 Not used: /home/wrosner/.arduino15/packages/esp8266/hardware/esp8266/2.5.0-beta2/libraries/Ethernet
 Not used: /home/wrosner/test/arduino/arduino-1.8.8/arduino-1.8.8/libraries/Ethernet
         virtual int connect(CONST IPAddress& ip, uint16_t port) =0;
                     ^
/home/wrosner/.arduino15/packages/esp8266/hardware/esp8266/2.5.0-beta2/cores/esp8266/Client.h:37:22: note: 	virtual bool Client::flush(unsigned int)
         virtual bool flush(unsigned int maxWaitMs = 0) = 0;
                      ^
/home/wrosner/.arduino15/packages/esp8266/hardware/esp8266/2.5.0-beta2/cores/esp8266/Client.h:38:22: note: 	virtual bool Client::stop(unsigned int)
         virtual bool stop(unsigned int maxWaitMs = 0) = 0;
                      ^
In file included from /home/wrosner/Arduino/libraries/Ethernet-2.0.0/src/EthernetServer.h:3:0,
                 from /home/wrosner/test/ESP8266/Ethernet-2.0.0/examples/DhcpAddressPrinter/DhcpAddressPrinter.ino:1:
/home/wrosner/Arduino/libraries/Ethernet-2.0.0/src/Ethernet.h:259:17: error: invalid abstract return type for member function 'EthernetClient EthernetServer::accept()'
  EthernetClient accept();
                 ^
/home/wrosner/Arduino/libraries/Ethernet-2.0.0/src/Ethernet.h:214:7: note:   since type 'EthernetClient' has pure virtual functions
 class EthernetClient : public Client {
       ^
/home/wrosner/Arduino/libraries/Ethernet-2.0.0/src/Ethernet.h:298:14: error: cannot declare field 'DhcpClass::_dhcpUdpSocket' to be of abstract type 'EthernetUDP'
  EthernetUDP _dhcpUdpSocket;
              ^
In file included from /home/wrosner/Arduino/libraries/Ethernet-2.0.0/src/EthernetServer.h:3:0,
                 from /home/wrosner/test/ESP8266/Ethernet-2.0.0/examples/DhcpAddressPrinter/DhcpAddressPrinter.ino:1:
/home/wrosner/Arduino/libraries/Ethernet-2.0.0/src/Ethernet.h:152:7: note:   because the following virtual functions are pure within 'EthernetUDP':
 class EthernetUDP : public UDP {
       ^
In file included from /home/wrosner/Arduino/libraries/Ethernet-2.0.0/src/Ethernet.h:54:0,
                 from /home/wrosner/Arduino/libraries/Ethernet-2.0.0/src/EthernetServer.h:3,
                 from /home/wrosner/test/ESP8266/Ethernet-2.0.0/examples/DhcpAddressPrinter/DhcpAddressPrinter.ino:1:
/home/wrosner/.arduino15/packages/esp8266/hardware/esp8266/2.5.0-beta2/cores/esp8266/Udp.h:81:27: note: 	virtual IPAddress UDP::remoteIP() const
         virtual IPAddress remoteIP() const =0;
                           ^
/home/wrosner/.arduino15/packages/esp8266/hardware/esp8266/2.5.0-beta2/cores/esp8266/Udp.h:83:26: note: 	virtual uint16_t UDP::remotePort() const
         virtual uint16_t remotePort() const =0;
                          ^
In file included from /home/wrosner/test/ESP8266/Ethernet-2.0.0/examples/DhcpAddressPrinter/DhcpAddressPrinter.ino:5:0:
/home/wrosner/Arduino/libraries/Ethernet-2.0.0/src/Dns.h:37:14: error: cannot declare field 'DNSClient::iUdp' to be of abstract type 'EthernetUDP'
  EthernetUDP iUdp;
              ^
In file included from /home/wrosner/Arduino/libraries/Ethernet-2.0.0/src/EthernetServer.h:3:0,
                 from /home/wrosner/test/ESP8266/Ethernet-2.0.0/examples/DhcpAddressPrinter/DhcpAddressPrinter.ino:1:
/home/wrosner/Arduino/libraries/Ethernet-2.0.0/src/Ethernet.h:152:7: note:   since type 'EthernetUDP' has pure virtual functions
 class EthernetUDP : public UDP {
       ^
exit status 1
Error compiling for board NodeMCU 0.9 (ESP-12 Module).

This report would have more information with
"Show verbose output during compilation"
option enabled in File -> Preferences.

MCVE Sketch

Nothing of my own.
I used the stock DhcpAddressPrinter Example, as shipped with current Arduino IDE, e.g. here
https://github.com/arduino-libraries/Ethernet/blob/master/examples/DhcpAddressPrinter/DhcpAddressPrinter.ino
I just changed the serial speed to 115200

It builds without issues for AVR platforms.

#include <EthernetServer.h>
#include <EthernetUdp.h>
#include <Dhcp.h>
#include <Ethernet.h>
#include <Dns.h>
#include <EthernetClient.h>

#include <EthernetServer.h>
#include <EthernetUdp.h>
#include <Dhcp.h>
#include <Ethernet.h>
#include <Dns.h>
#include <EthernetClient.h>

/*
  DHCP-based IP printer

  This sketch uses the DHCP extensions to the Ethernet library
  to get an IP address via DHCP and print the address obtained.
  using an Arduino Wiznet Ethernet shield.

  Circuit:
   Ethernet shield attached to pins 10, 11, 12, 13

  created 12 April 2011
  modified 9 Apr 2012
  by Tom Igoe
  modified 02 Sept 2015
  by Arturo Guadalupi

 */

#include <SPI.h>
#include <Ethernet.h>

// Enter a MAC address for your controller below.
// Newer Ethernet shields have a MAC address printed on a sticker on the shield
byte mac[] = {
  0x00, 0xAA, 0xBB, 0xCC, 0xDE, 0x02
};

void setup() {
  // You can use Ethernet.init(pin) to configure the CS pin
  //Ethernet.init(10);  // Most Arduino shields
  //Ethernet.init(5);   // MKR ETH shield
  //Ethernet.init(0);   // Teensy 2.0
  //Ethernet.init(20);  // Teensy++ 2.0
  //Ethernet.init(15);  // ESP8266 with Adafruit Featherwing Ethernet
  //Ethernet.init(33);  // ESP32 with Adafruit Featherwing Ethernet

  // Open serial communications and wait for port to open:
  Serial.begin(115200);
  while (!Serial) {
    ; // wait for serial port to connect. Needed for native USB port only
  }

  // start the Ethernet connection:
  Serial.println("Initialize Ethernet with DHCP:");
  if (Ethernet.begin(mac) == 0) {
    Serial.println("Failed to configure Ethernet using DHCP");
    if (Ethernet.hardwareStatus() == EthernetNoHardware) {
      Serial.println("Ethernet shield was not found.  Sorry, can't run without hardware. :(");
    } else if (Ethernet.linkStatus() == LinkOFF) {
      Serial.println("Ethernet cable is not connected.");
    }
    // no point in carrying on, so do nothing forevermore:
    while (true) {
      delay(1);
    }
  }
  // print your local IP address:
  Serial.print("My IP address: ");
  Serial.println(Ethernet.localIP());
}

void loop() {
  switch (Ethernet.maintain()) {
    case 1:
      //renewed fail
      Serial.println("Error: renewed fail");
      break;

    case 2:
      //renewed success
      Serial.println("Renewed success");
      //print your local IP address:
      Serial.print("My IP address: ");
      Serial.println(Ethernet.localIP());
      break;

    case 3:
      //rebind fail
      Serial.println("Error: rebind fail");
      break;

    case 4:
      //rebind success
      Serial.println("Rebind success");
      //print your local IP address:
      Serial.print("My IP address: ");
      Serial.println(Ethernet.localIP());
      break;

    default:
      //nothing happened
      break;
  }
}

[d-a-v]

@wolfgangr

You can add in Ethernet.h around L220

virtual bool flush(unsigned int maxWaitMs = 0) { (void)maxWaitMs; return flush(); }
virtual bool stop(unsigned int maxWaitMs = 0) { (void)maxWaitMs; return stop(); }

and

virtual int connect(const IPAddress& ip, uint16_t port) { return connect((IPAddress)ip, port) };

Compilation error or infinite loop may happen with this one, better change connect() signature to match this one instead (in .h and .cc)

virtual IPAddress remoteIP() const;
virtual IPAddress remotePort() const;

You may add const like above in Ethernet.h L239,L240

@PaulStoffregen What version of this arduino core did you use for your testings ?

We could also replace the old version with Paul's updated one, but it's better if we could
use it as a submodule like we did with SoftwareSerial.

edit: that was you :) d-a-v/W5500lwIP#4

[PaulStoffregen]

I am sad to report the testing I did last summer had DNS lookup fail on ESP8266. Here's a blog article I wrote at the time. Scroll to the end for test data.

https://www.pjrc.com/arduino-ethernet-library-2-0-0/

[wolfgangr]

@d-a-v

edit: that was you :) d-a-v/W5500lwIP#4

yepp. small world, eh :)...
Just wanted to test basic functionality, before I dived into your testing instructions - and got stuck even at that basic level...

@PaulStoffregen

I am sad to report the testing I did last summer had DNS lookup fail on ESP8266.

Ashes on my head: I have linked your article but not read it to the bottom?
So I'll try my test with a fixed IP?
But it does not even build...
So, if you just could point me to some git to ESP8266-arduino where you built your tests aginst, I might be happy.

Some remarks on the wider view angle

In your blog, you quest for support from the ESP community.

So, I assume, there is no easy solution at hand yet.
This begs the question whether it is worth the effort to fix it.

Looks like just another case of ordered network configuration is conflicting with opaque ESP8266 core .
ESP8266 always has WiFi on it, and even if I look for ESP32, modules without WiFi sell at a multifold price than the ones with WiFi included.

While I think it is safe to say that WiFi sticks to ESP (both ...8266 and ...32) like s...orry...,
not all use cases do so.

I see different constellations if it comes to ethernet on the ESP8266:

1. switch the onboard WiFi off, when Ethernet is active
2. independent but coexistent implementation of WiFI and Ethernet
3. integrate WiFi and Ethernet in a single network stack
4. rewrite the whole ESP core to overcome its quirks

regarding 4.)
While a real FOSS ESP core replacement would be the dream scenario, I don't feel elected to contribute, not even comment anything on that.

regarding 1.)
Disabling WiFi when I need ethernet would be OK for my test purpose currently at hand.

It also might fulfill the need e.g. of many tasmota and other IoT framework use cases:
Stay with the same platform and if possible with the same module, no matter whether a thingie (e.g. sensor interface, realy, button,....) is connected by wire or by thin WiFi air.

Of course it complicates code maintenance of such projects, that may end up with loads of #ifdefineds, and spoils the on-binary-for-all-purpose approach of e.g. tasmota.

Maybe, this might be implemented by a complete seperate fork of esp8266, which disables the onboard WiFi and just uses the MCU core.
Maybe this even could be designed without any proprietary ESP code at all?
Is something like that flowing around?

regarding 2.)
Use case examples of coexisting but independent WiFi and Ethernet:

• Have tasmota running it's web GUI over WiFI and MQTT over Ethernet
• running debug, config or whatsoever over one and the app on the other
• small proxy-like thinges operating at higher OSI layers, where the app has two different stacks open
  Are those cases really worth the effort?

regarding 3.)
WiFi and Ethernet in a single stack
The app does not care whether the core delivers its package by wire or by air.
I think this is what @d-a-v aims at with his lwip integration - correct?
Pls correct my when I'm wrong, but I think this is not possible with the Arduino ethernet Module, OK?
If it were possible to have ethernet in lwip, use cases 1.) and 2.) could be satisfied, too.

So, to make the point:
I opened this issue because I thought there was an easy solution around, that had just been overlooked.
If there is much work to do, I'd prefer to invest this in the lwip based architecture.

[d-a-v]

regarding 3.)
WiFi and Ethernet in a single stack
The app does not care whether the core delivers its package by wire or by air.
I think this is what @d-a-v aims at with his lwip integration - correct?

correct.

We could however propose a PR to make Paul's library compatible with esp8266 arduino core if he's opened with that. Small changes I guess, even for DNS not working. After that, his library could be submoduled here and replace the old Ethernet library.
For the sake of sketch portability without effort, the arduino way.

[PaulStoffregen]

I'm currently very busy with a hardware project that's far behind schedule, so limited time I can spent right now.

I can tell you Ethernet 2.0.0 (on Arduino's repository) works on all other boards and it uses only the SPI library to access hardware.

[PaulStoffregen]

So, if you just could point me to some git to ESP8266-arduino where you built your tests aginst, I might be happy.

Sorry, I've deleted and reinstalled Arduino many times since the summer, even restarted new preferences.txt files several times (partly working on a bug added in Arduino 1.8.8). ESP8266 is no longer in the boards manager at all on my machine.

There is a lingering copy if the esp8266 package on my machine with this pathname: ~/.arduino15/packages/esp8266/hardware/esp8266/2.4.1

[PaulStoffregen]

On this part:

We could however propose a PR to make Paul's library compatible with esp8266 arduino core if he's opened with that.

I'm no longer maintaining a separate fork of the Ethernet library. So what you're talking about is a patch to Ethernet 2.0.0 on Arduino's repository, meant for all Arduino compatible boards.

Adding more called to yield() in places that wait would probably be a very welcome patch. Except in the optimized chip select pin stuff, very ESP specific stuff probably isn't appropriate in Arduino's repository. The Wiznet chip is accessed using the SPI library, so in theory all you should need to do is make sure ESP8266 has a SPI library which follows Arduino's specs.

[JAndrassy]

esp8266 developers should not change the Arduino API base classes Client, Server, UDP

#5749

[Jason2866]

Any news for integrating wired ethernet?

[d-a-v]

Once arduino-libraries/Ethernet#96 is merged, we can set it up here as a submodule.

Another version for ethernet is also coming from my ethernet + ppp drivers branch, some days are still needed to finish this up.

[Smetanski]

Hi guys, what is going on with Ethernet lib integration ??

[d-a-v]

@Smetanski you can either try #6812 or try the 0.0.2 alpha release.
Please report whether it's working with you.

[devyte]

Or try #6680 which has a different Ethernet integrated with lwip and the wifi interfaces.

[d-a-v]

#6812 is merged