nghttp2/nghttp2.h: No such file or directory

[anteph]

Hi,

I'm trying to do some experiments with HTTP2 but I'm running into some troubles. From what I've been reading, we can use nghttp2, which is already included in IDF, for HTTP2 requests.

Nonetheless, I cannot seem to include the lib in the Arduino code. I've already pulled the latest version of the Arduino core and NGHTTP2 is indeed present in tools/sdk/includes.

I'm trying to include the lib pretty much like is done on the asio_http2.h file, with no luck so far:
#include <nghttp2/nghttp2.h>

It's probably something silly, but I really cannot make it work.

Additionally, if I try to include the nghttp2/asio_http2_client.h (I haven't yet analyzed in detail but seems to be a wrapper that maybe will be handy?) I also get an error in this include:

fatal error: boost/system/error_code.hpp: No such file or directory

Thank you very much,
Nuno Santos

Hardware:

Board: ESP32 Dev Module
Core Installation/update date: 13/11/2018
IDE name: Arduino IDE v1.8.5
Computer OS: Windows 8.1

Sketch:

#include <nghttp2/nghttp2.h>

void setup() {}

  
void loop() {}

[lbernstone]

I don't have any tips on how to use it, but the esp32 port seems to pack all the functionality into http_parser.h
#include <http_parser.h>

[chegewara]

There is no nghttp2.h in tools/sdk/include path.

[anteph]

Hi! Thanks for the feedback :)

@lbernstone thanks for pointing it. I'll take a look. But for now I was trying to somehow import the nghttp2.h. IDF has a wrapper called sh2lib.h that seemed very simple to use, but it imports the nghttp2.h.

@chegewara But in the asio_http2.h file, it is importing the nghttp2.h with the following include:
#include <nghttp2/nghttp2.h>

Shouldn't we also be able to import it, in case we want to use the nghttp2.h functionalities directly?

Best regards,
Nuno Santos

[chegewara]

Ok, i see the point. But now there we have something like that in esp-idf:
https://github.com/espressif/esp-idf/tree/master/components/nghttp

As you can see there is external library nghttp2. I dont know arduino very good, but i can see this library needs to be included in arduino-esp32 or somehow added into library.

[anteph]

I'm also not sure how these lower level includes work. I was thinking that, since there's a reference to the nghttp2.h lib, that it was already somehow included in the Arduino core and we could directly use it.

One thing I did find was a libnghttp.a in tools/sdk/lib, although I'm not sure if this is a different lib for HTTP 1 or if it is effectively the nghttp2 lib.

Hopefully this has a solution, would really like to run some tests with HTTP2 :)

[anteph]

It seems like the .a for the nghttp2 is indeed available in the Arduino core and what is missing is a .h file in the sdk folder so we can include the nghttp2 lib in our main code.

So I've copied the nghttp2.h file directly from the project git page:
https://github.com/nghttp2/nghttp2/blob/4aac05e1936ad14d66b994b789ab0abe0354a28c/lib/includes/nghttp2/nghttp2.h

And pasted it in the tools\sdk\include\nghttp\nghttp2 directory of the Arduino core. After that, I can now include it in my Arduino code.

I've been able to also import the sh2lib wrapper from IDF and use it as an Arduino library. Just pasted both the .c and .h files in my Arduino libraries folder.

Nonetheless, it needs to be included as external C:

extern "C"{
#include "sh2lib.h"
}

Link for the sh2lib wrapper here:
https://github.com/espressif/esp-idf/tree/master/examples/protocols/http2_request/components/sh2lib

Finally, with a little bit of tweaking I've managed to test a HTTP2 request with success. For reference, here is the code that worked for me (very similar to the IDF example, with some changes to make it easier):

#include "WiFi.h"

extern "C"{
#include "sh2lib.h"
}

const char* ssid = "yourNetworkName";
const char* password =  "yourNetworkPass";


#define HTTP2_SERVER_URI  "https://http2.golang.org"
#define HTTP2_STREAMING_GET_PATH  "/reqinfo"   //if tou use the sream endpoint, it keeps a connection open and the server pushes data to us "/clockstream"
#define HTTP2_PUT_PATH            "/ECHO"

int handle_get_response(struct sh2lib_handle *handle, const char *data, size_t len, int flags)
{
    if (len) {
        printf("[get-response] %.*s\n", len, data);
    }
    if (flags == DATA_RECV_FRAME_COMPLETE) {
        printf("[get-response] Frame fully received\n");
    }
    if (flags == DATA_RECV_RST_STREAM) {
        printf("[get-response] Stream Closed\n");
    }
    return 0;
}

int handle_echo_response(struct sh2lib_handle *handle, const char *data, size_t len, int flags)
{
    if (len) {
        printf("[echo-response] %.*s\n", len, data);
    }
    if (flags == DATA_RECV_FRAME_COMPLETE) {
        printf("[echo-response] Frame fully received\n");
    }
    if (flags == DATA_RECV_RST_STREAM) {
        printf("[echo-response] Stream Closed\n");
    }
    return 0;
}

int send_put_data(struct sh2lib_handle *handle, char *buf, size_t length, uint32_t *data_flags)
{
#define DATA_TO_SEND "Hello World"
    int copylen = strlen(DATA_TO_SEND);
    if (copylen < length) {
        printf("[data-prvd] Sending %d bytes\n", copylen);
        memcpy(buf, DATA_TO_SEND, copylen);
    } else {
        copylen = 0;
    }

    (*data_flags) |= NGHTTP2_DATA_FLAG_EOF;
    return copylen;
}

static void http2_task(void *args)
{

    /* HTTP2: one connection multiple requests. Do the TLS/TCP connection first */
    printf("Connecting to server\n");
    struct sh2lib_handle hd;
    if (sh2lib_connect(&hd, HTTP2_SERVER_URI) != 0) {
        printf("Failed to connect\n");
        return;
    }
    printf("Connection done\n");

    /* HTTP GET  */
    sh2lib_do_get(&hd, HTTP2_STREAMING_GET_PATH, handle_get_response);

    /* HTTP PUT  */
    sh2lib_do_put(&hd, HTTP2_PUT_PATH, send_put_data, handle_echo_response);

    while (1) {
        /* Process HTTP2 send/receive */
        if (sh2lib_execute(&hd) < 0) {
            printf("Error in send/receive\n");
            break;
        }
        vTaskDelay(2);
    }

    sh2lib_free(&hd);
    vTaskDelete(NULL);
}

void setup() {

  Serial.begin(115200);
 
  WiFi.begin(ssid, password);
 
  while (WiFi.status() != WL_CONNECTED) {
    delay(1000);
    Serial.println("Connecting to WiFi..");
  }
 
  Serial.println(WiFi.localIP());

  xTaskCreate(&http2_task, "http2_task", (1024 * 32), NULL, 5, NULL);

}

void loop() {
  delay(1000);
}

If I have some time I'll try to make a more detailed blog post to help others replicating the results and share the link here. There's also a lot of investigation to be done, as this was basically the first working thing I could achieve :)

Meanwhile, I would like to ask a couple of questions:

• Was the nghttp2.h file not included on purpose so we use some higher level abstraction library, or was it missed and will be included later?
• Are there any plans to create a nicer wrapper with Arduino like functions, for HTTP2?
• Is there any way to fix the error when including the asio_http2_client.h and asio_http2_server.h libs? Does anyone have used them and can comment about the functionalities it exposes?

Best regards,
Nuno Santos

[lbernstone]

Answers:

• Do not ascribe purpose to that which could easily be error
• If you want to make a library for HTTP2, it would be a great help to the community (in a couple years when we all catch up on the protocol)
• The asio_* headers seem to assume the boost libraries will be available. It is not part of the IDF, and so really these headers should not be included.

I'll put together a PR to add the main library and remove the asio stuff.
Also, it is not unusual to need an extern "C" around some IDF bits. That's part of what the library would wrap around the code.

[anteph]

@lbernstone thank you very much for your feedback and for the fix.

Regarding the first point I was just asking because there could have been some higher level wrapper in development and it could make sense to not expose the lower level lib, or it could not be ready yet for use. Sorry if it seemed to mean otherwise, I totally understand that it could be an error :)

For now I'm planning to do some simple tutorials on how to use the IDF wrapper on the Arduino core. It seems to be simple enough, but I've not yet had time to experiment much more. Let's see to what that leads :)

for the asio_* headers I was trying to make something similar to what I did by including the missing headers files, since there is also a libasio.a in the sdk/lib folder. I've started copying header files from here: https://github.com/espressif/asio/tree/55efc179b76139c8f9b44bf22a4aba4803f7a7bd

And could solve some include errors, but I started hitting on others in lwip, ssl, etc.. so I gave up.

IDF seems to have asio as a component, but I'm not sure how it depends on boost and if those dependencies are solved, so for now I'll probably leave that one and check what I can do with nghtt2 and the sh2lib.

IDF component:
https://github.com/espressif/esp-idf/tree/master/components/asio

Just as a final note, my purpose with using HTTP2 is just playing around, doing some tests and hopefully creating some content for others who also want to experiment with it. I really don't have any requirement of using asio or any other particular lib, I'm just trying to understand what is the state of the art and what is currently available in the Arduino core.

Thanks again for the support of for the awesome work on the Arduino core!

Best regards,
Nuno Santos

[lbernstone]

Thanks for your involvement. This will clearly be a needed feature in the near future, and any content and examples you can add to the community knowledge will be very helpful.
The required boost components do seem to be in asio, so maybe I can get this working with a little massaging of the includes.

[anteph]

Here are some initial examples about the use of sh2lib wrapper with the Arduino core:

• Connecting to a HTTP/2 server: https://techtutorialsx.com/2018/11/18/esp32-http-2-connecting-to-server/
• Making a HTTP/2 GET request: https://techtutorialsx.com/2018/11/19/esp32-arduino-http-2-get-request/

These tutorials are based on the IDF example, adapted for the Arduino core:
https://github.com/espressif/esp-idf/blob/master/examples/protocols/http2_request/main/http2_request_example_main.c

@lbernstone thanks for looking into the asio includes error :) Hopefully it is possible to make it work with some changes to the includes, since unfortunately the sh2lib wrapper has just functions for client requests. So the asio wrappers seem to be the best bet for server functionalities.

Meanwhile I will keep checking how far I can stretch this sh2lib wrapper and keep exploring more client functionalities. I'll post here when I have new tutorials or more information.

Best regards,
Nuno Santos