Skip to content

HTTPClient unable to read returned Headers #1368

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
supersjimmie opened this issue Jan 4, 2016 · 34 comments
Closed

HTTPClient unable to read returned Headers #1368

supersjimmie opened this issue Jan 4, 2016 · 34 comments

Comments

@supersjimmie
Copy link

HTTPClient http;
http.begin("http://server.com/url");
code = http.POST(PostData);
Serial.printf("[HTTP] POST... code: %d\r\n", code);
res = http.getString();

Variable res now contains the webpage, but how do I read the returned headers?
(specifically the Set-Cookie?)

I have tried combinations of http.collectHeaders(), http.headers(), http.headerName(), etc...

@supersjimmie
Copy link
Author

Thanks, but it still returns empty.
My current code:

  HTTPClient http;
  const char * headerkeys[] = {"User-Agent","Set-Cookie","Cookie","Date","Content-Type","Connection"} ;
    size_t headerkeyssize = sizeof(headerkeys)/sizeof(char*);
  String PostData = "password=xxxxx&userName=user123";

  http.begin("http://server.growatt.com/LoginAPI.do");
  http.setReuse(true);
  http.setUserAgent("ShinePhone 1.3 (iPhone; iOS 9.0.2; Scale/1.0)"); 
  http.addHeader("Content-type",   "application/x-www-form-urlencoded");

  code = http.POST(PostData);
  Serial.printf("[HTTP] POST... code: %d\r\n", code);
  res = http.getString(); 

  http.collectHeaders(headerkeys,headerkeyssize);
  Serial.printf("Header count: %d\r\n", http.headers());
  for (int i=0; i < http.headers(); i++) {
    Serial.printf("%s = %s\r\n", http.headerName(i).c_str(), http.header(i).c_str());
  }
  Serial.printf("Cookie: %s\r\n", http.header("Cookie").c_str());
  Serial.printf("Set-Cookie: %s\r\n", http.header("Set-Cookie").c_str());
  Serial.printf("Res 1 = %s\r\n", res.c_str());

When looking at a WireShark capture, there are a number of headers ahead of the "normal" data:

HTTP/1.1 200 OK
Date: Mon, 04 Jan 2016 12:31:41 GMT
Content-Type: application/json;charset=UTF-8
Transfer-Encoding: chunked
Connection: keep-alive
Set-Cookie: SERVERID=a0003cd0dfb000bae0dbe0000cb84141|1451910701|1451910204;Path=/

35
{"back":{"userId":1234,"userLevel":1,"success":true}}

But the code returns:

[HTTP] POST... code: 200
Header count: 6
 User-Agent =
Set-Cookie =
Cookie =
Date =
Content-Type =
Connection =
Cookie:
Set-Cookie:

Res 1 = {"back":{"userId":1234,"userLevel":1,"success":true}}

@igrr
Copy link
Member

igrr commented Jan 5, 2016

I suppose collectHeaders should be called before http.POST, because the
whole point of this method is to tell HTTP client which headers to collect.

On Tue, Jan 5, 2016 at 1:08 PM, supersjimmie [email protected]
wrote:

Thanks, but it still returns empty.
My current code:

HTTPClient http;
const char * headerkeys[] = {"User-Agent","Set-Cookie","Cookie","Date","Content-Type","Connection"} ;
size_t headerkeyssize = sizeof(headerkeys)/sizeof(char*);
String PostData = "password=xxxxx&userName=user123";

http.begin("http://server.growatt.com/LoginAPI.do");
http.setReuse(true);
http.setUserAgent("ShinePhone 1.3 (iPhone; iOS 9.0.2; Scale/1.0)");
http.addHeader("Content-type", "application/x-www-form-urlencoded");

code = http.POST(PostData);
Serial.printf("[HTTP] POST... code: %d\r\n", code);
res = http.getString();

http.collectHeaders(headerkeys,headerkeyssize);
Serial.printf("Header count: %d\r\n", http.headers());
for (int i=0; i < http.headers(); i++) {
Serial.printf("%s = %s\r\n", http.headerName(i).c_str(), http.header(i).c_str());
}
Serial.printf("Cookie: %s\r\n", http.header("Cookie").c_str());
Serial.printf("Set-Cookie: %s\r\n", http.header("Set-Cookie").c_str());
Serial.printf("Res 1 = %s\r\n", res);

When looking at a WireShark capture, there are a number of headers ahead
of the "normal" data:

HTTP/1.1 200 OK
Date: Mon, 04 Jan 2016 12:31:41 GMT
Content-Type: application/json;charset=UTF-8
Transfer-Encoding: chunked
Connection: keep-alive
Set-Cookie: SERVERID=a0003cd0dfb000bae0dbe0000cb84141|1451910701|1451910204;Path=/

35
{"back":{"userId":1234,"userLevel":1,"success":true}}

But the code returns:

[HTTP] POST... code: 200
Header count: 6
User-Agent =
Set-Cookie =
Cookie =
Date =
Content-Type =
Connection =
Cookie:
Set-Cookie:

Res 1 = {"back":{"userId":1141,"userLevel":1,"success":true}}


Reply to this email directly or view it on GitHub
#1368 (comment).

@supersjimmie
Copy link
Author

(our comments crossed) Oh, no... And then, just after reading it all again, I realized what I did wrong.
I had to call http.collectHeaders(headerkeys,headerkeyssize) before the POST, not after.
(my idea was that I had to do the Post and then collect the headers from the result, but now I see that he collectHeaders sets the headers that it will collect later)

@supersjimmie
Copy link
Author

Now I see that, in a next stage, the server responds with 2 identical named headers:
Set-Cookie: JSESSIONID=xxx
Set-Cookie: SERVERID=yyy
I can only find one of the Set-Cookie headers. Is it possible to handle multiple headers with same names?

@Links2004
Copy link
Collaborator

no, the handling is the same as the Webserver uses and is not designed of this usage.
a easy possibility where to combine the headers when it reads headers whit the same name, may whit some delimiter but this is not a good solution.
will think a bit over better possibilitys.
can you configure the server to send only one header as workaround?

@supersjimmie
Copy link
Author

Sorry, it is not my server. (I am the client in Europe, requesting solar data from a server in China)
Is there any other way to handle session cookies perhaps?

@supersjimmie
Copy link
Author

So, for now I hav done a quick and very dirty change to my local version to add:

                if(headerName.equalsIgnoreCase("Set-Cookie")) {
                    _cookies[_cookieCount++] = headerValue;
                }

(including about everything else needed)
Now I can use http.cookies() to get the number of Set-Cookie headers, and use http.cookie(num) to get them by number.
I guess it actually had to be 'setcookie', but this is just fine for me now.
I am a very uneperienced programmer, so I am now very happy that my Q&D solution works. :)

@Links2004
Copy link
Collaborator

the http client gives you no direct access to the header.

change:
https://github.com/esp8266/Arduino/blob/master/libraries/ESP8266HTTPClient/src/ESP8266HTTPClient.cpp#L846

to:

_currentHeaders[i].value += headerValue + "; ";

and you get at least all data, as workaround it will work but its no solution for the core.

Note:
you wirte a new messages when i type to :)
hope your _cookies array will not get a overflow in some case.

@supersjimmie
Copy link
Author

Yes sorry I sometimes type a second message after my previous message and then our messages cross. Since I do not understand dynamically allocating memory and things like that, I now assume only 2 _cookies. and to be safe I use String _cookies[4];

So, in the .h file, just below the size_t headerCount;

        String          _cookies[4];
        size_t          _cookieCount;

and in HTTPClient::handleHeaderResponse():

    _cookieCount = 0;
...
                if(headerName.equalsIgnoreCase("Set-Cookie")) {
                    _cookies[_cookieCount++] = headerValue;
                }

And also 2 new functions:

String HTTPClient::cookie(size_t i) {
    if(i < _cookieCount)
        return _cookies[i];
    return String();
}

int HTTPClient::cookies() {
    return _cookieCount;
}

I could try to figure out how to prevent an overflow, or just limit the cookieCount to a max of 4.
(and change cookies and cookieCount to setcookies and setcookiecount or so)
But it works now, so I don't want to change too much.
Thanks so far!

@Links2004
Copy link
Collaborator

may add:

if(_cookieCount < 4) 

to be sure not to use the array out of its borders,
this type of problem is really hard to finde later on ;)

@supersjimmie
Copy link
Author

Thanks, your change:
_currentHeaders[i].value += headerValue + "; ";
also works, it only gives me a bit more work because I have to strip the result afterwards.
This gives both cookies in one string, and in that string are also the "Path=" parts.
JSESSIONID=ADA6EBBF76894CAF08B477600FAC8A26; Path=/; SERVERID=08272d982c6c53b9a0d4a9b5172a0059|1002077403|1452077403;Path=/;

But because this solution only requires one small change in one file, I guest it is better than my solution to filter all Set-Cookies to a separate series of Strings.

@supersjimmie
Copy link
Author

Could there be something changed in the latest version that broke the cookies?
I have modified this again:
_currentHeaders[i].value += headerValue + "; ";
I do receive the cookies. That still works.
But in the next http session, where I add the cookies as a Header "Cookie", jut like I did before, I get http status -1 back. Don't know why?

When I return to the older version, it works fine again.
First difference I see, is that the latest has #include <base64.h>, while the older working one does not. Unfortunately I do not know from what date the older working one is.
(don't think it has to do with that include, but perhaps that makes clear what versions I use?)

@Links2004
Copy link
Collaborator

2.1.0-rc1 has SDK 1.5.0 this one has some bugs.
try using the git version, will work better.
(SDK 1.5.1)

@supersjimmie
Copy link
Author

Git version is just the zip that can be downloaded here?
https://github.com/esp8266/Arduino
That is the one I use, downloaded this morning.

@Links2004
Copy link
Collaborator

when you mean:
https://github.com/esp8266/Arduino/archive/master.zip
then yes its the latest possible version (on download time).

@supersjimmie
Copy link
Author

Confirm. I downloaded that today and only changed that one line.
Now the first part with a http.POST() still works, it recieves the Cookie part, but when I do a next part with a http.GET() and use the Cookies, it fails with -1.
Exactly the same code with the older (Stable 2.0.0 ?) works fine.

To be precise, I use Arduino IDE 1.6.5 with the old version and 1.6.7 with the new version.
(which is also the way it should be?)

@Links2004
Copy link
Collaborator

#define HTTPC_ERROR_CONNECTION_REFUSED  (-1)

the connect to the server failed, I think you change has no effect on the -1 result.

@supersjimmie
Copy link
Author

Ok, I will do a packet trace tomorrow to see what happens exactly.
Because something is different between old and new environments with the same .ino file.

@supersjimmie
Copy link
Author

It is very strange... When I look at wireshark:

1a. I send a POST with the login details
1b. Server responds with 200 OK and the session cookies.

2a. I send a GET, together with the session cookies and a data request
2b. The server sends a "200 OK" with the data back, but the code says code -1.
I also notice that this -1 is received very fast (looks like it gives -1 before it even receives the real response), like the code bails out without even trying.

If I then just try http.getString() it is empty, offcourse.

I do not know very much about packets, but I notice a small difference.
In the working sessions (first part on old and new, second part on old), I see:
ESP: GET ...
Server: ACK
Server: Data
Server: HTTP1/1 200 OK

In the non working part, the ESP sends a packet with FIN,ACK:
ESP: GET ...
Server: ACK
ESP: FIN,ACK
Server: Data
Server: HTTP1/1 200 OK
I suppose that is where the ESP stops, gives code -1 and does not look at the server anymore.

EDIT: In addition, the third and last session normally depends on the first two, but if I just fake the second session, the third session seems to be successful again.

This all tested many times over and over... Same results, old passes, new fails on the second session.

@Links2004
Copy link
Collaborator

the ESP closes the connection:
ESP: FIN,ACK
but why is the question.

try to enable the debug for the HTTPClient.
https://github.com/esp8266/Arduino/blob/master/doc/Troubleshooting/debugging.md

@supersjimmie
Copy link
Author

I would love to, but there is no Debug Port option in my IDE Tools-menu.
This may be because I did not install esp8266 (2.1.0-rc1) via the Boards Manager, but by just adding the latest GIT via unzipping it?
But if I now install it via Boards Manager, that will downgrade latest GIT to 2.1.0-rc1?

@Links2004
Copy link
Collaborator

GIT and 2.1.0-rc1 have the menu independent of the installing method.
may there are left overs from the older version try to delete them.

@supersjimmie
Copy link
Author

Figured it out, the debug options are only available with board "generic esp8266 module".
I was using "nodemcu 1.0 (esp12e)". Now trying to add the debugging to that board too...

@supersjimmie
Copy link
Author

Here is the complete list with all 3 requests:

 [HTTP-Client][begin] url: http://server.growatt.com/LoginAPI.do
 [HTTP-Client][begin] host: server.growatt.com port: 80 url: /LoginAPI.do https: 0 httpsFingerprint:
 [HTTP-Client] connect http...
 [HTTP-Client] connected to server.growatt.com:80
 [HTTP-Client][handleHeaderResponse] RX: 'HTTP/1.1 200 OK'
 [HTTP-Client][handleHeaderResponse] RX: 'Date: Thu, 14 Jan 2016 13:01:34 GMT'
 [HTTP-Client][handleHeaderResponse] RX: 'Content-Type: application/json;charset=UTF-8'
 [HTTP-Client][handleHeaderResponse] RX: 'Transfer-Encoding: chunked'
 [HTTP-Client][handleHeaderResponse] RX: 'Connection: keep-alive'
 [HTTP-Client][handleHeaderResponse] RX: 'Set-Cookie: JSESSIONID=849E08546DDF319EAB00485DABD450DD; Path=/'
 [HTTP-Client][handleHeaderResponse] RX: 'Set-Cookie: SERVERID=a4043cd5dfb873bae3dbe3124cb4141|145277693|1452776493;Path=/'
 [HTTP-Client][handleHeaderResponse] RX: ''
 [HTTP-Client][handleHeaderResponse] code: 200
 [HTTP-Client][handleHeaderResponse] Transfer-Encoding: chunked
                      [HTTP] POST... code: 200
 [HTTP-Client] read chunk len: 53
 [HTTP-Client][writeToStreamDataBlock] connection closed or file end (written: 53).
 [HTTP-Client] read chunk len: 0
 [HTTP-Client][end] tcp keep open for reuse
Header count: 1
Set-Cookie = JSESSIONID=849E08546DDF319EAB00485ABD450DD; Path=/; SERVERID=a4043cd5dfb873bae3dbe3124cb84141|1452776493|1452776493;Path=/;
Cookies: JSESSIONID=849E08546DDF319EAB0085DABD450DD; SERVERID=a4043cd5dfb873bae3dbe3124cb8411|1452776493|1452776493
UserID: 1234
 [HTTP-Client][begin] url: http://server.growatt.com/PlantListAPI.do?userId=1234
 [HTTP-Client][begin] host: server.growatt.com port: 80 url: /PlantListAPI.do?userId=1234 https: 0 httpsFingerprint:
 [HTTP-Client] connect. already connected, try reuse!
 [HTTP-Client][handleHeaderResponse] RX: '0'
 [HTTP-Client][handleHeaderResponse] RX: ''
 [HTTP-Client][handleHeaderResponse] code: -1
 [HTTP-Client][returnError] error(-1): connection refused
 [HTTP-Client][returnError] tcp stop
 [HTTP] GET... code: -1
 [HTTP-Client][returnError] error(-4): not connected
No PlantID.
 [HTTP-Client][begin] url: http://server.growatt.com/PlantDetailAPI.do?type=4&plantId=1234
 [HTTP-Client][begin] host: server.growatt.com port: 80 url: /PlantDetailAPI.do?type=4&plantId=1234 https: 0 httpsFingerprint:
 [HTTP-Client] connect http...
 [HTTP-Client] connected to server.growatt.com:80
 [HTTP-Client][handleHeaderResponse] RX: 'HTTP/1.1 200 OK'
 [HTTP-Client][handleHeaderResponse] RX: 'Date: Thu, 14 Jan 2016 13:01:34 GMT'
 [HTTP-Client][handleHeaderResponse] RX: 'Content-Type: application/json;charset=UTF-8'
 [HTTP-Client][handleHeaderResponse] RX: 'Transfer-Encoding: chunked'
 [HTTP-Client][handleHeaderResponse] RX: 'Connection: keep-alive'
 [HTTP-Client][handleHeaderResponse] RX: 'Set-Cookie: SERVERID=a4043cd5dfb873bae3dbe124cb84141|1452776494|142776493;Path=/'
 [HTTP-Client][handleHeaderResponse] RX: ''
 [HTTP-Client][handleHeaderResponse] code: 200
 [HTTP-Client][handleHeaderResponse] Transfer-Encoding: chunked
 [HTTP] GET... code: 200
 [HTTP-Client] read chunk len: 221
 [HTTP-Client][writeToStreamDataBlock] connection closed or file end (written: 221).
 [HTTP-Client] read chunk len: 0
 [HTTP-Client][end] tcp keep open for reuse

(don't look at the '1234' id's and don't compare the cookies, I removed some parts for safety)

@supersjimmie
Copy link
Author

Removed http.setReuse(true);, now it seems to work.
This did work with the older version, so why not anymore?

@Links2004
Copy link
Collaborator

the server sends two times a 0 byte chunk this is not good (not in the RFC like this)
try to add:

      while(_tcp->available() > 0) {
           _tcp->read();
      }

here:
https://github.com/esp8266/Arduino/blob/master/libraries/ESP8266HTTPClient/src/ESP8266HTTPClient.cpp#L713

to clear the buffer before next request

@supersjimmie
Copy link
Author

#1368 (comment)
Seemed to be working fine. :)

Links2004 added a commit that referenced this issue Jan 16, 2016
@supersjimmie
Copy link
Author

Thanks for #1368 with:
while(_tcp->available() > 0) { _tcp->read();
But you will not implement the other part?
_currentHeaders[i].value += headerValue + "; ";

@Links2004
Copy link
Collaborator

the += is not my Favorit way to do this, its more a workaround,
have currently no easy and good way,
the best concept i have in mind is using std::vector.

@igrr
Copy link
Member

igrr commented Jan 18, 2016

I think another way to do this is to change RequestArgument structure to maintain a linked list of values instead of one String value. However you would also have to provide a way to iterate over values for the given key, potentially changing the signatures of existing methods (i.e. headers(name))

@kiralikbeyin
Copy link

How to make this Cookie with ESP8266HTTPClient lib ?

 client.print(String("GET ") + url + " HTTP/1.1\r\n" +
               "Host: " + host + "\r\n" +
               "Cookie: " + yap + "\r\n" +
               "Connection: close\r\n\r\n");

@arnolde
Copy link

arnolde commented Sep 20, 2018

@supersjimmie @Links2004
This is the only issue I could find which addresses my current problem: To login to a captive portal (Telekom Wifi Hotspot) I need to get 2 cookies (from 2 separate Set-Cookie lines) from the first response and send them with the request for the second page (redirection).
I read all the above but since it's 2 years old, I wonder if the issue has not been permanently fixed in core yet?
I looked at this:

change:
https://github.com/esp8266/Arduino/blob/master/libraries/ESP8266HTTPClient/src/ESP8266HTTPClient.cpp#L846

to:

_currentHeaders[i].value += headerValue + "; ";

But it seems now that that referenced line is not correct anymore due to changes in the meantime. So I dont know where now to make that change... is it now line 1044? But if I change that line, it adds a semicolon to ALL header lines, corrupting i.e. the "Location" redirect URL.
Am I really the only person around who is trying to connect with an ESP to a public Telekom hotspot?
Does nobody else have trouble with reading multiple cookie header lines so that the limitation still persists after 2 years?

@sharifzadehsharif
Copy link

How to make this Cookie with ESP8266HTTPClient lib ?

 client.print(String("GET ") + url + " HTTP/1.1\r\n" +
               "Host: " + host + "\r\n" +
               "Cookie: " + yap + "\r\n" +
               "Connection: close\r\n\r\n");

Hi, could you solve this?
I receive this response, I need a library to set cookie automatically.
thanks
<html><body><script type="text/javascript" src="/aes.js" ></script><script>function toNumbers(d){var e=[];d.replace(/(..)/g,function(d){e.push(parseInt(d,16))});return e}function toHex(){for(var d=[],d=1==arguments.length&&arguments[0].constructor==Array?arguments[0]:arguments,e="",f=0;f<d.length;f++)e+=(16>d[f]?"0":"")+d[f].toString(16);return e.toLowerCase()}var a=toNumbers("f655ba9d09a112d4968c63579db590b4"),b=toNumbers("98344c2eee86c3994890592585b49f80"),c=toNumbers("80bcc4ef876a12efa5496ffb08a16190");document.cookie="__test="+toHex(slowAES.decrypt(c,2,a,b))+"; expires=Thu, 31-Dec-37 23:55:55 GMT; path=/"; location.href="http://sharifzadehsharif.gigfa.com/send.php?gram=1025.3&mode=timer&i=1";</script><noscript>This site requires Javascript to work, please enable Javascript in your browser or use a browser with Javascript support</noscript></body></html>

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

6 participants