HTTPClient https request works the first time, but fails the second time

[RosiersRobin]

Board

ESP32-Wroom-32UE

Device Description

Plain module un custom hardware with an ETH chip (LAN8720) connected to it, using that chip to cummunicate with internet intead of wifi.

Hardware Configuration

GPIO0 has 50Mhz crystal input for ETH -> disabled on boot, thus device boots

GPIO12 -> crystal power pin
GPIO2 -> I2C clock signal
GPIO17 -> I2C MDIO

Version

v2.0.2

IDE Name

Arduino IDE

Operating System

Windows 10

Flash frequency

80Mhz

PSRAM enabled

no

Upload speed

921600

Description

I updated from version 1.0.6 to 2.0.2 and changed nothing to the code. The code works still, however, it seems that the HTTPClient is broken since then.

I can do an API call to the server once, but not twice.

I have for example a heartbeat that runs every 5 minutes, the first time it runs just fine, the second time I get a -1 error. While this uses the exact same code...

Sketch

void setup() {
    ... Some other setup code...
    xTaskCreatePinnedToCore(sendHeartBeat, "Send Heartbeat", 20000, NULL, 25, &heartBeatTask, 0);
}

void sendHeartBeat(void *pvParameters) {
    // Heartbeat millis
    unsigned long previousHeartBeatMillis = 0;
    const long heartBeatInterval = 300000; // 5 minutes

    // Send a first heartbeat to let the server know it's alive
    sendHeartBeatApiCall();

    // Infinite loop (runs on another core)
    while (true) {
        unsigned long currentMillis = millis();

        // heartbeat thing
        if (currentMillis - previousHeartBeatMillis >= heartBeatInterval) {

            previousHeartBeatMillis = currentMillis;

            sendHeartBeatApiCall();
        }

        vTaskDelay(1); // to allow for resources to terminate and the task not to crash
    }
}


void sendHeartBeatApiCall() {
    StaticJsonDocument<256> doc;
    doc.clear(); // release the memory
    doc["data"] = "random";
    ... some more json data

    String heartbeatData;
    serializeJson(doc, heartbeatData);

    Serial.println("Sending Heartbeat");

    http.begin("https://my-endpoint.com/path/to/heartbeat");

    http.addHeader("Content-Type", "application/json");

    int responseCode = http.POST(heartbeatData); // just post, Don't care about the response.

    // If the response code was 200, store the current millis.
    if (responseCode == 200) {
        lastHeartbeat = millis();
    }

    Serial.print("ResponseCode from heartbeat: ");
    Serial.println(responseCode);

    // Free resources
    http.end();
}

Debug Message

This is the output (redacted) of a successfull API call;


15:34:26.270 -> Sending Heartbeat
15:34:26.270 -> [ 30759][V][HTTPClient.cpp:247] beginInternal(): url: https://mydomain.com/path/to/api
15:34:26.270 -> [ 30760][W][HTTPClient.cpp:258] beginInternal(): unexpected protocol: https, expected http
15:34:26.270 -> [ 30768][V][HTTPClient.cpp:247] beginInternal(): url: https://mydomain.com/path/to/api
15:34:26.270 -> [ 30776][D][HTTPClient.cpp:298] beginInternal(): protocol: https, host:mydomain.com port: 443 url: /path/to/api
15:34:26.270 -> [ 30787][D][HTTPClient.cpp:595] sendRequest(): request type: 'POST' redirCount: 0
15:34:26.339 -> 
15:34:26.339 -> [ 30794][D][HTTPClient.cpp:1101] connect(): already connected, reusing connection
15:34:26.339 -> [ 30802][V][ssl_client.cpp:351] send_ssl_data(): Writing HTTP request with 257 bytes...
15:34:26.339 -> [ 30810][V][ssl_client.cpp:351] send_ssl_data(): Writing HTTP request with 219 bytes...
15:34:26.408 -> [ 30874][V][HTTPClient.cpp:1234] handleHeaderResponse(): RX: 'HTTP/1.1 200 OK'
15:34:26.408 -> [ 30875][V][HTTPClient.cpp:1234] handleHeaderResponse(): RX: 'Server: nginx'
15:34:26.408 -> [ 30877][V][HTTPClient.cpp:1234] handleHeaderResponse(): RX: 'Content-Type: application/json'
15:34:26.408 -> [ 30885][V][HTTPClient.cpp:1234] handleHeaderResponse(): RX: 'Transfer-Encoding: chunked'
15:34:26.408 -> [ 30893][V][HTTPClient.cpp:1234] handleHeaderResponse(): RX: 'Connection: keep-alive'
15:34:26.408 -> [ 30901][V][HTTPClient.cpp:1234] handleHeaderResponse(): RX: 'Vary: Accept-Encoding'
15:34:26.442 -> [ 30908][V][HTTPClient.cpp:1234] handleHeaderResponse(): RX: 'Cache-Control: no-cache, private'
15:34:26.442 -> [ 30917][V][HTTPClient.cpp:1234] handleHeaderResponse(): RX: 'Date: Mon, 11 Apr 2022 13:34:26 GMT'
15:34:26.442 -> [ 30925][V][HTTPClient.cpp:1234] handleHeaderResponse(): RX: 'Access-Control-Allow-Origin: *'
15:34:26.442 -> [ 30934][V][HTTPClient.cpp:1234] handleHeaderResponse(): RX: 'X-Frame-Options: SAMEORIGIN'
15:34:26.442 -> [ 30942][V][HTTPClient.cpp:1234] handleHeaderResponse(): RX: 'X-XSS-Protection: 1; mode=block'
15:34:26.442 -> [ 30950][V][HTTPClient.cpp:1234] handleHeaderResponse(): RX: 'X-Content-Type-Options: nosniff'
15:34:26.477 -> [ 30958][V][HTTPClient.cpp:1234] handleHeaderResponse(): RX: ''
15:34:26.477 -> [ 30964][D][HTTPClient.cpp:1275] handleHeaderResponse(): code: 200
15:34:26.477 -> [ 30970][D][HTTPClient.cpp:1282] handleHeaderResponse(): Transfer-Encoding: chunked
15:34:26.477 -> [ 30977][D][HTTPClient.cpp:619] sendRequest(): sendRequest code=200
15:34:26.477 -> 
15:34:26.477 -> ResponseCode from heartbeat: 200
15:34:26.477 -> [ 30993][D][HTTPClient.cpp:383] disconnect(): still data in buffer (25), clean up.
15:34:26.477 -> 
15:34:26.477 -> [ 30994][D][HTTPClient.cpp:390] disconnect(): tcp keep open for reuse



This is the output (redacted) of the failed API call;

```cpp
15:38:55.509 -> Sending Heartbeat
15:38:55.509 -> [300001][V][HTTPClient.cpp:247] beginInternal(): url: https://mydomain.com/path/to/api
15:38:55.509 -> [300001][W][HTTPClient.cpp:258] beginInternal(): unexpected protocol: https, expected http
15:38:55.509 -> [300007][V][HTTPClient.cpp:247] beginInternal(): url: https://mydomain.com/path/to/api
15:38:55.509 -> [300016][D][HTTPClient.cpp:298] beginInternal(): protocol: https, host: mydomain.com port: 443 url: /path/to/api
15:38:55.509 -> [300026][D][HTTPClient.cpp:595] sendRequest(): request type: 'POST' redirCount: 0
15:38:55.509 -> 
15:38:55.509 -> [300035][V][ssl_client.cpp:311] stop_ssl_socket(): Cleaning SSL connection.
15:38:55.543 -> [300041][V][ssl_client.cpp:61] start_ssl_client(): Free internal heap before TLS 161764
15:38:55.543 -> [300048][V][ssl_client.cpp:67] start_ssl_client(): Starting socket
15:38:55.610 -> [300084][V][ssl_client.cpp:143] start_ssl_client(): Seeding the random number generator
15:38:55.610 -> [300086][V][ssl_client.cpp:152] start_ssl_client(): Setting up the SSL/TLS structure...
15:38:55.610 -> [300089][I][ssl_client.cpp:173] start_ssl_client(): WARNING: Skipping SSL Verification. INSECURE!
15:38:55.610 -> [300097][V][ssl_client.cpp:244] start_ssl_client(): Setting hostname for TLS session...
15:38:55.610 -> [300105][V][ssl_client.cpp:259] start_ssl_client(): Performing the SSL/TLS handshake...
15:38:55.610 -> [300116][E][WiFiClientSecure.cpp:135] connect(): start_ssl_client: -1
15:38:55.644 -> [300120][V][ssl_client.cpp:311] stop_ssl_socket(): Cleaning SSL connection.
15:38:55.644 -> [300126][D][HTTPClient.cpp:1134] connect(): failed connect to miqey.libaro.io:443
15:38:55.644 -> [300133][W][HTTPClient.cpp:1437] returnError(): error(-1): connection refused
15:38:55.644 -> ResponseCode from heartbeat: -1
15:38:55.644 -> [300150][D][HTTPClient.cpp:405] disconnect(): tcp is closed
15:38:55.644 -> 

All outputs were captured via the serial debug monitor setting the core debug level to verbose.

### Other Steps to Reproduce

I already have tried alot here.

Tried to add the CA cert, didn't work,

Used an wificlientsecure with that CA cert, didn't work

`client.setReuse(false);` Didn't work.

The stranges thing here is, I do API calls, but the second time I do the call, it doesn't work.

What I do is like the following;

1. Sync NTP time
2. Register device on https://domain.com/register
3. Connect to AWS IoT using it's own client and certificates
4. Send a heartbeat on https://domain.com/heartbeat
5. Send a heartbeat on https://domain.com/heartbeat --> fails with -1

If I do for example this;

1. Sync NTP time
2. Register device on https://domain.com/register
3. Connect to AWS IoT using it's own client and certificates
4. Send a heartbeat on https://domain.com/heartbeat
5. Send a API call to https://domain2.com/endpoint --> fails with -1
6. Send a heartbeat on https://domain.com/heartbeat --> fails with -1

So I thought, it's somewhere I didn't call `http.end()`, so I went looking, but every `http.begin()` has a `http.end()`.

I'm totally at a loss here now....

### I have checked existing issues, online documentation and the Troubleshooting Guide

- [X] I confirm I have checked existing issues, online documentation and Troubleshooting guide.

[vlastahajek]

This seems to be the same issue as #6165. It should work in 2.0.3-rc1

[VojtechBartoska]

Yes, please try to retest it on v2.0.3-rc1.

[RosiersRobin]

Tested it on v2.0.3-rc1, then I have the folloing issue;

First calls work as expectedd (just like above)

Second call fails with error code -2 BUT the thrid call of that same URL works just fine, it then alternates bewteen 200 and -2 of status codes returned.

So;

works
doesnt
works
doesnt
works
....

[RosiersRobin]

Update: If I do the call every 5 minutes, it breaks, if I do it every 5 seconds, it works just fine...

[TD-er]

Update: If I do the call every 5 minutes, it breaks, if I do it every 5 seconds, it works just fine...

Can you track the free heap?
Have not tested the latest code myself, but the older code had a memory leak in loading the TLS code.

See my (still pending) PR for my own project to include TLS support.
https://github.com/letscontrolit/ESPEasy/pull/3788/files
These files in particular (especially ESPEasy_ssl_client.h/.cpp) :

• src/src/Helpers/ESPEasy_WiFiClientSecure.h
• src/src/Helpers/ESPEasy_WiFiClientSecure.cpp
• src/src/Helpers/ESPEasy_ssl_client.cpp
• src/src/Helpers/ESPEasy_ssl_client.h

These are taken from this lib and adapted to prevent this memory leak as the mbettls context objects are not dealt with properly.
The pointers as members are not initialized correctly, thus if you destruct them you will see a crash and if you don't, you will run out of memory.

[VojtechBartoska]

This was added to our Roadmap, we will investigate it further. Thanks for your contribution.

[podaen]

A request would work, it s the content that just not right.

[RosiersRobin]

Update: If I do the call every 5 minutes, it breaks, if I do it every 5 seconds, it works just fine...

Can you track the free heap? Have not tested the latest code myself, but the older code had a memory leak in loading the TLS code.

See my (still pending) PR for my own project to include TLS support. https://github.com/letscontrolit/ESPEasy/pull/3788/files These files in particular (especially ESPEasy_ssl_client.h/.cpp) :

• src/src/Helpers/ESPEasy_WiFiClientSecure.h
• src/src/Helpers/ESPEasy_WiFiClientSecure.cpp
• src/src/Helpers/ESPEasy_ssl_client.cpp
• src/src/Helpers/ESPEasy_ssl_client.h

These are taken from this lib and adapted to prevent this memory leak as the mbettls context objects are not dealt with properly. The pointers as members are not initialized correctly, thus if you destruct them you will see a crash and if you don't, you will run out of memory.

Sorry for the late reply.

I have checked memory usage and that didn't seem to be the rpoblem. I even extracted the code into a blank project and tested it the exact same way. Same results.