Memory leak with WiFiClientSecure in 2.0rc2?

[bachi76]

When I continuously run the tests for my Firebase library, I see after a while / bunch of https requests:

Panic /home/bachi/.arduino15/packages/esp8266/hardware/esp8266/2.0.0-rc2/libraries/ESP8266WiFi/src/WiFiClientSecure.cpp:404 ax_port_realloc

This happens with the 2.0rc2 core release, but not with the 1.6.x release.

The project is here: https://bitbucket.org/bachi76/esp6288-framework/
(Note: the folders in /framework/ where symlinked to the libraries)

The debug output when running with 2.0rc2

Debug output when the same code is run with 1.6.x

It starts with something like this:

Firebase - Memory left before Wifi client: 33984
Firebase - Memory left before connect: 34056
Firebase - Connected
Firebase - Memory before sending request: 11096

And the last try before the crash looks like:

Firebase - Memory left before Wifi client: 31336
Firebase - Memory left before connect: 31288

Panic /home/bachi/.arduino15/packages/esp8266/hardware/esp8266/2.0.0-rc2/libraries/ESP8266WiFi/src/WiFiClientSecure.cpp:404 ax_port_realloc

Not 100% sure where the mem leak comes from (c++ apprentice here), or if it's just due to the new core requiring more memory. The https conn. takes around 22kb - is there a way to reduce this?

[Links2004]

the WiFiClientSecure itself has no memory leek.
tested at latest git.

tested with:
https://github.com/esp8266/Arduino/blob/master/libraries/ESP8266httpClient/examples/BasicHttpClient/BasicHttpClient.ino

if you deconstruct the WiFiClientSecure object correct (use delete for ptr) it free the 22K.
regarding the reduce may @igrr can help.

[igrr]

I took 2.0.0-rc2 and ran the following minimal example: https://gist.github.com/igrr/816523b463bd7d045b77

Here's the output: http://hastebin.com/ipuroqinuw.css

There are occasional changes in free heap, but overall there is no noticeable trend which would indicate a leak. I looked briefly at your code and couldn't find any obvious issues. Note that my sample only does a single write and then reads all response data, so perhaps that's not complicated enough to trigger the issue (if the issue is within WiFiClientSecure). I think we have to start slimming down your example to find the minimal test case which still demonstrates the leak.

[igrr]

Looking at your code a bit more, i think i understand what leads to this issue.

In my example, there is no state preserved between loop iterations. All memory which was allocated gets freed.

In your test case, things are a bit more complicated.

You call some Firebase function which takes string as an argument (getNode, setValue, addRow, etc.)
Your heap layout looks as follows:

[  ....  ] [ contents of string arg ] [ free ]

Firebase public functions call executeRequest function, and WiFiClientSecure object is created. It does some dynamic allocation, and heap layout looks like this:

[  ....  ] [ contents of string arg ] [ helpers for WiFiClientSecure ] [ free ]

Once you read the response, heap starts looking like this (response data gets allocated by Strings inside response):

[  ....  ] [ contents of string arg ] [ helpers for WiFiClientSecure ] [ response data ] [ free ]

Once you exit this function, things look as follows:

[  ....  ] [ contents of string arg ] [                     free                     ] [ response data ] [ free ]

Temporary Strings are deleted:

[  ....  ] [                                                free                     ] [ response data ] [ free ]

Okay, you move on to the next request in your test.
This one, unfortunately, takes Strings which are a bit longer.

[  ....  ] [ longer contents of string arg ] [               free                     ] [ response data ] [ free ]

Then WiFiClientSecure tries to allocate its helpers, and oups... there isn't enough space to fit all it needs, because the chunk of memory between args and response data is suddenly smaller than it was before, and the chunk after response data is not large enough.

I have simplified a few things in this explanation (i.e. WiFiClientSecure doesn't allocate a single chunk, there are multiple), but overall i think this is what is happening.

[igrr]

I think that the way to solve this would be to allocate WiFiClientSecure statically. You would have much less memory available, but this would guarantee that once you need WiFiClientSecure, memory will be available. Unfortunately at the moment a lot of things inside axTLS library are built around dynamic allocation. So making WiFiClientSecure a global static variable will not help immediately. We need to implement a way to allocate all axTLS variables statically as well.

[igrr]

Since memory allocation for axTLS is implemented via hooks in WiFiClientSecure.cpp, enabling static allocation is going to be pretty easy.

[bachi76]

First of all, @igrr , thank you for the awesome help!

I've reduced the tests, so only one runs repeatedly (does 2 requests):

t.test("Add to list", fb.addRow("somelist-and-some-extra-long-string-here", "{\"name\":\"value-with-some-extra-long-string\"}").success && fb.addRow("somelist", "{\"name\":\"value\"}").success);

Looking at the mem stats, I see that from the second run, the mem usage is constant. However, after 75 or so requests, and still the same available mem, the crash re-appears:

Firebase - Memory left before connect: 33904
Firebase - Memory left before connect: 33792
Firebase - Memory left before connect: 31720
Firebase - Memory left before connect: 31640
Firebase - Memory left before connect: 31720
Firebase - Memory left before connect: 31640
Firebase - Memory left before connect: 31720
Firebase - Memory left before connect: 31640
Firebase - Memory left before connect: 31720
Firebase - Memory left before connect: 31640
Firebase - Memory left before connect: 31720
...
Firebase - Memory left before connect: 31640
Firebase - Memory left before connect: 31720
Firebase - Memory left before connect: 31640
Firebase - Memory left before connect: 31720
Firebase - Memory left before connect: 31640
Firebase - Memory left before connect: 31720

Panic /home/bachi/.arduino15/packages/esp8266/hardware/esp8266/2.0.0-rc2/libraries/ESP8266WiFi/src/WiFiClientSecure.cpp:404 ax_port_realloc

ctx: cont
sp: 3fff1cb0 end: 3fff2290 offset: 01b0

So this probably indicates the heap fragmentation issue you mentioned (but then, the first two string args passed to fb.addRow() are the longest now, and as I've understood this should prevent the heap fragmentation?).

Anyway - thanks a lot for your help here!

[drmpf]

Slightly off topic,
but can we have char* versions of all the library functions that currently expect or return Strings.
This would allow for more robust programming by avoiding extra background heap allocations.
i.e.

 char * hostname(char *, int maxLen); // max 32 chars

where maxLen is the size of char array
as an alternative to

    String hostname(void);

[Links2004]

what you are searching is already there in the SDK.

Arduino/tools/sdk/include/user_interface.h

Line 244 in 342c4ae

char* wifi_station_get_hostname(void);

the SDK itself has no "String".

[drmpf]

Great Thanks. Glad to know the SDK does not use Strings.

@bachi76
How did you fix this issue?