espressif: `re` runs out of stack

[dhalbert]

The stack size on ESP32-xx may be too small. This code gets a stack overflow on an ESP32-S2 board running 7.2.5

>>> import re
>>> _REQUEST_RE = re.compile(r"(\S+)\s+(\S+)\s")
>>> _REQUEST_RE.match("GET /" + ("a" * 135) + " whatev")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
RuntimeError: maximum recursion depth exceeded
>>> _REQUEST_RE.match("GET /" + ("a" * 134) + " whatev")
<match num=3>

(originally noted in adafruit/Adafruit_CircuitPython_HTTPServer#3)

I tried this on an nRF52840 without difficulty.

[RetiredWizard]

Just a thought....

I routinely compile custom images with MICROPY_ENABLE_PYSTACK turned off and MICROPY_STACKESS turned on. I don't know what I'm trading off with these settings but I've been assuming I've just been making the garbage collector's job harder and reducing the amount of memory I'm able to allocate in my applications.

If the advantage of using the stacks is to keep larger chunks of contiguous memory available, I wonder if they are still necessary on the new multi megabyte microcontroller boards.

[RetiredWizard]

Well I just tried your example code on a Feather S2 and a DevKit S3 using my CircuitPython images with the stacks turned off and they both failed. The Feather returned the recursion depth error and the DevKit crashed into safe mode.

It would appear turning off the stacks isn't going to help with this error.

[dhalbert]

There is still a C stack that's running and is presumably used by the C-based re code. It think it may be that stack that is not big enough.

[richardanaya]

I'm having difficulties doing 512-bit rsa key parsing as well ( not creation )

[richardanaya]

https://gist.github.com/richardanaya/2158b09c52c3b2ea904aaeb48c524062

code.py output:
Traceback (most recent call last):
  File "code.py", line 9, in <module>
  File "/lib/adafruit_jwt.py", line 101, in generate
RuntimeError: pystack exhausted

[RetiredWizard]

So I know building with PyStack=0 and Stackless=1 probably means you can't tell much from my results, but just in case it helps...
I though I'd try running @richardanaya 's code on my build. As soon as I try and import adafruit_jwt I get the following:

 Traceback (most recent call last):
   File "<stdin>", line 1, in <module>
 RuntimeError: maximum recursion depth exceeded

[dhalbert]

There is a maximum FreeRTOS task stack size which is fairly low at the moment (2k??)

[dhalbert]

The size of the C stack for the main FreeRTOS task, CONFIG_ESP_MAIN_TASK_STACK_SIZE, is set to 8kB here:

circuitpython/ports/espressif/esp-idf-config/sdkconfig.defaults

Line 297 in d0953e8

CONFIG_ESP_MAIN_TASK_STACK_SIZE=8192

(Confusingly, note there is also an CONFIG_MAIN_TASK_STACK_SIZE, whose value does not seem to affect this.)

If I change this to 16384, then I don't get the error above. The re library we are using uses recursion, and suffers from this stack problem. Here's an older bug from MicroPython, mentions that trying to match longer strings can cause stack overflow: micropython#2451.

We could grow the stack; on other relatively RAM rich ports, we usually set the stack to 24kB, such as SAMD51 and nRF52480.

@tannewt I should discuss this with you. I'm confused about why CIRCUITPY_DEFAULT_STACK_SIZE does not affect this: I thought that determined the size of the C stack, but apparently not.

[tannewt]

Increasing the stack size sounds good to me.

CIRCUITPY_DEFAULT_STACK_SIZE doesn't impact it because the IDF is managing the stack size for us and I never connected the two.

Going from 8k to 24k seems a bit much to me but 16k would be fine. Remember that we don't use the external RAM for the IDF at all, only the MP heap.