Problem with mod_perl in master where crashes after any dialplan call (At least on Ubuntu)

[mitchcapper]

Describe the bug
Running the latest master I receive either a segfault or the error "Out of memory!" when mod_perl is called from a diaplan. Sometimes it needs to be called twice to get the error. The "Out of Memory!" error is generated by FS I believe (not an OS memkill) however a quick look at where FS throws that error in source do not show immediately anything related (mostly "src/mod/languages/mod_java/modjava.c" file and "libzrtp"). It is important to note the ! at the end as FS does throw generic "Out of memory" errors in several other places.

I have confirmed that building from source in a very similar manor on Debian Buster does NOT have the same issue (can call it several times at least). @briankwest mentioned this is likely an issue with perl, a perl module, or the perl script itself and not freeswitch. Brian also thought it was related to clone being unsafe to use in a threaded environment, from the BT I believe this is the FS threadpool worker that is calling clone so I am not sure what to do with that.

I am not sure where to go from here. Could potentially try to strace the freeswitch process for when perl is called into but

To Reproduce
Attached are two docker files (.txt annoying but GH requirement). Dockerfile_DEBIAN_Compiled.txt and Dockerfile_Ubuntu_Compiled.txt both follow a similar process for building freeswitch and setting up a dummy test. To build run docker build -f .\Dockerfile_Ubuntu_Compiled.txt -t test_ubuntu . then docker run -t -i --rm -p 5060 test_ubuntu. Once started connect and call extension 9198. Note for some reason Ubuntu in this setup requires it to be called twice to crash. The perl script is very basic just

#!/usr/bin/perl
use strict;
our $session;

1;

For reference, you do NOT need to have an ubuntu box to verify this on, any nix box running docker should be fine (the docker base will still be ubuntu).

I also attached the output of the perl build info from both platforms they are exceptionally similar (minus one perl 5.28 vs 5.30). Files: perl_debian.txt and perl_ubuntu.txt.

Expected behavior
Perl script to be executed and FS not to crash.

Package version or git hash
bb7bf86

Trace logs
Provide freeswitch logs w/ DEBUG and UUID logging enabled

backtrace from core file
Full set of gdb traces attached to ticket gdb_bts.log.

#0  0x00007fb746486593 in Perl__invlist_intersection_maybe_complement_2nd () from /lib/x86_64-linux-gnu/libperl.so.5.30
#1  0x00007fb746486bd5 in ?? () from /lib/x86_64-linux-gnu/libperl.so.5.30
#2  0x00007fb746496c7f in ?? () from /lib/x86_64-linux-gnu/libperl.so.5.30
#3  0x00007fb74649d268 in ?? () from /lib/x86_64-linux-gnu/libperl.so.5.30
#4  0x00007fb7464a1d03 in ?? () from /lib/x86_64-linux-gnu/libperl.so.5.30
#5  0x00007fb7464a22cf in ?? () from /lib/x86_64-linux-gnu/libperl.so.5.30
#6  0x00007fb74649d087 in ?? () from /lib/x86_64-linux-gnu/libperl.so.5.30
#7  0x00007fb7464a1d03 in ?? () from /lib/x86_64-linux-gnu/libperl.so.5.30
#8  0x00007fb7464a22cf in ?? () from /lib/x86_64-linux-gnu/libperl.so.5.30
#9  0x00007fb7464a736c in Perl_re_op_compile () from /lib/x86_64-linux-gnu/libperl.so.5.30
#10 0x00007fb74643bc35 in Perl_pmruntime () from /lib/x86_64-linux-gnu/libperl.so.5.30
#11 0x00007fb74647823b in Perl_yyparse () from /lib/x86_64-linux-gnu/libperl.so.5.30
#12 0x00007fb7465197a7 in ?? () from /lib/x86_64-linux-gnu/libperl.so.5.30
#13 0x00007fb74651f06d in Perl_pp_require () from /lib/x86_64-linux-gnu/libperl.so.5.30
#14 0x00007fb7464d44a6 in Perl_runops_standard () from /lib/x86_64-linux-gnu/libperl.so.5.30
#15 0x00007fb746441d24 in Perl_call_sv () from /lib/x86_64-linux-gnu/libperl.so.5.30
#16 0x00007fb746444950 in Perl_call_list () from /lib/x86_64-linux-gnu/libperl.so.5.30
#17 0x00007fb746421e00 in ?? () from /lib/x86_64-linux-gnu/libperl.so.5.30
#18 0x00007fb74643a9ff in Perl_newATTRSUB_x () from /lib/x86_64-linux-gnu/libperl.so.5.30
#19 0x00007fb74643df92 in Perl_utilize () from /lib/x86_64-linux-gnu/libperl.so.5.30
#20 0x00007fb7464786d9 in Perl_yyparse () from /lib/x86_64-linux-gnu/libperl.so.5.30
#21 0x00007fb7465197a7 in ?? () from /lib/x86_64-linux-gnu/libperl.so.5.30
#22 0x00007fb74651f06d in Perl_pp_require () from /lib/x86_64-linux-gnu/libperl.so.5.30
#23 0x00007fb7464d44a6 in Perl_runops_standard () from /lib/x86_64-linux-gnu/libperl.so.5.30
#24 0x00007fb746441d24 in Perl_call_sv () from /lib/x86_64-linux-gnu/libperl.so.5.30
#25 0x00007fb746444950 in Perl_call_list () from /lib/x86_64-linux-gnu/libperl.so.5.30
#26 0x00007fb746421e00 in ?? () from /lib/x86_64-linux-gnu/libperl.so.5.30
#27 0x00007fb74643a9ff in Perl_newATTRSUB_x () from /lib/x86_64-linux-gnu/libperl.so.5.30
#28 0x00007fb74643df92 in Perl_utilize () from /lib/x86_64-linux-gnu/libperl.so.5.30
#29 0x00007fb7464786d9 in Perl_yyparse () from /lib/x86_64-linux-gnu/libperl.so.5.30
#30 0x00007fb7465197a7 in ?? () from /lib/x86_64-linux-gnu/libperl.so.5.30
#31 0x00007fb74652476a in Perl_pp_entereval () from /lib/x86_64-linux-gnu/libperl.so.5.30
#32 0x00007fb746442605 in Perl_eval_sv () from /lib/x86_64-linux-gnu/libperl.so.5.30
#33 0x00007fb7464427fd in Perl_eval_pv () from /lib/x86_64-linux-gnu/libperl.so.5.30
#34 0x00007fb74673cbd3 in Perl_safe_eval (my_perl=my_perl@entry=0x7fb7381ad110, string=0x7fb7462293a0 "use lib '/usr/local/freeswitch/perl';\nuse freeswitch;\n$SWITCH_ENV{UUID} = \"773ec4b8-2168-11ec-be42-974253d8f472\";\n$session = new freeswitch::Session(\"773ec4b8-2168-11ec-be42-974253d8f472\")") at mod_perl.c:69
#35 0x00007fb74673d317 in perl_parse_and_execute (my_perl=0x7fb7381ad110, input_code=0x55f69c904d10 "/usr/local/freeswitch/scripts/test.pl", setup_code=<optimized out>) at mod_perl.c:90
#36 0x00007fb74673d9ab in perl_function (session=<optimized out>, data=0x55f69c904d10 "/usr/local/freeswitch/scripts/test.pl") at mod_perl.c:223
#37 0x00007fb75785ec05 in switch_core_session_exec (session=session@entry=0x55f69c8f49f8, application_interface=application_interface@entry=0x55f69c077770, arg=0x55f69c904d10 "/usr/local/freeswitch/scripts/test.pl") at src/switch_core_session.c:2900
#38 0x00007fb75785f344 in switch_core_session_execute_application_get_flags (session=session@entry=0x55f69c8f49f8, app=0x55f69c904d08 "perl", arg=<optimized out>, flags=flags@entry=0x0) at src/switch_core_session.c:2765
#39 0x00007fb757863662 in switch_core_standard_on_execute (session=0x55f69c8f49f8) at src/switch_core_state_machine.c:354
#40 switch_core_session_run (session=0x55f69c8f49f8) at src/switch_core_state_machine.c:651
#41 0x00007fb75785c572 in switch_core_session_thread (thread=<optimized out>, obj=0x55f69c8f49f8) at src/switch_core_session.c:1719
#42 0x00007fb757857adf in switch_core_session_thread_pool_worker (thread=0x55f69c919cb0, obj=<optimized out>) at src/switch_core_session.c:1782
#43 0x00007fb757b88de0 in dummy_worker (opaque=0x55f69c919cb0) at threadproc/unix/thread.c:151
#44 0x00007fb7577aa609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#45 0x00007fb7576d1293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

gdb_bts.log
perl_ubuntu.txt
perl_debian.txt
Dockerfile_Ubuntu_Compiled.txt
Dockerfile_DEBIAN_Compiled.txt

[mitchcapper]

Alright well in looking at something else I ran into an old bug I forgot about that was eerily similar:
https://freeswitch.org/jira/browse/FS-10071?filter=10071

This is likely due to a newer perl version causing the crash (just like then). At that time the problem was the proper way of calling clone was not happening. For clone's usage and then perl_parse you must call perl_alloc then call perl_construct and finally perl_parse with a null terminated array of parameters and the call must be valid (there we were not passing a correct value to -e). None of the mod_perl code has changed and we are segfaulting at a different spot. While clone is at the top of the stack in this case it is likely not the source of issue. From what I have found as well all the proper initialization is happening.

Perl/perl5#15917 is the original bug filed with perl when I thought it was a perl issue.

I did run some very old FS copies in the newer perl just to make sure it wasn't a FS change of some sort and it still crashes. I will see if I can track down what perl version broke it first.

[jimmyt86]

Hi @mitchcapper,

I have identified the same problem too. I upgraded from FS 1.6.20 on Ubuntu 16.04 to FS 1.10.7 on Ubuntu 20.04 using self compliled Freeswitch binaries. Ubuntu 16.04 provides perl 5.22, Ubuntu 20.04 provides perl 5.30.

The key I found in my trace is also in the first line of your backtrace, 'Perl__invlist_intersection_maybe_complement_2nd'. This relates directly to the perl bug here Perl/perl5#17154

The fix isn't available in perl 5.30, which is included in Ubuntu 20.04. Ubuntu also don't provide backports of perl as they use it too heavily for internal integrations. As my next steps I'm going to try moving to a later Ubuntu release which provides a later release of perl. See here for a list of which perl version is include in each Ubuntu release; https://packages.ubuntu.com/search?keywords=perl

Have you tried perl 5.32 or later?

[mitchcapper]

Confirmed fixed in ubuntu 21/22 along with 20.04 with custom compiled perl (attached sample dockerfile)
Dockerfile_Ubuntu_focal_perl_5_3_4.txt