Segmentation fault with Gitea on a Raspi 4 #3271

LuciferSam86 · 2019-10-06T14:47:09Z

Describe the bug
When I launch gitea I got "segmentation fault"

To reproduce

Download gitea binary 1.9.3 from here : https://dl.gitea.io/gitea/1.9.3/gitea-1.9.3-linux-arm-6
execute gitea web
Get "segmentation fault"

Expected behaviour
Gitea should executed and deploy his webserver.

Actual behaviour
I get segmentation fault

System
Copy and paste the results of the raspinfo command in to this section. Alternatively, copy and paste a pastebin link, or add answers to the following questions:

Which model of Raspberry Pi? 4B with 4 GB of RAM
Which OS and version (cat /etc/rpi-issue)?

Raspberry Pi reference 2019-09-26
Generated using pi-gen, https://github.com/RPi-Distro/pi-gen, 80d486687ea77d31fc3fc13cf3a2f8b464e129be, stage2

Which firmware version (vcgencmd version)?

Sep 24 2019 17:34:30
Copyright (c) 2012 Broadcom
version cd3add54955f8fa065b414d8fc07c525e7ddffc8 (clean) (release) (start)

Which kernel version (uname -a)?
Linux raspberrypi 4.19.75-v7l+ #1270 SMP Tue Sep 24 18:51:41 BST 2019 armv7l GNU/Linux

Logs
The only log I have is from his stack trace using dbg

Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x0118d598 in brk ()
warning: Missing auto-load script at offset 0 in section .debug_gdb_scripts
of file /mnt/bigdata/stuff/gitea.
Use `info auto-load python-scripts [REGEXP]' to list them.
(gdb) where
#0  0x0118d598 in brk ()
#1  0x01167c14 in sbrk ()
#2  0x01140378 in __libc_setup_tls ()
#3  0x0113ff2c in __libc_start_main ()
#4  0x0001022c in _start ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Additional context
This answer from the ticket I opened at gitea github say if you use an older version of raspi kernel it works without problem:
go-gitea/gitea#8373 (comment)

EDIT : https://github.molgen.mpg.de/git-mirror/glibc/blob/20003c49884422da7ffbc459cdeee768a6fee07b/csu/libc-tls.c#L105

This is the function where the segfault happens.

The text was updated successfully, but these errors were encountered:

pelwell · 2019-10-06T14:54:59Z

To save others following the link:

On my pi running kernel version: 4.19.66-v7+ gitea is working without problems. The other two pi´s - where gitea gave the "segmentation fault" - are running kernel version: 4.19.75-v7+.

UweHeber · 2019-10-09T19:06:54Z

Thanks for tracking this this issue. Have the same problem with gitea 1.9.2 to 1.9.4 (arm v6) on my RaspberryPI 4

$ cat /proc/device-tree/model
Raspberry Pi 4 Model B Rev 1.1p

$ uname -a
Linux raspberrypi 4.19.75-v7l+ #1270 SMP Tue Sep 24 18:51:41 BST 2019 armv7l GNU/Linux

$ cat /proc/version
Linux version 4.19.75-v7l+ (dom@buildbot) (gcc version 4.9.3 (crosstool-NG crosstool-ng-1.22.0-88-g8460611)) #1270 SMP Tue Sep 24 18:51:41 BST 2019

petergil · 2019-10-10T16:51:48Z

I can confirm that this also affects my Raspberry Pi 3 Model B Rev 1.2 so not restricted to rpi4.

LuciferSam86 · 2019-10-10T17:35:47Z

Another interesting stuff , when I execute gitea under dbg , it works and it can deploy its webserver. At least on archlinux for raspi 4. But outside dbg I got segfault . Why?

ciph · 2019-10-10T18:23:24Z

I have the same on my Raspberry Pi 3 B+.

Describe the bug
When I launch gitea I got "segmentation fault".

To reproduce
Download gitea binary 1.7.6, 1.8.0, 1.9.4 from here : https://dl.gitea.io/gitea/1.9.4/gitea-1.9.4-linux-arm-6
execute gitea web
Get "segmentation fault"

Expected behaviour
Gitea should execute and run the server

Actual behaviour
I get segmentation fault and the server does not start

System
Copy and paste the results of the raspinfo command in to this section. Alternatively, copy and paste a pastebin link, or add answers to the following questions:

Which model of Raspberry Pi? Pi3B+
Which OS and version (cat /etc/rpi-issue)? Raspberry Pi reference 2019-07-10 Generated using pi-gen, https://github.com/RPi-Distro/pi-gen, 175dfb027ffabd4b8d5080097af0e51ed9a4a56c, stage2
Which firmware version (vcgencmd version)? Oct 4 2019 15:20:55 Copyright (c) 2012 Broadcom version 8ebb91b5731d2595643426b007a9cee64217fc15 (tainted) (release) (start_cd)
Which kernel version (uname -a)? Linux 4.19.76-v7+ USB disconnects and reconnects cause kernel panics #1272 SMP Fri Oct 4 14:48:47 BST 2019 armv7l GNU/Linux

LuciferSam86 · 2019-10-10T19:47:23Z

Even under raspbian running gitea under gdb gitea it works and I can connect to gitea webserver.

For someone like me that doesn't understand this stuff it's pretty crazy :)

here the screenshots : https://postimg.cc/gallery/2ba368sw2/

pelwell · 2019-10-10T20:01:34Z

I haven't started looking at this yet, but you might be able to speed up the investigation if you can run Gitea under strace and capture the output.

LuciferSam86 · 2019-10-10T20:32:16Z

Hi pelwell, this is when I execute strace ./gitea web

execve("./gitea", ["./gitea", "web"], 0xbe908704 /* 21 vars */) = 0
brk(NULL)                               = 0x15f2000
brk(0x15f2d18)                          = 0x15f2000
--- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_MAPERR, si_addr=0x24} ---
+++ killed by SIGSEGV (core dumped) +++

this is when I execute strace ./gitea

execve("./gitea", ["./gitea"], 0xbeaf3710 /* 21 vars */) = 0
brk(NULL)                               = 0xed7000
brk(0xed7d18)                           = 0xed7000
--- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_MAPERR, si_addr=0x24} ---
+++ killed by SIGSEGV (core dumped) +++

I suppose you want I run gitea under gdb and run "strace" but I have no idea how to do.
Do you have some guide I can read for doing that?

Thanks!

focmb · 2019-10-11T07:31:41Z

Same here after the upgrade to kernel 4.19.75-v7l+ (SMP) armv7l. Before the upgrade I am running kernel 4.19.66 and gitea was working fine. After the upgrade with no other changes gitea won't start.

pelwell · 2019-10-11T08:42:14Z

I can reproduce this - it's on the list to look at.

holta · 2019-10-11T13:03:27Z

Is #3279 [raspberrypi-kernel-headers (1.20190925-2): Wrong exec format in scripts in buster] perhaps related?

SuNNjek · 2019-10-11T14:09:12Z

Is there a way to temporarily downgrade the kernel again so Gitea can be run again? It seems the old kernel was removed from the repository

pelwell · 2019-10-11T14:14:52Z

We never delete firmware builds - you can install the 4.19.66 kernel using sudo rpi-update 11ef4498.

This reverts commit c0ccb4d ("binfmt_elf: move brk out of mmap when doing direct loader exec") See: #3271 Signed-off-by: Phil Elwell <[email protected]>

pelwell · 2019-10-11T15:28:14Z

I've found the culprit: c0ccb4d

It's now reverted on rpi-4.19.y, and Gitea runs again.

@kees Do you have an idea why this particular executable is failing on Pi?

tmm1 · 2019-10-11T19:01:52Z

I'm experiencing the same issue on 4.19.75-v7l+ with a statically compiled comskip binary. Same binary works on 4.19.71-v7l+

I was able to get a backtrace by forcing core dump (ulimit -c unlimited), then loading that dump into gdb:

[New LWP 6139]
Core was generated by `/mnt/channels/channels-dvr/latest/comskip video.mpg'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x005b2090 in brk ()
(gdb) bt
#0  0x005b2090 in brk ()
#1  0x0058c07e in sbrk ()
#2  0x00564f26 in __libc_setup_tls ()
#3  0x0051d13c in __pthread_initialize_minimal_internal ()
#4  0x00564cfa in __libc_start_main ()
#5  0x0002fede in _start ()

popcornmix · 2019-10-11T19:02:30Z

Latest rpi-update kernel may have a fix for this.

holta · 2019-10-11T21:15:04Z

FYI the Gitea and Kiwix segmentation faults (kiwix/kiwix-tools#341) are NOT solved for me when I do apt update && apt -y dist-upgrade && reboot to upgrade raspberrypi-kernel (1.20190925+1-1) over (1.20190925-2).

CLARIF: the regression persists on (both) RPi 3 and RPi 4.

tmm1 · 2019-10-11T21:17:42Z

You need to run sudo rpi-update

holta · 2019-10-11T21:24:30Z

You need to run sudo rpi-update

Why is that specifically?
Given apt -y dist-upgrade says it already installed kernel (1.20190925+1-1) over (1.20190925-2) -- and then I rebooted.

tmm1 · 2019-10-11T21:26:45Z

Because the fix was made a few hours ago and is not contained in package from 9/25?

holta · 2019-10-11T21:31:37Z

Because the fix was made a few hours ago and is not contained in package from 9/25?

Ok then I was mistaken in assuming #3279 (comment) was the fix. Apologies.

Certainly applying kernel (1.20190925+1-1) over (1.20190925-2) is insufficient.

holta · 2019-10-11T22:56:50Z

I can confirm that running sudo rpi-update on a Raspberry Pi 4 fixed both problems -- allowing Gitea (iiab/iiab#1999) and Kiwix (iiab/iiab#1993) to start!

FYI kernel goes from...

root@box:~# uname -a
Linux box.lan 4.19.75-v7l+ #1270 SMP Tue Sep 24 18:51:41 BST 2019 armv7l GNU/Linux

To...

root@box:~# uname -a
Linux box.lan 4.19.79-v7l+ #1273 SMP Fri Oct 11 18:23:45 BST 2019 armv7l GNU/Linux

holta · 2019-10-13T00:39:01Z

Question: how many days is likely, until regular users get their kernel fixed here?

(via apt update etc)

ciph · 2019-10-13T15:22:06Z

I just run apt-get update && apt-get dist-upgrade, then rpi-update I am now on 4.19.79-v7+ #1273 SMP Fri Oct 11 18:13:16 BST 2019 armv7l GNU/Linux.

Now gitea is starting successfully with https://dl.gitea.io/gitea/1.9.4/gitea-1.9.4-linux-arm-6
Thank you so much!

LuciferSam86 · 2019-10-13T16:17:12Z

Welp, it's resolved.
I'm gonna close the ticket.

SteVader · 2019-11-08T10:56:36Z

Thank you all so much for this. I've been wrestling with the issue on and off for almost a month before finding this.

pdfruth · 2019-11-19T14:09:30Z

@pelwell
Any ideas when commit 68a2665 will make it's way into the mainstream kernel repositories so we can receive the fix via the normal apt update && apt upgrade process?

pelwell · 2019-11-19T14:23:04Z

@XECDesign When are we likely to next bump the kernel package?

XECDesign · 2019-11-19T14:26:13Z

@pelwell @popcornmix I'm waiting for something new to show up on https://github.com/raspberrypi/firmware/commits/stable/

nickbp · 2019-12-04T10:58:07Z

@pelwell @popcornmix I'm waiting for something new to show up on https://github.com/raspberrypi/firmware/commits/stable/

Looks like something new has since shown up? Anything else blocking a release? It's now been a few weeks. raspberrypi/firmware@878d27e

I'm hesitant to reach beyond the stable apt channel as the last rpi-update I had tried resulted in needing an sdcard wipe and reinstall from scratch.

In my case I am seeing this same segfault/stack with a build of Drone CI on an RPi4/4GB running 4.19.75-v7l+, specifically the build provided within this image: https://hub.docker.com/layers/drone/drone/latest/images/sha256-722f8062c3b683005ca1a1f96216b65bf0df1cc4a3a3aa0f2911d71307ce98ce .

nickbp · 2019-12-06T06:45:01Z

I gave up waiting for a kernel release to finally be cut and tried an rpi-update across a few boards. So far, the board I tried it on happily didn't need a reformat this time, so that's cool. With the new 4.19.86-v7l+ kernel the segfaults I was seeing have gone away too -- also cool.

Now off to see if I can successfully rpi-update the remaining boards...

aelth · 2019-12-23T20:33:54Z

@pelwell @popcornmix I'm waiting for something new to show up on https://github.com/raspberrypi/firmware/commits/stable/

Please, can you release the new firmware, so that the fix will be available without rpi-update procedure?

The month has passed since the bug has been fixed, but new kernel has not yet been released:(

JamesH65 · 2019-12-23T22:17:25Z

It will be released early in the new year. The guy who does it is on holiday.

If you need it urgently, AFAIK there are not any major issues with the code in rpi-update so that should be safe.

SuNNjek · 2020-01-17T09:40:31Z

@JamesH65 We're now half-way through January, any chance we'll get an update soon?

JamesH65 · 2020-01-17T09:47:49Z

Should be sometime next week. Been slightly delayed for various reasons (Xmas, holidays, illness etc).

SuNNjek · 2020-01-17T09:59:11Z

@JamesH65 Good to know, thanks 👍

This was referenced Oct 11, 2019

Gitea fails on Raspberry Pi iiab/iiab#1999

Closed

Kiwix install fails as part of install of current master [segmentation fault, due to Raspbian or Ubuntu 19.10 kernel on RPi] iiab/iiab#1993

Closed

kiwix-serve segmentation fault kiwix/kiwix-tools#341

Closed

LuciferSam86 closed this as completed Oct 13, 2019

5schatten mentioned this issue Oct 13, 2019

docker: updated to 19.03.3 LibreELEC/LibreELEC.tv#3889

Closed

Sunbread mentioned this issue Oct 14, 2019

Segmentation fault while trying executing binary go-gitea/gitea#8511

Closed

7 tasks

tha2015 mentioned this issue Oct 29, 2019

Segmentation fault on ARM with version 2.5.2 weaveworks/weave#3717

Closed

This was referenced Dec 9, 2019

DietPi-Software | Gitea: Fails on RPi2/3/4 with segmentation fault MichaIng/DietPi#3248

Closed

raspberry 4 raspbian buster install ERROR: Segmentation fault go-gitea/gitea#8909

Closed

Segmentation fault with Gitea on a Raspi 4 #3271

Segmentation fault with Gitea on a Raspi 4 #3271

Comments

LuciferSam86 commented Oct 6, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

pelwell commented Oct 6, 2019

Uh oh!

UweHeber commented Oct 9, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

petergil commented Oct 10, 2019

Uh oh!

LuciferSam86 commented Oct 10, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ciph commented Oct 10, 2019

Uh oh!

LuciferSam86 commented Oct 10, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pelwell commented Oct 10, 2019

Uh oh!

LuciferSam86 commented Oct 10, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

focmb commented Oct 11, 2019

Uh oh!

pelwell commented Oct 11, 2019

Uh oh!

holta commented Oct 11, 2019

Uh oh!

SuNNjek commented Oct 11, 2019

Uh oh!

pelwell commented Oct 11, 2019

Uh oh!

pelwell commented Oct 11, 2019

Uh oh!

tmm1 commented Oct 11, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

popcornmix commented Oct 11, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

holta commented Oct 11, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tmm1 commented Oct 11, 2019

Uh oh!

holta commented Oct 11, 2019

Uh oh!

tmm1 commented Oct 11, 2019

Uh oh!

holta commented Oct 11, 2019

Uh oh!

holta commented Oct 11, 2019

Uh oh!

holta commented Oct 13, 2019

Uh oh!

ciph commented Oct 13, 2019

Uh oh!

LuciferSam86 commented Oct 13, 2019

Uh oh!

SteVader commented Nov 8, 2019

Uh oh!

pdfruth commented Nov 19, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pelwell commented Nov 19, 2019

Uh oh!

XECDesign commented Nov 19, 2019

Uh oh!

nickbp commented Dec 4, 2019

Uh oh!

nickbp commented Dec 6, 2019

Uh oh!

aelth commented Dec 23, 2019

Uh oh!

JamesH65 commented Dec 23, 2019

Uh oh!

SuNNjek commented Jan 17, 2020

LuciferSam86 commented Oct 6, 2019 •

edited

Loading

UweHeber commented Oct 9, 2019 •

edited

Loading

LuciferSam86 commented Oct 10, 2019 •

edited

Loading

LuciferSam86 commented Oct 10, 2019 •

edited

Loading

LuciferSam86 commented Oct 10, 2019 •

edited

Loading

tmm1 commented Oct 11, 2019 •

edited

Loading

popcornmix commented Oct 11, 2019 •

edited

Loading

holta commented Oct 11, 2019 •

edited

Loading

pdfruth commented Nov 19, 2019 •

edited

Loading