Skip to content

Segmentation fault with Gitea on a Raspi 4 #3271

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
LuciferSam86 opened this issue Oct 6, 2019 · 36 comments
Closed

Segmentation fault with Gitea on a Raspi 4 #3271

LuciferSam86 opened this issue Oct 6, 2019 · 36 comments

Comments

@LuciferSam86
Copy link

LuciferSam86 commented Oct 6, 2019

Describe the bug
When I launch gitea I got "segmentation fault"

To reproduce

  1. Download gitea binary 1.9.3 from here : https://dl.gitea.io/gitea/1.9.3/gitea-1.9.3-linux-arm-6
  2. execute gitea web
  3. Get "segmentation fault"

Expected behaviour
Gitea should executed and deploy his webserver.

Actual behaviour
I get segmentation fault

System
Copy and paste the results of the raspinfo command in to this section. Alternatively, copy and paste a pastebin link, or add answers to the following questions:

  • Which model of Raspberry Pi? 4B with 4 GB of RAM
  • Which OS and version (cat /etc/rpi-issue)?
Raspberry Pi reference 2019-09-26
Generated using pi-gen, https://github.com/RPi-Distro/pi-gen, 80d486687ea77d31fc3fc13cf3a2f8b464e129be, stage2
  • Which firmware version (vcgencmd version)?
Sep 24 2019 17:34:30
Copyright (c) 2012 Broadcom
version cd3add54955f8fa065b414d8fc07c525e7ddffc8 (clean) (release) (start)
  • Which kernel version (uname -a)?
    Linux raspberrypi 4.19.75-v7l+ #1270 SMP Tue Sep 24 18:51:41 BST 2019 armv7l GNU/Linux

Logs
The only log I have is from his stack trace using dbg

Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x0118d598 in brk ()
warning: Missing auto-load script at offset 0 in section .debug_gdb_scripts
of file /mnt/bigdata/stuff/gitea.
Use `info auto-load python-scripts [REGEXP]' to list them.
(gdb) where
#0  0x0118d598 in brk ()
#1  0x01167c14 in sbrk ()
#2  0x01140378 in __libc_setup_tls ()
#3  0x0113ff2c in __libc_start_main ()
#4  0x0001022c in _start ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Additional context
This answer from the ticket I opened at gitea github say if you use an older version of raspi kernel it works without problem:
go-gitea/gitea#8373 (comment)

EDIT : https://github.molgen.mpg.de/git-mirror/glibc/blob/20003c49884422da7ffbc459cdeee768a6fee07b/csu/libc-tls.c#L105

This is the function where the segfault happens.

@pelwell
Copy link
Contributor

pelwell commented Oct 6, 2019

To save others following the link:

On my pi running kernel version: 4.19.66-v7+ gitea is working without problems. The other two pi´s - where gitea gave the "segmentation fault" - are running kernel version: 4.19.75-v7+.

@UweHeber
Copy link

UweHeber commented Oct 9, 2019

Thanks for tracking this this issue. Have the same problem with gitea 1.9.2 to 1.9.4 (arm v6) on my RaspberryPI 4

$ cat /proc/device-tree/model
Raspberry Pi 4 Model B Rev 1.1p

$ uname -a
Linux raspberrypi 4.19.75-v7l+ #1270 SMP Tue Sep 24 18:51:41 BST 2019 armv7l GNU/Linux

$ cat /proc/version
Linux version 4.19.75-v7l+ (dom@buildbot) (gcc version 4.9.3 (crosstool-NG crosstool-ng-1.22.0-88-g8460611)) #1270 SMP Tue Sep 24 18:51:41 BST 2019

@petergil
Copy link

I can confirm that this also affects my Raspberry Pi 3 Model B Rev 1.2 so not restricted to rpi4.

@LuciferSam86
Copy link
Author

LuciferSam86 commented Oct 10, 2019

Another interesting stuff , when I execute gitea under dbg , it works and it can deploy its webserver. At least on archlinux for raspi 4. But outside dbg I got segfault . Why?

@ciph
Copy link

ciph commented Oct 10, 2019

I have the same on my Raspberry Pi 3 B+.

Describe the bug
When I launch gitea I got "segmentation fault".

To reproduce
Download gitea binary 1.7.6, 1.8.0, 1.9.4 from here : https://dl.gitea.io/gitea/1.9.4/gitea-1.9.4-linux-arm-6
execute gitea web
Get "segmentation fault"

Expected behaviour
Gitea should execute and run the server

Actual behaviour
I get segmentation fault and the server does not start

System
Copy and paste the results of the raspinfo command in to this section. Alternatively, copy and paste a pastebin link, or add answers to the following questions:

  • Which model of Raspberry Pi? Pi3B+
  • Which OS and version (cat /etc/rpi-issue)? Raspberry Pi reference 2019-07-10 Generated using pi-gen, https://github.com/RPi-Distro/pi-gen, 175dfb027ffabd4b8d5080097af0e51ed9a4a56c, stage2
  • Which firmware version (vcgencmd version)? Oct 4 2019 15:20:55 Copyright (c) 2012 Broadcom version 8ebb91b5731d2595643426b007a9cee64217fc15 (tainted) (release) (start_cd)
  • Which kernel version (uname -a)? Linux 4.19.76-v7+ USB disconnects and reconnects cause kernel panics #1272 SMP Fri Oct 4 14:48:47 BST 2019 armv7l GNU/Linux

@LuciferSam86
Copy link
Author

LuciferSam86 commented Oct 10, 2019

Even under raspbian running gitea under gdb gitea it works and I can connect to gitea webserver.

For someone like me that doesn't understand this stuff it's pretty crazy :)

here the screenshots : https://postimg.cc/gallery/2ba368sw2/

@pelwell
Copy link
Contributor

pelwell commented Oct 10, 2019

I haven't started looking at this yet, but you might be able to speed up the investigation if you can run Gitea under strace and capture the output.

@LuciferSam86
Copy link
Author

LuciferSam86 commented Oct 10, 2019

Hi pelwell, this is when I execute strace ./gitea web

execve("./gitea", ["./gitea", "web"], 0xbe908704 /* 21 vars */) = 0
brk(NULL)                               = 0x15f2000
brk(0x15f2d18)                          = 0x15f2000
--- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_MAPERR, si_addr=0x24} ---
+++ killed by SIGSEGV (core dumped) +++

this is when I execute strace ./gitea

execve("./gitea", ["./gitea"], 0xbeaf3710 /* 21 vars */) = 0
brk(NULL)                               = 0xed7000
brk(0xed7d18)                           = 0xed7000
--- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_MAPERR, si_addr=0x24} ---
+++ killed by SIGSEGV (core dumped) +++

I suppose you want I run gitea under gdb and run "strace" but I have no idea how to do.
Do you have some guide I can read for doing that?

Thanks!

@focmb
Copy link

focmb commented Oct 11, 2019

Same here after the upgrade to kernel 4.19.75-v7l+ (SMP) armv7l. Before the upgrade I am running kernel 4.19.66 and gitea was working fine. After the upgrade with no other changes gitea won't start.

@pelwell
Copy link
Contributor

pelwell commented Oct 11, 2019

I can reproduce this - it's on the list to look at.

@holta
Copy link

holta commented Oct 11, 2019

Is #3279 [raspberrypi-kernel-headers (1.20190925-2): Wrong exec format in scripts in buster] perhaps related?

@SuNNjek
Copy link

SuNNjek commented Oct 11, 2019

Is there a way to temporarily downgrade the kernel again so Gitea can be run again? It seems the old kernel was removed from the repository

@pelwell
Copy link
Contributor

pelwell commented Oct 11, 2019

We never delete firmware builds - you can install the 4.19.66 kernel using sudo rpi-update 11ef4498.

pelwell pushed a commit that referenced this issue Oct 11, 2019
This reverts commit c0ccb4d
("binfmt_elf: move brk out of mmap when doing direct loader exec")

See: #3271

Signed-off-by: Phil Elwell <[email protected]>
@pelwell
Copy link
Contributor

pelwell commented Oct 11, 2019

I've found the culprit: c0ccb4d

It's now reverted on rpi-4.19.y, and Gitea runs again.

@kees Do you have an idea why this particular executable is failing on Pi?

@tmm1
Copy link

tmm1 commented Oct 11, 2019

I'm experiencing the same issue on 4.19.75-v7l+ with a statically compiled comskip binary. Same binary works on 4.19.71-v7l+

I was able to get a backtrace by forcing core dump (ulimit -c unlimited), then loading that dump into gdb:

[New LWP 6139]
Core was generated by `/mnt/channels/channels-dvr/latest/comskip video.mpg'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x005b2090 in brk ()
(gdb) bt
#0  0x005b2090 in brk ()
#1  0x0058c07e in sbrk ()
#2  0x00564f26 in __libc_setup_tls ()
#3  0x0051d13c in __pthread_initialize_minimal_internal ()
#4  0x00564cfa in __libc_start_main ()
#5  0x0002fede in _start ()

@popcornmix
Copy link
Collaborator

popcornmix commented Oct 11, 2019

Latest rpi-update kernel may have a fix for this.

@holta
Copy link

holta commented Oct 11, 2019

FYI the Gitea and Kiwix segmentation faults (kiwix/kiwix-tools#341) are NOT solved for me when I do apt update && apt -y dist-upgrade && reboot to upgrade raspberrypi-kernel (1.20190925+1-1) over (1.20190925-2).

CLARIF: the regression persists on (both) RPi 3 and RPi 4.

@tmm1
Copy link

tmm1 commented Oct 11, 2019

You need to run sudo rpi-update

@holta
Copy link

holta commented Oct 11, 2019

You need to run sudo rpi-update

Why is that specifically?
Given apt -y dist-upgrade says it already installed kernel (1.20190925+1-1) over (1.20190925-2) -- and then I rebooted.

@tmm1
Copy link

tmm1 commented Oct 11, 2019

Because the fix was made a few hours ago and is not contained in package from 9/25?

@holta
Copy link

holta commented Oct 11, 2019

Because the fix was made a few hours ago and is not contained in package from 9/25?

Ok then I was mistaken in assuming #3279 (comment) was the fix. Apologies.

Certainly applying kernel (1.20190925+1-1) over (1.20190925-2) is insufficient.

@holta
Copy link

holta commented Oct 11, 2019

I can confirm that running sudo rpi-update on a Raspberry Pi 4 fixed both problems -- allowing Gitea (iiab/iiab#1999) and Kiwix (iiab/iiab#1993) to start!

FYI kernel goes from...

root@box:~# uname -a
Linux box.lan 4.19.75-v7l+ #1270 SMP Tue Sep 24 18:51:41 BST 2019 armv7l GNU/Linux

To...

root@box:~# uname -a
Linux box.lan 4.19.79-v7l+ #1273 SMP Fri Oct 11 18:23:45 BST 2019 armv7l GNU/Linux

@holta
Copy link

holta commented Oct 13, 2019

Question: how many days is likely, until regular users get their kernel fixed here?

(via apt update etc)

@ciph
Copy link

ciph commented Oct 13, 2019

I just run apt-get update && apt-get dist-upgrade, then rpi-update I am now on 4.19.79-v7+ #1273 SMP Fri Oct 11 18:13:16 BST 2019 armv7l GNU/Linux.

Now gitea is starting successfully with https://dl.gitea.io/gitea/1.9.4/gitea-1.9.4-linux-arm-6
Thank you so much!

@LuciferSam86
Copy link
Author

Welp, it's resolved.
I'm gonna close the ticket.

@SteVader
Copy link

SteVader commented Nov 8, 2019

Thank you all so much for this. I've been wrestling with the issue on and off for almost a month before finding this.

@pdfruth
Copy link

pdfruth commented Nov 19, 2019

@pelwell
Any ideas when commit 68a2665 will make it's way into the mainstream kernel repositories so we can receive the fix via the normal apt update && apt upgrade process?

@pelwell
Copy link
Contributor

pelwell commented Nov 19, 2019

@XECDesign When are we likely to next bump the kernel package?

@XECDesign
Copy link
Contributor

@pelwell @popcornmix I'm waiting for something new to show up on https://github.com/raspberrypi/firmware/commits/stable/

@nickbp
Copy link

nickbp commented Dec 4, 2019

@pelwell @popcornmix I'm waiting for something new to show up on https://github.com/raspberrypi/firmware/commits/stable/

Looks like something new has since shown up? Anything else blocking a release? It's now been a few weeks. raspberrypi/firmware@878d27e

I'm hesitant to reach beyond the stable apt channel as the last rpi-update I had tried resulted in needing an sdcard wipe and reinstall from scratch.

In my case I am seeing this same segfault/stack with a build of Drone CI on an RPi4/4GB running 4.19.75-v7l+, specifically the build provided within this image: https://hub.docker.com/layers/drone/drone/latest/images/sha256-722f8062c3b683005ca1a1f96216b65bf0df1cc4a3a3aa0f2911d71307ce98ce .

@nickbp
Copy link

nickbp commented Dec 6, 2019

I gave up waiting for a kernel release to finally be cut and tried an rpi-update across a few boards. So far, the board I tried it on happily didn't need a reformat this time, so that's cool. With the new 4.19.86-v7l+ kernel the segfaults I was seeing have gone away too -- also cool.

Now off to see if I can successfully rpi-update the remaining boards...

@aelth
Copy link

aelth commented Dec 23, 2019

@pelwell @popcornmix I'm waiting for something new to show up on https://github.com/raspberrypi/firmware/commits/stable/

Please, can you release the new firmware, so that the fix will be available without rpi-update procedure?

The month has passed since the bug has been fixed, but new kernel has not yet been released:(

@JamesH65
Copy link
Contributor

It will be released early in the new year. The guy who does it is on holiday.

If you need it urgently, AFAIK there are not any major issues with the code in rpi-update so that should be safe.

@SuNNjek
Copy link

SuNNjek commented Jan 17, 2020

@JamesH65 We're now half-way through January, any chance we'll get an update soon?

@JamesH65
Copy link
Contributor

Should be sometime next week. Been slightly delayed for various reasons (Xmas, holidays, illness etc).

@SuNNjek
Copy link

SuNNjek commented Jan 17, 2020

@JamesH65 Good to know, thanks 👍

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests