Please enable CONFIG_PSTORE for ramoops support

[hailfinger]

Describe the bug

CONFIG_PSTORE=y allows storing messages in a RAM buffer which survives reboot/reset/watchdog.
Together with CONFIG_PSTORE_CONSOLE=y (kernel messages in that RAM buffer) and CONFIG_PSTORE_RAM=y (Oops/Panic messages in that RAM buffer) most of the relevant info before a crash/hang can survive a reboot.
As a nice bonus, enabling CONFIG_PSTORE_DEFLATE_COMPRESS=y will compress the data before storing it, making more efficient use of the 10 kBytes set aside for this purpose.

This helps a lot if you want to debug kernel crashes/hangs without a serial console attached.

Steps to reproduce the behaviour

The full Howto is here https://forums.raspberrypi.com/viewtopic.php?t=199047

Device (s)

Raspberry Pi 4 Mod. B

System

pi@raspberrypi:~ $ cat /etc/rpi-issue
Raspberry Pi reference 2022-04-04
Generated using pi-gen, https://github.com/RPi-Distro/pi-gen, 27a8050c3c06e567c794620394a8c2d74262a516, stage4
pi@raspberrypi:~ $ vcgencmd version
Mar 24 2022 13:19:26
Copyright (c) 2012 Broadcom
version e5a963efa66a1974127860b42e913d2374139ff5 (clean) (release) (start)
pi@raspberrypi:~ $ uname -a
Linux raspberrypi 5.15.32-v8+ #1538 SMP PREEMPT Thu Mar 31 19:40:39 BST 2022 aarch64 GNU/Linux

Logs

No response

Additional context

Thanks!

[pelwell]

I started to look at this before starting the weekend early. My intention is to build the pstore support as a module, with an overlay to reserve the memory and cause the module to be loaded, but the first attempt crashed and burned - probably because of a change in the base DTBs since 2017 making them incompatible with @notro's overlay. I'll have another look if I get a moment.

[hailfinger]

Thank you for working on this.
Good point about making this a module. Given that the memory is reserved via device tree, loading this as module will still allow data retrieval.
The only downside about compiling ramoops as a module is that it can only capture data starting at the point where it is loaded, i.e. crashes before mounting the root filesystem will be lost. That might be unlikely in the general case, but I regularly use the awesome "tryboot" functionality for testing possibly unstable kernel features and device tree overlays. Compiling in ramoops would be more convenient for me, but my primary goal is to have it available at all.

[pelwell]

Building ramoops and pstore as modules running on a 3B (what I happened to have connected and booted):

[   11.741317] ramoops: ramoops_probe:
[   11.810353] pstore: Using crash dump compression: deflate
[   11.810400] pstore: Registered ramoops as persistent store backend
[   11.810417] ramoops: using 0x10000@0xb000000, ecc: 0

and with them built-in:

[    0.059230] ramoops: ramoops_probe:
[    0.059382] pstore: Registered ramoops as persistent store backend
[    0.059417] ramoops: using 0x10000@0xb000000, ecc: 0
[    3.469833] pstore: Using crash dump compression: deflate

That's quite a difference in terms of the point at which ramoops becomes usable.

In terms of memory usage, the built-in version takes up an extra 16kB when not in use (i.e. how much are we "taxing" the millions of users who will never use ramoops), which isn't great - I'm not decided yet.

The awkward aspect of the ramoops mechanism is that the memory buffer must be declared with an explicit address as well as the size, which could make it tricky to create a single, simple overlay to work with all configurations. An alternative would be to give each base DTB file a disabled ramoops declaration, with a dtoverlay to enable it. This is more work, but more likely to give a satisfactory result.

[pelwell]

That's enabled now - see cf28d57 and eda0160.

In the end, the fixed address blob seems to work OK, although I did add a parameter to allow it to be moved. The Pi 4 family required a modified overlay because #address-cells is 2 rather than 1, but the overlay_map file should take care of that - just use dtoverlay=ramoops on all devices.

[popcornmix]

Latest rpi-update kernel contains this feature

[hailfinger]

Thank you so much! I'm using this with the following snippet in config.txt:
dtoverlay=ramoops,console-size=0x4000
It works pretty well (last 16k of dmesg available). The log of the previous dmesg will be in /sys/fs/pstore/console-ramoops-0 .

Please note that the settings in /etc/sysctl.d/98-rpi.conf may need to be adjusted to capture all relevant dmesg data after setting sysctls. (Specifically, kernel.printk needs 7 as first number.) An alternative would be to add "max-reason" to the dts which allows configuring this filter without spamming the console.

Oh, and if you want somewhat reliable logging of previous dmesg, do not use systemd-pstore. The first (cold) boot will be logged correctly, but any subsequent reboots will only have the last line of dmesg available because systemd clears the pstore at the wrong time.

[hailfinger]

There is also an ecc parameter which enables automatic error correction in case some bits flip in RAM during reboot/reset. Not a requirement for me, but I thought I'd mention it in case someone ever wants to find all related info in one place.

The good thing about the additional dts parameters ("max-reason" and "ecc") is that they don't need a kernel recompile.

[pelwell]

OK to close?

[hailfinger]

Yes. Thank you!