5.10.17-v8+ Kernel Panic while poweroff - Null Pointer Exception - with TPM-Module SLB 9670

[LloydS42]

Describe the bug
Kernel Panic - Null Pointer Exception - System freeze

To reproduce
Reboot or Shutdown the System

Expected behaviour
Kernel upgrade in January 2021 from Kernel Version 5.4.83 to 5.10.17

Actual behaviour
Raspberry Pi4 with currently RaspianOS

System

• Which model of Raspberry Pi? e.g. Pi3B+, PiZeroW (dmesg | grep Machine)
  Raspberry Pi 4 Model B Rev 1.2
• Which OS and version (cat /etc/rpi-issue)?
  Raspberry Pi reference 2020-08-20
  Generated using pi-gen, https://github.com/RPi-Distro/pi-gen, 7252c154838ec5b4576f29c996ac8fe3750cae12, stage4
• Which firmware version (vcgencmd version)?
  Feb 25 2021 12:10:40
  Copyright (c) 2012 Broadcom
  version 564e5f9b852b23a330b1764bcf0b2d022a20afd0 (clean) (release) (start)
• Which kernel version (uname -a)?
  Linux blueberry 5.10.17-v8+ vchiq_arm: do not use page_cache_release(page) macro #1403 SMP PREEMPT Mon Feb 22 11:37:54 GMT 2021 aarch64 GNU/Linux

Logs
Take a look under additional content

Additional context
Find discusion and more details in forum: Raspberrypi.org/forum

[paulenuta]

Same for 32-bit architecture Linux raspberrypi 5.10.17-v7+ #1403 SMP Mon Feb 22 11:29:51 GMT 2021 armv7l.

[pelwell]

We don't have TPMs and don't really support them, but it looks as though the "ops" field of tpm_chip is NULL when tpm_clk_enable is called at the beginning of tpm_start.

I wonder if tpm_del_char_device and tpm_class_shutdown are being called from different threads? Try adding some debug output to log entry to and exit from those functions, along with tpm_chip_unregister.

[LloydS42]

The system information described here comes from a 64bit kernel and userland. I have the image of: downloads.raspberry.org, which is the official RaspianOS 64Bit beta version.
After installation, the system was updated with apt update/upgrade, not anymore.
I have another SD card with the official RaspbinOS, where the 64bit kernel runs with a 32bit userland. The result is also the kernel panic. I'm sure the error has been coming from 5.4 to 5.10 since the kernel update, whether under 32bit or 64bit.

[paulenuta]

We don't have TPMs and don't really support them, but it looks as though the "ops" field of tpm_chip is NULL when tpm_clk_enable is called at the beginning of tpm_start.

I wonder if tpm_del_char_device and tpm_class_shutdown are being called from different threads? Try adding some debug output to log entry to and exit from those functions, along with tpm_chip_unregister.

Hi Phil, you can test on the custom board sent by me in May 2018, it has same TPM chip on SPI CE1.
Don't forget to use config.txt as here https://www.raspberrypi.org/forums/viewtopic.php?f=75&t=303564&p=1817702#p1839340

[pelwell]

That was several office move/reorganisations ago - I'm not finding it at the moment.

This looks like a simple shutdown problem in the upstream driver. Just add the debug I suggested.

If you want to try a quick fix, insert the following at the beginning of tpm_chip_start:

int tpm_chip_start(struct tpm_chip *chip)
{
	int ret;

	if (!chip->ops)
		return -EIO;
	tpm_clk_enable(chip);

[paulenuta]

int tpm_chip_start(struct tpm_chip *chip)

Are you refer here ?
And recompile kernel?

[pelwell]

That's the one - insert those two new lines before the call to tpm_clk_enable. It will result in a change to the tpm driver module, which will have to be installed, but our building instructions (https://www.raspberrypi.org/documentation/linux/kernel/building.md) explain how to do that.

[paulenuta]

Try adding some debug output to log entry to and exit from those functions, along with tpm_chip_unregister.

@pelwell Any quick hand (as guidance) for doing this?

[LloydS42]

Not yet. I have picked out a new sd card and will put on a fresh system image tonight. Compiling the kernel I will do on the RaspberryPi. Presumably it will run all night. First results, therefore, will not take place until tomorrow morning at the earliest.

[pelwell]

Try this:

index ddaeceb7e109..0d55235f0822 100644
--- a/drivers/char/tpm/tpm-chip.c
+++ b/drivers/char/tpm/tpm-chip.c
@@ -101,6 +101,11 @@ int tpm_chip_start(struct tpm_chip *chip)
 {
        int ret;
 
+       if (!chip->ops) {
+               pr_err("%s: Avoiding NULL ops pointer\n", __func__);
+               return -EIO;
+       }
+
        tpm_clk_enable(chip);
 
        if (chip->locality == -1) {
@@ -295,6 +300,7 @@ static int tpm_class_shutdown(struct device *dev)
 {
        struct tpm_chip *chip = container_of(dev, struct tpm_chip, dev);
 
+       pr_err("%s start\n", __func__);
        down_write(&chip->ops_sem);
        if (chip->flags & TPM_CHIP_FLAG_TPM2) {
                if (!tpm_chip_start(chip)) {
@@ -304,6 +310,7 @@ static int tpm_class_shutdown(struct device *dev)
        }
        chip->ops = NULL;
        up_write(&chip->ops_sem);
+       pr_err("%s end\n", __func__);
 
        return 0;
 }
@@ -465,6 +472,7 @@ static int tpm_add_char_device(struct tpm_chip *chip)
 
 static void tpm_del_char_device(struct tpm_chip *chip)
 {
+       pr_err("%s start\n", __func__);
        cdev_device_del(&chip->cdev, &chip->dev);
 
        /* Make the chip unavailable. */
@@ -482,6 +490,7 @@ static void tpm_del_char_device(struct tpm_chip *chip)
        }
        chip->ops = NULL;
        up_write(&chip->ops_sem);
+       pr_err("%s end\n", __func__);
 }
 
 static void tpm_del_legacy_sysfs(struct tpm_chip *chip)
@@ -636,6 +645,7 @@ EXPORT_SYMBOL_GPL(tpm_chip_register);
  */
 void tpm_chip_unregister(struct tpm_chip *chip)
 {
+       pr_err("%s start\n", __func__);
        tpm_del_legacy_sysfs(chip);
        if (IS_ENABLED(CONFIG_HW_RANDOM_TPM))
                hwrng_unregister(&chip->hwrng);
@@ -643,5 +653,6 @@ void tpm_chip_unregister(struct tpm_chip *chip)
        if (chip->flags & TPM_CHIP_FLAG_TPM2)
                cdev_device_del(&chip->cdevs, &chip->devs);
        tpm_del_char_device(chip);
+       pr_err("%s end\n", __func__);
 }
 EXPORT_SYMBOL_GPL(tpm_chip_unregister);

I've used pr_err so they stand out in the coloured dmesg output to a console, but only the first is a real error.

[pelwell]

Compiling a kernel on a Pi 4 shouldn't take more than about an hour.

[paulenuta]

if (!chip->ops)
return -EIO;

Now I'm rebuilding with this, but on 3B+. I'll add debug info and rebuild later.

[paulenuta]

That was several office move/reorganisations ago - I'm not finding it at the moment.

This looks like a simple shutdown problem in the upstream driver. Just add the debug I suggested.

If you want to try a quick fix, insert the following at the beginning of tpm_chip_start:

int tpm_chip_start(struct tpm_chip *chip)
{
	int ret;

	if (!chip->ops)
		return -EIO;
	tpm_clk_enable(chip);

Except the kernel rebuild time on 3B+, this is truly quick fix :). No more hang on reeboot/poweroff.
I've started recompile with debug activated.

[paulenuta]

Also now there is no more tpm tpm0: A TPM error (256) occurred attempting the self test on dmesg
dmesg | grep tpm [ 10.592971] tpm_tis_spi spi0.1: 2.0 TPM (device-id 0x1B, rev-id 16).

[paulenuta]

@pelwell How can we solve this upstream driver problem on 5.10.y and avoid custom kernel build?
Thanks a lot!