RPi2 kernel 4.1.13-v7+ freezes with green LED on

[McTrk]

Registering new issue as the one suggested in https://www.raspberrypi.org/forums/viewtopic.php?f=28&t=122620 by yy502 on Fri Oct 30, 2015 8:15 pm:

There was an upstream kernel bug that caused freezes in rpi-update builds since v4.1.7 but that has been fixed for a while:
#481

If you still have the issue with latest (4.1.12) kernel and want to help solve it, then please create a github issue.

I tested 4.1.12 this morning and encountered another freeze after 4 hours of idling. Guess the issue is not resolved... will raise a ticket in GitHub. Thanks for your info.

Last observed on Linux version 4.1.13-v7+ (dc4@dc4-XPS13-9333) (gcc version 4.8.3 20140303 (prerelease) (crosstool-NG linaro-1.13.1+bzr2650 - Linaro GCC 2014.03) ) #826 SMP PREEMPT Fri Nov 13 20:19:03 GMT 2015.
Updated to: c213b61355d6ebcd805157b57440b00b4e3de04d today; kernel version same as above, uptime up 2:19, 1 user, load average: 0.00, 0.04, 0.05, observing.

grep -v '^ *#' config.txt

disable_overscan=0

hdmi_force_hotplug=1
config_hdmi_boost=4
overscan_left=24
overscan_right=24
overscan_top=16
overscan_bottom=16
disable_overscan=0
dtparam=spi=off
dtparam=i2c_arm=off
gpu_mem=64

[popcornmix]

Can you confirm when this issue arose? You can revert to older firmware/kernel with
sudo rpi-update <hash>
Use the commits from here: https://github.com/Hexxeh/rpi-update/commits/master
Then end of the URL of each commit is the hash you can pass to rpi-update. Reboot after updating and try to identify the first update that causes the freeze.

[McTrk]

First observed on Nov 7 on this RPi2. Recall seeing a log of rpi-updates just today, forgetting which file.

[popcornmix]

So are you saying the preceding firmware/kernel:
Hexxeh/rpi-firmware@d0ab2f4
is completely stable?

[McTrk]

Not sure. Exact history of rip-updates would help. Do you know where that file is located on the RPi2?

[popcornmix]

I linked to the complete list of firmware updates in my first post.
Click on that link and you'll see every firmware update every released.
Click on a firmware update and you'll see a git hash at the end of the URL.
Use rpi-update to revert to an older version.
Try and find the first firmware update that is not stable.

[McTrk]

Thanks, I understand, but have limited time. What would help speed up is the local log file listing the time and hash of rbi-update invocations on my local RPi2 filesystem, which I saw an hour ago but forgot the name/location.

[popcornmix]

You can try history to see all commands you have entered recently.
But really you need to confirm if a recent specific firmware update introduced the instability.
Once we know that point, we can see exactly what changed and narrow it down.

Just pick a point when you believe it was stable (a week ago? a months ago?) and update to that point. Then just carry on with what you normally do with the pi. If you get a crash then pick an older firmware. If you believe it is stable then pick a newer one. It may take a few days to narrow down (depending on how frequent the hangs tend to be).

[McTrk]

Ok, I remember the log file. I was saving the uname printouts manually. Here it is:

Wed, 18 Nov 2015 10:11:39 -0600
Linux rpi2 4.1.12-v7+ #825 SMP PREEMPT Fri Nov 6 18:36:38 GMT 2015 armv7l GNU/Linux
Fri, 20 Nov 2015 11:12:03 -0600
Linux rpi2 4.1.13-v7+ #826 SMP PREEMPT Fri Nov 13 20:19:03 GMT 2015 armv7l GNU/Linux
Thu, 26 Nov 2015 11:09:58 -0600
Linux rpi2 4.1.13-v7+ #826 SMP PREEMPT Fri Nov 13 20:19:03 GMT 2015 armv7l GNU/Linux
Thu, 26 Nov 2015 11:10:10 -0600
Linux rpi2 4.1.13-v7+ #826 SMP PREEMPT Fri Nov 13 20:19:03 GMT 2015 armv7l GNU/Linux
Sun, 29 Nov 2015 15:29:44 -0600
Linux rpi2 4.1.13-v7+ #826 SMP PREEMPT Fri Nov 13 20:19:03 GMT 2015 armv7l GNU/Linux
Mon, 30 Nov 2015 09:23:20 -0600
Linux rpi2 4.1.13-v7+ #826 SMP PREEMPT Fri Nov 13 20:19:03 GMT 2015 armv7l GNU/Linux
Tue, 01 Dec 2015 12:16:17 -0600
Linux rpi2 4.1.13-v7+ #826 SMP PREEMPT Fri Nov 13 20:19:03 GMT 2015 armv7l GNU/Linux

Unfortunately, I started doing this only recently, and did not know the commit hash number.
It would be nice to have a /var/log/rpi-update.log to save such info, by the way.

This RPi2 was stable until Nov 7. The latest commit before Nov 7 appears to be done on Sep 4 2015, Hexxeh/rpi-update@3d0ce93 , which apparently had the fault.

Being out-of-town, I detected the freeze seven days later, have been using rpi-update more frequently since then, with results I do not quite understand. For example:

1. for a given kernel version / build number, sometimes skipped update saying it was already up-to-date, other times downloaded and updated the currently running build number.
2. /boot.bak contents are not changed even when an update occurs (still has files from April 4)
   These seem to occur more often since the crashes appeared, may be because of having to use rpi-update more often, expecting a fix.

[popcornmix]

I believe rpi-update <hash> will always update whatever the current version.
rpi-update won't update if /boot/.firmware_revision matches the latest version. You can delete /boot/.firmware_revision to force an update.

[McTrk]

How is the commit <hash> related to the kernel build number, e.g. 4.1.13-v7+ #826?
Is it possible to have several commit hashes for the same kernel build?

[popcornmix]

The commit messages will describe any kernel version bumps. Some commits will be firmware only so the kernel may not change every update.

[McTrk]

Thanks. The fault has been triggered with the commit from yesterday.
Uptime was ~24h. Cannot restart since remote.

[Phelonas]

I experienced a similar issue with Version 4.1.12 v7+ #824 when the Pi is totally idle. It froze with red and green led on and blue light of Edimax WiFi dongle constant on. But when the Pi has something to do, in my case answering on TCP Requests and /or writing every minute the status of my wifi connection to a logfile. Now i have a uptime of over 14 days. Maybe this helps a bit.

[popcornmix]

@Phelonas was the pi actually hung if interacting with it locally?
It sounds more like a wifi issue (possibly the device not waking from a low powered sleep mode).