Skip to content

RPi2 kernel 4.1.13-v7+ freezes with green LED on #510

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
McTrk opened this issue Dec 1, 2015 · 24 comments
Closed

RPi2 kernel 4.1.13-v7+ freezes with green LED on #510

McTrk opened this issue Dec 1, 2015 · 24 comments

Comments

@McTrk
Copy link

McTrk commented Dec 1, 2015

Registering new issue as the one suggested in https://www.raspberrypi.org/forums/viewtopic.php?f=28&t=122620 by yy502 on Fri Oct 30, 2015 8:15 pm:

There was an upstream kernel bug that caused freezes in rpi-update builds since v4.1.7 but that has been fixed for a while:
#481

If you still have the issue with latest (4.1.12) kernel and want to help solve it, then please create a github issue.

I tested 4.1.12 this morning and encountered another freeze after 4 hours of idling. Guess the issue is not resolved... will raise a ticket in GitHub. Thanks for your info.

Last observed on Linux version 4.1.13-v7+ (dc4@dc4-XPS13-9333) (gcc version 4.8.3 20140303 (prerelease) (crosstool-NG linaro-1.13.1+bzr2650 - Linaro GCC 2014.03) ) #826 SMP PREEMPT Fri Nov 13 20:19:03 GMT 2015.
Updated to: c213b61355d6ebcd805157b57440b00b4e3de04d today; kernel version same as above, uptime up 2:19, 1 user, load average: 0.00, 0.04, 0.05, observing.

grep -v '^ *#' config.txt

disable_overscan=0

hdmi_force_hotplug=1
config_hdmi_boost=4
overscan_left=24
overscan_right=24
overscan_top=16
overscan_bottom=16
disable_overscan=0
dtparam=spi=off
dtparam=i2c_arm=off
gpu_mem=64
@McTrk McTrk changed the title RPi 2 kernel 4.1.13.v7+ freezes with green LED on RPi2 kernel 4.1.13-v7+ freezes with green LED on Dec 1, 2015
@popcornmix
Copy link
Contributor

Can you confirm when this issue arose? You can revert to older firmware/kernel with
sudo rpi-update <hash>
Use the commits from here: https://github.com/Hexxeh/rpi-update/commits/master
Then end of the URL of each commit is the hash you can pass to rpi-update. Reboot after updating and try to identify the first update that causes the freeze.

@McTrk
Copy link
Author

McTrk commented Dec 1, 2015

First observed on Nov 7 on this RPi2. Recall seeing a log of rpi-updates just today, forgetting which file.

@popcornmix
Copy link
Contributor

So are you saying the preceding firmware/kernel:
Hexxeh/rpi-firmware@d0ab2f4
is completely stable?

@McTrk
Copy link
Author

McTrk commented Dec 1, 2015

Not sure. Exact history of rip-updates would help. Do you know where that file is located on the RPi2?

@popcornmix
Copy link
Contributor

I linked to the complete list of firmware updates in my first post.
Click on that link and you'll see every firmware update every released.
Click on a firmware update and you'll see a git hash at the end of the URL.
Use rpi-update to revert to an older version.
Try and find the first firmware update that is not stable.

@McTrk
Copy link
Author

McTrk commented Dec 1, 2015

Thanks, I understand, but have limited time. What would help speed up is the local log file listing the time and hash of rbi-update invocations on my local RPi2 filesystem, which I saw an hour ago but forgot the name/location.

@popcornmix
Copy link
Contributor

You can try history to see all commands you have entered recently.
But really you need to confirm if a recent specific firmware update introduced the instability.
Once we know that point, we can see exactly what changed and narrow it down.

Just pick a point when you believe it was stable (a week ago? a months ago?) and update to that point. Then just carry on with what you normally do with the pi. If you get a crash then pick an older firmware. If you believe it is stable then pick a newer one. It may take a few days to narrow down (depending on how frequent the hangs tend to be).

@McTrk
Copy link
Author

McTrk commented Dec 1, 2015

Ok, I remember the log file. I was saving the uname printouts manually. Here it is:

Wed, 18 Nov 2015 10:11:39 -0600
Linux rpi2 4.1.12-v7+ #825 SMP PREEMPT Fri Nov 6 18:36:38 GMT 2015 armv7l GNU/Linux
Fri, 20 Nov 2015 11:12:03 -0600
Linux rpi2 4.1.13-v7+ #826 SMP PREEMPT Fri Nov 13 20:19:03 GMT 2015 armv7l GNU/Linux
Thu, 26 Nov 2015 11:09:58 -0600
Linux rpi2 4.1.13-v7+ #826 SMP PREEMPT Fri Nov 13 20:19:03 GMT 2015 armv7l GNU/Linux
Thu, 26 Nov 2015 11:10:10 -0600
Linux rpi2 4.1.13-v7+ #826 SMP PREEMPT Fri Nov 13 20:19:03 GMT 2015 armv7l GNU/Linux
Sun, 29 Nov 2015 15:29:44 -0600
Linux rpi2 4.1.13-v7+ #826 SMP PREEMPT Fri Nov 13 20:19:03 GMT 2015 armv7l GNU/Linux
Mon, 30 Nov 2015 09:23:20 -0600
Linux rpi2 4.1.13-v7+ #826 SMP PREEMPT Fri Nov 13 20:19:03 GMT 2015 armv7l GNU/Linux
Tue, 01 Dec 2015 12:16:17 -0600
Linux rpi2 4.1.13-v7+ #826 SMP PREEMPT Fri Nov 13 20:19:03 GMT 2015 armv7l GNU/Linux

Unfortunately, I started doing this only recently, and did not know the commit hash number.
It would be nice to have a /var/log/rpi-update.log to save such info, by the way.

This RPi2 was stable until Nov 7. The latest commit before Nov 7 appears to be done on Sep 4 2015, Hexxeh/rpi-update@3d0ce93 , which apparently had the fault.

Being out-of-town, I detected the freeze seven days later, have been using rpi-update more frequently since then, with results I do not quite understand. For example:

  1. for a given kernel version / build number, sometimes skipped update saying it was already up-to-date, other times downloaded and updated the currently running build number.
  2. /boot.bak contents are not changed even when an update occurs (still has files from April 4)
    These seem to occur more often since the crashes appeared, may be because of having to use rpi-update more often, expecting a fix.

@popcornmix
Copy link
Contributor

I believe rpi-update <hash> will always update whatever the current version.
rpi-update won't update if /boot/.firmware_revision matches the latest version. You can delete /boot/.firmware_revision to force an update.

@McTrk
Copy link
Author

McTrk commented Dec 2, 2015

How is the commit <hash> related to the kernel build number, e.g. 4.1.13-v7+ #826?
Is it possible to have several commit hashes for the same kernel build?

@popcornmix
Copy link
Contributor

The commit messages will describe any kernel version bumps. Some commits will be firmware only so the kernel may not change every update.

@McTrk
Copy link
Author

McTrk commented Dec 2, 2015

Thanks. The fault has been triggered with the commit from yesterday.
Uptime was ~24h. Cannot restart since remote.

@Phelonas
Copy link

Phelonas commented Dec 3, 2015

I experienced a similar issue with Version 4.1.12 v7+ #824 when the Pi is totally idle. It froze with red and green led on and blue light of Edimax WiFi dongle constant on. But when the Pi has something to do, in my case answering on TCP Requests and /or writing every minute the status of my wifi connection to a logfile. Now i have a uptime of over 14 days. Maybe this helps a bit.

@popcornmix
Copy link
Contributor

@Phelonas was the pi actually hung if interacting with it locally?
It sounds more like a wifi issue (possibly the device not waking from a low powered sleep mode).

@Phelonas
Copy link

Phelonas commented Dec 3, 2015

@popcornmix i went the extra mile to be sure the dongle wont enter sleep mode. This included turning power management off in /etc/network/interfaces , writing into the drivers (8192cu.conf file) to not go into sleep more /powermgnt off and pinging the router every 10 seconds (and try a reconnect as long as theres no connection made)

@McTrk
Copy link
Author

McTrk commented Dec 3, 2015

In my rpi2 the freeze happens with eth0 only and no other peripheral connected. Installed watchdog package today, hoping it triggers a restart when freeze occurs, thus restoring remote access.

@McTrk
Copy link
Author

McTrk commented Dec 3, 2015

Suggesting patch for rpi-update, not tested, addressing the following:

  1. Introduce logging of firmware revision update in /var/log/rpi-update
  2. Factor out recurring "$(cat ${FW_REVFILE})"
*** /usr/bin/rpi-update~    2015-12-03 14:58:02.783631930 -0600
--- /usr/bin/rpi-update 2015-12-03 14:58:02.843631796 -0600
***************
*** 40,45 ****
--- 40,46 ----
  FW_MODPATH="${ROOT_PATH}/lib/modules"
  FW_REV=${1:-""}
  FW_REVFILE="${FW_PATH}/.firmware_revision"
+ LOGFILE="/var/log/rpi-update";
  [ "${RPI_UPDATE_UNSUPPORTED}" -ne 0 ] && echo -e "You appear to be trying to update firmware on an incompatible distribution. To force update, run the following:\nsudo -E RPI_UPDATE_UNSUPPORTED=0 rpi-update" && exit 1

  function update_self() {
***************
*** 189,196 ****
--- 190,199 ----
        echo " *** Running ldconfig"
        ldconfig -r "${ROOT_PATH}"
    fi
+   local logmsg="$(date -R) : revision change from { $(ls --full-time ${FW_REVFILE}) : { $(cat ${FW_REVFILE}) } } to ${FW_REV} }";
    echo " *** Storing current firmware revision"
    echo "${FW_REV}" > "${FW_REVFILE}"
+   echo "$logmsg" | tee -a "$LOGFILE"; 
  }

  function do_backup {
***************
*** 352,364 ****
    fi
    do_backup
  else
!   if [[ $(cat "${FW_REVFILE}") == "${FW_REV}" ]]; then
        echo " *** Your firmware is already up to date"
        exit 0
    fi
    if [[ ${JUST_CHECK} -ne 0 ]]; then
        echo " *** Firmware update required. New commits available:"
!       DIFF_API=${REPO_URI/github.com/api.github.com\/repos}/compare/$(cat "${FW_REVFILE}")...${BRANCH}
        SEPARATOR="======================================================"
        curl -Ls ${DIFF_API} | awk -v SEPARATOR="${SEPARATOR}" -F\" ' { if ($2 == "commits") {commits=1} if (commits && $2 == "message") {print SEPARATOR "\nCommit: " $4} }' | sed 's/\\n\\n/\nCommit:\ /g'
        exit 2
--- 355,368 ----
    fi
    do_backup
  else
!   FW_OLDREV=$(cat "${FW_REVFILE}");
!   if [[ "${FW_OLDREV}" == "${FW_REV}" ]]; then
        echo " *** Your firmware is already up to date"
        exit 0
    fi
    if [[ ${JUST_CHECK} -ne 0 ]]; then
        echo " *** Firmware update required. New commits available:"
!       DIFF_API=${REPO_URI/github.com/api.github.com\/repos}/compare/"${FW_OLDREV}...${BRANCH}"
        SEPARATOR="======================================================"
        curl -Ls ${DIFF_API} | awk -v SEPARATOR="${SEPARATOR}" -F\" ' { if ($2 == "commits") {commits=1} if (commits && $2 == "message") {print SEPARATOR "\nCommit: " $4} }' | sed 's/\\n\\n/\nCommit:\ /g'
        exit 2

Will let know how log looks when tested.

@McTrk
Copy link
Author

McTrk commented Dec 3, 2015

Watchdog config file needed minor correction: a closing single quote was missing.
Cannot log in to SF although have valid account, web page seems looping back to log in prompt. Including patch here in case it helps recovery/testing:

*** /lib/systemd/system/watchdog.service    2015/12/03 21:20:22 1.1
--- /lib/systemd/system/watchdog.service    2015/12/03 21:20:32
***************
*** 7,13 ****
  [Service]
  Type=forking
  EnvironmentFile=/etc/default/watchdog
! ExecStartPre=/bin/sh -c '[ -z "${watchdog_module}" ] || [ "${watchdog_module}" = "none" ] || /sbin/modprobe $watchdog_module
  ExecStart=/bin/sh -c '[ $run_watchdog != 1 ] || exec /usr/sbin/watchdog $watchdog_options'
  ExecStopPost=/bin/sh -c '[ $run_wd_keepalive != 1 ] || false'

--- 7,13 ----
  [Service]
  Type=forking
  EnvironmentFile=/etc/default/watchdog
! ExecStartPre=/bin/sh -c '[ -z "${watchdog_module}" ] || [ "${watchdog_module}" = "none" ] || /sbin/modprobe $watchdog_module'
  ExecStart=/bin/sh -c '[ $run_watchdog != 1 ] || exec /usr/sbin/watchdog $watchdog_options'
  ExecStopPost=/bin/sh -c '[ $run_wd_keepalive != 1 ] || false'

@popcornmix
Copy link
Contributor

I've just done sudo apt-get install watchdog on Ubuntu 15.10. I get the same ExecStartPre line with the missing quote. Looks like it's been reported upstream: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=798294 so hopefully the fix will trickle through eventually.

@McTrk
Copy link
Author

McTrk commented Dec 13, 2015

Have not observed the freeze for a while, with watchdog running. Running the modified rpi-update locally to log firmware revision when update occurs. Log looks a bit ugly, need to change ls option to show mod time only.

rpi2 ~ $ date -R; uptime; cat /var/log/rpi-update 
Sun, 13 Dec 2015 13:06:56 -0600
 13:06:56 up 4 days,  4:57,  0 users,  load average: 0.03, 0.07, 0.06
Wed, 09 Dec 2015 07:35:52 -0600 : revision change from { -rwxr-xr-x 1 root root 41 2015-12-01 11:11:28.000000000 -0600 /boot/.firmware_revision : { c213b61355d6ebcd805157b57440b00b4e3de04d } } to 9b6f9a13a7eebcdf0e03cb8a5e14ce4f2e890a16 }

@McTrk McTrk closed this as completed Dec 13, 2015
@McTrk McTrk reopened this Dec 13, 2015
@McTrk
Copy link
Author

McTrk commented Dec 13, 2015

Need to run a while without watchdog module inserted in the kernel to reverify. This is to make sure insmod'ing a kernel module is not masking the issue.

@Ruffio
Copy link

Ruffio commented Jun 29, 2016

@McTrk How did your test go? Can this issue be closed?

@McTrk
Copy link
Author

McTrk commented Jun 30, 2016

The issue was not triggered until the next firmware upgrade, which was through aptitude to the next raspbian firmware, "testing" branch, if I recall correctly.
I stayed with the distributed kernel & firmware and did not get a chance to use rpi-update since then.
Other lockups occurred, which I suspect were due to storage shortage, recovered by rebooting and have not had a chance to troubleshoot.

@McTrk
Copy link
Author

McTrk commented Jun 30, 2016

So far issue seems resolved when using distributed firmware in raspbian. Closing.

@McTrk McTrk closed this as completed Jun 30, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants