(pcf8523) hwclock: ioctl(RTC_RD_TIME) to /dev/rtc to read the time failed: Invalid argument

[brubbel]

TL;DR: The Adafruit PCF8523 Real Time Clock for Raspberry Pi loses time due to a combination of issues in (1) the pcf8523 chip, (2) the pcb design and the (3) rtc-pcf8523 kernel module. Here is why:

(Note: this is a technical explanation, so that anyone with similar experiences won't have to reinvent the wheel.)

Hardware (multiple devices were tested)

1. PCF8523 RTC plugin module
2. RPI3B+ (rev. a020d3), new raspbian (kernel 4.14.71-v7+).
3. Raspberry pi official power supply (5.1V, 2.5A)
4. No other peripherals, apart from ethernet cable.

Setup
The module is loaded either with dtoverlay i2c-rtc pcf8523=true or at boot time using dtoverlay=i2c-rtc,pcf8523 in /boot/config.txt, as explained here.

Result
Then, the RTC is initialized using hwclock --systohc --utc or hwclock -w.
At this point the RTC works and can be checked with timedatectl or hwclock -r. So far so good.

Problem
The pcf8523 clock module looses time every once in a while (read time failed). On a reboot, on unplugging the power or even just by waiting long enough (10 minutes, sometimes days).
The purpose of a battery-backupped hw clock however is to keep time, especially on power loss.

Diagnosis
Origin of the ioctl(RTC_RD_TIME) to /dev/rtc to read the time failed: Invalid argument error.
The origin can be tracked down to this commit and these 2 lines of code:
https://github.com/raspberrypi/linux/blob/961aa2356a25ad04528e91761e7081a25dc983dc/drivers/rtc/rtc-pcf8523.c#L181-L182

In short: the rtc-pcf8523 driver checks if the REG_SECONDS_OS flag is set. If it is set, the Invalid argument error is produced.
According to the datasheet, the REG_SECONDS_OS flag indicates that

The OS flag is set whenever the oscillator is stopped.

So, now the question is why it stops on multiple devices and intermittent times. According to the datasheet:

... that VDD is monitored every 1 ms, the battery switch-over works properly in all cases where VDD falls with a rate lower than 0.7 V/ms.

So, glitches on the power supply rails are not warmly welcomed by this IC. For this, the datasheet also has a solution:

A series resistor of 1k and a capacitor of 3.3 uF assure the proper functionality of the battery switch-over even with very fast VDD slope

Back to the Adafruit module:

According to the schematic there should be a 10uF capacitor on this board, however in reality it looks rather 100nF, i.e. not enough to guarantee the 0.7 V/ms and thus causing a VDD-to-battery failover failure.

Workaround

1. Recompiling the kernel module to remove the REG_SECONDS_OS is one possible workaround.
2. Soldering an extra decoupling capacitor is another workaround. However, the latter one is still disaster waiting to happen: even the shortest glitch in several months (e.g. EMI) causes a lockup of the driver, while the internal time may only be off a few milliseconds.

Further info
Detect the pcf_8523 module: i2cdetect -y 1
Read the raw register contents: i2cdump -y -f -r 0-15 1 0x68
Example output:

     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f    0123456789abcdef
00: 80 00 08 83 38 16 26 05 10 18 80 80 80 80 00 00    ?.??8?&???????..

Register 0x03 contains the REG_SECONDS_OS flag at bit 7. When it is set the Invalid argument error occurs, hwclock -r stops working while the timestamp provided by the IC is still ok.

[pelwell]

That's a comprehensive explanation of the problem. My summary would be:

Some PCF8523 RTC modules lose time, probably due to electrical issues when switching between external power and backup battery. When this happens, the read function returns an error rather than return an incorrect value, even though the error may be small.

The device obviously has no way of knowing how much time elapses during the down time, hence the boolean right/wrong status. I have some sympathy for the author of the patch that introduced this behaviour, but if the oscillator stopping briefly is a regular occurrence on some boards then perhaps the error should be made optional.

I'm prepared to add a Device Tree property to revert to the old behaviour, with an overlay parameter to enable it, if that is acceptable.

[brubbel]

I've been thinking some more about this, checking how other RTC kernel drivers implement the 'clock integrity' thing. Most of them return an error when the integrity cannot be assured (or the battery is low), as it is currently implemented in the rtc-pfc8523 driver.

Theory
There should be decent decoupling, shielding and no EMI. The clock LV/OS-flag would be set at low battery.

Reality
RTC plugin modules for RPI are not shielded and will most likely suffer from a power glitch one point in time. For the end user no clock (or fake-hwclock) is worse than a clock with offset.

Removing the OS check fixed my problem in short term, so I'm open for any solution.

Edit: if the rtc-pfc8523 driver is loaded, the fake-hwclock is ignored it seems, so the issue is worse than a fake-hwclock-only situation.

[pelwell]

For the end user no clock (or fake-hwclock) is worse than a clock with offset.

I'm inclined to agree. There are at least two other RTC drivers that just log a warning under similar conditions, but there are others that return -EINVAL. As you are currently the only person to have reported this problem I'll leave the current (error) behaviour as the default.

[brubbel]

My understanding (and observation) is that the RTC module is more sensitive at the high-Z pins of the oscillator than to corruption of internal registers. So any disturbance may skip or add a few cycles in the oscillator, but register contents are not affected by the 0.7 V/ms requirement.
Hence: ignoring the OS-flag while battery is ok has more advantages than the current situation.

Below I'll add the instructions to recompile the module so anyone with similar findings may check the workaround.

[brubbel]

Details for recompiling the rtc-pfc8523 module without OS (oscillator stop) check.

1. Install kernel sources as described here:

rpi-source --skip-gcc

2. edit /root/linux/drivers/rtc/rtc-pcf8523.c, comment out the following lines around L181:

        //if (regs[0] & REG_SECONDS_OS)
        //      return -EINVAL;

3. Compile:

cd /root/linux/
make prepare
make M=scripts
make -j4 M=drivers/rtc

4. Temp link to test driver:

cd /lib/modules/`uname -r`/kernel/drivers/rtc/
mv rtc-pcf8523.ko rtc-pcf8523.ko.orig
ln -s /root/linux/drivers/rtc/rtc-pcf8523.ko

5. Reload module (dtoverlay i2c-rtc pcf8523=true must be loaded):

rmmod rtc_pcf8523; modprobe rtc_pcf8523

[pelwell]

After oscillating a few times between DT property and module parameter I settled for reverting the original commit.

[brubbel]

After oscillating a few times

That is due to negative feedback, or a second pole below the unity gain freq. ;-)

[popcornmix]

Latest rpi-update kernel contains this revert.