pcie stalled using rtl8192ce (default wifi card)

[waby38b]

pcie Wifi card included in 4kopen box is a rtl8192ce

Seem, that our DWC pcie driver generate randomly some HANG
Here some logs related to this HANG (stacktrace)
=> seem related to interrupt

[  184.243426] rcu: INFO: rcu_sched self-detected stall on CPU
[  184.249008] rcu: 	0-....: (8405 ticks this GP) idle=df74/1/0x40000002 softirq=3607/3607 fqs=2972
[  184.257790] 	(t=8407 jiffies g=6613 q=56899 ncpus=4)
[  184.257803] CPU: 0 PID: 1059 Comm: ifconfig Not tainted 6.2.7 #3
[  184.257813] Hardware name: STi SoC with Flattened Device Tree
[  184.257818] PC is at pci_read32_sync+0x10/0x14 [rtl_pci]
[  184.257860] LR is at rtl92ce_enable_interrupt+0x4c/0x8c [rtl8192ce]
[  184.257921] pc : [<bf46d03c>]    lr : [<bf478e44>]    psr: 20060013
[  184.257928] sp : f1375ce0  ip : bf451b34  fp : c14f3c00
[  184.257933] r10: c2c4b780  r9 : c28f4000  r8 : c15b18bc
[  184.257938] r7 : 00000000  r6 : c15bd880  r5 : c15bc880  r4 : c15b1880
[  184.257944] r3 : f09e8000  r2 : 00000730  r1 : f09e8120  r0 : 00001b5f
[  184.257950] Flags: nzCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment none
[  184.257959] Control: 10c5387d  Table: 86c0804a  DAC: 00000051
[  184.257970]  pci_read32_sync [rtl_pci] from rtl92ce_enable_interrupt+0x4c/0x8c [rtl8192ce]
[  184.258018]  rtl92ce_enable_interrupt [rtl8192ce] from rtl_pci_start+0xbc/0x18c [rtl_pci]
[  184.258059]  rtl_pci_start [rtl_pci] from rtl_op_start+0x58/0x7c [rtlwifi]
[  184.258269]  rtl_op_start [rtlwifi] from drv_start+0x4c/0x10c [mac80211]
[  184.259512]  drv_start [mac80211] from ieee80211_do_open+0x370/0x6d8 [mac80211]
[  184.260444]  ieee80211_do_open [mac80211] from ieee80211_open+0x60/0x84 [mac80211]
[  184.261374]  ieee80211_open [mac80211] from __dev_open+0xfc/0x1d0
[  184.261860]  __dev_open from __dev_change_flags+0x190/0x214
[  184.261878]  __dev_change_flags from dev_change_flags+0x20/0x5c
[  184.261894]  dev_change_flags from devinet_ioctl+0x7a8/0x7dc
[  184.261918]  devinet_ioctl from inet_ioctl+0x1a4/0x244
[  184.261937]  inet_ioctl from sock_ioctl+0x4dc/0x5b8
[  184.261962]  sock_ioctl from sys_ioctl+0x510/0xbd8
[  184.261985]  sys_ioctl from ret_fast_syscall+0x0/0x1c
[  184.262000] Exception stack(0xf1375fa8 to 0xf1375ff0)
[  184.262011] 5fa0:                   005a8bcd beedbb10 00000003 00008914 beedbb10 beedbaa8
[  184.262020] 5fc0: 005a8bcd beedbb10 beedbe18 00000036 beedbcf8 00000003 b6f34d78 b6f35048
[  184.262027] 5fe0: 005c81f8 beedba90 004fe2d8 b6e4c758

HowTo reproduce

# Wake up interface
ifconfig wlan0 up
# (optional) can check if wifi is working for you (=> show AP nearby)
iw dev wlan0 scan 

# loop up/down wifi interface (...wait a bit...)
while : ; do ifconfig wlan0 up; ifconfig wlan0 down; echo -n "."; done

[waby38b]

So, I have a first ~working solution, check this branch

drivers/net/wireless/realtek/rtlwifi/rtl8192ce/hw.c |    7 +++++++
1 file changed, 7 insertions(+)

Seem that stacktrace show an interrupt issue.... So, as a first try, I just make protection on enable/disable interrupt
Now, we can use NetworkManager ~without any issue, and no more hard crash!
Others rtlwifi drivers should also be affected.... (I don't have HW to check...)

Anyway, even with this patch, we may (low rate) got some not expected msg from kernel
when switching from AP discover and AP usage. This is exactly what NetworkManager do in background
So, this patch need to be tuned

    [ 1011.101387] ------------[ cut here ]------------
    [ 1011.101399] WARNING: CPU: 0 PID: 528 at net/mac80211/rx.c:5146 ieee80211_rx_list+0x6bc/0xd38 [mac80211]
    [ 1011.102736] CPU: 0 PID: 528 Comm: apps.plugin Not tainted 6.2.9 #17
    [ 1011.102751] Hardware name: STi SoC with Flattened Device Tree
    [ 1011.102763]  unwind_backtrace from show_stack+0x18/0x1c
    [ 1011.102794]  show_stack from dump_stack_lvl+0x58/0x70
    [ 1011.102823]  dump_stack_lvl from __warn+0x84/0x12c
    [ 1011.102857]  __warn from warn_slowpath_fmt+0xb4/0x18c
    [ 1011.102886]  warn_slowpath_fmt from ieee80211_rx_list+0x6bc/0xd38 [mac80211]
    [ 1011.103835]  ieee80211_rx_list [mac80211] from ieee80211_rx_napi+0x38/0xb0 [mac80211]
    [ 1011.105676]  ieee80211_rx_napi [mac80211] from ieee80211_tasklet_handler+0xd8/0xdc [mac80211]
    [ 1011.107055]  ieee80211_tasklet_handler [mac80211] from tasklet_action_common.constprop.0+0xf8/0x128
    [ 1011.107875]  tasklet_action_common.constprop.0 from __do_softirq+0x168/0x400
    [ 1011.107912]  __do_softirq from __irq_exit_rcu+0xa4/0xc8
    [ 1011.107939]  __irq_exit_rcu from irq_exit+0x10/0x30
    [ 1011.107969]  irq_exit from call_with_stack+0x18/0x20
    [ 1011.108004]  call_with_stack from __irq_svc+0x9c/0xb8
    [ 1011.108027] Exception stack(0xf0c29ed0 to 0xf0c29f18)
    [ 1011.108045] 9ec0:                                     c01a4788 00000000 2c629000 60000093
    [ 1011.108054] 9ee0: 00000000 c4d9d778 c4d9d100 c2adbd08 0000069c fffffe30 5ac3c35a c2a04d8c
    [ 1011.108062] 9f00: 00000000 f0c29f20 c014b488 c071105c 60000013 ffffffff
    [ 1011.108068]  __irq_svc from __cond_resched+0x0/0x54
    [ 1011.108082]  __cond_resched from task_work_run+0xa4/0xcc
    [ 1011.108100]  task_work_run from do_work_pending+0x448/0x51c
    [ 1011.108116]  do_work_pending from slow_work_pending+0xc/0x20
    [ 1011.108126] Exception stack(0xf0c29fb0 to 0xf0c29ff8)
    [ 1011.108134] 9fa0:                                     fffffffe 005467f0 00028000 00000000
    [ 1011.108142] 9fc0: 004a99e8 00000001 004a1400 00000142 005467f0 00000000 0049ec48 00000000
    [ 1011.108150] 9fe0: 00000001 be9650b8 00000000 b6b02660 80000010 ffffff9c
    [ 1011.108210] ---[ end trace 0000000000000000 ]---
    [ 1011.108229] ------------[ cut here ]------------

[waby38b]

Here second version of my patch.

 drivers/net/wireless/realtek/rtlwifi/core.c |    4 +++-
 drivers/net/wireless/realtek/rtlwifi/pci.c  |   12 ++++++++++--
 drivers/net/wireless/realtek/rtlwifi/ps.c   |    6 ++++++
 3 files changed, 19 insertions(+), 3 deletions(-)

No more localized in rtl92ce_{enable|disable}_interrupt() but in rtlwifi framework.
All realteck drivers follow sames rules, so every RTLxxx card should raise same locking issue

I do not (yet!) see any drawback like with patch n°1, so seem safe ! (cross finger)

We may reduced later this patch, but let's take some time to test !

[waby38b]

Here third version of the rtlwifi patch (hope that it's the last one!)

 drivers/net/wireless/realtek/rtlwifi/pci.c |    5 +++++
 drivers/net/wireless/realtek/rtlwifi/ps.c  |    3 +++
 2 files changed, 8 insertions(+)

Patch v2 still generate some WARN (but no error or hang)

[297627.187798] ------------[ cut here ]------------
[297627.187831] WARNING: CPU: 3 PID: 0 at net/mac80211/rx.c:5146 ieee80211_rx_list+0xb0/0xae0 [mac80211]
[297627.190485] Modules linked in: nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip_set nf_tables nfnetlink qrtr sunrpc vfat fat rtl8192ce rtl_pci rtl8192c_common rtlwifi mac80211 libarc4 cfg80211 snd_soc_sti_sas snd_soc_simple_card ni_usb_gpib(OE) snd_soc_sti snd_soc_simple_card_utils gpib_common(OE) snd_soc_core rfkill ssd130x_i2c ac97_bus ssd130x snd_seq leds_gpio snd_pcm_dmaengine snd_seq_device ssd1307fb st_remoteproc snd_pcm st_delta videobuf2_dma_contig ofpart videobuf2_memops v4l2_mem2mem videobuf2_v4l2 st_spi_fsm videobuf2_common mtd snd_timer st_thermal_memmap videodev st_thermal snd soundcore st_rng mc rpmsg_core cpufreq_dt fuse zram dwmac_generic xhci_plat_hcd dwc3 dwmac_sti stmmac_platform ulpi mmc_block udc_core stmmac pcs_xpcs ehci_st pcie_sti ohci_st ahci_st ohci_hcd phylink libahci_platform sdhci_st i2c_st pwm_sti sdhci_pltfm sdhci sti_drm st_fdma
[297627.191474]  st_slim_rproc st_lpc_wdt drm_dma_helper rtc_st_lpc dwc3_st cec mailbox_sti phy_miphy28lp phy_stih407_usb sti_cpufreq
[297627.191591] CPU: 3 PID: 0 Comm: swapper/3 Tainted: G        W  OE      6.2.10-100.4kopen.fc36.armv7hl #1
[297627.191613] Hardware name: STi SoC with Flattened Device Tree
[297627.191633]  unwind_backtrace from show_stack+0x18/0x1c
[297627.191675]  show_stack from dump_stack_lvl+0x40/0x4c
[297627.191716]  dump_stack_lvl from __warn+0x78/0x114
[297627.191748]  __warn from warn_slowpath_fmt+0xa8/0x12c
[297627.191773]  warn_slowpath_fmt from ieee80211_rx_list+0xb0/0xae0 [mac80211]
[297627.193854]  ieee80211_rx_list [mac80211] from ieee80211_rx_napi+0x34/0xa4 [mac80211]
[297627.196705]  ieee80211_rx_napi [mac80211] from ieee80211_tasklet_handler+0x9c/0xc0 [mac80211]
[297627.199489]  ieee80211_tasklet_handler [mac80211] from tasklet_action_common.constprop.0+0x80/0xb8
[297627.200810]  tasklet_action_common.constprop.0 from __do_softirq+0x1e0/0x2a4
[297627.200850]  __do_softirq from __irq_exit_rcu+0x84/0xfc
[297627.200880]  __irq_exit_rcu from irq_exit+0x10/0x18
[297627.200915]  irq_exit from call_with_stack+0x18/0x20
[297627.200963]  call_with_stack from __irq_svc+0x98/0xb0
[297627.200992] Exception stack(0xf08b1f50 to 0xf08b1f98)
[297627.201009] 1f40:                                     eede997c 2d593000 00000003 c031b700
[297627.201026] 1f60: c2164800 00000000 c2164800 c1857a70 00000003 c190a0f8 00000000 00000000
[297627.201041] 1f80: f0853200 f08b1fa0 c0307e2c c0307e10 60000013 ffffffff
[297627.201052]  __irq_svc from arch_cpu_idle+0x24/0x4c
[297627.201088]  arch_cpu_idle from default_idle_call+0x38/0x54
[297627.201122]  default_idle_call from do_idle+0x10c/0x284
[297627.201159]  do_idle from cpu_startup_entry+0x20/0x24
[297627.201191]  cpu_startup_entry from secondary_start_kernel+0x1c0/0x1e8
[297627.201230]  secondary_start_kernel from 0x80301640
[297627.201323] ---[ end trace 0000000000000000 ]---

So, tweak again and reduce as low as possible locking and remove unneeded changes (mutex)

Let's again wait for a final conclusion

Here my updated script for stress testing ( waby38 is my Wifi AP)
while : ; do /etc/init.d/S45network-manager stop; for ((i=1;i<=1000;i++)); do ifconfig wlp1s0 up; ifconfig wlp1s0 down; echo -n "."; done; /etc/init.d/S45network-manager start; for ((i=1;i<=20;i++)); do sleep 5; nmcli connection up way38; echo -n "u"; nmcli device wifi rescan && nmcli device wifi list; sleep 5; nmcli connection down waby38; echo -n "d"; nmcli device wifi rescan && nmcli device wifi list; done; done

And ply provide some useful stats
ply 'kprobe:rtl92ce_*_interrupt { @[comm, uid, pid, kpid, caller] = count(); if (pid == 0) print(stack); } '

[waby38b]

Patchv4 (CodeName: I'm back!)

 drivers/net/wireless/realtek/rtlwifi/core.c | 18 +++++++++++++++++-
 drivers/net/wireless/realtek/rtlwifi/pci.c  |  5 +++++
 drivers/net/wireless/realtek/rtlwifi/ps.c   |  3 +++
 3 files changed, 25 insertions(+), 1 deletion(-)

Tested a new path in ieee80211 stack: Access Point!
And this raise (again, low fhz) new locking issue...

[  126.781074]  pci_read32_sync [rtl_pci] from rtl92ce_enable_interrupt+0x24/0x44 [rtl8192ce]
[  126.781126]  rtl92ce_enable_interrupt [rtl8192ce] from rtl_op_bss_info_changed+0x1a0/0x6f0 [rtlwifi]
[  126.781321]  rtl_op_bss_info_changed [rtlwifi] from drv_link_info_changed+0x1cc/0x1e4 [mac80211]
[  126.783142]  drv_link_info_changed [mac80211] from ieee80211_link_info_change_notify+0xd8/0xec [mac80211]
[  126.784637]  ieee80211_link_info_change_notify [mac80211] from ieee80211_start_ap+0x60c/0x68c [mac80211]
[  126.786143]  ieee80211_start_ap [mac80211] from nl80211_start_ap+0xa2c/0xb24 [cfg80211]
[  126.788642]  nl80211_start_ap [cfg80211] from genl_rcv_msg+0x220/0x368
[  126.789308]  genl_rcv_msg from netlink_rcv_skb+0xac/0x114
[  126.789327]  netlink_rcv_skb from genl_rcv+0x28/0x3c
[  126.789342]  genl_rcv from netlink_unicast+0x12c/0x1d4
[  126.789357]  netlink_unicast from netlink_sendmsg+0x374/0x3a4
[  126.789372]  netlink_sendmsg from sock_sendmsg_nosec+0x14/0x24
[  126.789398]  sock_sendmsg_nosec from ____sys_sendmsg+0x174/0x21c
[  126.789417]  ____sys_sendmsg from ___sys_sendmsg+0x9c/0xd4
[  126.789434]  ___sys_sendmsg from __sys_sendmsg+0x74/0xb0
[  126.789445]  __sys_sendmsg from ret_fast_syscall+0x0/0x1c

Now, issue is detected in rtlwifi/core.c (rtl_op_bss_info_changed) when we enable AP (nl80211_start_ap)

After analysis, I confirm that 3 new entries may be call interrupt_{enable|disable} without any protection

  .update_interrupt_mask = rtl92ce_update_interrupt_mask,
  .set_bcn_intv = rtl92ce_set_beacon_interval,
  .set_bcn_reg = rtl92ce_set_beacon_related_registers,

rtl_op_add_interface()
  1 x update_interrupt_mask()
  2 x set_bcn_reg()

rtl_op_remove_interface()
  1 x update_interrupt_mask()

rtl_op_bss_info_changed()
  2 x update_interrupt_mask()
  1 x set_bcn_intv()

HowToCheck:
while : ; do nmcli c u "Wi-Fi AP"; echo -n "."; nmcli c d "Wi-Fi AP"; done