Skip to content

SSH over USB stops working after Pi reboot #4890

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
alevchuk opened this issue Feb 14, 2022 · 20 comments
Open

SSH over USB stops working after Pi reboot #4890

alevchuk opened this issue Feb 14, 2022 · 20 comments

Comments

@alevchuk
Copy link

alevchuk commented Feb 14, 2022

Describe the bug

I followed the guide to enable SSH over USB: https://artivis.github.io/post/2020/pi-zero/
SSHing into the Pi worked but after rebooting the Pi from the host I cannot SSH into the Pi.

The USB Ethernet connection IPv4 is stuck on "Connecting...".

Looks like the problem is that the Pi can no longer Arp the hardware address of the host.

Steps to reproduce the behaviour

  1. Install fresh OS on the Pi and a laptop/desktop.
  2. Follow SSH over UBS instructions https://artivis.github.io/post/2020/pi-zero/
  3. Run sudo reboot on the Pi
  4. Try SSH'ing again

On Host Computer:

  • Ubuntu 20.04.3 LTS, fresh install
  • Ubuntu host has the USB Ethernet connection IPv4 set to "Shared to other computers"

On the Pi:

  • Rasbian 11 (bullseye), fresh install
  • /boot/config.txt has dtoverlay=dwc2
  • /boot/cmdline.txt ends with rootwait modules-load=dwc2,g_ether g_ether.host_addr=82:54:68:48:e4:87
  • In /etc/dhcpcd.conf put

interface usb0
static ip_address=10.42.0.42
static routers=10.42.0.1

Device (s)

Raspberry Pi Zero

System

pi@raspberrypi:~ $ cat /etc/rpi-issue
Raspberry Pi reference 2022-01-28
Generated using pi-gen, https://github.com/RPi-Distro/pi-gen, f01430c9d8f67a4b9719cc00e74a2079d3834d5d, stage2

pi@raspberrypi:~ $ vcgencmd version
Jan 20 2022 13:58:22 
Copyright (c) 2012 Broadcom
version bd88f66f8952d34e4e0613a85c7a6d3da49e13e2 (clean) (release) (start)

pi@raspberrypi:~ $ uname -a
Linux raspberrypi 5.10.92+ #1514 Mon Jan 17 17:35:21 GMT 2022 armv6l GNU/Linux

Logs

On the Pi (when things work):

dmesg  | grep dwc2
[    0.000000] Kernel command line: coherent_pool=1M 8250.nr_uarts=0 snd_bcm2835.enable_compat_alsa=0 snd_bcm2835.enable_hdmi=1 video=Composite-1:720x480@60i smsc95xx.macaddr=B8:27:EB:5A:0E:9B vc_mem.mem_base=0x1ec00000 vc_mem.mem_size=0x20000000  dwc_otg.lpm_enable=0 dwc_otg.speed=1 console=ttyS0,115200 console=tty1 root=PARTUUID=80be026c-02 rootfstype=ext4 fsck.repair=yes rootwait modules-load=dwc2,g_ether
[    2.989856]     modules-load=dwc2,g_ether
[    9.655965] dwc2 20980000.usb: supply vusb_d not found, using dummy regulator
[    9.660432] dwc2 20980000.usb: supply vusb_a not found, using dummy regulator
[    9.932844] dwc2 20980000.usb: EPs: 8, dedicated fifos, 4080 entries in SPRAM
[    9.953381] dwc2 20980000.usb: DWC OTG Controller
[    9.957344] dwc2 20980000.usb: new USB bus registered, assigned bus number 1
[    9.961262] dwc2 20980000.usb: irq 33, io mem 0x20980000
[   10.033313] usb usb1: Manufacturer: Linux 5.10.92+ dwc2_hsotg
[   10.446481] dwc2 20980000.usb: bound driver g_ether
[   10.574255] dwc2 20980000.usb: new device is high-speed
[   10.846476] dwc2 20980000.usb: new device is high-speed
[   10.978497] dwc2 20980000.usb: new device is high-speed
[   11.051539] dwc2 20980000.usb: new address 9

On the Pi when SSH stops working:

dmesg  | grep dwc2
[    0.000000] Kernel command line: coherent_pool=1M 8250.nr_uarts=0 snd_bcm2835.enable_compat_alsa=0 snd_bcm2835.enable_hdmi=1 video=Composite-1:720x480@60i smsc95xx.macaddr=B8:27:EB:5A:0E:9B vc_mem.mem_base=0x1ec00000 vc_mem.mem_size=0x20000000  dwc_otg.lpm_enable=0 dwc_otg.speed=1 console=ttyS0,115200 console=tty1 root=PARTUUID=80be026c-02 rootfstype=ext4 fsck.repair=yes rootwait modules-load=dwc2,g_ether
[    2.990618]     modules-load=dwc2,g_ether
[    9.702605] dwc2 20980000.usb: supply vusb_d not found, using dummy regulator
[    9.707165] dwc2 20980000.usb: supply vusb_a not found, using dummy regulator
[    9.952832] dwc2 20980000.usb: EPs: 8, dedicated fifos, 4080 entries in SPRAM
[    9.966937] dwc2 20980000.usb: DWC OTG Controller
[   10.006210] dwc2 20980000.usb: new USB bus registered, assigned bus number 1
[   10.032801] dwc2 20980000.usb: irq 33, io mem 0x20980000
[   10.063128] usb usb1: Manufacturer: Linux 5.10.92+ dwc2_hsotg
[   10.456476] dwc2 20980000.usb: bound driver g_ether
[   10.580662] dwc2 20980000.usb: new device is high-speed
[   10.848792] dwc2 20980000.usb: new device is high-speed
[   10.976975] dwc2 20980000.usb: new device is high-speed
[   11.053953] dwc2 20980000.usb: new address 11

The Ubuntu host where the Pi is plugged in as a USB gadget shows:

[ 1799.836438] usb 1-4: new full-speed USB device number 8 using xhci_hcd
[ 1800.520438] usb 1-4: new high-speed USB device number 9 using xhci_hcd
[ 1800.669903] usb 1-4: New USB device found, idVendor=0525, idProduct=a4a2, bcdDevice= 5.10
[ 1800.669908] usb 1-4: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[ 1800.669911] usb 1-4: Product: RNDIS/Ethernet Gadget
[ 1800.669914] usb 1-4: Manufacturer: Linux 5.10.92+ with 20980000.usb
[ 1800.701969] cdc_ether 1-4:1.0 usb0: register 'cdc_ether' at usb-0000:00:14.0-4, CDC Ethernet Device, 92:0e:10:16:3d:3e
[ 1800.702072] usbcore: registered new interface driver cdc_ether
[ 1800.702116] usbcore: registered new interface driver cdc_subset
[ 1800.723964] cdc_ether 1-4:1.0 enx920e10163d3e: renamed from usb0

After changing the Ubuntu Settings GUI for the network connection to "IPv4 Shared to Other computers" and turning the connection On and Off:

[ 1893.038229] bpfilter: Loaded bpfilter_umh pid 5030
[ 1893.038664] Started bpfilter

Additional context

I put all the details here: https://raspberrypi.stackexchange.com/questions/136166/ssh-over-usb-stops-working-after-pi-reboot

@pelwell
Copy link
Contributor

pelwell commented Feb 14, 2022

Does unplugging the USB cable and reconnecting it make the interface work again?

Perhaps the problem is with the host computer not handling the reboot of the Zero.

@alevchuk
Copy link
Author

Yes, same issue when unplugging and reconnecting the USB cable.

@alevchuk
Copy link
Author

It works on Debian 11.

(Both work: reboot and yanking out the USB cord)

@pelwell
Copy link
Contributor

pelwell commented Feb 16, 2022

On the host PC or the Zero?

@pelwell
Copy link
Contributor

pelwell commented Feb 16, 2022

Probably a duplicate of #4872. Try updating the kernel to 5.15. 24 with sudo rpi-update.

@macmpi
Copy link

macmpi commented Feb 17, 2022

unfortunately available firmware from yesterday is only @.23 for now
raspberrypi/rpi-firmware@9e99d81

@alevchuk
Copy link
Author

@pelwell it works for Debian on host PC. Yet, a problem with Ubuntu on the host PC. On the Pi Zero I always have Rasbian 11 (bullseye)

@pelwell
Copy link
Contributor

pelwell commented Feb 17, 2022

unfortunately available firmware from yesterday is only @.23 for now

That's true, but a comment in #4872 suggests it might be worth updating anyway.

@alevchuk
Copy link
Author

alevchuk commented Feb 17, 2022

I did the Pi Zero kernel update. Same problem.

before: Linux raspberrypi 5.10.92+ #1514
after: Linux raspberrypi 5.15.23+ #1525

@macmpi
Copy link

macmpi commented Feb 18, 2022

Lucky you.
For me 5.15.18 fixed it, then 5.15.21 & 5.15.23 broke it again, but I'm hopeful 5.15.24 may fix it again...

@pelwell
Copy link
Contributor

pelwell commented Feb 19, 2022

5.15.24 is now in rpi-firmware and available for rpi-update.

@alevchuk
Copy link
Author

Sorry my last comment was incorrect, updating to 5.15.23 does not work after rebooting Pi and host PC. The usb0 interface does not have an IP address.

I'm going to try re-running sudo rpi-update which will bring me to 5.15.24

@alevchuk alevchuk reopened this Feb 19, 2022
@alevchuk
Copy link
Author

5.15.24 has the same issue. The host PC no longer sees the USB Ethernet connection

@macmpi
Copy link

macmpi commented Feb 19, 2022

some progress with 5.15.24 (xhci USB ok now, no more crash), but still some issue when starting bpfilter: Host dmesg below.

[13645.073942] usb 1-2: new full-speed USB device number 9 using xhci_hcd
[13645.690122] usb 1-2: new high-speed USB device number 10 using xhci_hcd
[13645.839593] usb 1-2: New USB device found, idVendor=0525, idProduct=a4a2, bcdDevice= 5.15
[13645.839607] usb 1-2: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[13645.839613] usb 1-2: Product: RNDIS/Ethernet Gadget
[13645.839618] usb 1-2: Manufacturer: Linux 5.15.24+ with 20980000.usb
[13645.873636] cdc_ether 1-2:1.0 usb0: register 'cdc_ether' at usb-0000:00:14.0-2, CDC Ethernet Device, 96:fd:35:2d:7b:5c
[13645.874251] usbcore: registered new interface driver cdc_subset
[13645.876111] usbcore: registered new interface driver cdc_ether
[13645.886613] cdc_ether 1-2:1.0 enx96fd352d7b5c: renamed from usb0
[13756.270525] bpfilter: Loaded bpfilter_umh pid 14971
[13756.270754] Started bpfilter
[13802.243003] ------------[ cut here ]------------
[13802.243012] NETDEV WATCHDOG: enx96fd352d7b5c (cdc_ether): transmit queue 0 timed out
[13802.243060] WARNING: CPU: 0 PID: 0 at net/sched/sch_generic.c:467 dev_watchdog+0x24f/0x260
[13802.243075] Modules linked in: xt_MASQUERADE xt_state xt_conntrack ipt_REJECT nf_reject_ipv4 iptable_filter xt_tcpudp bpfilter nf_nat_h323 nf_conntrack_h323 nf_nat_pptp nf_conntrack_pptp nf_nat_tftp nf_conntrack_tftp nf_nat_sip nf_conntrack_sip nf_nat_irc nf_conntrack_irc nf_nat_ftp nf_conntrack_ftp iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c cdc_subset cdc_ether usbnet mii uas usb_storage ccm rfcomm snd_sof_pci_intel_cnl snd_sof_intel_hda_common cmac algif_hash soundwire_intel soundwire_generic_allocation soundwire_cadence snd_sof_intel_hda snd_sof_pci snd_sof_xtensa_dsp snd_sof algif_skcipher snd_soc_hdac_hda snd_hda_ext_core snd_soc_acpi_intel_match af_alg snd_hda_codec_hdmi bnep snd_soc_acpi soundwire_bus snd_soc_core snd_compress ac97_bus snd_pcm_dmaengine snd_hda_codec_realtek ipheth snd_hda_codec_generic ledtrig_audio snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi snd_hda_codec snd_hda_core snd_hwdep snd_pcm snd_seq_midi snd_seq_midi_event uvcvideo
[13802.243208]  snd_rawmidi intel_rapl_msr intel_rapl_common btusb intel_tcc_cooling videobuf2_vmalloc btrtl videobuf2_memops btbcm videobuf2_v4l2 x86_pkg_temp_thermal intel_powerclamp btintel videobuf2_common snd_seq bluetooth coretemp iwlmvm videodev mac80211 snd_seq_device libarc4 mc ecdh_generic rtsx_usb_ms ecc joydev apple_mfi_fastcharge snd_timer iwlwifi kvm_intel kvm memstick rapl nls_iso8859_1 snd intel_cstate cfg80211 input_leds serio_raw efi_pstore soundcore hid_multitouch intel_pch_thermal acpi_pad acpi_tad mac_hid intel_hid soc_button_array sparse_keymap sch_fq_codel ipmi_devintf ipmi_msghandler msr parport_pc ppdev binfmt_misc lp parport ip_tables x_tables autofs4 dm_crypt rtsx_usb_sdmmc rtsx_usb hid_generic i915 crct10dif_pclmul i2c_algo_bit crc32_pclmul ghash_clmulni_intel nvme drm_kms_helper syscopyarea aesni_intel sysfillrect sysimgblt fb_sys_fops cec crypto_simd rc_core cryptd nvme_core sdhci_pci drm intel_lpss_pci cqhci i2c_i801 intel_lpss sdhci i2c_smbus idma64 ahci
[13802.243413]  xhci_pci libahci i2c_hid_acpi xhci_pci_renesas i2c_hid hid video pinctrl_cannonlake
[13802.243436] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 5.13.0-30-generic #33~20.04.1-Ubuntu
[13802.243449] Hardware name: Star Labs LabTop/LabTop, BIOS 2.6.0 09/30/2021
[13802.243454] RIP: 0010:dev_watchdog+0x24f/0x260
[13802.243467] Code: 87 36 fd ff eb ab 4c 89 ff c6 05 34 d8 6e 01 01 e8 96 0e fa ff 44 89 e9 4c 89 fe 48 c7 c7 00 b2 ca 88 48 89 c2 e8 8c 27 17 00 <0f> 0b eb 8c 66 66 2e 0f 1f 84 00 00 00 00 00 66 90 0f 1f 44 00 00
[13802.243477] RSP: 0018:ffffaede40003e88 EFLAGS: 00010282
[13802.243486] RAX: 0000000000000000 RBX: ffff8d8b92486600 RCX: 0000000000000027
[13802.243493] RDX: 0000000000000027 RSI: 00000000ffffdfff RDI: ffff8d8d4e4209c8
[13802.243499] RBP: ffffaede40003eb8 R08: ffff8d8d4e4209c0 R09: ffffaede40003c60
[13802.243505] R10: 0000000000000001 R11: 0000000000000001 R12: 0000000000000001
[13802.243510] R13: 0000000000000000 R14: ffff8d8ba5118480 R15: ffff8d8ba5118000
[13802.243516] FS:  0000000000000000(0000) GS:ffff8d8d4e400000(0000) knlGS:0000000000000000
[13802.243524] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[13802.243530] CR2: 00000d0cc03b3000 CR3: 000000043f410003 CR4: 00000000003706f0
[13802.243537] Call Trace:
[13802.243541]  <IRQ>
[13802.243550]  ? pfifo_fast_enqueue+0x150/0x150
[13802.243561]  call_timer_fn+0x2c/0x100
[13802.243574]  run_timer_softirq+0x3d7/0x480
[13802.243585]  ? lapic_next_deadline+0x2c/0x40
[13802.243593]  ? clockevents_program_event+0x8f/0xe0
[13802.243605]  __do_softirq+0xdd/0x29b
[13802.243619]  irq_exit_rcu+0xa4/0xb0
[13802.243626]  sysvec_apic_timer_interrupt+0x7c/0x90
[13802.243634]  </IRQ>
[13802.243636]  <TASK>
[13802.243639]  asm_sysvec_apic_timer_interrupt+0x12/0x20
[13802.243648] RIP: 0010:cpuidle_enter_state+0xcc/0x360
[13802.243657] Code: 3d 01 a3 0b 78 e8 f4 85 78 ff 49 89 c6 0f 1f 44 00 00 31 ff e8 c5 90 78 ff 80 7d d7 00 0f 85 01 01 00 00 fb 66 0f 1f 44 00 00 <45> 85 ff 0f 88 0d 01 00 00 49 63 c7 4c 2b 75 c8 48 8d 14 40 48 8d
[13802.243663] RSP: 0018:ffffffff89403dc8 EFLAGS: 00000246
[13802.243669] RAX: ffff8d8d4e434e80 RBX: 0000000000000006 RCX: 000000000000001f
[13802.243673] RDX: 0000000000000000 RSI: 000000003c9b2a8c RDI: 0000000000000000
[13802.243677] RBP: ffffffff89403e00 R08: 00000c8d95ca4be7 R09: 0000000000000001
[13802.243681] R10: ffff8d8d4e4338a4 R11: ffff8d8d4e433884 R12: ffff8d8d4e440700
[13802.243684] R13: ffffffff89650e40 R14: 00000c8d95ca4be7 R15: 0000000000000006
[13802.243692]  ? cpuidle_enter_state+0xbb/0x360
[13802.243700]  cpuidle_enter+0x2e/0x40
[13802.243706]  call_cpuidle+0x23/0x40
[13802.243715]  do_idle+0x1e6/0x260
[13802.243724]  cpu_startup_entry+0x20/0x30
[13802.243732]  rest_init+0xb8/0xba
[13802.243739]  arch_call_rest_init+0xe/0x1b
[13802.243748]  start_kernel+0x6ae/0x6d4
[13802.243756]  x86_64_start_reservations+0x24/0x26
[13802.243763]  x86_64_start_kernel+0x8b/0x8f
[13802.243769]  secondary_startup_64_no_verify+0xc2/0xcb
[13802.243781]  </TASK>
[13802.243784] ---[ end trace 21e2b1a3b4e2ab0e ]---

@alevchuk
Copy link
Author

Summary: upgrading the Pi Zero kernel to 5.15.23 Or 5.15.24 does not fix the issue. It makes it worse. After rebooting the host PC the network interface no longer shows up on the host PC.

@macmpi
Copy link

macmpi commented Feb 23, 2022

Interestingly it works fine with my Host on Mint 19.2, kernel 4.15.0-167
But still issue remains with Host on Ubuntu 20.04.1, kernel 5.13.0-30 (elementary 6.1).
Not same PC hardware either in both case btw...

@macmpi
Copy link

macmpi commented Mar 2, 2022

5.15.26 seems to have yet-a-further fix for the issue:

commit 943a914d3dab545c1496298a193a49130213806b
Author: Fabrice Gasnier <[email protected]>
Date:   Wed Feb 16 09:12:15 2022 +0100

    usb: dwc2: drd: fix soft connect when gadget is unconfigured
    
    commit 32fde84362c40961726a5c91f35ad37355ccc0c6 upstream.
    
    When the gadget driver hasn't been (yet) configured, and the cable is
    connected to a HOST, the SFTDISCON gets cleared unconditionally, so the
    HOST tries to enumerate it.
    At the host side, this can result in a stuck USB port or worse. When
    getting lucky, some dmesg can be observed at the host side:
     new high-speed USB device number ...
     device descriptor read/64, error -110
    
    Fix it in drd, by checking the enabled flag before calling
    dwc2_hsotg_core_connect(). It will be called later, once configured,
    by the normal flow:
    - udc_bind_to_driver
     - usb_gadget_connect
       - dwc2_hsotg_pullup
         - dwc2_hsotg_core_connect
    
    Fixes: 17f934024e84 ("usb: dwc2: override PHY input signals with usb role switch support")
    Cc: stable <[email protected]>
    Signed-off-by: Fabrice Gasnier <[email protected]>
    Link: https://lore.kernel.org/r/[email protected]
    Signed-off-by: Greg Kroah-Hartman <[email protected]>

@macmpi
Copy link

macmpi commented Mar 5, 2022

I confirm 5.15.26 with todays rpi-update does fix the issue: I hope it will last for a good while now...

@EzYDark
Copy link

EzYDark commented Mar 15, 2022

I can also confirm 5.15.26 works, but version 5.15.27 and 5.15.28 no longer work...

I tested with the help of this tutorial: https://www.isticktoit.net/?p=1383
and then updated the kernel with rpi-update.

@EzYDark
Copy link

EzYDark commented Mar 15, 2022

I can also confirm 5.15.26 works, but version 5.15.27 and 5.15.28 no longer work...

I tested with the help of this tutorial: https://www.isticktoit.net/?p=1383 and then updated the kernel with rpi-update.

Never mind. I just had broken cable... I guess...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants