Pi4 crash when using 2nd USB3 device when booting from USB3 SSD

[poliboy]

I use the Pi4 booting with a USB3 connected SSD.
This works nicely, but when I connect in addition an external USB3 HDD to the USB3 port, then the system crashes with "IO errors". Any commands like "dmesg" are not available any longer.
Connection of the HDD to USB2 port works. Booting from SD card and connecting the USB3 SSD and the USB3 HDD at the same time works.

Actual behaviour
The GUI disappears (only background is visible). Some "IO errors" are indicated.

System
System Information

Raspberry Pi 4 Model B Rev 1.1
PRETTY_NAME="Raspbian GNU/Linux 10 (buster)"
NAME="Raspbian GNU/Linux"
VERSION_ID="10"
VERSION="10 (buster)"

Raspberry Pi reference 2019-09-26
Generated using pi-gen, https://github.com/RPi-Distro/pi-gen, 80d486687ea77d31fc
3fc13cf3a2f8b464e129be, stage5

Linux toni4 4.19.97-v7l+ #1294 SMP Thu Jan 30 13:21:14 GMT 2020 armv7l GNU/Linux
Revision : c03111
Serial : 100000000824fe4c
Model : Raspberry Pi 4 Model B Rev 1.1
Throttled flag : throttled=0x0
Camera : supported=0 detected=0

• Which model of Raspberry Pi? Pi4

• Which OS and version (cat /etc/rpi-issue)?
  Raspberry Pi reference 2019-09-26
  Generated using pi-gen, https://github.com/RPi-Distro/pi-gen, 80d486687ea77d31fc3fc13cf3a2f8b464e129be, stage5

• Which firmware version (vcgencmd version)?
  Feb 12 2020 12:36:21
  Copyright (c) 2012 Broadcom
  version c3c8dbdf147686fb0c3f32aece709d0653368810 (clean) (release) (start)

• Which kernel version (uname -a)?
  Linux toni4 4.19.97-v7l+ problem with device tree overlay at86rf233  #1294 SMP Thu Jan 30 13:21:14 GMT 2020 armv7l GNU/Linux

Logs
unfortunately demsg is not available any longer

Additional context
Add any other relevant context for the problem.

[tiagoboldt]

Have you tried connecting the USB disks using a powered USB hub? The pi4 might now have enough juice to power 2 external disks. I've struggled with this in the past and solved it with a powered hub. Not sure how this relates to it working on the USB2 ports, not sure if the USB3 port is pulling more power.

[poliboy]

Good point. The SSD and its USB3 interface are bus powered and working standalone when booting from it. The additional USB3 HDD has a power supply attached. My expectation is, that there is not more current drawn from USB3 port when plugging in a powered device.
I tried with a powered USB3 hub and both disks worked also in USB3 boot mode. I assume that the driver software used is a different one, because only one USB3 port from the Pi4 is used. Unfortunately I can't check with a non-powered USB3 hub.
Therefore: I'm pretty sure that there is no issue with the power consumption, but full evidence is missing.

[popcornmix]

The additional USB3 HDD has a power supply attached. My expectation is, that there is not more current drawn from USB3 port when plugging in a powered device.

I don't think that is a valid expectation. Often the power supply to a drive just supplies, say 12V for the motor. There may be significant 5V still drawn from the Pi over USB.

[poliboy]

Thanks for your replies so far.
In order to remove all doubts, I purchased a passive USB3 hub (therefore my answer took some time).
The result: If I attach the passive hub to one USB3 port of the Pi4, and the SSD to the passive hub, then it works (as expected). If I then attach the HDD to the passive hub, it works as well!
If I instead attach the HDD to the 2nd USB3 port of the Pi4, then the crash happens. This is reproducible also with another HDD (I used Seagate and WD).
I consider this as evidence that the problem is not the too high current drawn from the Pi4.
Some other experiment: I booted with SSD directly in USB3 port of Pi4, and after plugging in the passive USB hub, then the crash happened. Nothing was attached to the passive hub.
Let me know if I could support the bug tracing by some SW analytics...

[tiagoboldt]

I don't think there's any bug here to be solved. It's an electronic limitation. When you draw more current then the pi can provide, it restarts. No way around that. Maybe this should be better documented -- this issue might do just that for those who have the same problem in the future.

I guess this can be closed, but I'll leave that feedback to @popcornmix who's the expert here.

[poliboy]

Maybe my formulation was too complicated: I don't draw more current than the Pi can provide... Otherwise I would see the same issue with the passive hub, which is not the case.

[P33M]

Plug an optical mouse into one of the spare USB ports. When the crash occurs, does the LED on the bottom of the mouse turn off?

[poliboy]

Thanks for the hint.
I attached this:
Bottom USB3: SSD
Bottom USB2: optical mouse
After booting, everything was fine.
Then I plugged the SSD (actively supplied) into the Top USB3.
The crash occurred, but the optical mouse still was working perfectly and the LED was still on.

[poliboy]

I did another experiment, maybe this refers better to your question:
Starting with
Bottom USB3: SSD
I booted without mouse and keyboard.
Then I plugged the optical mouse into the Top USB3. The crash happened, the mouse LED was on, but the mouse didn't work.

[P33M]

Do you have a powered USB3.0 hub, and if so, what is the behaviour with one or both SSDs plugged into the hub?

[poliboy]

I have a powered USB3 hub (combined with a HDD).
If I plug this into the Pi4 instead of the passive hub, and attach the drives to the powered hub, then everything works fine.

I did some more investigations on the issue and found out, that a message appears at the boot screen, which was not visible in dmesg. It took several trials to take a photo at the right point in time.
The message said:
"Failed to start persistent change of wireless mouse interrupt to avoid conflict with USB3 SSD. See 'systemctl status mouse_irq.service' for details."
The message didn't disappear when I used a wired mouse and keyboard.

The output of the mentioned "systemctl" command was:

● mouse_irq.service - persistent change of wireless mouse interrupt to avoid conflict with USB3 SSD
   Loaded: loaded (/etc/systemd/system/mouse_irq.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Sat 2020-03-28 03:32:50 CET; 8h ago
  Process: 519 ExecStart=/bin/bash -c echo 2 | sudo tee /proc/irq/55/smp_affinity (code=exited, status=1/FAILURE)
 Main PID: 519 (code=exited, status=1/FAILURE)

Mär 28 03:32:49 toni4 systemd[1]: Starting persistent change of wireless mouse interrupt to avoid conflict with USB3 SSD...
Mär 28 03:32:50 toni4 sudo[523]:     root : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/bin/tee /proc/irq/55/smp_affinity
Mär 28 03:32:50 toni4 sudo[523]: pam_unix(sudo:session): session opened for user root by (uid=0)
Mär 28 03:32:50 toni4 bash[519]: tee: /proc/irq/55/smp_affinity: Datei oder Verzeichnis nicht gefunden
Mär 28 03:32:50 toni4 bash[519]: 2
Mär 28 03:32:50 toni4 sudo[523]: pam_unix(sudo:session): session closed for user root
Mär 28 03:32:50 toni4 systemd[1]: mouse_irq.service: Main process exited, code=exited, status=1/FAILURE
Mär 28 03:32:50 toni4 systemd[1]: mouse_irq.service: Failed with result 'exit-code'.
Mär 28 03:32:50 toni4 systemd[1]: Failed to start persistent change of wireless mouse interrupt to avoid conflict with USB3 SSD.

"Datei oder Verzeichnis nicht gefunden" is translated: "File or directory not found".

Maybe this is related to the issue I described at the beginning and you know what's going on there...

[m-t-smith]

Hey Poliboy,

Did you ever determine the exact issue? I experienced a similar problem; I booted and ran an HP laptop from a USB3 attached external SSD Ubuntu 20.04 installation. This setup worked fine until I plugged a second SSD into the other USB3 port, at which point Ubuntu crashed and I got a black screen full of a rapidly repeating I/O error. If this is the same problem, I'm glad to hear that using a USB2 port for one of the drives or booting from an SD card works (though I'm unsure is this means you're booting the OS on your SSD with a bootloader on your SD card, or booting fully from the SD card and leaving the SSD-based OS to languish.).

I plan to use the SSD-based OS on a pi4, but have you tried recreating the issue on any other type of computer? I share your suspicion that something besides power is causing the problem, though that's probably usually a fair bet for a pi, and hence why everyone seems so hung up on it even after you ran both through a passive hub. My gut feeling is that the problem lies with the usb3 driver (probably UAS) integrated into the Linux kernel. USB3s cause interference; I'm guessing part of the solution to that was a bit of a hack job that involved blocking all/the other USB3 ports temporarily whenever something new was plugged into one. That would obviously be a problem for this kind of setup, because it cuts of the OS, so the interrupt handler is never reached and the request just keeps running because it's supposed to be non-blocking.

That's just my guess, would like to hear if you or anyone else figured it out.

[clssn]

Hi everyone,
looks like I have a very similar issue, though I don't boot from the SSD (only have /var mounted from one of them, which makes a crash equally destructive). I initially thought it was power related but after some weeks of trying around similar things as @poliboy power doesn't appear to be the problem. In my setup the crash happens after a couple of hours of flawless operation which makes it harder to inspect, together with journal disappearing with /var.
For now, I just wanted to confirm the problem. I'll check next, whether using only the bottom USB3 port works for me as well, and whether by plugging in arbitrary devices in the top port I can provoke the crash.
@m-t-smith, according to what you tell, it's about the point in time of connection, while I don't connect anything new when the crash happens.

Good to know I'm not alone, though it's a bit quiet in here while I can't believe we're the only ones with that particular use-case and problem.

Edit: Looks like I finally found my problem. It seems to be well known (see for example #3070) that some JMicron USB3 chipsets cause problems like mine and can be circumvented using a "quirk". Searching for the term "usb uas JMicron" there's a lot more information about the incompatibility (both Linux and Windows seems to be affected). Shopping for USB3 adapters you should search for UASP compatible devices - or disable UASP with the quirks.

[sourabhjaiswal]

Hi,

I am also facing the same problem. My setup is :-
Raspberry pi 4 4gb
USB SSD: https://www.amazon.in/gp/product/B08N187N6N/
USB HDD: https://www.amazon.in/gp/product/B01N07NBLA

I am booting system with USB SSD and connecting HDD later on. but as soon as I connect the HDD it crashes.
I tried with one Seagate HDD too but same result.

Any feedback on resolving it would be greatly appreciated.

Hi,

I am also facing the same problem. My setup is :-
Raspberry pi 4 4gb
USB SSD: https://www.amazon.in/gp/product/B08N187N6N/
USB HDD: https://www.amazon.in/gp/product/B01N07NBLA

I am booting system with USB SSD and connecting HDD later on. but as soon as I connect the HDD it crashes.
I tried with one Seagate HDD too but same result.

Any feedback on resolving it would be greatly appreciated.

Use a decent quality powered hub. That applies to anyone with USB problems with drives attached, even just one drive.

An example of a decent quality powered USB 3.0 hub is the D-Link DUB-1340, which is what I use and it works great. See https://eu.dlink.com/uk/en/products/dub-1340-4-port-superspeed-usb-3-hub.

The rest of the problems with using USB drives with the Pi can be traced to poor quality drives and enclosures - usually it's the enclosure. I recommend startech.com enclosures. There are very few problems which can be attributed to the Pi itself.