More, then 6 of USB 4G modems does not work correct. Process [kworker] have high iowait load.

[wind3style]

When I connected 7 of USB 4G modems, system began to hang (to freeze), but If I connected only 6 USB 4G modems, system work stable.
Please repair this issue..

System begin work stable if one of 7 interfaces USB modems set to "down". (Example: "# ifconfig eth5 down")

I mean, that it issue in the cdc_ether module of kernel.
I tried to connect 7 of USB 4G modems to laptop with CentOS 7.1.1503 - system worked stable.

In Example, when i connected 7 of USB 4G modems:

And, whan I connected 6 of USB 4G modems:

Add information:
root@raspberrypi:~# uname -a
Linux raspberrypi 4.1.19-v7+ #858 SMP Tue Mar 15 15:56:00 GMT 2016 armv7l GNU/Linux

"dmesg" after connected 7th USB 4G modem:
root@raspberrypi:~# dmesg
[ 2367.510026] usb 1-1.2.1: new high-speed USB device number 15 using dwc_otg
[ 2367.631845] usb 1-1.2.1: New USB device found, idVendor=19d2, idProduct=1225
[ 2367.631872] usb 1-1.2.1: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[ 2367.631889] usb 1-1.2.1: Product: ZTE Technologies MSM
[ 2367.631905] usb 1-1.2.1: Manufacturer: ZTE,Incorporated
[ 2367.631922] usb 1-1.2.1: SerialNumber: MF8250ZTED000000CP261718HNL0ZV5OM9A4JX8_8_7D74&&&&&&&&&&&&&&&&&0
[ 2367.650934] usb-storage 1-1.2.1:1.0: USB Mass Storage device detected
[ 2367.651645] usb-storage 1-1.2.1:1.0: Quirks match for vid 19d2 pid 1225: 1
[ 2367.651772] scsi host8: usb-storage 1-1.2.1:1.0
[ 2368.668932] scsi 8:0:0:0: CD-ROM CWID USB SCSI CD-ROM 2.31 PQ: 0 ANSI: 2
[ 2368.852676] sr 8:0:0:0: [sr2] scsi-1 drive
[ 2368.853860] sr 8:0:0:0: Attached scsi CD-ROM sr2
[ 2368.856080] sr 8:0:0:0: Attached scsi generic sg8 type 5
[ 2373.850736] usb 1-1.2.1: USB disconnect, device number 15
[ 2374.140012] usb 1-1.2.1: new high-speed USB device number 16 using dwc_otg
[ 2374.263630] usb 1-1.2.1: New USB device found, idVendor=19d2, idProduct=1405
[ 2374.263651] usb 1-1.2.1: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[ 2374.263662] usb 1-1.2.1: Product: ZTE Technologies MSM
[ 2374.263673] usb 1-1.2.1: Manufacturer: ZTE,Incorporated
[ 2374.263684] usb 1-1.2.1: SerialNumber: MF825C0ZTED00000CP2261718HNL0ZV5OM9A4JX8_8_7D74&&&&&&&&&&&&&&&&0
[ 2374.298674] cdc_ether 1-1.2.1:1.0 usb2: register 'cdc_ether' at usb-3f980000.usb-1.2.1, CDC Ethernet Device, 36:4b:50:b7:ef:2d
[ 2374.305436] usb-storage 1-1.2.1:1.2: USB Mass Storage device detected
[ 2374.305838] scsi host9: usb-storage 1-1.2.1:1.2
[ 2376.313399] scsi 9:0:0:0: CD-ROM CWID USB SCSI CD-ROM 2.31 PQ: 0 ANSI: 2
[ 2378.194383] sr 9:0:0:0: [sr2] scsi-1 drive
[ 2378.197427] sr 9:0:0:0: Attached scsi CD-ROM sr2
[ 2378.200768] sr 9:0:0:0: Attached scsi generic sg8 type 5

[pelwell]

I'm impressed it works for 6. How many do you think you should be able to use with your RPi?

Load average is going to be artificially high because Raspbian distributions enable the io_is_busy feature of the on-demand governor in order to get good SD card performance (the SD card clock varies with the core clock). However, commits have just gone into the kernel and firmware repos that remove the need to set io_is_busy, along with a temporary hack to ignore io_is_busy to get some meaningful testing, so if you rpi-update now you should find your load average is lower.

[popcornmix]

You'll need sudo BRANCH=next rpi-update to get the io_is_busy removal and lower load - it's not on master branch yet.

[wind3style]

so if you rpi-update now you should find your load average is lower.
I have just updated (rpi-update) software, but it didn't help.
After update software, param io_is_busy was '1':
root@raspberrypi:~# cat /sys/devices/system/cpu/cpufreq/ondemand/io_is_busy
1

root@raspberrypi:~# uname -a
Linux raspberrypi 4.1.20-v7+ #867 SMP Wed Mar 23 20:12:32 GMT 2016 armv7l GNU/Linux

I have tried to disable io_is_busy:
root@raspberrypi:~# echo 0 > /sys/devices/system/cpu/cpufreq/ondemand/io_is_busy
But it didn't help too.

I have noticed, that other processes(exclude "kworker" ) start to take high load iowait:

I guess, that it issue in the cdc_ether module of kernel and it have artificially limitation by some params.

You'll need sudo BRANCH=next rpi-update to get the io_is_busy removal and lower load - it's not on master branch yet.

I have tried to update software by BRANCH=next rpi-update:
root@raspberrypi:~# uname -a
Linux raspberrypi 4.4.6-v7+ #871 SMP Mon Apr 4 19:59:50 BST 2016 armv7l GNU/Linux
But it didn't help, after add 7th USB 4G modem, iowait have rised to high load.

[cleverca22]

it may help to get a kernel backtrace
echo l > /proc/sysrq-trigger
then check dmesg, it should show the call stack for every core

[P33M]

Can you post the results of the following:

• Run a root shell (sudo -s)
• navigate to /sys/devices/platform/soc/3f980000.usb
• echo 0x418 > regoffset
• cat regvalue - run this several times and post results.

Example:

root@raspberrypi:/sys/devices/platform/soc/3f980000.usb# echo 0x418 > regoffset
root@raspberrypi:/sys/devices/platform/soc/3f980000.usb# cat regvalue
Reg@0x000418 = 0x00000020

[Ruffio]

@wind3style is this still an issue? Could you post the results from @P33M commands?

[wind3style]

Yes, this is an issue still.
I have tried all of the above recomendation.

[P33M]

Then what were the results of the commands I posted above?

[wind3style]

root@wayzz:/sys/devices/platform/soc/3f980000.usb#
root@wayzz:/sys/devices/platform/soc/3f980000.usb# echo 0x418 > regoffset
root@wayzz:/sys/devices/platform/soc/3f980000.usb# cat regvalue
Reg@0x000418 = 0x000000cc

[P33M]

How many USB modems are connected and active when you ran this command? Running the command only makes sense when you're in the high iowait case.

I need the results of multiple readouts of "regvalue" - try this script (save as e.g. test.sh then chmod +x test.sh)

#!/bin/bash

echo 0x418 > /sys/devices/platform/soc/3f980000.usb/regoffset
i="0"
while [ $i -lt 100 ]
do
        cat /sys/devices/platform/soc/3f980000.usb/regvalue >> test.txt
        i=$[$i+1]
done

and post the contents of the file test.txt here.

[wind3style]

6 USB modems, iowait was high:
root@wayzz:/sys/devices/platform/soc/3f980000.usb# echo 0x418 > regoffset
root@wayzz:/sys/devices/platform/soc/3f980000.usb# cat regvalue
Reg@0x000418 = 0x000000ff

muiltiple readouts of "regvalue" always is:
Reg@0x000418 = 0x000000ff
(100 lines of file test.txt)

[P33M]

You've run out of host channels. There are 8 host channels (i.e. state machines capable of carrying out a single USB transfer to/from a single endpoint) which means there can be a maximum of 8 outstanding transfers. The respective processes trying to access each USB device are blocked waiting for host channels to become available - in the resource-starved case then endpoints will get serviced semi-randomly.

This number of devices is not supportable on a single Pi.

[wind3style]

I have tried use 8 USB modems on the laptop with CentOS 7, and I didn't have problem.
Is it posible increase host channels by recompile Linux kernel?
That kernel params used for limitation maximum number of host channels?

[P33M]

No, this is a hardware limitation.

[wind3style]

Is it limitation of SoC or USB hub?

[P33M]

The limitation is in the SoC USB peripheral.

[jkuek]

@P33M can you explain what that regvalue means? What values are considered normal? Does a value of 0xff indicate all host channels are busy? Is a lower value better?

[ghollingworth]

The value is a bitfield of the channels in use, if it is 0xff that means
all eight channels are in use...

On Tue, Jul 26, 2016 at 3:55 AM, jkuek [email protected] wrote:

@P33M https://github.com/P33M can you explain what that regvalue means?
What values are considered normal? Does a value of 0xff indicate all host
channels are busy? Is a lower value better?

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
#582 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AB9CHLMd4eTHGf8Ecwr2UknIw9kXoWizks5qZXcwgaJpZM4H_zAq
.

[vmspike]

Two years passed. As I remember old versions of Raspbian was unable to recognize more than 5 LTE dongles at a time. Who knows maybe now is possible to tune some dwc_otg (or other) options to decrease iowait rate with higher amount of USB devices?

[JamesH65]

No, the HW limitation is still there, the USB controller has not changed.

[vmspike]

I understand that the same HW is still there, I'm asking if it's possible to balance load for existing 8 host channels between all USB devices in more smart way to decrease iowait, for example tune interrupts, fiq, etc; maybe increase dwc_otg.int_ep_interval_min and/or change other dwc_otg.* options.

BTW what will be the limitation for xhci/ehci/ohci USB controllers?

[JamesH65]

I have absolutely no idea. You are more than welcome to try stuff out yourself, not sure we have any engineering time available for this internally. @P33M?

[P33M]

It's never going to work. Host channels are effectively tied up for indefinitely if a device never responds with any data on a bulk endpoint. There's no way you can guarantee that there won't be data loss if you force a host channel halt, either.

xHCI/EHCI controllers use an entirely different hardware architecture. This limitation is specific to the DWC OTG core.