More, then 6 of USB 4G modems does not work correct. Process [kworker] have high iowait load.

[wind3style]

When I connected 7 of USB 4G modems, system began to hang (to freeze), but If I connected only 6 USB 4G modems, system work stable.
Please repair this issue..

System begin work stable if one of 7 interfaces USB modems set to "down". (Example: "# ifconfig eth5 down")

I mean, that it issue in the cdc_ether module of kernel.
I tried to connect 7 of USB 4G modems to laptop with CentOS 7.1.1503 - system worked stable.

In Example, when i connected 7 of USB 4G modems:

And, whan I connected 6 of USB 4G modems:

Add information:
root@raspberrypi:~# uname -a
Linux raspberrypi 4.1.19-v7+ #858 SMP Tue Mar 15 15:56:00 GMT 2016 armv7l GNU/Linux

"dmesg" after connected 7th USB 4G modem:
root@raspberrypi:~# dmesg
[ 2367.510026] usb 1-1.2.1: new high-speed USB device number 15 using dwc_otg
[ 2367.631845] usb 1-1.2.1: New USB device found, idVendor=19d2, idProduct=1225
[ 2367.631872] usb 1-1.2.1: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[ 2367.631889] usb 1-1.2.1: Product: ZTE Technologies MSM
[ 2367.631905] usb 1-1.2.1: Manufacturer: ZTE,Incorporated
[ 2367.631922] usb 1-1.2.1: SerialNumber: MF8250ZTED000000CP261718HNL0ZV5OM9A4JX8_8_7D74&&&&&&&&&&&&&&&&&0
[ 2367.650934] usb-storage 1-1.2.1:1.0: USB Mass Storage device detected
[ 2367.651645] usb-storage 1-1.2.1:1.0: Quirks match for vid 19d2 pid 1225: 1
[ 2367.651772] scsi host8: usb-storage 1-1.2.1:1.0
[ 2368.668932] scsi 8:0:0:0: CD-ROM CWID USB SCSI CD-ROM 2.31 PQ: 0 ANSI: 2
[ 2368.852676] sr 8:0:0:0: [sr2] scsi-1 drive
[ 2368.853860] sr 8:0:0:0: Attached scsi CD-ROM sr2
[ 2368.856080] sr 8:0:0:0: Attached scsi generic sg8 type 5
[ 2373.850736] usb 1-1.2.1: USB disconnect, device number 15
[ 2374.140012] usb 1-1.2.1: new high-speed USB device number 16 using dwc_otg
[ 2374.263630] usb 1-1.2.1: New USB device found, idVendor=19d2, idProduct=1405
[ 2374.263651] usb 1-1.2.1: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[ 2374.263662] usb 1-1.2.1: Product: ZTE Technologies MSM
[ 2374.263673] usb 1-1.2.1: Manufacturer: ZTE,Incorporated
[ 2374.263684] usb 1-1.2.1: SerialNumber: MF825C0ZTED00000CP2261718HNL0ZV5OM9A4JX8_8_7D74&&&&&&&&&&&&&&&&0
[ 2374.298674] cdc_ether 1-1.2.1:1.0 usb2: register 'cdc_ether' at usb-3f980000.usb-1.2.1, CDC Ethernet Device, 36:4b:50:b7:ef:2d
[ 2374.305436] usb-storage 1-1.2.1:1.2: USB Mass Storage device detected
[ 2374.305838] scsi host9: usb-storage 1-1.2.1:1.2
[ 2376.313399] scsi 9:0:0:0: CD-ROM CWID USB SCSI CD-ROM 2.31 PQ: 0 ANSI: 2
[ 2378.194383] sr 9:0:0:0: [sr2] scsi-1 drive
[ 2378.197427] sr 9:0:0:0: Attached scsi CD-ROM sr2
[ 2378.200768] sr 9:0:0:0: Attached scsi generic sg8 type 5

[pelwell]

I'm impressed it works for 6. How many do you think you should be able to use with your RPi?

Load average is going to be artificially high because Raspbian distributions enable the io_is_busy feature of the on-demand governor in order to get good SD card performance (the SD card clock varies with the core clock). However, commits have just gone into the kernel and firmware repos that remove the need to set io_is_busy, along with a temporary hack to ignore io_is_busy to get some meaningful testing, so if you rpi-update now you should find your load average is lower.

[popcornmix]

You'll need sudo BRANCH=next rpi-update to get the io_is_busy removal and lower load - it's not on master branch yet.

[wind3style]

so if you rpi-update now you should find your load average is lower.
I have just updated (rpi-update) software, but it didn't help.
After update software, param io_is_busy was '1':
root@raspberrypi:~# cat /sys/devices/system/cpu/cpufreq/ondemand/io_is_busy
1

root@raspberrypi:~# uname -a
Linux raspberrypi 4.1.20-v7+ #867 SMP Wed Mar 23 20:12:32 GMT 2016 armv7l GNU/Linux

I have tried to disable io_is_busy:
root@raspberrypi:~# echo 0 > /sys/devices/system/cpu/cpufreq/ondemand/io_is_busy
But it didn't help too.

I have noticed, that other processes(exclude "kworker" ) start to take high load iowait:

I guess, that it issue in the cdc_ether module of kernel and it have artificially limitation by some params.

You'll need sudo BRANCH=next rpi-update to get the io_is_busy removal and lower load - it's not on master branch yet.

I have tried to update software by BRANCH=next rpi-update:
root@raspberrypi:~# uname -a
Linux raspberrypi 4.4.6-v7+ #871 SMP Mon Apr 4 19:59:50 BST 2016 armv7l GNU/Linux
But it didn't help, after add 7th USB 4G modem, iowait have rised to high load.

[cleverca22]

it may help to get a kernel backtrace
echo l > /proc/sysrq-trigger
then check dmesg, it should show the call stack for every core

[P33M]

Can you post the results of the following:

• Run a root shell (sudo -s)
• navigate to /sys/devices/platform/soc/3f980000.usb
• echo 0x418 > regoffset
• cat regvalue - run this several times and post results.

Example:

root@raspberrypi:/sys/devices/platform/soc/3f980000.usb# echo 0x418 > regoffset
root@raspberrypi:/sys/devices/platform/soc/3f980000.usb# cat regvalue
Reg@0x000418 = 0x00000020

[Ruffio]

@wind3style is this still an issue? Could you post the results from @P33M commands?

[wind3style]

Yes, this is an issue still.
I have tried all of the above recomendation.

[P33M]

Then what were the results of the commands I posted above?

[wind3style]

root@wayzz:/sys/devices/platform/soc/3f980000.usb#
root@wayzz:/sys/devices/platform/soc/3f980000.usb# echo 0x418 > regoffset
root@wayzz:/sys/devices/platform/soc/3f980000.usb# cat regvalue
Reg@0x000418 = 0x000000cc

[P33M]

How many USB modems are connected and active when you ran this command? Running the command only makes sense when you're in the high iowait case.

I need the results of multiple readouts of "regvalue" - try this script (save as e.g. test.sh then chmod +x test.sh)

#!/bin/bash

echo 0x418 > /sys/devices/platform/soc/3f980000.usb/regoffset
i="0"
while [ $i -lt 100 ]
do
        cat /sys/devices/platform/soc/3f980000.usb/regvalue >> test.txt
        i=$[$i+1]
done

and post the contents of the file test.txt here.

[wind3style]

6 USB modems, iowait was high:
root@wayzz:/sys/devices/platform/soc/3f980000.usb# echo 0x418 > regoffset
root@wayzz:/sys/devices/platform/soc/3f980000.usb# cat regvalue
Reg@0x000418 = 0x000000ff

muiltiple readouts of "regvalue" always is:
Reg@0x000418 = 0x000000ff
(100 lines of file test.txt)

[P33M]

You've run out of host channels. There are 8 host channels (i.e. state machines capable of carrying out a single USB transfer to/from a single endpoint) which means there can be a maximum of 8 outstanding transfers. The respective processes trying to access each USB device are blocked waiting for host channels to become available - in the resource-starved case then endpoints will get serviced semi-randomly.

This number of devices is not supportable on a single Pi.