raspbian: kernel OOPS following page allocation failures under USB/network load #189
Comments
|
Can you quote your traces, or link to pastbin? |
|
Fixed with code blocks added |
|
Possibly a duplicate of: |
|
All the above lines were done with the following command line: |
|
I went one better and recompiled with smsc95xx compiled as a module, then blacklisted it. The only thing plugged into the system on boot were 2x mass storage devices. Same slab corruption, though with different text in the corrupt entries. I have gotten kmemleak working (required larger kmemleak early buffer size of 1500 entries) so will attempt to probe further. |
|
After boot, kmemleak reports Can these init calls be ignored safely? But subsequently, kmemleak breaks when I try to run a scan again during the "critical phase" i.e. video playback on remote computer. Note the time differences are due to a reboot in between. |
|
Found it (I think). I changed the kernel allocator to SLUB which allowed for extensive tracing. Every single corrupted memory is a result of a handle_hc_xfercomp_intr call from completed USB transactions. Additionally I get smbd page allocation failures but without any associated SLUB backtraces. |
|
I assume you are running with CMA enabled? Do you get same failure with it disabled? |
|
Crashed even sooner with CMA disabled. Found the problem on adding a QTD to a QH, the endpoint handle was being assigned after an atomic section. In the case of heavy USB activity, a transfer complete interrupt could happen during the QTD insertion which would then get serviced immediately after the spinlock was released. A transfer complete interrupt removes a QTD from the list which then makes the pointer used by dwc_otg_hcd_urb_enqueue point to something entirely random, usually freed memory. Zero crashes or SLUB failures with this patch applied. Edit: as an added bonus, smsc95xx.turbo_mode=y now results in zero kevent 2 dropped messages. |
|
Good catch, can you push the patch? Gordon On 9 January 2013 14:27, P33M [email protected] wrote:
|
|
Addendum to previous: kevent 2 messages are still being generated sporadically with turbo_mode=Y, though the trigger appears to be a much higher level of network activity than previously. I was getting them before the bugfix during video playback at about 1MB/s but afterwards it now requires a straight file copy operation over the network (i.e. max speed). Performance with the transfer from USB to network is typically 3.5MB/s with turbo_mode=Y and wierdly 5.1MB/s with turbo_mode=N. |
|
Presumably the 5.1MB/s with turbo_mode=N has no kevent 2 messages, and the 3.5MB/s with turbo_mode=Y has kevent 2 messages? |
|
Yes, correct. The copy operation does not fail, presumably due to TCP retries but performance suffers. |
|
Last time I investigated the dropped messages were much more common with CMA enabled (even getting them when just web browsing sites like engadget). I wasn't seeing them at all with CMA disabled. Are you seeing this, or is CMA not causing any greater problems? |
|
It seems this bug may not be fully squashed. I have seen scarce page allocation failures in a near-identical manner with slub_debug turned off after transferring about 4GB of a 6GB file but frustratingly when I turn on the slub_debug option that gives me a proper call stack it makes the bug even more scarce due to the slowdown it adds. I will play with various CMA/turbo/USB usage options to see if I can reliably replicate it. |
|
Definitely not squashed. Tested with stock kernel pulled from rpi-update with the patch incorporated. I dug out my old zd1211b wireless adapter and attempted to perform the same test (network copy). I can ping flood the device OK but as soon as I initiate multiple USB activity either with video viewing or file copy then the transfer grinds to a halt with a pitifully slow transfer rate with associated allocation failures. I think the inherent lag and variability of a wireless network interface are suddenly enough to tickle a similar bug. |
|
Progress, of a sort: I need to eliminate the zd1211 driver as a cause - I've ordered a different stick on Ebay to cross-compare. |
|
RTL5370 chipset produces no comparable slowdown or allocation failures. Bug most likely within the zd1211rw driver. |
popcornmix
pushed a commit
that referenced
this issue
Apr 24, 2014
commit 3a41c5d upstream. Following commits: 50e244c fb: rework locking to fix lock ordering on takeover e93a9a8 fb: Yet another band-aid for fixing lockdep mess 054430e fbcon: fix locking harder reworked locking to fix related lock ordering on takeover, and introduced console_lock into fbmem, but it seems that the new lock sequence(fb_info->lock ---> console_lock) is against with the one in console_callback(console_lock ---> fb_info->lock), and leads to a potential dead lock as following: [ 601.079000] ====================================================== [ 601.079000] [ INFO: possible circular locking dependency detected ] [ 601.079000] 3.11.0 #189 Not tainted [ 601.079000] ------------------------------------------------------- [ 601.079000] kworker/0:3/619 is trying to acquire lock: [ 601.079000] (&fb_info->lock){+.+.+.}, at: [<ffffffff81397566>] lock_fb_info+0x26/0x60 [ 601.079000] but task is already holding lock: [ 601.079000] (console_lock){+.+.+.}, at: [<ffffffff8141aae3>] console_callback+0x13/0x160 [ 601.079000] which lock already depends on the new lock. [ 601.079000] the existing dependency chain (in reverse order) is: [ 601.079000] -> #1 (console_lock){+.+.+.}: [ 601.079000] [<ffffffff810dc971>] lock_acquire+0xa1/0x140 [ 601.079000] [<ffffffff810c6267>] console_lock+0x77/0x80 [ 601.079000] [<ffffffff81399448>] register_framebuffer+0x1d8/0x320 [ 601.079000] [<ffffffff81cfb4c8>] efifb_probe+0x408/0x48f [ 601.079000] [<ffffffff8144a963>] platform_drv_probe+0x43/0x80 [ 601.079000] [<ffffffff8144853b>] driver_probe_device+0x8b/0x390 [ 601.079000] [<ffffffff814488eb>] __driver_attach+0xab/0xb0 [ 601.079000] [<ffffffff814463bd>] bus_for_each_dev+0x5d/0xa0 [ 601.079000] [<ffffffff81447e6e>] driver_attach+0x1e/0x20 [ 601.079000] [<ffffffff81447a07>] bus_add_driver+0x117/0x290 [ 601.079000] [<ffffffff81448fea>] driver_register+0x7a/0x170 [ 601.079000] [<ffffffff8144a10a>] __platform_driver_register+0x4a/0x50 [ 601.079000] [<ffffffff8144a12d>] platform_driver_probe+0x1d/0xb0 [ 601.079000] [<ffffffff81cfb0a1>] efifb_init+0x273/0x292 [ 601.079000] [<ffffffff81002132>] do_one_initcall+0x102/0x1c0 [ 601.079000] [<ffffffff81cb80a6>] kernel_init_freeable+0x15d/0x1ef [ 601.079000] [<ffffffff8166d2de>] kernel_init+0xe/0xf0 [ 601.079000] [<ffffffff816914ec>] ret_from_fork+0x7c/0xb0 [ 601.079000] -> #0 (&fb_info->lock){+.+.+.}: [ 601.079000] [<ffffffff810dc1d8>] __lock_acquire+0x1e18/0x1f10 [ 601.079000] [<ffffffff810dc971>] lock_acquire+0xa1/0x140 [ 601.079000] [<ffffffff816835ca>] mutex_lock_nested+0x7a/0x3b0 [ 601.079000] [<ffffffff81397566>] lock_fb_info+0x26/0x60 [ 601.079000] [<ffffffff813a4aeb>] fbcon_blank+0x29b/0x2e0 [ 601.079000] [<ffffffff81418658>] do_blank_screen+0x1d8/0x280 [ 601.079000] [<ffffffff8141ab34>] console_callback+0x64/0x160 [ 601.079000] [<ffffffff8108d855>] process_one_work+0x1f5/0x540 [ 601.079000] [<ffffffff8108e04c>] worker_thread+0x11c/0x370 [ 601.079000] [<ffffffff81095fbd>] kthread+0xed/0x100 [ 601.079000] [<ffffffff816914ec>] ret_from_fork+0x7c/0xb0 [ 601.079000] other info that might help us debug this: [ 601.079000] Possible unsafe locking scenario: [ 601.079000] CPU0 CPU1 [ 601.079000] ---- ---- [ 601.079000] lock(console_lock); [ 601.079000] lock(&fb_info->lock); [ 601.079000] lock(console_lock); [ 601.079000] lock(&fb_info->lock); [ 601.079000] *** DEADLOCK *** so we reorder the lock sequence the same as it in console_callback() to avoid this issue. And following Tomi's suggestion, fix these similar issues all in fb subsystem. Signed-off-by: Gu Zheng <[email protected]> Signed-off-by: Tomi Valkeinen <[email protected]> Signed-off-by: Jiri Slaby <[email protected]>
popcornmix
pushed a commit
that referenced
this issue
Sep 14, 2016
I got this:
divide error: 0000 [#1] PREEMPT SMP KASAN
CPU: 1 PID: 1327 Comm: a.out Not tainted 4.8.0-rc2+ #189
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.9.3-0-ge2fc41e-prebuilt.qemu-project.org 04/01/2014
task: ffff8801120a9580 task.stack: ffff8801120b0000
RIP: 0010:[<ffffffff82c8bd9a>] [<ffffffff82c8bd9a>] snd_hrtimer_callback+0x1da/0x3f0
RSP: 0018:ffff88011aa87da8 EFLAGS: 00010006
RAX: 0000000000004f76 RBX: ffff880112655e88 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff880112655ea0 RDI: 0000000000000001
RBP: ffff88011aa87e00 R08: ffff88013fff905c R09: ffff88013fff9048
R10: ffff88013fff9050 R11: 00000001050a7b8c R12: ffff880114778a00
R13: ffff880114778ab4 R14: ffff880114778b30 R15: 0000000000000000
FS: 00007f071647c700(0000) GS:ffff88011aa80000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000603001 CR3: 0000000112021000 CR4: 00000000000006e0
Stack:
0000000000000000 ffff880114778ab8 ffff880112655ea0 0000000000004f76
ffff880112655ec8 ffff880112655e80 ffff880112655e88 ffff88011aa98fc0
00000000b97ccf2b dffffc0000000000 ffff88011aa98fc0 ffff88011aa87ef0
Call Trace:
<IRQ>
[<ffffffff813abce7>] __hrtimer_run_queues+0x347/0xa00
[<ffffffff82c8bbc0>] ? snd_hrtimer_close+0x130/0x130
[<ffffffff813ab9a0>] ? retrigger_next_event+0x1b0/0x1b0
[<ffffffff813ae1a6>] ? hrtimer_interrupt+0x136/0x4b0
[<ffffffff813ae220>] hrtimer_interrupt+0x1b0/0x4b0
[<ffffffff8120f91e>] local_apic_timer_interrupt+0x6e/0xf0
[<ffffffff81227ad3>] ? kvm_guest_apic_eoi_write+0x13/0xc0
[<ffffffff83c35086>] smp_apic_timer_interrupt+0x76/0xa0
[<ffffffff83c3416c>] apic_timer_interrupt+0x8c/0xa0
<EOI>
[<ffffffff83c3239c>] ? _raw_spin_unlock_irqrestore+0x2c/0x60
[<ffffffff82c8185d>] snd_timer_start1+0xdd/0x670
[<ffffffff82c87015>] snd_timer_continue+0x45/0x80
[<ffffffff82c88100>] snd_timer_user_ioctl+0x1030/0x2830
[<ffffffff8159f3a0>] ? __follow_pte.isra.49+0x430/0x430
[<ffffffff82c870d0>] ? snd_timer_pause+0x80/0x80
[<ffffffff815a26fa>] ? do_wp_page+0x3aa/0x1c90
[<ffffffff815aa4f8>] ? handle_mm_fault+0xbc8/0x27f0
[<ffffffff815a9930>] ? __pmd_alloc+0x370/0x370
[<ffffffff82c870d0>] ? snd_timer_pause+0x80/0x80
[<ffffffff816b0733>] do_vfs_ioctl+0x193/0x1050
[<ffffffff816b05a0>] ? ioctl_preallocate+0x200/0x200
[<ffffffff81002f2f>] ? syscall_trace_enter+0x3cf/0xdb0
[<ffffffff815045ba>] ? __context_tracking_exit.part.4+0x9a/0x1e0
[<ffffffff81002b60>] ? exit_to_usermode_loop+0x190/0x190
[<ffffffff82001a97>] ? check_preemption_disabled+0x37/0x1e0
[<ffffffff81d93889>] ? security_file_ioctl+0x89/0xb0
[<ffffffff816b167f>] SyS_ioctl+0x8f/0xc0
[<ffffffff816b15f0>] ? do_vfs_ioctl+0x1050/0x1050
[<ffffffff81005524>] do_syscall_64+0x1c4/0x4e0
[<ffffffff83c32b2a>] entry_SYSCALL64_slow_path+0x25/0x25
Code: e8 fc 42 7b fe 8b 0d 06 8a 50 03 49 0f af cf 48 85 c9 0f 88 7c 01 00 00 48 89 4d a8 e8 e0 42 7b fe 48 8b 45 c0 48 8b 4d a8 48 99 <48> f7 f9 49 01 c7 e8 cb 42 7b fe 48 8b 55 d0 48 b8 00 00 00 00
RIP [<ffffffff82c8bd9a>] snd_hrtimer_callback+0x1da/0x3f0
RSP <ffff88011aa87da8>
---[ end trace 6aa380f756a21074 ]---
The problem happens when you call ioctl(SNDRV_TIMER_IOCTL_CONTINUE) on a
completely new/unused timer -- it will have ->sticks == 0, which causes a
divide by 0 in snd_hrtimer_callback().
Signed-off-by: Vegard Nossum <[email protected]>
Cc: <[email protected]>
Signed-off-by: Takashi Iwai <[email protected]>
popcornmix
pushed a commit
that referenced
this issue
Sep 15, 2016
commit 6b760bb upstream. I got this: divide error: 0000 [#1] PREEMPT SMP KASAN CPU: 1 PID: 1327 Comm: a.out Not tainted 4.8.0-rc2+ #189 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.9.3-0-ge2fc41e-prebuilt.qemu-project.org 04/01/2014 task: ffff8801120a9580 task.stack: ffff8801120b0000 RIP: 0010:[<ffffffff82c8bd9a>] [<ffffffff82c8bd9a>] snd_hrtimer_callback+0x1da/0x3f0 RSP: 0018:ffff88011aa87da8 EFLAGS: 00010006 RAX: 0000000000004f76 RBX: ffff880112655e88 RCX: 0000000000000000 RDX: 0000000000000000 RSI: ffff880112655ea0 RDI: 0000000000000001 RBP: ffff88011aa87e00 R08: ffff88013fff905c R09: ffff88013fff9048 R10: ffff88013fff9050 R11: 00000001050a7b8c R12: ffff880114778a00 R13: ffff880114778ab4 R14: ffff880114778b30 R15: 0000000000000000 FS: 00007f071647c700(0000) GS:ffff88011aa80000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000000000603001 CR3: 0000000112021000 CR4: 00000000000006e0 Stack: 0000000000000000 ffff880114778ab8 ffff880112655ea0 0000000000004f76 ffff880112655ec8 ffff880112655e80 ffff880112655e88 ffff88011aa98fc0 00000000b97ccf2b dffffc0000000000 ffff88011aa98fc0 ffff88011aa87ef0 Call Trace: <IRQ> [<ffffffff813abce7>] __hrtimer_run_queues+0x347/0xa00 [<ffffffff82c8bbc0>] ? snd_hrtimer_close+0x130/0x130 [<ffffffff813ab9a0>] ? retrigger_next_event+0x1b0/0x1b0 [<ffffffff813ae1a6>] ? hrtimer_interrupt+0x136/0x4b0 [<ffffffff813ae220>] hrtimer_interrupt+0x1b0/0x4b0 [<ffffffff8120f91e>] local_apic_timer_interrupt+0x6e/0xf0 [<ffffffff81227ad3>] ? kvm_guest_apic_eoi_write+0x13/0xc0 [<ffffffff83c35086>] smp_apic_timer_interrupt+0x76/0xa0 [<ffffffff83c3416c>] apic_timer_interrupt+0x8c/0xa0 <EOI> [<ffffffff83c3239c>] ? _raw_spin_unlock_irqrestore+0x2c/0x60 [<ffffffff82c8185d>] snd_timer_start1+0xdd/0x670 [<ffffffff82c87015>] snd_timer_continue+0x45/0x80 [<ffffffff82c88100>] snd_timer_user_ioctl+0x1030/0x2830 [<ffffffff8159f3a0>] ? __follow_pte.isra.49+0x430/0x430 [<ffffffff82c870d0>] ? snd_timer_pause+0x80/0x80 [<ffffffff815a26fa>] ? do_wp_page+0x3aa/0x1c90 [<ffffffff815aa4f8>] ? handle_mm_fault+0xbc8/0x27f0 [<ffffffff815a9930>] ? __pmd_alloc+0x370/0x370 [<ffffffff82c870d0>] ? snd_timer_pause+0x80/0x80 [<ffffffff816b0733>] do_vfs_ioctl+0x193/0x1050 [<ffffffff816b05a0>] ? ioctl_preallocate+0x200/0x200 [<ffffffff81002f2f>] ? syscall_trace_enter+0x3cf/0xdb0 [<ffffffff815045ba>] ? __context_tracking_exit.part.4+0x9a/0x1e0 [<ffffffff81002b60>] ? exit_to_usermode_loop+0x190/0x190 [<ffffffff82001a97>] ? check_preemption_disabled+0x37/0x1e0 [<ffffffff81d93889>] ? security_file_ioctl+0x89/0xb0 [<ffffffff816b167f>] SyS_ioctl+0x8f/0xc0 [<ffffffff816b15f0>] ? do_vfs_ioctl+0x1050/0x1050 [<ffffffff81005524>] do_syscall_64+0x1c4/0x4e0 [<ffffffff83c32b2a>] entry_SYSCALL64_slow_path+0x25/0x25 Code: e8 fc 42 7b fe 8b 0d 06 8a 50 03 49 0f af cf 48 85 c9 0f 88 7c 01 00 00 48 89 4d a8 e8 e0 42 7b fe 48 8b 45 c0 48 8b 4d a8 48 99 <48> f7 f9 49 01 c7 e8 cb 42 7b fe 48 8b 55 d0 48 b8 00 00 00 00 RIP [<ffffffff82c8bd9a>] snd_hrtimer_callback+0x1da/0x3f0 RSP <ffff88011aa87da8> ---[ end trace 6aa380f756a21074 ]--- The problem happens when you call ioctl(SNDRV_TIMER_IOCTL_CONTINUE) on a completely new/unused timer -- it will have ->sticks == 0, which causes a divide by 0 in snd_hrtimer_callback(). Signed-off-by: Vegard Nossum <[email protected]> Signed-off-by: Takashi Iwai <[email protected]> Signed-off-by: Greg Kroah-Hartman <[email protected]>
popcornmix
pushed a commit
that referenced
this issue
Sep 15, 2016
commit 6b760bb upstream. I got this: divide error: 0000 [#1] PREEMPT SMP KASAN CPU: 1 PID: 1327 Comm: a.out Not tainted 4.8.0-rc2+ #189 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.9.3-0-ge2fc41e-prebuilt.qemu-project.org 04/01/2014 task: ffff8801120a9580 task.stack: ffff8801120b0000 RIP: 0010:[<ffffffff82c8bd9a>] [<ffffffff82c8bd9a>] snd_hrtimer_callback+0x1da/0x3f0 RSP: 0018:ffff88011aa87da8 EFLAGS: 00010006 RAX: 0000000000004f76 RBX: ffff880112655e88 RCX: 0000000000000000 RDX: 0000000000000000 RSI: ffff880112655ea0 RDI: 0000000000000001 RBP: ffff88011aa87e00 R08: ffff88013fff905c R09: ffff88013fff9048 R10: ffff88013fff9050 R11: 00000001050a7b8c R12: ffff880114778a00 R13: ffff880114778ab4 R14: ffff880114778b30 R15: 0000000000000000 FS: 00007f071647c700(0000) GS:ffff88011aa80000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000000000603001 CR3: 0000000112021000 CR4: 00000000000006e0 Stack: 0000000000000000 ffff880114778ab8 ffff880112655ea0 0000000000004f76 ffff880112655ec8 ffff880112655e80 ffff880112655e88 ffff88011aa98fc0 00000000b97ccf2b dffffc0000000000 ffff88011aa98fc0 ffff88011aa87ef0 Call Trace: <IRQ> [<ffffffff813abce7>] __hrtimer_run_queues+0x347/0xa00 [<ffffffff82c8bbc0>] ? snd_hrtimer_close+0x130/0x130 [<ffffffff813ab9a0>] ? retrigger_next_event+0x1b0/0x1b0 [<ffffffff813ae1a6>] ? hrtimer_interrupt+0x136/0x4b0 [<ffffffff813ae220>] hrtimer_interrupt+0x1b0/0x4b0 [<ffffffff8120f91e>] local_apic_timer_interrupt+0x6e/0xf0 [<ffffffff81227ad3>] ? kvm_guest_apic_eoi_write+0x13/0xc0 [<ffffffff83c35086>] smp_apic_timer_interrupt+0x76/0xa0 [<ffffffff83c3416c>] apic_timer_interrupt+0x8c/0xa0 <EOI> [<ffffffff83c3239c>] ? _raw_spin_unlock_irqrestore+0x2c/0x60 [<ffffffff82c8185d>] snd_timer_start1+0xdd/0x670 [<ffffffff82c87015>] snd_timer_continue+0x45/0x80 [<ffffffff82c88100>] snd_timer_user_ioctl+0x1030/0x2830 [<ffffffff8159f3a0>] ? __follow_pte.isra.49+0x430/0x430 [<ffffffff82c870d0>] ? snd_timer_pause+0x80/0x80 [<ffffffff815a26fa>] ? do_wp_page+0x3aa/0x1c90 [<ffffffff815aa4f8>] ? handle_mm_fault+0xbc8/0x27f0 [<ffffffff815a9930>] ? __pmd_alloc+0x370/0x370 [<ffffffff82c870d0>] ? snd_timer_pause+0x80/0x80 [<ffffffff816b0733>] do_vfs_ioctl+0x193/0x1050 [<ffffffff816b05a0>] ? ioctl_preallocate+0x200/0x200 [<ffffffff81002f2f>] ? syscall_trace_enter+0x3cf/0xdb0 [<ffffffff815045ba>] ? __context_tracking_exit.part.4+0x9a/0x1e0 [<ffffffff81002b60>] ? exit_to_usermode_loop+0x190/0x190 [<ffffffff82001a97>] ? check_preemption_disabled+0x37/0x1e0 [<ffffffff81d93889>] ? security_file_ioctl+0x89/0xb0 [<ffffffff816b167f>] SyS_ioctl+0x8f/0xc0 [<ffffffff816b15f0>] ? do_vfs_ioctl+0x1050/0x1050 [<ffffffff81005524>] do_syscall_64+0x1c4/0x4e0 [<ffffffff83c32b2a>] entry_SYSCALL64_slow_path+0x25/0x25 Code: e8 fc 42 7b fe 8b 0d 06 8a 50 03 49 0f af cf 48 85 c9 0f 88 7c 01 00 00 48 89 4d a8 e8 e0 42 7b fe 48 8b 45 c0 48 8b 4d a8 48 99 <48> f7 f9 49 01 c7 e8 cb 42 7b fe 48 8b 55 d0 48 b8 00 00 00 00 RIP [<ffffffff82c8bd9a>] snd_hrtimer_callback+0x1da/0x3f0 RSP <ffff88011aa87da8> ---[ end trace 6aa380f756a21074 ]--- The problem happens when you call ioctl(SNDRV_TIMER_IOCTL_CONTINUE) on a completely new/unused timer -- it will have ->sticks == 0, which causes a divide by 0 in snd_hrtimer_callback(). Signed-off-by: Vegard Nossum <[email protected]> Signed-off-by: Takashi Iwai <[email protected]> Signed-off-by: Greg Kroah-Hartman <[email protected]>
anholt
pushed a commit
to anholt/linux
that referenced
this issue
Apr 23, 2018
syzbot reported a use-after-free of shm_file_data(file)->file->f_op in shm_get_unmapped_area(), called via sys_remap_file_pages(). Unfortunately it couldn't generate a reproducer, but I found a bug which I think caused it. When remap_file_pages() is passed a full System V shared memory segment, the memory is first unmapped, then a new map is created using the ->vm_file. Between these steps, the shm ID can be removed and reused for a new shm segment. But, shm_mmap() only checks whether the ID is currently valid before calling the underlying file's ->mmap(); it doesn't check whether it was reused. Thus it can use the wrong underlying file, one that was already freed. Fix this by making the "outer" shm file (the one that gets put in ->vm_file) hold a reference to the real shm file, and by making __shm_open() require that the file associated with the shm ID matches the one associated with the "outer" file. Taking the reference to the real shm file is needed to fully solve the problem, since otherwise sfd->file could point to a freed file, which then could be reallocated for the reused shm ID, causing the wrong shm segment to be mapped (and without the required permission checks). Commit 1ac0b6d ("ipc/shm: handle removed segments gracefully in shm_mmap()") almost fixed this bug, but it didn't go far enough because it didn't consider the case where the shm ID is reused. The following program usually reproduces this bug: #include <stdlib.h> #include <sys/shm.h> #include <sys/syscall.h> #include <unistd.h> int main() { int is_parent = (fork() != 0); srand(getpid()); for (;;) { int id = shmget(0xF00F, 4096, IPC_CREAT|0700); if (is_parent) { void *addr = shmat(id, NULL, 0); usleep(rand() % 50); while (!syscall(__NR_remap_file_pages, addr, 4096, 0, 0, 0)); } else { usleep(rand() % 50); shmctl(id, IPC_RMID, NULL); } } } It causes the following NULL pointer dereference due to a 'struct file' being used while it's being freed. (I couldn't actually get a KASAN use-after-free splat like in the syzbot report. But I think it's possible with this bug; it would just take a more extraordinary race...) BUG: unable to handle kernel NULL pointer dereference at 0000000000000058 PGD 0 P4D 0 Oops: 0000 [#1] SMP NOPTI CPU: 9 PID: 258 Comm: syz_ipc Not tainted 4.16.0-05140-gf8cf2f16a7c95 raspberrypi#189 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.11.0-20171110_100015-anatol 04/01/2014 RIP: 0010:d_inode include/linux/dcache.h:519 [inline] RIP: 0010:touch_atime+0x25/0xd0 fs/inode.c:1724 [...] Call Trace: file_accessed include/linux/fs.h:2063 [inline] shmem_mmap+0x25/0x40 mm/shmem.c:2149 call_mmap include/linux/fs.h:1789 [inline] shm_mmap+0x34/0x80 ipc/shm.c:465 call_mmap include/linux/fs.h:1789 [inline] mmap_region+0x309/0x5b0 mm/mmap.c:1712 do_mmap+0x294/0x4a0 mm/mmap.c:1483 do_mmap_pgoff include/linux/mm.h:2235 [inline] SYSC_remap_file_pages mm/mmap.c:2853 [inline] SyS_remap_file_pages+0x232/0x310 mm/mmap.c:2769 do_syscall_64+0x64/0x1a0 arch/x86/entry/common.c:287 entry_SYSCALL_64_after_hwframe+0x42/0xb7 [[email protected]: add comment] Link: http://lkml.kernel.org/r/[email protected] Link: http://lkml.kernel.org/r/[email protected] Reported-by: syzbot+d11f321e7f1923157eac80aa990b446596f46439@syzkaller.appspotmail.com Fixes: c8d78c1 ("mm: replace remap_file_pages() syscall with emulation") Signed-off-by: Eric Biggers <[email protected]> Acked-by: Kirill A. Shutemov <[email protected]> Acked-by: Davidlohr Bueso <[email protected]> Cc: Manfred Spraul <[email protected]> Cc: "Eric W . Biederman" <[email protected]> Cc: <[email protected]> Signed-off-by: Andrew Morton <[email protected]> Signed-off-by: Linus Torvalds <[email protected]>
nathanchance
pushed a commit
to nathanchance/pi-kernel
that referenced
this issue
Apr 24, 2018
commit 3f05317 upstream. syzbot reported a use-after-free of shm_file_data(file)->file->f_op in shm_get_unmapped_area(), called via sys_remap_file_pages(). Unfortunately it couldn't generate a reproducer, but I found a bug which I think caused it. When remap_file_pages() is passed a full System V shared memory segment, the memory is first unmapped, then a new map is created using the ->vm_file. Between these steps, the shm ID can be removed and reused for a new shm segment. But, shm_mmap() only checks whether the ID is currently valid before calling the underlying file's ->mmap(); it doesn't check whether it was reused. Thus it can use the wrong underlying file, one that was already freed. Fix this by making the "outer" shm file (the one that gets put in ->vm_file) hold a reference to the real shm file, and by making __shm_open() require that the file associated with the shm ID matches the one associated with the "outer" file. Taking the reference to the real shm file is needed to fully solve the problem, since otherwise sfd->file could point to a freed file, which then could be reallocated for the reused shm ID, causing the wrong shm segment to be mapped (and without the required permission checks). Commit 1ac0b6d ("ipc/shm: handle removed segments gracefully in shm_mmap()") almost fixed this bug, but it didn't go far enough because it didn't consider the case where the shm ID is reused. The following program usually reproduces this bug: #include <stdlib.h> #include <sys/shm.h> #include <sys/syscall.h> #include <unistd.h> int main() { int is_parent = (fork() != 0); srand(getpid()); for (;;) { int id = shmget(0xF00F, 4096, IPC_CREAT|0700); if (is_parent) { void *addr = shmat(id, NULL, 0); usleep(rand() % 50); while (!syscall(__NR_remap_file_pages, addr, 4096, 0, 0, 0)); } else { usleep(rand() % 50); shmctl(id, IPC_RMID, NULL); } } } It causes the following NULL pointer dereference due to a 'struct file' being used while it's being freed. (I couldn't actually get a KASAN use-after-free splat like in the syzbot report. But I think it's possible with this bug; it would just take a more extraordinary race...) BUG: unable to handle kernel NULL pointer dereference at 0000000000000058 PGD 0 P4D 0 Oops: 0000 [raspberrypi#1] SMP NOPTI CPU: 9 PID: 258 Comm: syz_ipc Not tainted 4.16.0-05140-gf8cf2f16a7c95 raspberrypi#189 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.11.0-20171110_100015-anatol 04/01/2014 RIP: 0010:d_inode include/linux/dcache.h:519 [inline] RIP: 0010:touch_atime+0x25/0xd0 fs/inode.c:1724 [...] Call Trace: file_accessed include/linux/fs.h:2063 [inline] shmem_mmap+0x25/0x40 mm/shmem.c:2149 call_mmap include/linux/fs.h:1789 [inline] shm_mmap+0x34/0x80 ipc/shm.c:465 call_mmap include/linux/fs.h:1789 [inline] mmap_region+0x309/0x5b0 mm/mmap.c:1712 do_mmap+0x294/0x4a0 mm/mmap.c:1483 do_mmap_pgoff include/linux/mm.h:2235 [inline] SYSC_remap_file_pages mm/mmap.c:2853 [inline] SyS_remap_file_pages+0x232/0x310 mm/mmap.c:2769 do_syscall_64+0x64/0x1a0 arch/x86/entry/common.c:287 entry_SYSCALL_64_after_hwframe+0x42/0xb7 [[email protected]: add comment] Link: http://lkml.kernel.org/r/[email protected] Link: http://lkml.kernel.org/r/[email protected] Reported-by: syzbot+d11f321e7f1923157eac80aa990b446596f46439@syzkaller.appspotmail.com Fixes: c8d78c1 ("mm: replace remap_file_pages() syscall with emulation") Signed-off-by: Eric Biggers <[email protected]> Acked-by: Kirill A. Shutemov <[email protected]> Acked-by: Davidlohr Bueso <[email protected]> Cc: Manfred Spraul <[email protected]> Cc: "Eric W . Biederman" <[email protected]> Cc: <[email protected]> Signed-off-by: Andrew Morton <[email protected]> Signed-off-by: Linus Torvalds <[email protected]> Signed-off-by: Greg Kroah-Hartman <[email protected]>
popcornmix
pushed a commit
that referenced
this issue
Apr 26, 2018
commit 3f05317 upstream. syzbot reported a use-after-free of shm_file_data(file)->file->f_op in shm_get_unmapped_area(), called via sys_remap_file_pages(). Unfortunately it couldn't generate a reproducer, but I found a bug which I think caused it. When remap_file_pages() is passed a full System V shared memory segment, the memory is first unmapped, then a new map is created using the ->vm_file. Between these steps, the shm ID can be removed and reused for a new shm segment. But, shm_mmap() only checks whether the ID is currently valid before calling the underlying file's ->mmap(); it doesn't check whether it was reused. Thus it can use the wrong underlying file, one that was already freed. Fix this by making the "outer" shm file (the one that gets put in ->vm_file) hold a reference to the real shm file, and by making __shm_open() require that the file associated with the shm ID matches the one associated with the "outer" file. Taking the reference to the real shm file is needed to fully solve the problem, since otherwise sfd->file could point to a freed file, which then could be reallocated for the reused shm ID, causing the wrong shm segment to be mapped (and without the required permission checks). Commit 1ac0b6d ("ipc/shm: handle removed segments gracefully in shm_mmap()") almost fixed this bug, but it didn't go far enough because it didn't consider the case where the shm ID is reused. The following program usually reproduces this bug: #include <stdlib.h> #include <sys/shm.h> #include <sys/syscall.h> #include <unistd.h> int main() { int is_parent = (fork() != 0); srand(getpid()); for (;;) { int id = shmget(0xF00F, 4096, IPC_CREAT|0700); if (is_parent) { void *addr = shmat(id, NULL, 0); usleep(rand() % 50); while (!syscall(__NR_remap_file_pages, addr, 4096, 0, 0, 0)); } else { usleep(rand() % 50); shmctl(id, IPC_RMID, NULL); } } } It causes the following NULL pointer dereference due to a 'struct file' being used while it's being freed. (I couldn't actually get a KASAN use-after-free splat like in the syzbot report. But I think it's possible with this bug; it would just take a more extraordinary race...) BUG: unable to handle kernel NULL pointer dereference at 0000000000000058 PGD 0 P4D 0 Oops: 0000 [#1] SMP NOPTI CPU: 9 PID: 258 Comm: syz_ipc Not tainted 4.16.0-05140-gf8cf2f16a7c95 #189 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.11.0-20171110_100015-anatol 04/01/2014 RIP: 0010:d_inode include/linux/dcache.h:519 [inline] RIP: 0010:touch_atime+0x25/0xd0 fs/inode.c:1724 [...] Call Trace: file_accessed include/linux/fs.h:2063 [inline] shmem_mmap+0x25/0x40 mm/shmem.c:2149 call_mmap include/linux/fs.h:1789 [inline] shm_mmap+0x34/0x80 ipc/shm.c:465 call_mmap include/linux/fs.h:1789 [inline] mmap_region+0x309/0x5b0 mm/mmap.c:1712 do_mmap+0x294/0x4a0 mm/mmap.c:1483 do_mmap_pgoff include/linux/mm.h:2235 [inline] SYSC_remap_file_pages mm/mmap.c:2853 [inline] SyS_remap_file_pages+0x232/0x310 mm/mmap.c:2769 do_syscall_64+0x64/0x1a0 arch/x86/entry/common.c:287 entry_SYSCALL_64_after_hwframe+0x42/0xb7 [[email protected]: add comment] Link: http://lkml.kernel.org/r/[email protected] Link: http://lkml.kernel.org/r/[email protected] Reported-by: syzbot+d11f321e7f1923157eac80aa990b446596f46439@syzkaller.appspotmail.com Fixes: c8d78c1 ("mm: replace remap_file_pages() syscall with emulation") Signed-off-by: Eric Biggers <[email protected]> Acked-by: Kirill A. Shutemov <[email protected]> Acked-by: Davidlohr Bueso <[email protected]> Cc: Manfred Spraul <[email protected]> Cc: "Eric W . Biederman" <[email protected]> Cc: <[email protected]> Signed-off-by: Andrew Morton <[email protected]> Signed-off-by: Linus Torvalds <[email protected]> Signed-off-by: Greg Kroah-Hartman <[email protected]>
popcornmix
pushed a commit
that referenced
this issue
Jun 4, 2019
[ Upstream commit a4b7013 ] BUG: KASAN: slab-out-of-bounds in rxe_mem_init_user+0x6c1/0x740 [rdma_rxe] Read of size 8 at addr ffff88805c01a608 by task ib_send_bw/573 CPU: 24 PID: 573 Comm: ib_send_bw Not tainted 5.0.0-rc5+ #189 Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.11.0-0-g63451fca13-prebuilt.qemu-project.org 04/01/2014 Call Trace: rxe_mem_init_user+0x6c1/0x740 [rdma_rxe] rxe_reg_user_mr+0x9b/0x110 [rdma_rxe] ib_uverbs_reg_mr+0x428/0x9c0 [ib_uverbs] ib_uverbs_handler_UVERBS_METHOD_INVOKE_WRITE+0x2b0/0x410 [ib_uverbs] ib_uverbs_run_method+0x79c/0x1da0 [ib_uverbs] rxe_mem_init_user+0x6c1/0x740 [rdma_rxe] rxe_reg_user_mr+0x9b/0x110 [rdma_rxe] ib_uverbs_reg_mr+0x428/0x9c0 [ib_uverbs] ib_uverbs_handler_UVERBS_METHOD_INVOKE_WRITE+0x2b0/0x410 [ib_uverbs] ib_uverbs_run_method+0x79c/0x1da0 [ib_uverbs] ib_uverbs_cmd_verbs+0x5f2/0xf20 [ib_uverbs] ib_uverbs_ioctl+0x202/0x310 [ib_uverbs] do_vfs_ioctl+0x193/0x1440 ksys_ioctl+0x3a/0x70 __x64_sys_ioctl+0x6f/0xb0 do_syscall_64+0x13f/0x570 entry_SYSCALL_64_after_hwframe+0x49/0xbe Allocated by task 573: __kasan_kmalloc.constprop.5+0xc1/0xd0 __kmalloc+0x161/0x310 rxe_mem_alloc+0x52/0x470 [rdma_rxe] rxe_mem_init_user+0x113/0x740 [rdma_rxe] rxe_reg_user_mr+0x9b/0x110 [rdma_rxe] ib_uverbs_reg_mr+0x428/0x9c0 [ib_uverbs] ib_uverbs_handler_UVERBS_METHOD_INVOKE_WRITE+0x2b0/0x410 [ib_uverbs] ib_uverbs_run_method+0x79c/0x1da0 [ib_uverbs] ib_uverbs_cmd_verbs+0x5f2/0xf20 [ib_uverbs] ib_uverbs_ioctl+0x202/0x310 [ib_uverbs] do_vfs_ioctl+0x193/0x1440 ksys_ioctl+0x3a/0x70 __x64_sys_ioctl+0x6f/0xb0 do_syscall_64+0x13f/0x570 entry_SYSCALL_64_after_hwframe+0x49/0xbe Freed by task 0: __kasan_slab_free+0x12e/0x180 kfree+0x10a/0x2c0 rcu_process_callbacks+0xa77/0x1260 __do_softirq+0x2ad/0xacb Test scenario: ib_send_bw -x 1 -d rxe0 -a & ib_send_bw -x 1 -d rxe0 -a localhost Fixes: 8700e3e ("Soft RoCE driver") Reported-by: Parav Pandit <[email protected]> Reviewed-by: Zhu Yanjun <[email protected]> Tested-by: Zhu Yanjun <[email protected]> Signed-off-by: Leon Romanovsky <[email protected]> Signed-off-by: Jason Gunthorpe <[email protected]> Signed-off-by: Sasha Levin <[email protected]>
herrnst
pushed a commit
to herrnst/linux-raspberrypi
that referenced
this issue
Jan 2, 2023
…add() [ Upstream commit 78316e9 ] In mpt3sas_transport_port_add(), if sas_rphy_add() returns error, sas_rphy_free() needs be called to free the resource allocated in sas_end_device_alloc(). Otherwise a kernel crash will happen: Unable to handle kernel NULL pointer dereference at virtual address 0000000000000108 CPU: 45 PID: 37020 Comm: bash Kdump: loaded Tainted: G W 6.1.0-rc1+ raspberrypi#189 pstate: 60000005 (nZCv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--) pc : device_del+0x54/0x3d0 lr : device_del+0x37c/0x3d0 Call trace: device_del+0x54/0x3d0 attribute_container_class_device_del+0x28/0x38 transport_remove_classdev+0x6c/0x80 attribute_container_device_trigger+0x108/0x110 transport_remove_device+0x28/0x38 sas_rphy_remove+0x50/0x78 [scsi_transport_sas] sas_port_delete+0x30/0x148 [scsi_transport_sas] do_sas_phy_delete+0x78/0x80 [scsi_transport_sas] device_for_each_child+0x68/0xb0 sas_remove_children+0x30/0x50 [scsi_transport_sas] sas_rphy_remove+0x38/0x78 [scsi_transport_sas] sas_port_delete+0x30/0x148 [scsi_transport_sas] do_sas_phy_delete+0x78/0x80 [scsi_transport_sas] device_for_each_child+0x68/0xb0 sas_remove_children+0x30/0x50 [scsi_transport_sas] sas_remove_host+0x20/0x38 [scsi_transport_sas] scsih_remove+0xd8/0x420 [mpt3sas] Because transport_add_device() is not called when sas_rphy_add() fails, the device is not added. When sas_rphy_remove() is subsequently called to remove the device in the remove() path, a NULL pointer dereference happens. Fixes: f92363d ("[SCSI] mpt3sas: add new driver supporting 12GB SAS") Signed-off-by: Yang Yingliang <[email protected]> Link: https://lore.kernel.org/r/[email protected] Signed-off-by: Martin K. Petersen <[email protected]> Signed-off-by: Sasha Levin <[email protected]>
popcornmix
pushed a commit
that referenced
this issue
Jan 3, 2023
…add() [ Upstream commit 78316e9 ] In mpt3sas_transport_port_add(), if sas_rphy_add() returns error, sas_rphy_free() needs be called to free the resource allocated in sas_end_device_alloc(). Otherwise a kernel crash will happen: Unable to handle kernel NULL pointer dereference at virtual address 0000000000000108 CPU: 45 PID: 37020 Comm: bash Kdump: loaded Tainted: G W 6.1.0-rc1+ #189 pstate: 60000005 (nZCv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--) pc : device_del+0x54/0x3d0 lr : device_del+0x37c/0x3d0 Call trace: device_del+0x54/0x3d0 attribute_container_class_device_del+0x28/0x38 transport_remove_classdev+0x6c/0x80 attribute_container_device_trigger+0x108/0x110 transport_remove_device+0x28/0x38 sas_rphy_remove+0x50/0x78 [scsi_transport_sas] sas_port_delete+0x30/0x148 [scsi_transport_sas] do_sas_phy_delete+0x78/0x80 [scsi_transport_sas] device_for_each_child+0x68/0xb0 sas_remove_children+0x30/0x50 [scsi_transport_sas] sas_rphy_remove+0x38/0x78 [scsi_transport_sas] sas_port_delete+0x30/0x148 [scsi_transport_sas] do_sas_phy_delete+0x78/0x80 [scsi_transport_sas] device_for_each_child+0x68/0xb0 sas_remove_children+0x30/0x50 [scsi_transport_sas] sas_remove_host+0x20/0x38 [scsi_transport_sas] scsih_remove+0xd8/0x420 [mpt3sas] Because transport_add_device() is not called when sas_rphy_add() fails, the device is not added. When sas_rphy_remove() is subsequently called to remove the device in the remove() path, a NULL pointer dereference happens. Fixes: f92363d ("[SCSI] mpt3sas: add new driver supporting 12GB SAS") Signed-off-by: Yang Yingliang <[email protected]> Link: https://lore.kernel.org/r/[email protected] Signed-off-by: Martin K. Petersen <[email protected]> Signed-off-by: Sasha Levin <[email protected]>
popcornmix
pushed a commit
that referenced
this issue
Jan 13, 2023
…add() [ Upstream commit 78316e9 ] In mpt3sas_transport_port_add(), if sas_rphy_add() returns error, sas_rphy_free() needs be called to free the resource allocated in sas_end_device_alloc(). Otherwise a kernel crash will happen: Unable to handle kernel NULL pointer dereference at virtual address 0000000000000108 CPU: 45 PID: 37020 Comm: bash Kdump: loaded Tainted: G W 6.1.0-rc1+ #189 pstate: 60000005 (nZCv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--) pc : device_del+0x54/0x3d0 lr : device_del+0x37c/0x3d0 Call trace: device_del+0x54/0x3d0 attribute_container_class_device_del+0x28/0x38 transport_remove_classdev+0x6c/0x80 attribute_container_device_trigger+0x108/0x110 transport_remove_device+0x28/0x38 sas_rphy_remove+0x50/0x78 [scsi_transport_sas] sas_port_delete+0x30/0x148 [scsi_transport_sas] do_sas_phy_delete+0x78/0x80 [scsi_transport_sas] device_for_each_child+0x68/0xb0 sas_remove_children+0x30/0x50 [scsi_transport_sas] sas_rphy_remove+0x38/0x78 [scsi_transport_sas] sas_port_delete+0x30/0x148 [scsi_transport_sas] do_sas_phy_delete+0x78/0x80 [scsi_transport_sas] device_for_each_child+0x68/0xb0 sas_remove_children+0x30/0x50 [scsi_transport_sas] sas_remove_host+0x20/0x38 [scsi_transport_sas] scsih_remove+0xd8/0x420 [mpt3sas] Because transport_add_device() is not called when sas_rphy_add() fails, the device is not added. When sas_rphy_remove() is subsequently called to remove the device in the remove() path, a NULL pointer dereference happens. Fixes: f92363d ("[SCSI] mpt3sas: add new driver supporting 12GB SAS") Signed-off-by: Yang Yingliang <[email protected]> Link: https://lore.kernel.org/r/[email protected] Signed-off-by: Martin K. Petersen <[email protected]> Signed-off-by: Sasha Levin <[email protected]>
popcornmix
pushed a commit
that referenced
this issue
Apr 10, 2025
[ Upstream commit 5ed3b0c ] When cur_qp isn't NULL, in order to avoid fetching the QP from the radix tree again we check if the next cqe QP is identical to the one we already have. The bug however is that we are checking if the QP is identical by checking the QP number inside the CQE against the QP number inside the mlx5_ib_qp, but that's wrong since the QP number from the CQE is from FW so it should be matched against mlx5_core_qp which is our FW QP number. Otherwise we could use the wrong QP when handling a CQE which could cause the kernel trace below. This issue is mainly noticeable over QPs 0 & 1, since for now they are the only QPs in our driver whereas the QP number inside mlx5_ib_qp doesn't match the QP number inside mlx5_core_qp. BUG: kernel NULL pointer dereference, address: 0000000000000012 #PF: supervisor read access in kernel mode #PF: error_code(0x0000) - not-present page PGD 0 P4D 0 Oops: Oops: 0000 [#1] SMP CPU: 0 UID: 0 PID: 7927 Comm: kworker/u62:1 Not tainted 6.14.0-rc3+ #189 Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.16.3-0-ga6ed6b701f0a-prebuilt.qemu.org 04/01/2014 Workqueue: ib-comp-unb-wq ib_cq_poll_work [ib_core] RIP: 0010:mlx5_ib_poll_cq+0x4c7/0xd90 [mlx5_ib] Code: 03 00 00 8d 58 ff 21 cb 66 39 d3 74 39 48 c7 c7 3c 89 6e a0 0f b7 db e8 b7 d2 b3 e0 49 8b 86 60 03 00 00 48 c7 c7 4a 89 6e a0 <0f> b7 5c 98 02 e8 9f d2 b3 e0 41 0f b7 86 78 03 00 00 83 e8 01 21 RSP: 0018:ffff88810511bd60 EFLAGS: 00010046 RAX: 0000000000000010 RBX: 0000000000000000 RCX: 0000000000000000 RDX: 0000000000000000 RSI: ffff88885fa1b3c0 RDI: ffffffffa06e894a RBP: 00000000000000b0 R08: 0000000000000000 R09: ffff88810511bc10 R10: 0000000000000001 R11: 0000000000000001 R12: ffff88810d593000 R13: ffff88810e579108 R14: ffff888105146000 R15: 00000000000000b0 FS: 0000000000000000(0000) GS:ffff88885fa00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000000000000012 CR3: 00000001077e6001 CR4: 0000000000370eb0 Call Trace: <TASK> ? __die+0x20/0x60 ? page_fault_oops+0x150/0x3e0 ? exc_page_fault+0x74/0x130 ? asm_exc_page_fault+0x22/0x30 ? mlx5_ib_poll_cq+0x4c7/0xd90 [mlx5_ib] __ib_process_cq+0x5a/0x150 [ib_core] ib_cq_poll_work+0x31/0x90 [ib_core] process_one_work+0x169/0x320 worker_thread+0x288/0x3a0 ? work_busy+0xb0/0xb0 kthread+0xd7/0x1f0 ? kthreads_online_cpu+0x130/0x130 ? kthreads_online_cpu+0x130/0x130 ret_from_fork+0x2d/0x50 ? kthreads_online_cpu+0x130/0x130 ret_from_fork_asm+0x11/0x20 </TASK> Fixes: e126ba9 ("mlx5: Add driver for Mellanox Connect-IB adapters") Signed-off-by: Patrisious Haddad <[email protected]> Reviewed-by: Edward Srouji <[email protected]> Link: https://patch.msgid.link/4ada09d41f1e36db62c44a9b25c209ea5f054316.1741875692.git.leon@kernel.org Signed-off-by: Leon Romanovsky <[email protected]> Signed-off-by: Sasha Levin <[email protected]>
popcornmix
pushed a commit
that referenced
this issue
Apr 14, 2025
[ Upstream commit 5ed3b0c ] When cur_qp isn't NULL, in order to avoid fetching the QP from the radix tree again we check if the next cqe QP is identical to the one we already have. The bug however is that we are checking if the QP is identical by checking the QP number inside the CQE against the QP number inside the mlx5_ib_qp, but that's wrong since the QP number from the CQE is from FW so it should be matched against mlx5_core_qp which is our FW QP number. Otherwise we could use the wrong QP when handling a CQE which could cause the kernel trace below. This issue is mainly noticeable over QPs 0 & 1, since for now they are the only QPs in our driver whereas the QP number inside mlx5_ib_qp doesn't match the QP number inside mlx5_core_qp. BUG: kernel NULL pointer dereference, address: 0000000000000012 #PF: supervisor read access in kernel mode #PF: error_code(0x0000) - not-present page PGD 0 P4D 0 Oops: Oops: 0000 [#1] SMP CPU: 0 UID: 0 PID: 7927 Comm: kworker/u62:1 Not tainted 6.14.0-rc3+ #189 Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.16.3-0-ga6ed6b701f0a-prebuilt.qemu.org 04/01/2014 Workqueue: ib-comp-unb-wq ib_cq_poll_work [ib_core] RIP: 0010:mlx5_ib_poll_cq+0x4c7/0xd90 [mlx5_ib] Code: 03 00 00 8d 58 ff 21 cb 66 39 d3 74 39 48 c7 c7 3c 89 6e a0 0f b7 db e8 b7 d2 b3 e0 49 8b 86 60 03 00 00 48 c7 c7 4a 89 6e a0 <0f> b7 5c 98 02 e8 9f d2 b3 e0 41 0f b7 86 78 03 00 00 83 e8 01 21 RSP: 0018:ffff88810511bd60 EFLAGS: 00010046 RAX: 0000000000000010 RBX: 0000000000000000 RCX: 0000000000000000 RDX: 0000000000000000 RSI: ffff88885fa1b3c0 RDI: ffffffffa06e894a RBP: 00000000000000b0 R08: 0000000000000000 R09: ffff88810511bc10 R10: 0000000000000001 R11: 0000000000000001 R12: ffff88810d593000 R13: ffff88810e579108 R14: ffff888105146000 R15: 00000000000000b0 FS: 0000000000000000(0000) GS:ffff88885fa00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000000000000012 CR3: 00000001077e6001 CR4: 0000000000370eb0 Call Trace: <TASK> ? __die+0x20/0x60 ? page_fault_oops+0x150/0x3e0 ? exc_page_fault+0x74/0x130 ? asm_exc_page_fault+0x22/0x30 ? mlx5_ib_poll_cq+0x4c7/0xd90 [mlx5_ib] __ib_process_cq+0x5a/0x150 [ib_core] ib_cq_poll_work+0x31/0x90 [ib_core] process_one_work+0x169/0x320 worker_thread+0x288/0x3a0 ? work_busy+0xb0/0xb0 kthread+0xd7/0x1f0 ? kthreads_online_cpu+0x130/0x130 ? kthreads_online_cpu+0x130/0x130 ret_from_fork+0x2d/0x50 ? kthreads_online_cpu+0x130/0x130 ret_from_fork_asm+0x11/0x20 </TASK> Fixes: e126ba9 ("mlx5: Add driver for Mellanox Connect-IB adapters") Signed-off-by: Patrisious Haddad <[email protected]> Reviewed-by: Edward Srouji <[email protected]> Link: https://patch.msgid.link/4ada09d41f1e36db62c44a9b25c209ea5f054316.1741875692.git.leon@kernel.org Signed-off-by: Leon Romanovsky <[email protected]> Signed-off-by: Sasha Levin <[email protected]>
popcornmix
pushed a commit
that referenced
this issue
Apr 26, 2025
[ Upstream commit 5ed3b0c ] When cur_qp isn't NULL, in order to avoid fetching the QP from the radix tree again we check if the next cqe QP is identical to the one we already have. The bug however is that we are checking if the QP is identical by checking the QP number inside the CQE against the QP number inside the mlx5_ib_qp, but that's wrong since the QP number from the CQE is from FW so it should be matched against mlx5_core_qp which is our FW QP number. Otherwise we could use the wrong QP when handling a CQE which could cause the kernel trace below. This issue is mainly noticeable over QPs 0 & 1, since for now they are the only QPs in our driver whereas the QP number inside mlx5_ib_qp doesn't match the QP number inside mlx5_core_qp. BUG: kernel NULL pointer dereference, address: 0000000000000012 #PF: supervisor read access in kernel mode #PF: error_code(0x0000) - not-present page PGD 0 P4D 0 Oops: Oops: 0000 [#1] SMP CPU: 0 UID: 0 PID: 7927 Comm: kworker/u62:1 Not tainted 6.14.0-rc3+ #189 Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.16.3-0-ga6ed6b701f0a-prebuilt.qemu.org 04/01/2014 Workqueue: ib-comp-unb-wq ib_cq_poll_work [ib_core] RIP: 0010:mlx5_ib_poll_cq+0x4c7/0xd90 [mlx5_ib] Code: 03 00 00 8d 58 ff 21 cb 66 39 d3 74 39 48 c7 c7 3c 89 6e a0 0f b7 db e8 b7 d2 b3 e0 49 8b 86 60 03 00 00 48 c7 c7 4a 89 6e a0 <0f> b7 5c 98 02 e8 9f d2 b3 e0 41 0f b7 86 78 03 00 00 83 e8 01 21 RSP: 0018:ffff88810511bd60 EFLAGS: 00010046 RAX: 0000000000000010 RBX: 0000000000000000 RCX: 0000000000000000 RDX: 0000000000000000 RSI: ffff88885fa1b3c0 RDI: ffffffffa06e894a RBP: 00000000000000b0 R08: 0000000000000000 R09: ffff88810511bc10 R10: 0000000000000001 R11: 0000000000000001 R12: ffff88810d593000 R13: ffff88810e579108 R14: ffff888105146000 R15: 00000000000000b0 FS: 0000000000000000(0000) GS:ffff88885fa00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000000000000012 CR3: 00000001077e6001 CR4: 0000000000370eb0 Call Trace: <TASK> ? __die+0x20/0x60 ? page_fault_oops+0x150/0x3e0 ? exc_page_fault+0x74/0x130 ? asm_exc_page_fault+0x22/0x30 ? mlx5_ib_poll_cq+0x4c7/0xd90 [mlx5_ib] __ib_process_cq+0x5a/0x150 [ib_core] ib_cq_poll_work+0x31/0x90 [ib_core] process_one_work+0x169/0x320 worker_thread+0x288/0x3a0 ? work_busy+0xb0/0xb0 kthread+0xd7/0x1f0 ? kthreads_online_cpu+0x130/0x130 ? kthreads_online_cpu+0x130/0x130 ret_from_fork+0x2d/0x50 ? kthreads_online_cpu+0x130/0x130 ret_from_fork_asm+0x11/0x20 </TASK> Fixes: e126ba9 ("mlx5: Add driver for Mellanox Connect-IB adapters") Signed-off-by: Patrisious Haddad <[email protected]> Reviewed-by: Edward Srouji <[email protected]> Link: https://patch.msgid.link/4ada09d41f1e36db62c44a9b25c209ea5f054316.1741875692.git.leon@kernel.org Signed-off-by: Leon Romanovsky <[email protected]> Signed-off-by: Sasha Levin <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Hello.
I have a stock raspbian + kernel 3.6.11+ #350, and can reliably oops the kernel by doing the following:
Over time multiple page allocation failures occur, all with the typical stack trace:
Leading to an OOPS with a constant originator, failure of memcpy to allocate memory.
This behaviour can also be replicated by eliminating the network stack entirely from proceedings. Copying from one USB device to another results in similar failures:
Although it seems to be because there was network activity also.
I compiled a debugging kernel from the latest pull of this repository with slab allocation debugging enabled plus kmemleak debugging.
On boot I get this:
Something is corrupting memory on boot. Given that it's on USB data access during enumeration of a device, my feeling is that it's the USB driver.
The text was updated successfully, but these errors were encountered: