DLPX-84985 target: iscsi: fix deadlock in the iSCSI login code #23
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
pcd1193182
approved these changes
Mar 7, 2023
sumedhbala-delphix
approved these changes
Mar 7, 2023
sdimitro
added a commit
that referenced
this pull request
Mar 8, 2023
prakashsurya
pushed a commit
that referenced
this pull request
Mar 14, 2023
prakashsurya
pushed a commit
that referenced
this pull request
Mar 14, 2023
prakashsurya
pushed a commit
that referenced
this pull request
Mar 14, 2023
delphix-devops-bot
pushed a commit
that referenced
this pull request
Mar 30, 2023
delphix-devops-bot
pushed a commit
that referenced
this pull request
Apr 20, 2023
delphix-devops-bot
pushed a commit
that referenced
this pull request
Apr 28, 2023
delphix-devops-bot
pushed a commit
that referenced
this pull request
May 26, 2023
delphix-devops-bot
pushed a commit
that referenced
this pull request
Jun 3, 2023
delphix-devops-bot
pushed a commit
that referenced
this pull request
Jun 4, 2023
delphix-devops-bot
pushed a commit
that referenced
this pull request
Jun 5, 2023
prakashsurya
pushed a commit
that referenced
this pull request
Aug 8, 2023
delphix-devops-bot
pushed a commit
that referenced
this pull request
Aug 24, 2023
delphix-devops-bot
pushed a commit
that referenced
this pull request
Aug 25, 2023
delphix-devops-bot
pushed a commit
that referenced
this pull request
Aug 26, 2023
delphix-devops-bot
pushed a commit
that referenced
this pull request
Aug 27, 2023
delphix-devops-bot
pushed a commit
that referenced
this pull request
Sep 2, 2023
delphix-devops-bot
pushed a commit
that referenced
this pull request
Sep 3, 2023
delphix-devops-bot
pushed a commit
that referenced
this pull request
Sep 4, 2023
delphix-devops-bot
pushed a commit
that referenced
this pull request
Sep 5, 2023
delphix-devops-bot
pushed a commit
that referenced
this pull request
Sep 6, 2023
delphix-devops-bot
pushed a commit
that referenced
this pull request
Sep 7, 2023
delphix-devops-bot
pushed a commit
that referenced
this pull request
Sep 8, 2023
prakashsurya
pushed a commit
that referenced
this pull request
Sep 12, 2023
delphix-devops-bot
pushed a commit
that referenced
this pull request
Oct 21, 2024
palash-gandhi
pushed a commit
that referenced
this pull request
Oct 24, 2024
palash-gandhi
added a commit
that referenced
this pull request
Oct 24, 2024
delphix-devops-bot
pushed a commit
that referenced
this pull request
Nov 10, 2024
delphix-devops-bot
pushed a commit
that referenced
this pull request
Nov 11, 2024
delphix-devops-bot
pushed a commit
that referenced
this pull request
Dec 18, 2024
delphix-devops-bot
pushed a commit
that referenced
this pull request
Dec 19, 2024
delphix-devops-bot
pushed a commit
that referenced
this pull request
Dec 20, 2024
delphix-devops-bot
pushed a commit
that referenced
this pull request
Dec 21, 2024
delphix-devops-bot
pushed a commit
that referenced
this pull request
Dec 22, 2024
delphix-devops-bot
pushed a commit
that referenced
this pull request
Dec 23, 2024
delphix-devops-bot
pushed a commit
that referenced
this pull request
Dec 24, 2024
delphix-devops-bot
pushed a commit
that referenced
this pull request
Dec 25, 2024
delphix-devops-bot
pushed a commit
that referenced
this pull request
Dec 26, 2024
delphix-devops-bot
pushed a commit
that referenced
this pull request
Dec 27, 2024
delphix-devops-bot
pushed a commit
that referenced
this pull request
Dec 28, 2024
delphix-devops-bot
pushed a commit
that referenced
this pull request
Dec 29, 2024
delphix-devops-bot
pushed a commit
that referenced
this pull request
Dec 30, 2024
delphix-devops-bot
pushed a commit
that referenced
this pull request
Dec 31, 2024
delphix-devops-bot
pushed a commit
that referenced
this pull request
Jan 1, 2025
delphix-devops-bot
pushed a commit
that referenced
this pull request
Jan 2, 2025
delphix-devops-bot
pushed a commit
that referenced
this pull request
Jan 3, 2025
delphix-devops-bot
pushed a commit
that referenced
this pull request
Jan 4, 2025
delphix-devops-bot
pushed a commit
that referenced
this pull request
Jan 5, 2025
delphix-devops-bot
pushed a commit
that referenced
this pull request
Jan 6, 2025
delphix-devops-bot
pushed a commit
that referenced
this pull request
Jan 7, 2025
delphix-devops-bot
pushed a commit
that referenced
this pull request
Jan 31, 2025
delphix-devops-bot
pushed a commit
that referenced
this pull request
Feb 27, 2025
BugLink: https://bugs.launchpad.net/bugs/2089272 [ Upstream commit ca9984c5f0ab3690d98b13937b2485a978c8dd73 ] rxq contains a pointer to the device from where the redirect happened. Currently, the BPF program that was executed after a redirect via BPF_MAP_TYPE_DEVMAP* does not have it set. This is particularly bad since accessing ingress_ifindex, e.g. SEC("xdp") int prog(struct xdp_md *pkt) { return bpf_redirect_map(&dev_redirect_map, 0, 0); } SEC("xdp/devmap") int prog_after_redirect(struct xdp_md *pkt) { bpf_printk("ifindex %i", pkt->ingress_ifindex); return XDP_PASS; } depends on access to rxq, so a NULL pointer gets dereferenced: <1>[ 574.475170] BUG: kernel NULL pointer dereference, address: 0000000000000000 <1>[ 574.475188] #PF: supervisor read access in kernel mode <1>[ 574.475194] #PF: error_code(0x0000) - not-present page <6>[ 574.475199] PGD 0 P4D 0 <4>[ 574.475207] Oops: Oops: 0000 [#1] PREEMPT SMP NOPTI <4>[ 574.475217] CPU: 4 UID: 0 PID: 217 Comm: kworker/4:1 Not tainted 6.11.0-rc5-reduced-00859-g780801200300 #23 <4>[ 574.475226] Hardware name: Intel(R) Client Systems NUC13ANHi7/NUC13ANBi7, BIOS ANRPL357.0026.2023.0314.1458 03/14/2023 <4>[ 574.475231] Workqueue: mld mld_ifc_work <4>[ 574.475247] RIP: 0010:bpf_prog_5e13354d9cf5018a_prog_after_redirect+0x17/0x3c <4>[ 574.475257] Code: cc cc cc cc cc cc cc 80 00 00 00 cc cc cc cc cc cc cc cc f3 0f 1e fa 0f 1f 44 00 00 66 90 55 48 89 e5 f3 0f 1e fa 48 8b 57 20 <48> 8b 52 00 8b 92 e0 00 00 00 48 bf f8 a6 d5 c4 5d a0 ff ff be 0b <4>[ 574.475263] RSP: 0018:ffffa62440280c98 EFLAGS: 00010206 <4>[ 574.475269] RAX: ffffa62440280cd8 RBX: 0000000000000001 RCX: 0000000000000000 <4>[ 574.475274] RDX: 0000000000000000 RSI: ffffa62440549048 RDI: ffffa62440280ce0 <4>[ 574.475278] RBP: ffffa62440280c98 R08: 0000000000000002 R09: 0000000000000001 <4>[ 574.475281] R10: ffffa05dc8b98000 R11: ffffa05f577fca40 R12: ffffa05dcab24000 <4>[ 574.475285] R13: ffffa62440280ce0 R14: ffffa62440549048 R15: ffffa62440549000 <4>[ 574.475289] FS: 0000000000000000(0000) GS:ffffa05f4f700000(0000) knlGS:0000000000000000 <4>[ 574.475294] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 <4>[ 574.475298] CR2: 0000000000000000 CR3: 000000025522e000 CR4: 0000000000f50ef0 <4>[ 574.475303] PKRU: 55555554 <4>[ 574.475306] Call Trace: <4>[ 574.475313] <IRQ> <4>[ 574.475318] ? __die+0x23/0x70 <4>[ 574.475329] ? page_fault_oops+0x180/0x4c0 <4>[ 574.475339] ? skb_pp_cow_data+0x34c/0x490 <4>[ 574.475346] ? kmem_cache_free+0x257/0x280 <4>[ 574.475357] ? exc_page_fault+0x67/0x150 <4>[ 574.475368] ? asm_exc_page_fault+0x26/0x30 <4>[ 574.475381] ? bpf_prog_5e13354d9cf5018a_prog_after_redirect+0x17/0x3c <4>[ 574.475386] bq_xmit_all+0x158/0x420 <4>[ 574.475397] __dev_flush+0x30/0x90 <4>[ 574.475407] veth_poll+0x216/0x250 [veth] <4>[ 574.475421] __napi_poll+0x28/0x1c0 <4>[ 574.475430] net_rx_action+0x32d/0x3a0 <4>[ 574.475441] handle_softirqs+0xcb/0x2c0 <4>[ 574.475451] do_softirq+0x40/0x60 <4>[ 574.475458] </IRQ> <4>[ 574.475461] <TASK> <4>[ 574.475464] __local_bh_enable_ip+0x66/0x70 <4>[ 574.475471] __dev_queue_xmit+0x268/0xe40 <4>[ 574.475480] ? selinux_ip_postroute+0x213/0x420 <4>[ 574.475491] ? alloc_skb_with_frags+0x4a/0x1d0 <4>[ 574.475502] ip6_finish_output2+0x2be/0x640 <4>[ 574.475512] ? nf_hook_slow+0x42/0xf0 <4>[ 574.475521] ip6_finish_output+0x194/0x300 <4>[ 574.475529] ? __pfx_ip6_finish_output+0x10/0x10 <4>[ 574.475538] mld_sendpack+0x17c/0x240 <4>[ 574.475548] mld_ifc_work+0x192/0x410 <4>[ 574.475557] process_one_work+0x15d/0x380 <4>[ 574.475566] worker_thread+0x29d/0x3a0 <4>[ 574.475573] ? __pfx_worker_thread+0x10/0x10 <4>[ 574.475580] ? __pfx_worker_thread+0x10/0x10 <4>[ 574.475587] kthread+0xcd/0x100 <4>[ 574.475597] ? __pfx_kthread+0x10/0x10 <4>[ 574.475606] ret_from_fork+0x31/0x50 <4>[ 574.475615] ? __pfx_kthread+0x10/0x10 <4>[ 574.475623] ret_from_fork_asm+0x1a/0x30 <4>[ 574.475635] </TASK> <4>[ 574.475637] Modules linked in: veth br_netfilter bridge stp llc iwlmvm x86_pkg_temp_thermal iwlwifi efivarfs nvme nvme_core <4>[ 574.475662] CR2: 0000000000000000 <4>[ 574.475668] ---[ end trace 0000000000000000 ]--- Therefore, provide it to the program by setting rxq properly. Fixes: cb261b5 ("bpf: Run devmap xdp_prog on flush instead of bulk enqueue") Reviewed-by: Toke Høiland-Jørgensen <[email protected]> Signed-off-by: Florian Kauer <[email protected]> Acked-by: Jakub Kicinski <[email protected]> Link: https://lore.kernel.org/r/20240911-devel-koalo-fix-ingress-ifindex-v4-1-5c643ae10258@linutronix.de Signed-off-by: Martin KaFai Lau <[email protected]> Signed-off-by: Sasha Levin <[email protected]> CVE-2024-50162 Signed-off-by: Manuel Diewald <[email protected]> Signed-off-by: Mehmet Basaran <[email protected]>
delphix-devops-bot
pushed a commit
that referenced
this pull request
Apr 27, 2025
BugLink: https://bugs.launchpad.net/bugs/2095283 [ Upstream commit 146b6f1112eb30a19776d6c323c994e9d67790db ] Under certain kernel configurations when building with Clang/LLVM, the compiler does not generate a return or jump as the terminator instruction for ip_vs_protocol_init(), triggering the following objtool warning during build time: vmlinux.o: warning: objtool: ip_vs_protocol_init() falls through to next function __initstub__kmod_ip_vs_rr__935_123_ip_vs_rr_init6() At runtime, this either causes an oops when trying to load the ipvs module or a boot-time panic if ipvs is built-in. This same issue has been reported by the Intel kernel test robot previously. Digging deeper into both LLVM and the kernel code reveals this to be a undefined behavior problem. ip_vs_protocol_init() uses a on-stack buffer of 64 chars to store the registered protocol names and leaves it uninitialized after definition. The function calls strnlen() when concatenating protocol names into the buffer. With CONFIG_FORTIFY_SOURCE strnlen() performs an extra step to check whether the last byte of the input char buffer is a null character (commit 3009f89 ("fortify: Allow strlen() and strnlen() to pass compile-time known lengths")). This, together with possibly other configurations, cause the following IR to be generated: define hidden i32 @ip_vs_protocol_init() local_unnamed_addr #5 section ".init.text" align 16 !kcfi_type !29 { %1 = alloca [64 x i8], align 16 ... 14: ; preds = %11 %15 = getelementptr inbounds i8, ptr %1, i64 63 %16 = load i8, ptr %15, align 1 %17 = tail call i1 @llvm.is.constant.i8(i8 %16) %18 = icmp eq i8 %16, 0 %19 = select i1 %17, i1 %18, i1 false br i1 %19, label %20, label %23 20: ; preds = %14 %21 = call i64 @strlen(ptr noundef nonnull dereferenceable(1) %1) #23 ... 23: ; preds = %14, %11, %20 %24 = call i64 @strnlen(ptr noundef nonnull dereferenceable(1) %1, i64 noundef 64) #24 ... } The above code calculates the address of the last char in the buffer (value %15) and then loads from it (value %16). Because the buffer is never initialized, the LLVM GVN pass marks value %16 as undefined: %13 = getelementptr inbounds i8, ptr %1, i64 63 br i1 undef, label %14, label %17 This gives later passes (SCCP, in particular) more DCE opportunities by propagating the undef value further, and eventually removes everything after the load on the uninitialized stack location: define hidden i32 @ip_vs_protocol_init() local_unnamed_addr #0 section ".init.text" align 16 !kcfi_type !11 { %1 = alloca [64 x i8], align 16 ... 12: ; preds = %11 %13 = getelementptr inbounds i8, ptr %1, i64 63 unreachable } In this way, the generated native code will just fall through to the next function, as LLVM does not generate any code for the unreachable IR instruction and leaves the function without a terminator. Zero the on-stack buffer to avoid this possible UB. Fixes: 1da177e ("Linux-2.6.12-rc2") Reported-by: kernel test robot <[email protected]> Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/ Co-developed-by: Ruowen Qin <[email protected]> Signed-off-by: Ruowen Qin <[email protected]> Signed-off-by: Jinghao Jia <[email protected]> Acked-by: Julian Anastasov <[email protected]> Signed-off-by: Pablo Neira Ayuso <[email protected]> Signed-off-by: Sasha Levin <[email protected]> Signed-off-by: Koichiro Den <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Until I write a polished root-cause in jira here is the root-cause and overall correspondence with upstream for this patch: https://gist.github.com/sdimitro/16e33e0d87dc2ba4fa2e7dfa55545792
Only modification I did was s/iscsit_conn/iscsi_conn because the structure has been renamed in newer kernels but not our kernel yet.
ab-pre-push: http://selfservice.jenkins.delphix.com/job/appliance-build-orchestrator-pre-push/4661/
Manual testing only done in ESX only so far
Update: Full RCA can be found here: https://delphix.atlassian.net/browse/ESCL-4240?focusedCommentId=682736