Skip to content

Commit 2e4d475

Browse files
FelixMcFelixFelixMcFelix
authored
Datapath overhaul: zero-copy metadata with ingot, 'Compiled' UFTs (#585)
This PR rewrites the core of OPTE's packet model to use zero-copy packet parsing/modification via the [`ingot`](oxidecomputer/ingot#1) library. This enables a few changes which get us just shy of the 3Gbps mark. * **[2.36 -> 2.7]** The use of ingot for modifying packets in both the slowpath (UFT miss) and existing fastpath (UFT hit). * Parsing is faster -- we no longer copy out all packet header bytes onto the stack, and we do not allocate a vector to decompose an `mblk_t` into individual links. * Packet field modifications are applied directly to the `mblk_t` as they happen, and field reads are made from the same source. * Non-encap layers are not copied back out. * **[2.7 -> 2.75]** Packet L4 hashes are cached as part of the UFT, speeding up multipath route selection over the underlay. * **[2.75 -> 2.8]** Incremental Internet checksum recalculation is only performed when applicable fields change on inner flow headers (e.g., NAT'd packets). * VM-to-VM / intra-VPC traffic is the main use case here. * **[2.8 -> 3.05]** `NetworkParser`s now have the concept of inbound & outbound `LightweightMeta` formats. These support the key operations needed to execute all our UFT flows today (`FlowId` lookup, inner headers modification, encap push/pop, cksum update). * This also allows us to pre-serialize any bytes to be pushed in front of a packet, speeding up `EmitSpec`. * This is crucial for outbound traffic in particular, which has far smaller (in `struct`-size) metadata. * UFT misses or incompatible flows fallback to using the full metadata. * **[3.05 -> 2.95]** TCP state tracking uses a separate per-flow lock and does not require any lookup from a UFT. * I do not have numbers on how large the performance loss would be if we held the `Port` lock for the whole time. * (Not measured) Packet/UFT L4 Hashes are used as the Geneve source port, spreading inbound packets over NIC Rx queues based on the inner flow. * This is now possible because of #504 -- software classification would have limited us to the default inbound queue/group. * I feel bad for removing one more FF7 reference, but that is the way of these things. RIP port `7777`. * Previously, Rx queue affinity was derived solely from `(Src Sled, Dst Sled)`. There are several other changes here made to how OPTE functions which are needed to support the zero-copy model. * Each collected set of header transforms are `Arc<>`'d, such that we can apply them outside of the `Port` lock. * `FlowTable<S>`s now store `Arc<FlowEntry<S>>`, rather than `FlowEntry<S>`. * This enables the UFT entry for any flow to store its matched TCP flow, update its hit count and timestamp, and then update the TCP state without reacquring the `Port` lock. * This also drastically simplifies TCP state handling in fast path cases to not rely on post-transformation packets for lookup. * `Opte::process` returns an `EmitSpec` which is needed to finalise a packet before it can be used. * I'm not too happy about the ergonomics, but we have this problem because otherwise we'd need `Packet` to have some self-referential fields when supporting other key parts of XDE (e.g., parse -> use fields -> select port -> process). Closes #571, closes #481, closes #460. Slightly alleviates #435.
1 parent f3002b3 commit 2e4d475

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

88 files changed

+9936
-11164
lines changed

‎.github/buildomat/jobs/opte-api.sh

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@
33
#: name = "opte-api"
44
#: variety = "basic"
55
#: target = "helios-2.0"
6-
#: rust_toolchain = "nightly-2024-05-12"
6+
#: rust_toolchain = "nightly-2024-11-18"
77
#: output_rules = []
88
#:
99

@@ -24,7 +24,7 @@ header "check API_VERSION"
2424
./check-api-version.sh
2525

2626
header "check style"
27-
ptime -m cargo +nightly-2024-05-12 fmt -- --check
27+
ptime -m cargo +nightly-2024-11-18 fmt -- --check
2828

2929
header "analyze std"
3030
ptime -m cargo clippy --all-targets

‎.github/buildomat/jobs/opte-ioctl.sh

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@
33
#: name = "opte-ioctl"
44
#: variety = "basic"
55
#: target = "helios-2.0"
6-
#: rust_toolchain = "nightly-2024-05-12"
6+
#: rust_toolchain = "nightly-2024-11-18"
77
#: output_rules = []
88
#:
99

@@ -21,7 +21,7 @@ rustc --version
2121
cd lib/opte-ioctl
2222

2323
header "check style"
24-
ptime -m cargo +nightly-2024-05-12 fmt -- --check
24+
ptime -m cargo +nightly-2024-11-18 fmt -- --check
2525

2626
header "analyze"
2727
ptime -m cargo clippy --all-targets

‎.github/buildomat/jobs/opte.sh

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@
33
#: name = "opte"
44
#: variety = "basic"
55
#: target = "helios-2.0"
6-
#: rust_toolchain = "nightly-2024-05-12"
6+
#: rust_toolchain = "nightly-2024-11-18"
77
#: output_rules = []
88
#:
99

@@ -21,7 +21,7 @@ rustc --version
2121
cd lib/opte
2222

2323
header "check style"
24-
ptime -m cargo +nightly-2024-05-12 fmt -- --check
24+
ptime -m cargo +nightly-2024-11-18 fmt -- --check
2525

2626
header "check docs"
2727
#
@@ -30,13 +30,13 @@ header "check docs"
3030
#
3131
# Use nightly which is needed for the `kernel` feature.
3232
RUSTDOCFLAGS="-D warnings" ptime -m \
33-
cargo +nightly-2024-05-12 doc --no-default-features --features=api,std,engine,kernel
33+
cargo +nightly-2024-11-18 doc --no-default-features --features=api,std,engine,kernel
3434

3535
header "analyze std + api"
3636
ptime -m cargo clippy --all-targets
3737

3838
header "analyze no_std + engine + kernel"
39-
ptime -m cargo +nightly-2024-05-12 clippy --no-default-features --features engine,kernel
39+
ptime -m cargo +nightly-2024-11-18 clippy --no-default-features --features engine,kernel
4040

4141
header "test"
4242
ptime -m cargo test

‎.github/buildomat/jobs/opteadm.sh

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@
33
#: name = "opteadm"
44
#: variety = "basic"
55
#: target = "helios-2.0"
6-
#: rust_toolchain = "nightly-2024-05-12"
6+
#: rust_toolchain = "nightly-2024-11-18"
77
#: output_rules = [
88
#: "=/work/debug/opteadm",
99
#: "=/work/debug/opteadm.debug.sha256",
@@ -30,7 +30,7 @@ rustc --version
3030
pushd bin/opteadm
3131

3232
header "check style"
33-
ptime -m cargo +nightly-2024-05-12 fmt -- --check
33+
ptime -m cargo +nightly-2024-11-18 fmt -- --check
3434

3535
header "analyze"
3636
ptime -m cargo clippy --all-targets

‎.github/buildomat/jobs/oxide-vpc.sh

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@
33
#: name = "oxide-vpc"
44
#: variety = "basic"
55
#: target = "helios-2.0"
6-
#: rust_toolchain = "nightly-2024-05-12"
6+
#: rust_toolchain = "nightly-2024-11-18"
77
#: output_rules = []
88
#:
99

@@ -21,7 +21,7 @@ rustc --version
2121
cd lib/oxide-vpc
2222

2323
header "check style"
24-
ptime -m cargo +nightly-2024-05-12 fmt -- --check
24+
ptime -m cargo +nightly-2024-11-18 fmt -- --check
2525

2626
header "check docs"
2727
#
@@ -30,13 +30,13 @@ header "check docs"
3030
#
3131
# Use nightly which is needed for the `kernel` feature.
3232
RUSTDOCFLAGS="-D warnings" ptime -m \
33-
cargo +nightly-2024-05-12 doc --no-default-features --features=api,std,engine,kernel
33+
cargo +nightly-2024-11-18 doc --no-default-features --features=api,std,engine,kernel
3434

3535
header "analyze std + api + usdt"
3636
ptime -m cargo clippy --features usdt --all-targets
3737

3838
header "analyze no_std + engine + kernel"
39-
ptime -m cargo +nightly-2024-05-12 clippy --no-default-features --features engine,kernel
39+
ptime -m cargo +nightly-2024-11-18 clippy --no-default-features --features engine,kernel
4040

4141
header "test"
4242
ptime -m cargo test

‎.github/buildomat/jobs/p5p.sh

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@
33
#: name = "opte-p5p"
44
#: variety = "basic"
55
#: target = "helios-2.0"
6-
#: rust_toolchain = "nightly-2024-05-12"
6+
#: rust_toolchain = "nightly-2024-11-18"
77
#: output_rules = [
88
#: "=/out/opte.p5p",
99
#: "=/out/opte.p5p.sha256",

‎.github/buildomat/jobs/xde.sh

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@
33
#: name = "opte-xde"
44
#: variety = "basic"
55
#: target = "helios-2.0"
6-
#: rust_toolchain = "nightly-2024-05-12"
6+
#: rust_toolchain = "nightly-2024-11-18"
77
#: output_rules = [
88
#: "=/work/debug/xde.dbg",
99
#: "=/work/debug/xde.dbg.sha256",
@@ -75,7 +75,7 @@ pushd xde
7575
cp xde.conf /work/xde.conf
7676

7777
header "check style"
78-
ptime -m cargo +nightly-2024-05-12 fmt -p xde -p xde-link -- --check
78+
ptime -m cargo +nightly-2024-11-18 fmt -p xde -p xde-link -- --check
7979

8080
header "analyze"
8181
ptime -m cargo clippy -- \
@@ -123,7 +123,7 @@ sha256sum $REL_TGT/xde_link.so > $REL_TGT/xde_link.so.sha256
123123

124124
header "build xde integration tests"
125125
pushd xde-tests
126-
cargo +nightly-2024-05-12 fmt -- --check
126+
cargo +nightly-2024-11-18 fmt -- --check
127127
cargo clippy --all-targets
128128
cargo build --test loopback
129129
loopback_test=$(

0 commit comments

Comments
 (0)