Skip to content

Conversation

c-r33d
Copy link
Contributor

@c-r33d c-r33d commented May 7, 2025

Proposed Changes

1.) Add platform key indexer to be used for new key management architecture.

Checklist

  • I have added or updated unit tests
  • I have added or updated integration tests (if appropriate)
  • I have added or updated documentation

Testing Instructions

@c-r33d c-r33d requested a review from a team as a code owner May 7, 2025 18:03
@c-r33d c-r33d force-pushed the feat/DSPX-1062-key-indexer branch from ac28da0 to 22d35d3 Compare May 7, 2025 18:05
Copy link
Contributor

github-actions bot commented May 7, 2025

Benchmark results, click to expand

Benchmark Results:

Metric Value
Approved Decision Requests 5000
Denied Decision Requests 0
Total Time 459.41087ms

Bulk Benchmark Results

Metric Value
Total Decrypts 100
Successful Decrypts 100
Failed Decrypts 0
Total Time 374.020512ms
Throughput 267.37 requests/second

TDF3 Benchmark Results:

Metric Value
Total Requests 5000
Successful Requests 5000
Failed Requests 0
Concurrent Requests 50
Total Time 1m15.667515814s
Average Latency 754.394159ms
Throughput 66.08 requests/second

NANOTDF Benchmark Results:

Metric Value
Total Requests 5000
Successful Requests 4948
Failed Requests 52
Concurrent Requests 50
Total Time 1m7.176461297s
Average Latency 667.075187ms
Throughput 73.66 requests/second

Error Summary:

Error Message Occurrences
ReadNanoTDF error: getNanoRewrapKey: rewrapError: internal: internal error
rpc error: code = Internal desc = could not perform access
52 occurrences

Standard Benchmark Metrics Skipped or Failed

Copy link
Contributor

github-actions bot commented May 7, 2025

Benchmark results, click to expand

Benchmark Results:

Metric Value
Approved Decision Requests 5000
Denied Decision Requests 0
Total Time 438.90578ms

Bulk Benchmark Results

Metric Value
Total Decrypts 100
Successful Decrypts 100
Failed Decrypts 0
Total Time 368.515663ms
Throughput 271.36 requests/second

TDF3 Benchmark Results:

Metric Value
Total Requests 5000
Successful Requests 5000
Failed Requests 0
Concurrent Requests 50
Total Time 1m16.340625988s
Average Latency 761.15847ms
Throughput 65.50 requests/second

NANOTDF Benchmark Results:

Metric Value
Total Requests 5000
Successful Requests 4940
Failed Requests 60
Concurrent Requests 50
Total Time 1m7.491769494s
Average Latency 672.097255ms
Throughput 73.19 requests/second

Error Summary:

Error Message Occurrences
ReadNanoTDF error: getNanoRewrapKey: rewrapError: internal: internal error
rpc error: code = Internal desc = could not perform access
60 occurrences

Standard Benchmark Metrics Skipped or Failed

@c-r33d c-r33d force-pushed the feat/DSPX-1062-key-indexer branch from 22d35d3 to c362a31 Compare May 7, 2025 18:12
Copy link
Contributor

github-actions bot commented May 7, 2025

Benchmark results, click to expand

Benchmark Results:

Metric Value
Approved Decision Requests 5000
Denied Decision Requests 0
Total Time 478.005287ms

Bulk Benchmark Results

Metric Value
Total Decrypts 100
Successful Decrypts 100
Failed Decrypts 0
Total Time 415.331907ms
Throughput 240.77 requests/second

TDF3 Benchmark Results:

Metric Value
Total Requests 5000
Successful Requests 5000
Failed Requests 0
Concurrent Requests 50
Total Time 1m18.428179595s
Average Latency 782.086215ms
Throughput 63.75 requests/second

NANOTDF Benchmark Results:

Metric Value
Total Requests 5000
Successful Requests 4943
Failed Requests 57
Concurrent Requests 50
Total Time 1m8.584183467s
Average Latency 682.605948ms
Throughput 72.07 requests/second

Error Summary:

Error Message Occurrences
ReadNanoTDF error: getNanoRewrapKey: rewrapError: internal: internal error
rpc error: code = Internal desc = could not perform access
57 occurrences

Standard Benchmark Metrics Skipped or Failed

@c-r33d c-r33d force-pushed the feat/DSPX-1062-key-indexer branch from c362a31 to 79971f5 Compare May 7, 2025 18:27
Copy link
Contributor

github-actions bot commented May 7, 2025

Benchmark results, click to expand

Benchmark Results:

Metric Value
Approved Decision Requests 5000
Denied Decision Requests 0
Total Time 484.253724ms

Bulk Benchmark Results

Metric Value
Total Decrypts 100
Successful Decrypts 100
Failed Decrypts 0
Total Time 371.50805ms
Throughput 269.17 requests/second

TDF3 Benchmark Results:

Metric Value
Total Requests 5000
Successful Requests 5000
Failed Requests 0
Concurrent Requests 50
Total Time 1m22.111277558s
Average Latency 818.998699ms
Throughput 60.89 requests/second

NANOTDF Benchmark Results:

Metric Value
Total Requests 5000
Successful Requests 4949
Failed Requests 51
Concurrent Requests 50
Total Time 1m11.868052968s
Average Latency 712.851827ms
Throughput 68.86 requests/second

Error Summary:

Error Message Occurrences
ReadNanoTDF error: getNanoRewrapKey: rewrapError: internal: internal error
rpc error: code = Internal desc = could not perform access
51 occurrences

Standard Benchmark Metrics Skipped or Failed

remove comments.

linting.

linting.

fix test.
@c-r33d c-r33d force-pushed the feat/DSPX-1062-key-indexer branch from 79971f5 to 6413d34 Compare May 7, 2025 18:37
Copy link
Contributor

github-actions bot commented May 7, 2025

Benchmark results, click to expand

Benchmark Results:

Metric Value
Approved Decision Requests 5000
Denied Decision Requests 0
Total Time 443.130214ms

Bulk Benchmark Results

Metric Value
Total Decrypts 100
Successful Decrypts 100
Failed Decrypts 0
Total Time 377.242379ms
Throughput 265.08 requests/second

TDF3 Benchmark Results:

Metric Value
Total Requests 5000
Successful Requests 5000
Failed Requests 0
Concurrent Requests 50
Total Time 1m19.376967208s
Average Latency 791.036857ms
Throughput 62.99 requests/second

NANOTDF Benchmark Results:

Metric Value
Total Requests 5000
Successful Requests 4963
Failed Requests 37
Concurrent Requests 50
Total Time 1m10.143794777s
Average Latency 697.793571ms
Throughput 70.75 requests/second

Error Summary:

Error Message Occurrences
ReadNanoTDF error: getNanoRewrapKey: rewrapError: internal: internal error
rpc error: code = Internal desc = could not perform access
37 occurrences

Standard Benchmark Metrics Skipped or Failed

Copy link
Contributor

github-actions bot commented May 7, 2025

Benchmark results, click to expand

Benchmark Results:

Metric Value
Approved Decision Requests 5000
Denied Decision Requests 0
Total Time 449.58943ms

Bulk Benchmark Results

Metric Value
Total Decrypts 100
Successful Decrypts 100
Failed Decrypts 0
Total Time 361.006166ms
Throughput 277.00 requests/second

TDF3 Benchmark Results:

Metric Value
Total Requests 5000
Successful Requests 5000
Failed Requests 0
Concurrent Requests 50
Total Time 1m16.741011743s
Average Latency 765.508646ms
Throughput 65.15 requests/second

NANOTDF Benchmark Results:

Metric Value
Total Requests 5000
Successful Requests 4948
Failed Requests 52
Concurrent Requests 50
Total Time 1m6.780232635s
Average Latency 663.59386ms
Throughput 74.09 requests/second

Error Summary:

Error Message Occurrences
ReadNanoTDF error: getNanoRewrapKey: rewrapError: internal: internal error
rpc error: code = Internal desc = could not perform access
52 occurrences

Standard Benchmark Metrics Skipped or Failed

Copy link
Contributor

Benchmark results, click to expand

Benchmark Results:

Metric Value
Approved Decision Requests 5000
Denied Decision Requests 0
Total Time 480.427366ms

Bulk Benchmark Results

Metric Value
Total Decrypts 100
Successful Decrypts 100
Failed Decrypts 0
Total Time 360.938088ms
Throughput 277.06 requests/second

TDF3 Benchmark Results:

Metric Value
Total Requests 5000
Successful Requests 5000
Failed Requests 0
Concurrent Requests 50
Total Time 1m23.348544727s
Average Latency 831.908138ms
Throughput 59.99 requests/second

NANOTDF Benchmark Results:

Metric Value
Total Requests 5000
Successful Requests 4947
Failed Requests 53
Concurrent Requests 50
Total Time 1m11.857263852s
Average Latency 715.690339ms
Throughput 68.84 requests/second

Error Summary:

Error Message Occurrences
ReadNanoTDF error: getNanoRewrapKey: rewrapError: internal: internal error
rpc error: code = Internal desc = could not perform access
53 occurrences

Standard Benchmark Metrics Skipped or Failed

Copy link
Contributor

Benchmark results, click to expand

Benchmark Results:

Metric Value
Approved Decision Requests 5000
Denied Decision Requests 0
Total Time 476.352092ms

Bulk Benchmark Results

Metric Value
Total Decrypts 100
Successful Decrypts 100
Failed Decrypts 0
Total Time 363.209191ms
Throughput 275.32 requests/second

TDF3 Benchmark Results:

Metric Value
Total Requests 5000
Successful Requests 5000
Failed Requests 0
Concurrent Requests 50
Total Time 1m16.650765668s
Average Latency 764.459219ms
Throughput 65.23 requests/second

NANOTDF Benchmark Results:

Metric Value
Total Requests 5000
Successful Requests 4943
Failed Requests 57
Concurrent Requests 50
Total Time 1m6.946289235s
Average Latency 665.349141ms
Throughput 73.84 requests/second

Error Summary:

Error Message Occurrences
ReadNanoTDF error: getNanoRewrapKey: rewrapError: internal: internal error
rpc error: code = Internal desc = could not perform access
57 occurrences

Standard Benchmark Metrics Skipped or Failed

@c-r33d c-r33d requested review from a team as code owners May 20, 2025 14:24
Copy link
Contributor

Benchmark results, click to expand

Benchmark Results:

Metric Value
Approved Decision Requests 5000
Denied Decision Requests 0
Total Time 466.707204ms

Bulk Benchmark Results

Metric Value
Total Decrypts 100
Successful Decrypts 100
Failed Decrypts 0
Total Time 378.554516ms
Throughput 264.16 requests/second

TDF3 Benchmark Results:

Metric Value
Total Requests 5000
Successful Requests 5000
Failed Requests 0
Concurrent Requests 50
Total Time 1m22.710391291s
Average Latency 824.886512ms
Throughput 60.45 requests/second

NANOTDF Benchmark Results:

Metric Value
Total Requests 5000
Successful Requests 4944
Failed Requests 56
Concurrent Requests 50
Total Time 1m12.827115727s
Average Latency 725.926776ms
Throughput 67.89 requests/second

Error Summary:

Error Message Occurrences
ReadNanoTDF error: getNanoRewrapKey: rewrapError: internal: internal error
rpc error: code = Internal desc = could not perform access
56 occurrences

Standard Benchmark Metrics Skipped or Failed

Copy link
Contributor

Benchmark results, click to expand

Benchmark Results:

Metric Value
Approved Decision Requests 5000
Denied Decision Requests 0
Total Time 476.439251ms

Bulk Benchmark Results

Metric Value
Total Decrypts 100
Successful Decrypts 100
Failed Decrypts 0
Total Time 376.414995ms
Throughput 265.66 requests/second

TDF3 Benchmark Results:

Metric Value
Total Requests 5000
Successful Requests 5000
Failed Requests 0
Concurrent Requests 50
Total Time 1m19.906503139s
Average Latency 796.374912ms
Throughput 62.57 requests/second

NANOTDF Benchmark Results:

Metric Value
Total Requests 5000
Successful Requests 4970
Failed Requests 30
Concurrent Requests 50
Total Time 1m10.805167222s
Average Latency 704.871291ms
Throughput 70.19 requests/second

Error Summary:

Error Message Occurrences
ReadNanoTDF error: getNanoRewrapKey: rewrapError: internal: internal error
rpc error: code = Internal desc = could not perform access
30 occurrences

Standard Benchmark Metrics Skipped or Failed

@c-r33d c-r33d requested a review from dmihalcik-virtru May 20, 2025 15:10
@strantalis strantalis requested a review from Copilot May 20, 2025 19:36
Copy link
Contributor

@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR introduces a new platform key indexer for the updated key management architecture. The key changes include the implementation of PlatformKeyIndexer and KasKeyAdapter, along with corresponding unit tests for key retrieval and export.

Reviewed Changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File Description
service/policy/keymanagement/platform_key_indexer.go Implements the key indexer with functions for key lookup and public key export in various formats
service/policy/keymanagement/platform_key_indexer_test.go Provides unit tests for verifying key details and public key export functionality

Copy link
Contributor

Benchmark results, click to expand

Benchmark Results:

Metric Value
Approved Decision Requests 5000
Denied Decision Requests 0
Total Time 455.797205ms

Bulk Benchmark Results

Metric Value
Total Decrypts 100
Successful Decrypts 100
Failed Decrypts 0
Total Time 379.424586ms
Throughput 263.56 requests/second

TDF3 Benchmark Results:

Metric Value
Total Requests 5000
Successful Requests 5000
Failed Requests 0
Concurrent Requests 50
Total Time 1m20.738578375s
Average Latency 804.78367ms
Throughput 61.93 requests/second

NANOTDF Benchmark Results:

Metric Value
Total Requests 5000
Successful Requests 4950
Failed Requests 50
Concurrent Requests 50
Total Time 1m10.765425934s
Average Latency 704.079704ms
Throughput 69.95 requests/second

Error Summary:

Error Message Occurrences
ReadNanoTDF error: getNanoRewrapKey: rewrapError: internal: internal error
rpc error: code = Internal desc = could not perform access
50 occurrences

Standard Benchmark Metrics Skipped or Failed

@c-r33d c-r33d requested a review from a team as a code owner May 20, 2025 19:57
Copy link
Contributor

Benchmark results, click to expand

Benchmark Results:

Metric Value
Approved Decision Requests 5000
Denied Decision Requests 0
Total Time 455.319327ms

Bulk Benchmark Results

Metric Value
Total Decrypts 100
Successful Decrypts 100
Failed Decrypts 0
Total Time 366.120981ms
Throughput 273.13 requests/second

TDF3 Benchmark Results:

Metric Value
Total Requests 5000
Successful Requests 5000
Failed Requests 0
Concurrent Requests 50
Total Time 1m19.40949426s
Average Latency 790.702608ms
Throughput 62.96 requests/second

NANOTDF Benchmark Results:

Metric Value
Total Requests 5000
Successful Requests 4950
Failed Requests 50
Concurrent Requests 50
Total Time 1m9.567748109s
Average Latency 691.406234ms
Throughput 71.15 requests/second

Error Summary:

Error Message Occurrences
ReadNanoTDF error: getNanoRewrapKey: rewrapError: internal: internal error
rpc error: code = Internal desc = could not perform access
50 occurrences

Standard Benchmark Metrics Skipped or Failed

@c-r33d c-r33d added this pull request to the merge queue May 20, 2025
Merged via the queue into main with commit 861ef8d May 20, 2025
28 checks passed
@c-r33d c-r33d deleted the feat/DSPX-1062-key-indexer branch May 20, 2025 20:32
github-merge-queue bot pushed a commit that referenced this pull request May 22, 2025
🤖 I have created a release *beep* *boop*
---


##
[0.5.3](service/v0.5.2...service/v0.5.3)
(2025-05-22)


### Features

* **authz:** authz v2 versioning implementation
([#2173](#2173))
([557fc21](557fc21))
* **authz:** authz v2, ers v2 protos and gencode for ABAC with actions &
registered resource
([#2124](#2124))
([ea7992a](ea7992a))
* **authz:** export entity id prefix constant from entity instead of
authorization service v1
([#2261](#2261))
([94079a9](94079a9))
* **authz:** subject mapping plugin support for ABAC with actions
([#2223](#2223))
([d08b939](d08b939))
* bulk keycloak provisioning
([#2205](#2205))
([59e4485](59e4485))
* **core:** add otel to opentdf services
([#1858](#1858))
([53a7aa0](53a7aa0))
* **core:** Adds EC withSalt options
([#2126](#2126))
([67b6fb8](67b6fb8))
* **core:** enhance db configuration options
([#2285](#2285))
([ed9ff59](ed9ff59))
* **core:** New Key Index and Manager Plugin SPI
([#2095](#2095))
([eb446fc](eb446fc))
* **core:** support onConfigUpdate hook when registering services
([#1992](#1992))
([366d4dc](366d4dc))
* **core:** v2 ERS with proto updates
([#2210](#2210))
([a161ef8](a161ef8))
* **policy:** actions crud service endpoints and proto validation
([#2037](#2037))
([e933fa9](e933fa9))
* **policy:** actions service RPCs should actually hit storage layer
CRUD ([#2063](#2063))
([da4faf5](da4faf5))
* **policy:** add enhanced standard/custom actions protos
([#2020](#2020))
([bbac53f](bbac53f))
* **policy:** Add platform key indexer.
([#2189](#2189))
([861ef8d](861ef8d))
* **policy:** consume lib/identifier parse function
([#2181](#2181))
([1cef22b](1cef22b))
* **policy:** DSPX-1018 NDR retrieval by FQN support
([#2131](#2131))
([0001041](0001041))
* **policy:** DSPX-1057 registered resource action attribute values (DB
+ Service implementation)
([#2191](#2191))
([6bf1b2e](6bf1b2e))
* **policy:** DSPX-1057 registered resource action attribute values
(protos only) ([#2217](#2217))
([6375596](6375596))
* **policy:** DSPX-893 NDR define crud protos
([#2056](#2056))
([55a5c27](55a5c27))
* **policy:** DSPX-898 NDR database schema
([#2055](#2055))
([2a10a6a](2a10a6a))
* **policy:** DSPX-901 NDR database crud
([#2071](#2071))
([20e0a5f](20e0a5f))
* **policy:** DSPX-902 NDR service crud implementation (2/2)
([#2066](#2066))
([030ad33](030ad33))
* **policy:** DSPX-902 NDR service crud protos only (1/2)
([#2092](#2092))
([24b6cb5](24b6cb5))
* **policy:** Finish resource mapping groups
([#2224](#2224))
([5ff754e](5ff754e))
* **policy:** GetMatchedSubjectMappings should provide value FQN
([#2151](#2151))
([ad80044](ad80044))
* **policy:** key management crud
([#2110](#2110))
([4c3d53d](4c3d53d))
* **policy:** Key management proto
([#2115](#2115))
([561f853](561f853))
* **policy:** Modify get request to search for keys by kasid with keyid.
([#2147](#2147))
([780d2e4](780d2e4))
* **policy:** Restrict KAS deletion when tied to Key
([#2144](#2144))
([4c4ab13](4c4ab13))
* **policy:** Return KAS Key structure
([#2172](#2172))
([7f97b99](7f97b99))
* **policy:** rotate keys rpc
([#2180](#2180))
([0d00743](0d00743))
* **policy:** stored enhanced actions database migration, CRUD queries,
SM updates ([#2040](#2040))
([e6b7c79](e6b7c79))
* **sdk:** Add a KAS allowlist
([#2085](#2085))
([d7cfdf3](d7cfdf3))
* **sdk:** add nanotdf plaintext policy
([#2182](#2182))
([e5c56db](e5c56db))
* **sdk:** Use ConnectRPC in the go client
([#2200](#2200))
([fc34ee6](fc34ee6))


### Bug Fixes

* **core:** access pdp cleanup before actions in ABAC decisioning
([#2123](#2123))
([9b38a3c](9b38a3c))
* **core:** Autobump service
([#2080](#2080))
([006c724](006c724))
* **core:** Autobump service
([#2104](#2104))
([1f72cc7](1f72cc7))
* **core:** Autobump service
([#2108](#2108))
([be5b7d7](be5b7d7))
* **core:** bump to go 1.24 and bump service proto module dependencies
([#2064](#2064))
([94891a0](94891a0))
* **core:** Fix DPoP with grpc-gateway
([#2044](#2044))
([4483ef2](4483ef2))
* **core:** fix service go.mod
([#2141](#2141))
([3b98f6d](3b98f6d))
* **core:** Improves errors when under heavy load
([#2132](#2132))
([4490a14](4490a14))
* **core:** Let legacy KAOs use new trust plugins
([#2218](#2218))
([5aa6916](5aa6916))
* **core:** migrate from mitchellh/mapstructure to go-viper/mapstructure
([#2087](#2087))
([0a3a82e](0a3a82e))
* **core:** update viper to 1.20.1
([#2088](#2088))
([09099e9](09099e9))
* **core:** Updates vulnerable dep go/x/net
([#2072](#2072))
([11c02cd](11c02cd))
* **deps:** bump github.com/creasty/defaults from 1.7.0 to 1.8.0 in
/service ([#2242](#2242))
([86a9b46](86a9b46))
* **deps:** bump github.com/jackc/pgx/v5 from 5.5.5 to 5.7.5 in /service
([#2249](#2249))
([d8f3b67](d8f3b67))
* **deps:** bump the internal group across 1 directory with 2 updates
([#2296](#2296))
([7f92c70](7f92c70))
* **deps:** bump toolchain in /lib/fixtures and /examples to resolve CVE
GO-2025-3563 ([#2061](#2061))
([9c16843](9c16843))
* handle empty private and public key ctx structs
([#2272](#2272))
([f3fc647](f3fc647))
* **policy:** remove predefined rules in actions protos
([#2069](#2069))
([060f059](060f059))
* **policy:** return kas uri on keys for definition, namespace and
values ([#2186](#2186))
([6c55fb8](6c55fb8))
* update key_mode to provide more context
([#2226](#2226))
([44d0805](44d0805))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).

Co-authored-by: opentdf-automation[bot] <149537512+opentdf-automation[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants