feat(policy): DSPX-901 NDR database crud #2071

ryanulit · 2025-04-15T21:09:50Z

Proposed Changes

add db crud

Checklist

I have added or updated unit tests
I have added or updated integration tests (if appropriate)
I have added or updated documentation

Testing Instructions

github-actions · 2025-04-15T21:13:56Z

Benchmark Statistics

Name	№ Requests	Avg Duration	Min Duration	Max Duration
publickey	3	0.059 ms	0.035 ms	0.091 ms
rewrap-nanotdf	4937	413.544 ms	48.950 ms	883.574 ms
rewrap-tdf3	5001	444.364 ms	47.804 ms	1048.155 ms

Bulk Benchmark Results

Metric	Value
Total Decrypts	100
Successful Decrypts	100
Failed Decrypts	0
Total Time	359.569989ms
Throughput	278.11 requests/second

github-actions · 2025-04-16T16:30:34Z

Benchmark Statistics

Name	№ Requests	Avg Duration	Min Duration	Max Duration
publickey	3	0.070 ms	0.040 ms	0.122 ms
rewrap-nanotdf	5000	402.872 ms	51.783 ms	886.786 ms
rewrap-tdf3	5001	438.629 ms	49.277 ms	1042.218 ms

Bulk Benchmark Results

Metric	Value
Total Decrypts	100
Successful Decrypts	100
Failed Decrypts	0
Total Time	363.975503ms
Throughput	274.74 requests/second

github-actions · 2025-04-16T17:40:46Z

Benchmark Statistics

Name	№ Requests	Avg Duration	Min Duration	Max Duration
publickey	3	0.064 ms	0.044 ms	0.099 ms
rewrap-nanotdf	4541	398.018 ms	29.389 ms	972.835 ms
rewrap-tdf3	5001	436.892 ms	50.968 ms	1154.357 ms

Bulk Benchmark Results

Metric	Value
Total Decrypts	100
Successful Decrypts	100
Failed Decrypts	0
Total Time	361.796613ms
Throughput	276.40 requests/second

github-actions · 2025-04-16T18:19:15Z

Benchmark Statistics

Name	№ Requests	Avg Duration	Min Duration	Max Duration
publickey	3	0.064 ms	0.038 ms	0.101 ms
rewrap-nanotdf	4566	425.870 ms	42.488 ms	876.059 ms
rewrap-tdf3	5001	466.384 ms	37.827 ms	1125.538 ms

Bulk Benchmark Results

Metric	Value
Total Decrypts	100
Successful Decrypts	100
Failed Decrypts	0
Total Time	376.024596ms
Throughput	265.94 requests/second

github-actions · 2025-04-16T18:41:46Z

Benchmark Statistics

Name	№ Requests	Avg Duration	Min Duration	Max Duration
publickey	3	0.067 ms	0.034 ms	0.112 ms
rewrap-nanotdf	5000	438.332 ms	59.017 ms	961.590 ms
rewrap-tdf3	5001	461.231 ms	52.226 ms	1036.152 ms

Bulk Benchmark Results

Metric	Value
Total Decrypts	100
Successful Decrypts	100
Failed Decrypts	0
Total Time	350.317653ms
Throughput	285.46 requests/second

github-actions · 2025-04-16T21:07:30Z

Benchmark Statistics

Name	№ Requests	Avg Duration	Min Duration	Max Duration
publickey	3	0.081 ms	0.056 ms	0.116 ms
rewrap-nanotdf	4650	405.977 ms	26.354 ms	859.934 ms
rewrap-tdf3	5001	436.191 ms	44.340 ms	987.841 ms

Bulk Benchmark Results

Metric	Value
Total Decrypts	100
Successful Decrypts	100
Failed Decrypts	0
Total Time	349.978364ms
Throughput	285.73 requests/second

github-actions · 2025-04-16T21:07:56Z

Benchmark Statistics

Name	№ Requests	Avg Duration	Min Duration	Max Duration
publickey	3	0.066 ms	0.037 ms	0.109 ms
rewrap-nanotdf	4622	437.870 ms	32.686 ms	918.920 ms
rewrap-tdf3	5001	447.429 ms	48.483 ms	1127.537 ms

Bulk Benchmark Results

Metric	Value
Total Decrypts	100
Successful Decrypts	100
Failed Decrypts	0
Total Time	358.588363ms
Throughput	278.87 requests/second

github-actions · 2025-04-16T21:20:40Z

Benchmark Statistics

Name	№ Requests	Avg Duration	Min Duration	Max Duration
publickey	3	0.065 ms	0.047 ms	0.103 ms
rewrap-nanotdf	5000	431.208 ms	60.871 ms	986.980 ms
rewrap-tdf3	5001	462.366 ms	59.450 ms	1074.630 ms

Bulk Benchmark Results

Metric	Value
Total Decrypts	100
Successful Decrypts	100
Failed Decrypts	0
Total Time	358.279662ms
Throughput	279.11 requests/second

github-actions · 2025-04-16T21:49:31Z

Benchmark Statistics

Name	№ Requests	Avg Duration	Min Duration	Max Duration
publickey	3	0.064 ms	0.036 ms	0.100 ms
rewrap-nanotdf	5000	409.157 ms	23.062 ms	1027.140 ms
rewrap-tdf3	5001	444.269 ms	43.996 ms	999.567 ms

Bulk Benchmark Results

Metric	Value
Total Decrypts	100
Successful Decrypts	100
Failed Decrypts	0
Total Time	366.674528ms
Throughput	272.72 requests/second

github-actions · 2025-04-17T14:49:15Z

Benchmark Statistics

Name	№ Requests	Avg Duration	Min Duration	Max Duration
publickey	3	0.063 ms	0.027 ms	0.101 ms
rewrap-nanotdf	4946	409.793 ms	55.347 ms	1027.529 ms
rewrap-tdf3	5001	443.565 ms	49.669 ms	982.769 ms

Bulk Benchmark Results

Metric	Value
Total Decrypts	100
Successful Decrypts	100
Failed Decrypts	0
Total Time	371.457136ms
Throughput	269.21 requests/second

github-actions · 2025-04-17T18:52:15Z

Benchmark Statistics

Name	№ Requests	Avg Duration	Min Duration	Max Duration
publickey	3	0.108 ms	0.061 ms	0.154 ms
rewrap-nanotdf	4788	421.289 ms	71.035 ms	920.511 ms
rewrap-tdf3	5001	446.788 ms	59.235 ms	1083.823 ms

Bulk Benchmark Results

Metric	Value
Total Decrypts	100
Successful Decrypts	100
Failed Decrypts	0
Total Time	383.061631ms
Throughput	261.05 requests/second

github-actions · 2025-04-17T18:52:32Z

Benchmark Statistics

Name	№ Requests	Avg Duration	Min Duration	Max Duration
publickey	3	0.071 ms	0.040 ms	0.108 ms
rewrap-nanotdf	4867	440.390 ms	48.025 ms	1001.353 ms
rewrap-tdf3	5001	459.982 ms	38.393 ms	1125.929 ms

Bulk Benchmark Results

Metric	Value
Total Decrypts	100
Successful Decrypts	100
Failed Decrypts	0
Total Time	356.353696ms
Throughput	280.62 requests/second

github-actions · 2025-04-17T20:15:08Z

Benchmark Statistics

Name	№ Requests	Avg Duration	Min Duration	Max Duration
publickey	3	0.066 ms	0.036 ms	0.113 ms
rewrap-nanotdf	4751	424.322 ms	25.461 ms	959.410 ms
rewrap-tdf3	5001	443.341 ms	52.398 ms	1103.435 ms

Bulk Benchmark Results

Metric	Value
Total Decrypts	100
Successful Decrypts	100
Failed Decrypts	0
Total Time	362.214144ms
Throughput	276.08 requests/second

github-actions · 2025-04-17T20:38:31Z

Benchmark Statistics

Name	№ Requests	Avg Duration	Min Duration	Max Duration
publickey	3	0.077 ms	0.027 ms	0.144 ms
rewrap-nanotdf	4923	435.255 ms	64.466 ms	1023.223 ms
rewrap-tdf3	5001	462.457 ms	43.205 ms	1064.420 ms

Bulk Benchmark Results

Metric	Value
Total Decrypts	100
Successful Decrypts	100
Failed Decrypts	0
Total Time	368.583655ms
Throughput	271.31 requests/second

service/policy/db/registered_resources.go

service/integration/registered_resources_test.go

service/internal/fixtures/policy_fixtures.yaml

github-actions · 2025-04-21T19:22:07Z

Standard Benchmark Skipped or Failed

Bulk Benchmark Results

Metric	Value
Total Decrypts	100
Successful Decrypts	100
Failed Decrypts	0
Total Time	360.423985ms
Throughput	277.45 requests/second

github-actions · 2025-04-21T19:46:01Z

Standard Benchmark Skipped or Failed

Bulk Benchmark Results

Metric	Value
Total Decrypts	100
Successful Decrypts	100
Failed Decrypts	0
Total Time	363.88998ms
Throughput	274.81 requests/second

github-actions · 2025-04-21T21:10:40Z

Standard Benchmark Skipped or Failed

Bulk Benchmark Results

Metric	Value
Total Decrypts	100
Successful Decrypts	100
Failed Decrypts	0
Total Time	370.917762ms
Throughput	269.60 requests/second

github-actions · 2025-04-22T17:42:32Z

Standard Benchmark Skipped or Failed

Bulk Benchmark Results

Metric	Value
Total Decrypts	100
Successful Decrypts	100
Failed Decrypts	0
Total Time	360.038095ms
Throughput	277.75 requests/second

service/integration/registered_resources_test.go

github-actions · 2025-04-22T17:51:27Z

Standard Benchmark Skipped or Failed

Bulk Benchmark Results

Metric	Value
Total Decrypts	100
Successful Decrypts	100
Failed Decrypts	0
Total Time	365.946764ms
Throughput	273.26 requests/second

### Proposed Changes * add db crud ### Checklist - [ ] I have added or updated unit tests - [ ] I have added or updated integration tests (if appropriate) - [ ] I have added or updated documentation ### Testing Instructions

🤖 I have created a release *beep* *boop* --- ## [0.5.3](service/v0.5.2...service/v0.5.3) (2025-05-22) ### Features * **authz:** authz v2 versioning implementation ([#2173](#2173)) ([557fc21](557fc21)) * **authz:** authz v2, ers v2 protos and gencode for ABAC with actions & registered resource ([#2124](#2124)) ([ea7992a](ea7992a)) * **authz:** export entity id prefix constant from entity instead of authorization service v1 ([#2261](#2261)) ([94079a9](94079a9)) * **authz:** subject mapping plugin support for ABAC with actions ([#2223](#2223)) ([d08b939](d08b939)) * bulk keycloak provisioning ([#2205](#2205)) ([59e4485](59e4485)) * **core:** add otel to opentdf services ([#1858](#1858)) ([53a7aa0](53a7aa0)) * **core:** Adds EC withSalt options ([#2126](#2126)) ([67b6fb8](67b6fb8)) * **core:** enhance db configuration options ([#2285](#2285)) ([ed9ff59](ed9ff59)) * **core:** New Key Index and Manager Plugin SPI ([#2095](#2095)) ([eb446fc](eb446fc)) * **core:** support onConfigUpdate hook when registering services ([#1992](#1992)) ([366d4dc](366d4dc)) * **core:** v2 ERS with proto updates ([#2210](#2210)) ([a161ef8](a161ef8)) * **policy:** actions crud service endpoints and proto validation ([#2037](#2037)) ([e933fa9](e933fa9)) * **policy:** actions service RPCs should actually hit storage layer CRUD ([#2063](#2063)) ([da4faf5](da4faf5)) * **policy:** add enhanced standard/custom actions protos ([#2020](#2020)) ([bbac53f](bbac53f)) * **policy:** Add platform key indexer. ([#2189](#2189)) ([861ef8d](861ef8d)) * **policy:** consume lib/identifier parse function ([#2181](#2181)) ([1cef22b](1cef22b)) * **policy:** DSPX-1018 NDR retrieval by FQN support ([#2131](#2131)) ([0001041](0001041)) * **policy:** DSPX-1057 registered resource action attribute values (DB + Service implementation) ([#2191](#2191)) ([6bf1b2e](6bf1b2e)) * **policy:** DSPX-1057 registered resource action attribute values (protos only) ([#2217](#2217)) ([6375596](6375596)) * **policy:** DSPX-893 NDR define crud protos ([#2056](#2056)) ([55a5c27](55a5c27)) * **policy:** DSPX-898 NDR database schema ([#2055](#2055)) ([2a10a6a](2a10a6a)) * **policy:** DSPX-901 NDR database crud ([#2071](#2071)) ([20e0a5f](20e0a5f)) * **policy:** DSPX-902 NDR service crud implementation (2/2) ([#2066](#2066)) ([030ad33](030ad33)) * **policy:** DSPX-902 NDR service crud protos only (1/2) ([#2092](#2092)) ([24b6cb5](24b6cb5)) * **policy:** Finish resource mapping groups ([#2224](#2224)) ([5ff754e](5ff754e)) * **policy:** GetMatchedSubjectMappings should provide value FQN ([#2151](#2151)) ([ad80044](ad80044)) * **policy:** key management crud ([#2110](#2110)) ([4c3d53d](4c3d53d)) * **policy:** Key management proto ([#2115](#2115)) ([561f853](561f853)) * **policy:** Modify get request to search for keys by kasid with keyid. ([#2147](#2147)) ([780d2e4](780d2e4)) * **policy:** Restrict KAS deletion when tied to Key ([#2144](#2144)) ([4c4ab13](4c4ab13)) * **policy:** Return KAS Key structure ([#2172](#2172)) ([7f97b99](7f97b99)) * **policy:** rotate keys rpc ([#2180](#2180)) ([0d00743](0d00743)) * **policy:** stored enhanced actions database migration, CRUD queries, SM updates ([#2040](#2040)) ([e6b7c79](e6b7c79)) * **sdk:** Add a KAS allowlist ([#2085](#2085)) ([d7cfdf3](d7cfdf3)) * **sdk:** add nanotdf plaintext policy ([#2182](#2182)) ([e5c56db](e5c56db)) * **sdk:** Use ConnectRPC in the go client ([#2200](#2200)) ([fc34ee6](fc34ee6)) ### Bug Fixes * **core:** access pdp cleanup before actions in ABAC decisioning ([#2123](#2123)) ([9b38a3c](9b38a3c)) * **core:** Autobump service ([#2080](#2080)) ([006c724](006c724)) * **core:** Autobump service ([#2104](#2104)) ([1f72cc7](1f72cc7)) * **core:** Autobump service ([#2108](#2108)) ([be5b7d7](be5b7d7)) * **core:** bump to go 1.24 and bump service proto module dependencies ([#2064](#2064)) ([94891a0](94891a0)) * **core:** Fix DPoP with grpc-gateway ([#2044](#2044)) ([4483ef2](4483ef2)) * **core:** fix service go.mod ([#2141](#2141)) ([3b98f6d](3b98f6d)) * **core:** Improves errors when under heavy load ([#2132](#2132)) ([4490a14](4490a14)) * **core:** Let legacy KAOs use new trust plugins ([#2218](#2218)) ([5aa6916](5aa6916)) * **core:** migrate from mitchellh/mapstructure to go-viper/mapstructure ([#2087](#2087)) ([0a3a82e](0a3a82e)) * **core:** update viper to 1.20.1 ([#2088](#2088)) ([09099e9](09099e9)) * **core:** Updates vulnerable dep go/x/net ([#2072](#2072)) ([11c02cd](11c02cd)) * **deps:** bump github.com/creasty/defaults from 1.7.0 to 1.8.0 in /service ([#2242](#2242)) ([86a9b46](86a9b46)) * **deps:** bump github.com/jackc/pgx/v5 from 5.5.5 to 5.7.5 in /service ([#2249](#2249)) ([d8f3b67](d8f3b67)) * **deps:** bump the internal group across 1 directory with 2 updates ([#2296](#2296)) ([7f92c70](7f92c70)) * **deps:** bump toolchain in /lib/fixtures and /examples to resolve CVE GO-2025-3563 ([#2061](#2061)) ([9c16843](9c16843)) * handle empty private and public key ctx structs ([#2272](#2272)) ([f3fc647](f3fc647)) * **policy:** remove predefined rules in actions protos ([#2069](#2069)) ([060f059](060f059)) * **policy:** return kas uri on keys for definition, namespace and values ([#2186](#2186)) ([6c55fb8](6c55fb8)) * update key_mode to provide more context ([#2226](#2226)) ([44d0805](44d0805)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). Co-authored-by: opentdf-automation[bot] <149537512+opentdf-automation[bot]@users.noreply.github.com>

ryanulit added 3 commits April 15, 2025 09:03

add protos and generate

9592e69

fix missed rename

d47cdc4

add reg res db client methods except list and initial test setup

4c57dfa

add list sql queries and client methods

62ddb7a

add integration tests for create db client methods

ba348b5

Merge branch 'main' into DSPX-901-ndr-database-crud

ff8596e

add delete integration tests

093c19d

ryanulit added 2 commits April 16, 2025 15:02

add update integration tests

5dd05d2

Merge branch 'main' into DSPX-901-ndr-database-crud

e13fc7d

fix proto lint errors

569d0fb

more lint fixes

1f8bd21

Merge branch 'main' into DSPX-901-ndr-database-crud

dd571fb

ryanulit added 2 commits April 17, 2025 12:48

add get integration tests

f1101fc

Merge branch 'main' into DSPX-901-ndr-database-crud

c73118a

Merge branch 'main' into DSPX-901-ndr-database-crud

56d6e5a

lint fix

9647b24

jakedoublev reviewed Apr 21, 2025

View reviewed changes

service/policy/db/registered_resources.go Show resolved Hide resolved

service/integration/registered_resources_test.go Show resolved Hide resolved

jakedoublev reviewed Apr 21, 2025

View reviewed changes

service/integration/registered_resources_test.go Show resolved Hide resolved

jakedoublev reviewed Apr 21, 2025

View reviewed changes

service/internal/fixtures/policy_fixtures.yaml Outdated Show resolved Hide resolved

jakedoublev previously approved these changes Apr 21, 2025

View reviewed changes

PR change requests

6e818b7

ryanulit dismissed jakedoublev’s stale review via 6e818b7 April 21, 2025 19:18

Merge branch 'main' into DSPX-901-ndr-database-crud

40e0bb3

jakedoublev previously approved these changes Apr 21, 2025

View reviewed changes

fix flaky tests

876559f

ryanulit dismissed jakedoublev’s stale review via 876559f April 21, 2025 21:06

refactor get methods to use request instead of any

8373be9

jakedoublev previously approved these changes Apr 22, 2025

View reviewed changes

service/integration/registered_resources_test.go Outdated Show resolved Hide resolved

remove stale comment

364fa07

ryanulit dismissed jakedoublev’s stale review via 364fa07 April 22, 2025 17:47

jakedoublev approved these changes Apr 22, 2025

View reviewed changes

ryanulit enabled auto-merge April 22, 2025 17:49

ryanulit added this pull request to the merge queue Apr 22, 2025

github-merge-queue bot removed this pull request from the merge queue due to failed status checks Apr 22, 2025

ryanulit added this pull request to the merge queue Apr 22, 2025

Merged via the queue into main with commit 20e0a5f Apr 22, 2025
23 of 24 checks passed

ryanulit deleted the DSPX-901-ndr-database-crud branch April 22, 2025 18:21

opentdf-automation bot mentioned this pull request Apr 21, 2025

chore(main): release service 0.5.3 #2046

Merged

feat(policy): DSPX-901 NDR database crud #2071

feat(policy): DSPX-901 NDR database crud #2071

Uh oh!

Conversation

ryanulit commented Apr 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed Changes

Checklist

Testing Instructions

Uh oh!

github-actions bot commented Apr 15, 2025

Benchmark Statistics

Bulk Benchmark Results

Uh oh!

github-actions bot commented Apr 16, 2025

Benchmark Statistics

Bulk Benchmark Results

Uh oh!

github-actions bot commented Apr 16, 2025

Benchmark Statistics

Bulk Benchmark Results

Uh oh!

github-actions bot commented Apr 16, 2025

Benchmark Statistics

Bulk Benchmark Results

Uh oh!

github-actions bot commented Apr 16, 2025

Benchmark Statistics

Bulk Benchmark Results

Uh oh!

github-actions bot commented Apr 16, 2025

Benchmark Statistics

Bulk Benchmark Results

Uh oh!

github-actions bot commented Apr 16, 2025

Benchmark Statistics

Bulk Benchmark Results

Uh oh!

github-actions bot commented Apr 16, 2025

Benchmark Statistics

Bulk Benchmark Results

Uh oh!

github-actions bot commented Apr 16, 2025

Benchmark Statistics

Bulk Benchmark Results

Uh oh!

github-actions bot commented Apr 17, 2025

Benchmark Statistics

Bulk Benchmark Results

Uh oh!

github-actions bot commented Apr 17, 2025

Benchmark Statistics

Bulk Benchmark Results

Uh oh!

github-actions bot commented Apr 17, 2025

Benchmark Statistics

Bulk Benchmark Results

Uh oh!

github-actions bot commented Apr 17, 2025

Benchmark Statistics

Bulk Benchmark Results

Uh oh!

github-actions bot commented Apr 17, 2025

Benchmark Statistics

Bulk Benchmark Results

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Apr 21, 2025

Standard Benchmark Skipped or Failed

Bulk Benchmark Results

Uh oh!

github-actions bot commented Apr 21, 2025

Standard Benchmark Skipped or Failed

Bulk Benchmark Results

Uh oh!

github-actions bot commented Apr 21, 2025

Standard Benchmark Skipped or Failed

ryanulit commented Apr 15, 2025 •

edited

Loading