Skip to content

feat(policy): add obligation tables #2532

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 34 commits into from
Jul 21, 2025
Merged

feat(policy): add obligation tables #2532

merged 34 commits into from
Jul 21, 2025

Conversation

alkalescent
Copy link
Contributor

Proposed Changes

Checklist

  • I have added or updated unit tests
  • I have added or updated integration tests (if appropriate)
  • I have added or updated documentation

Testing Instructions

@alkalescent alkalescent requested review from a team as code owners July 8, 2025 01:31
@github-actions github-actions bot added comp:db DB component comp:policy Policy Configuration ( attributes, subject mappings, resource mappings, kas registry) size/m labels Jul 8, 2025
gemini-code-assist[bot]
gemini-code-assist bot reviewed Jul 8, 2025
View reviewed changes
Copy link
Contributor

@gemini-code-assist gemini-code-assist bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Summary of Changes

Hello @alkalescent, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces the foundational database schema for managing policy obligations within the system. It establishes the necessary tables and relationships to define, store, and manage various aspects of obligations, including their definitions, values, triggers, and associated actions and attributes. This is a significant step towards enabling more granular and dynamic policy enforcement.

Highlights

  • New Database Tables for Policy Obligations: I've introduced five new database tables to support policy obligations: obligation_definitions, obligation_values_standard, obligation_triggers, obligation_fulfillers, and obligation_action_attribute_values. These tables are designed to store the structure, values, triggers, fulfillers, and action-attribute relationships for policy obligations.
  • Database Schema Definition: The schema for these new tables includes UUID primary keys, foreign key relationships to existing tables like attribute_namespaces, attribute_values, and actions, as well as unique constraints to ensure data integrity. Standard metadata, created_at, and updated_at columns are also added to most tables.
  • SQL Migration Script: A new SQL migration script (20250703000000_obligations.sql) has been added to create these tables. This script also includes helper functions (get_obligation_tables, standardize_table, standardize_tables, drop_tables) to streamline the creation and standardization of these tables, and to facilitate their cleanup during a goose Down operation.
  • Schema ER Diagram Update: The overall database schema ER diagram (schema_erd.md) has been updated to visually represent the newly added obligation tables and their relationships with other existing entities in the database.
Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment @gemini-code-assist Responds in comments when explicitly tagged, both in issue comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist is currently in preview and may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments to provide feedback.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

New tables arise, Policy's rules now take form, Data finds its home.

Footnotes

  1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist[bot]
gemini-code-assist bot reviewed Jul 8, 2025
View reviewed changes
Copy link
Contributor

@gemini-code-assist gemini-code-assist bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces new database tables and relationships to support 'obligations' in the policy service. The SQL migration script has a few issues: incorrect UNIQUE constraints, a missing UNIQUE constraint, and an inefficient step where a column is added and then immediately dropped. Suggestions have been provided to fix these issues.

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Jul 8, 2025

Benchmark results, click to expand

Benchmark authorization.GetDecisions Results:

Metric Value
Approved Decision Requests 1000
Denied Decision Requests 0
Total Time 170.910035ms

Benchmark authorization.v2.GetMultiResourceDecision Results:

Metric Value
Approved Decision Requests 1000
Denied Decision Requests 0
Total Time 87.034912ms

Standard Benchmark Metrics Skipped or Failed

Bulk Benchmark Results

Metric Value
Total Decrypts 100
Successful Decrypts 100
Failed Decrypts 0
Total Time 344.121247ms
Throughput 290.60 requests/second

TDF3 Benchmark Results:

Metric Value
Total Requests 5000
Successful Requests 5000
Failed Requests 0
Concurrent Requests 50
Total Time 36.992554592s
Average Latency 367.719452ms
Throughput 135.16 requests/second

NANOTDF Benchmark Results:

Metric Value
Total Requests 5000
Successful Requests 5000
Failed Requests 0
Concurrent Requests 50
Total Time 25.84330653s
Average Latency 257.470613ms
Throughput 193.47 requests/second

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Jul 8, 2025

Benchmark results, click to expand

Benchmark authorization.GetDecisions Results:

Metric Value
Approved Decision Requests 1000
Denied Decision Requests 0
Total Time 177.621466ms

Benchmark authorization.v2.GetMultiResourceDecision Results:

Metric Value
Approved Decision Requests 1000
Denied Decision Requests 0
Total Time 100.588739ms

Standard Benchmark Metrics Skipped or Failed

Bulk Benchmark Results

Metric Value
Total Decrypts 100
Successful Decrypts 100
Failed Decrypts 0
Total Time 356.316265ms
Throughput 280.65 requests/second

TDF3 Benchmark Results:

Metric Value
Total Requests 5000
Successful Requests 5000
Failed Requests 0
Concurrent Requests 50
Total Time 37.748980112s
Average Latency 375.52978ms
Throughput 132.45 requests/second

NANOTDF Benchmark Results:

Metric Value
Total Requests 5000
Successful Requests 5000
Failed Requests 0
Concurrent Requests 50
Total Time 26.405487573s
Average Latency 263.154432ms
Throughput 189.35 requests/second

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Jul 8, 2025

Benchmark results, click to expand

Benchmark authorization.GetDecisions Results:

Metric Value
Approved Decision Requests 1000
Denied Decision Requests 0
Total Time 169.112206ms

Benchmark authorization.v2.GetMultiResourceDecision Results:

Metric Value
Approved Decision Requests 1000
Denied Decision Requests 0
Total Time 99.703605ms

Standard Benchmark Metrics Skipped or Failed

Bulk Benchmark Results

Metric Value
Total Decrypts 100
Successful Decrypts 100
Failed Decrypts 0
Total Time 353.090325ms
Throughput 283.21 requests/second

TDF3 Benchmark Results:

Metric Value
Total Requests 5000
Successful Requests 5000
Failed Requests 0
Concurrent Requests 50
Total Time 37.525393026s
Average Latency 373.494395ms
Throughput 133.24 requests/second

NANOTDF Benchmark Results:

Metric Value
Total Requests 5000
Successful Requests 5000
Failed Requests 0
Concurrent Requests 50
Total Time 26.436484249s
Average Latency 263.496117ms
Throughput 189.13 requests/second

@jakedoublev
Copy link
Contributor

/gemini review

gemini-code-assist[bot]
gemini-code-assist bot reviewed Jul 9, 2025
View reviewed changes
Copy link
Contributor

@gemini-code-assist gemini-code-assist bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces new database tables and related functions for handling obligations. The changes include a SQL migration script and an ERD diagram for the new tables. A critical issue was identified in the SQL migration script that would cause it to fail, related to an invalid format specifier in a CREATE TRIGGER statement. A medium-severity issue was also found in the ERD documentation, where a relationship pointed to a non-existent table. Suggestions are provided to fix both issues.

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Jul 9, 2025

Benchmark results, click to expand

Benchmark authorization.GetDecisions Results:

Metric Value
Approved Decision Requests 1000
Denied Decision Requests 0
Total Time 173.247877ms

Benchmark authorization.v2.GetMultiResourceDecision Results:

Metric Value
Approved Decision Requests 1000
Denied Decision Requests 0
Total Time 88.81621ms

Standard Benchmark Metrics Skipped or Failed

Bulk Benchmark Results

Metric Value
Total Decrypts 100
Successful Decrypts 100
Failed Decrypts 0
Total Time 338.606592ms
Throughput 295.33 requests/second

TDF3 Benchmark Results:

Metric Value
Total Requests 5000
Successful Requests 5000
Failed Requests 0
Concurrent Requests 50
Total Time 37.299005519s
Average Latency 370.583085ms
Throughput 134.05 requests/second

NANOTDF Benchmark Results:

Metric Value
Total Requests 5000
Successful Requests 5000
Failed Requests 0
Concurrent Requests 50
Total Time 26.384414452s
Average Latency 262.646985ms
Throughput 189.51 requests/second

jakedoublev
jakedoublev previously approved these changes Jul 10, 2025
View reviewed changes
@github-actions GitHub Actions
Copy link
Contributor

Benchmark results, click to expand

Benchmark authorization.GetDecisions Results:

Metric Value
Approved Decision Requests 1000
Denied Decision Requests 0
Total Time 179.512284ms

Benchmark authorization.v2.GetMultiResourceDecision Results:

Metric Value
Approved Decision Requests 1000
Denied Decision Requests 0
Total Time 99.565256ms

Standard Benchmark Metrics Skipped or Failed

Bulk Benchmark Results

Metric Value
Total Decrypts 100
Successful Decrypts 100
Failed Decrypts 0
Total Time 455.325782ms
Throughput 219.62 requests/second

TDF3 Benchmark Results:

Metric Value
Total Requests 5000
Successful Requests 5000
Failed Requests 0
Concurrent Requests 50
Total Time 37.248926835s
Average Latency 370.799235ms
Throughput 134.23 requests/second

NANOTDF Benchmark Results:

Metric Value
Total Requests 5000
Successful Requests 5000
Failed Requests 0
Concurrent Requests 50
Total Time 26.229464044s
Average Latency 261.259347ms
Throughput 190.63 requests/second

@github-actions GitHub Actions
Copy link
Contributor

Benchmark results, click to expand

Benchmark authorization.GetDecisions Results:

Metric Value
Approved Decision Requests 1000
Denied Decision Requests 0
Total Time 190.102208ms

Benchmark authorization.v2.GetMultiResourceDecision Results:

Metric Value
Approved Decision Requests 1000
Denied Decision Requests 0
Total Time 94.426686ms

Standard Benchmark Metrics Skipped or Failed

Bulk Benchmark Results

Metric Value
Total Decrypts 100
Successful Decrypts 100
Failed Decrypts 0
Total Time 345.948508ms
Throughput 289.06 requests/second

TDF3 Benchmark Results:

Metric Value
Total Requests 5000
Successful Requests 5000
Failed Requests 0
Concurrent Requests 50
Total Time 37.112625446s
Average Latency 368.521977ms
Throughput 134.73 requests/second

NANOTDF Benchmark Results:

Metric Value
Total Requests 5000
Successful Requests 5000
Failed Requests 0
Concurrent Requests 50
Total Time 25.53755981s
Average Latency 253.908417ms
Throughput 195.79 requests/second

@github-actions GitHub Actions
Copy link
Contributor

Benchmark results, click to expand

Benchmark authorization.GetDecisions Results:

Metric Value
Approved Decision Requests 1000
Denied Decision Requests 0
Total Time 173.72346ms

Benchmark authorization.v2.GetMultiResourceDecision Results:

Metric Value
Approved Decision Requests 1000
Denied Decision Requests 0
Total Time 103.172885ms

Standard Benchmark Metrics Skipped or Failed

Bulk Benchmark Results

Metric Value
Total Decrypts 100
Successful Decrypts 100
Failed Decrypts 0
Total Time 406.238358ms
Throughput 246.16 requests/second

TDF3 Benchmark Results:

Metric Value
Total Requests 5000
Successful Requests 5000
Failed Requests 0
Concurrent Requests 50
Total Time 37.019827185s
Average Latency 368.702686ms
Throughput 135.06 requests/second

NANOTDF Benchmark Results:

Metric Value
Total Requests 5000
Successful Requests 5000
Failed Requests 0
Concurrent Requests 50
Total Time 25.743259939s
Average Latency 256.066396ms
Throughput 194.23 requests/second

@github-actions GitHub Actions
Copy link
Contributor

Benchmark results, click to expand

Benchmark authorization.GetDecisions Results:

Metric Value
Approved Decision Requests 1000
Denied Decision Requests 0
Total Time 284.102213ms

Benchmark authorization.v2.GetMultiResourceDecision Results:

Metric Value
Approved Decision Requests 1000
Denied Decision Requests 0
Total Time 98.426394ms

Standard Benchmark Metrics Skipped or Failed

Bulk Benchmark Results

Metric Value
Total Decrypts 100
Successful Decrypts 100
Failed Decrypts 0
Total Time 409.326694ms
Throughput 244.30 requests/second

TDF3 Benchmark Results:

Metric Value
Total Requests 5000
Successful Requests 5000
Failed Requests 0
Concurrent Requests 50
Total Time 37.169421291s
Average Latency 369.755452ms
Throughput 134.52 requests/second

NANOTDF Benchmark Results:

Metric Value
Total Requests 5000
Successful Requests 5000
Failed Requests 0
Concurrent Requests 50
Total Time 26.136601429s
Average Latency 260.383164ms
Throughput 191.30 requests/second

ryanulit
ryanulit requested changes Jul 21, 2025
View reviewed changes
Copy link
Contributor

@ryanulit ryanulit left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One last minor fix for consistency and then we're good to merge

@github-actions GitHub Actions
Copy link
Contributor

Benchmark results, click to expand

Benchmark authorization.GetDecisions Results:

Metric Value
Approved Decision Requests 1000
Denied Decision Requests 0
Total Time 171.264173ms

Benchmark authorization.v2.GetMultiResourceDecision Results:

Metric Value
Approved Decision Requests 1000
Denied Decision Requests 0
Total Time 90.155972ms

Standard Benchmark Metrics Skipped or Failed

Bulk Benchmark Results

Metric Value
Total Decrypts 100
Successful Decrypts 100
Failed Decrypts 0
Total Time 343.876188ms
Throughput 290.80 requests/second

TDF3 Benchmark Results:

Metric Value
Total Requests 5000
Successful Requests 5000
Failed Requests 0
Concurrent Requests 50
Total Time 36.143572643s
Average Latency 359.795243ms
Throughput 138.34 requests/second

NANOTDF Benchmark Results:

Metric Value
Total Requests 5000
Successful Requests 5000
Failed Requests 0
Concurrent Requests 50
Total Time 25.89601679s
Average Latency 257.861532ms
Throughput 193.08 requests/second

ryanulit
ryanulit previously approved these changes Jul 21, 2025
View reviewed changes
@policy-bot-opentdf policy-bot-opentdf bot dismissed stale reviews from ryanulit July 21, 2025 19:00

Invalidated by push of 59718e9

@github-actions GitHub Actions
Copy link
Contributor

Benchmark results, click to expand

Benchmark authorization.GetDecisions Results:

Metric Value
Approved Decision Requests 1000
Denied Decision Requests 0
Total Time 173.939905ms

Benchmark authorization.v2.GetMultiResourceDecision Results:

Metric Value
Approved Decision Requests 1000
Denied Decision Requests 0
Total Time 87.28586ms

Standard Benchmark Metrics Skipped or Failed

Bulk Benchmark Results

Metric Value
Total Decrypts 100
Successful Decrypts 100
Failed Decrypts 0
Total Time 337.016447ms
Throughput 296.72 requests/second

TDF3 Benchmark Results:

Metric Value
Total Requests 5000
Successful Requests 5000
Failed Requests 0
Concurrent Requests 50
Total Time 36.998974877s
Average Latency 368.303083ms
Throughput 135.14 requests/second

NANOTDF Benchmark Results:

Metric Value
Total Requests 5000
Successful Requests 5000
Failed Requests 0
Concurrent Requests 50
Total Time 26.358866955s
Average Latency 262.835275ms
Throughput 189.69 requests/second

@ryanulit ryanulit added this pull request to the merge queue Jul 21, 2025
Merged via the queue into main with commit c7d7aa4 Jul 21, 2025
29 checks passed
@ryanulit ryanulit deleted the feature/obligations-tables branch July 21, 2025 19:54
@opentdf-automation opentdf-automation bot mentioned this pull request Jul 21, 2025
Merged
github-merge-queue bot pushed a commit that referenced this pull request Jul 31, 2025
@opentdf-automation
chore(main): release service 0.8.0 (#2487)
b95d4b3 
🤖 I have created a release *beep* *boop*
---


##
[0.8.0](service/v0.7.0...service/v0.8.0)
(2025-07-29)


### Features

* **authz:** RR GetDecision improvements
([#2479](#2479))
([443cedb](443cedb))
* **authz:** sensible request limit upper bounds
([#2526](#2526))
([b3093cc](b3093cc))
* **core:** Add the ability to configure the http server settings
([#2522](#2522))
([b1472df](b1472df))
* **policy:** Add list key mappings rpc.
([#2533](#2533))
([fbc2724](fbc2724))
* **policy:** add obligation protos
([#2579](#2579))
([50882e1](50882e1))
* **policy:** add obligation tables
([#2532](#2532))
([c7d7aa4](c7d7aa4))
* **policy:** Add validation to delete keys
([#2576](#2576))
([cc169d9](cc169d9))
* **policy:** Allow the deletion of a key.
([#2575](#2575))
([82b96f0](82b96f0))
* **policy:** Change return type for delete key proto.
([#2566](#2566))
([c1ae924](c1ae924))
* **policy:** sqlc queries refactor
([#2541](#2541))
([e34680e](e34680e))


### Bug Fixes

* add back grants to listAttributesByDefOrValueFqns
([#2493](#2493))
([2b47095](2b47095))
* **authz:** access pdp should use proto getter
([#2530](#2530))
([f856212](f856212))
* **core:** Allow 521 curve to be used
([#2485](#2485))
([aaf43dc](aaf43dc))
* **core:** resolve 'built-in' typos
([#2548](#2548))
([ccdfa96](ccdfa96))
* **deps:** bump github.com/opentdf/platform/lib/ocrypto from 0.2.0 to
0.3.0 in /service
([#2504](#2504))
([a9cc4dd](a9cc4dd))
* **sdk:** Prefer KID and Algorithm selection from key maps
([#2475](#2475))
([98fd392](98fd392))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).

Co-authored-by: opentdf-automation[bot] <149537512+opentdf-automation[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ryanulit ryanulit ryanulit approved these changes

@jakedoublev jakedoublev jakedoublev left review comments

+1 more reviewer

@gemini-code-assist gemini-code-assist[bot] gemini-code-assist[bot] left review comments

Reviewers whose approvals may not affect merge requirements
Assignees
No one assigned
Labels
comp:db DB component comp:policy Policy Configuration ( attributes, subject mappings, resource mappings, kas registry) size/m
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@alkalescent @jakedoublev @ryanulit