feat: bulk keycloak provisioning

[pflynn-virtru]

Introduce Support for Multiple Users and Clients in Keycloak Setup

Overview

This PR enhances the Keycloak setup functionality by introducing support for creating multiple users and clients dynamically. It modifies the SetupCustomKeycloak function and updates the Keycloak data structure to allow specifying a count for users and clients, ensuring scalability and flexibility in managing Keycloak resources.

- user
  copies: 100
- client
  copies: 10

results in an additional 100 users with -000 -- -999 prepended

Key Changes

1. Updated RealmToCreate Structure:

   • Replaced Users field type from []gocloak.User to []User.
   • Introduced a new User struct that extends gocloak.User with an additional Count field.
   • Added a Count field to the Client struct to allow creating multiple clients.

2. Dynamic User and Client Creation:

   • Modified the SetupCustomKeycloak function:
     • Added logic to iterate and create multiple users and clients based on the provided Count value.
     • Implemented padding for generated user IDs and usernames, as well as client IDs, to ensure proper formatting.

3. Mathematical Calculations:

   • Introduced math package for calculating the number of digits required for padding user and client identifiers.

4. Configuration Changes:

   • Updated keycloak_data.yaml to include the new count field for clients and users under realms.

5. Code Annotations:

   • Added a //nolint:nestif comment to suppress linting errors for nested if-statements in the client creation logic.

Code Example

• Dynamic Client Creation:

  numDigits := int(math.Log10(float64(customClient.Count-1))) + 1
  padFormat := fmt.Sprintf("%%s-%%%dd", numDigits)
  for i := 0; i < customClient.Count; i++ {
      customClient.Client.ClientID = gocloak.StringP(fmt.Sprintf(padFormat, customClient.Client.ClientID, i))
      _, err = createClient(ctx, client, token, &kcConnectParams, customClient.Client, realmRoles, clientRoleMap)
      if err != nil {
          return err
      }
  }

• Dynamic User Creation:

  numDigits := int(math.Log10(float64(customUser.Count-1))) + 1
  padFormat := fmt.Sprintf("%%s-%%%dd", numDigits)
  for i := 0; i < customUser.Count; i++ {
      customUser.User.ID = gocloak.StringP(fmt.Sprintf(padFormat, customUser.User.ID, i))
      customUser.User.Username = gocloak.StringP(fmt.Sprintf(padFormat, customUser.User.Username, i))
      _, err = createUser(ctx, client, token, &kcConnectParams, customUser.User)
      if err != nil {
          return err
      }
  }

Benefits

• Scalability: Enables creating multiple users and clients programmatically, reducing manual configuration overhead.
• Flexibility: Simplifies Keycloak setup for scenarios requiring a large number of users or clients.
• Improved Configuration: Introduces a more streamlined and dynamic approach to managing Keycloak data through YAML configuration.

Files Modified

1. lib/fixtures/keycloak.go:
   • Updated structures and logic for user and client creation.
2. service/cmd/keycloak_data.yaml:
   • Added count fields to sample configuration.

Testing

• Verified the creation of multiple users and clients with correct padding using updated YAML configuration.
• Confirmed backward compatibility for existing setups without the count field.

Notes

• The count field defaults to 1 if not provided.
• Linting exceptions were added where necessary to maintain readability and functionality.

Checklist

• I have added or updated unit tests
• I have added or updated integration tests (if appropriate)
• I have added or updated documentation

Testing Instructions

[github-actions]

Benchmark results, click to expand

Benchmark Results:

Metric	Value
Approved Decision Requests	5000
Denied Decision Requests	0
Total Time	465.300044ms

Bulk Benchmark Results

Metric	Value
Total Decrypts	100
Successful Decrypts	100
Failed Decrypts	0
Total Time	363.650071ms
Throughput	274.99 requests/second

TDF3 Benchmark Results:

Metric	Value
Total Requests	5000
Successful Requests	5000
Failed Requests	0
Concurrent Requests	50
Total Time	1m16.474132939s
Average Latency	761.585982ms
Throughput	65.38 requests/second

NANOTDF Benchmark Results:

Metric	Value
Total Requests	5000
Successful Requests	4942
Failed Requests	58
Concurrent Requests	50
Total Time	1m7.827147235s
Average Latency	674.308843ms
Throughput	72.86 requests/second

Error Summary:

Error Message	Occurrences
ReadNanoTDF error: getNanoRewrapKey: rewrapError: internal: internal error rpc error: code = Internal desc = could not perform access	58 occurrences

Standard Benchmark Metrics Skipped or Failed

[github-actions]

Benchmark results, click to expand

Benchmark Results:

Metric	Value
Error	rpc error: code = Internal desc = resource retrieval failed
Total Time	62.16917ms

Bulk Benchmark Results

Metric	Value
Total Decrypts	100
Successful Decrypts	0
Failed Decrypts	100
Total Time	167.156086ms
Throughput	0.00 requests/second

Error Summary

Error Message	Occurrences
splitKey.unable to reconstruct split key: map[{http://localhost:8080 }:kao unwrap failed for split {http://localhost:8080 }: internal: internal error
rpc error: code = Internal desc = could not perform access]
kao unwrap failed for split {http://localhost:8080 }: internal: internal error
rpc error: code = Internal desc = could not perform access	100 occurrences

TDF3 Benchmark Results:

Metric	Value
Total Requests	5000
Successful Requests	0
Failed Requests	5000
Concurrent Requests	50
Total Time	31.336268095s
Throughput	0.00 requests/second

Error Summary:

Error Message	Occurrences
read error: reader.WriteTo failed: splitKey.unable to reconstruct split key: map[{http://localhost:8080 }:kao unwrap failed for split {http://localhost:8080 }: internal: internal error rpc error: code = Internal desc = could not perform access] kao unwrap failed for split {http://localhost:8080 }: internal: internal error rpc error: code = Internal desc = could not perform access	5000 occurrences

NANOTDF Benchmark Results:

Metric	Value
Total Requests	5000
Successful Requests	0
Failed Requests	5000
Concurrent Requests	50
Total Time	24.145925149s
Throughput	0.00 requests/second

Error Summary:

Error Message	Occurrences
ReadNanoTDF error: getNanoRewrapKey: rewrapError: internal: internal error rpc error: code = Internal desc = could not perform access	5000 occurrences

Standard Benchmark Metrics Skipped or Failed

[github-actions]

Benchmark results, click to expand

Benchmark Results:

Metric	Value
Error	rpc error: code = Internal desc = resource retrieval failed
Total Time	67.627679ms

Bulk Benchmark Results

Metric	Value
Total Decrypts	100
Successful Decrypts	0
Failed Decrypts	100
Total Time	164.676037ms
Throughput	0.00 requests/second

Error Summary

Error Message	Occurrences
splitKey.unable to reconstruct split key: map[{http://localhost:8080 }:kao unwrap failed for split {http://localhost:8080 }: internal: internal error
rpc error: code = Internal desc = could not perform access]
kao unwrap failed for split {http://localhost:8080 }: internal: internal error
rpc error: code = Internal desc = could not perform access	100 occurrences

TDF3 Benchmark Results:

Metric	Value
Total Requests	5000
Successful Requests	0
Failed Requests	5000
Concurrent Requests	50
Total Time	35.194080888s
Throughput	0.00 requests/second

Error Summary:

Error Message	Occurrences
read error: reader.WriteTo failed: splitKey.unable to reconstruct split key: map[{http://localhost:8080 }:kao unwrap failed for split {http://localhost:8080 }: internal: internal error rpc error: code = Internal desc = could not perform access] kao unwrap failed for split {http://localhost:8080 }: internal: internal error rpc error: code = Internal desc = could not perform access	5000 occurrences

NANOTDF Benchmark Results:

Metric	Value
Total Requests	5000
Successful Requests	0
Failed Requests	5000
Concurrent Requests	50
Total Time	26.498638549s
Throughput	0.00 requests/second

Error Summary:

Error Message	Occurrences
ReadNanoTDF error: getNanoRewrapKey: rewrapError: internal: internal error rpc error: code = Internal desc = could not perform access	5000 occurrences

Standard Benchmark Metrics Skipped or Failed

[github-actions]

Benchmark results, click to expand

Benchmark Results:

Metric	Value
Approved Decision Requests	5000
Denied Decision Requests	0
Total Time	468.994904ms

Bulk Benchmark Results

Metric	Value
Total Decrypts	100
Successful Decrypts	100
Failed Decrypts	0
Total Time	353.420283ms
Throughput	282.95 requests/second

TDF3 Benchmark Results:

Metric	Value
Total Requests	5000
Successful Requests	5000
Failed Requests	0
Concurrent Requests	50
Total Time	1m15.348762392s
Average Latency	749.742508ms
Throughput	66.36 requests/second

NANOTDF Benchmark Results:

Metric	Value
Total Requests	5000
Successful Requests	4948
Failed Requests	52
Concurrent Requests	50
Total Time	1m7.712678499s
Average Latency	671.024739ms
Throughput	73.07 requests/second

Error Summary:

Error Message	Occurrences
ReadNanoTDF error: getNanoRewrapKey: rewrapError: internal: internal error rpc error: code = Internal desc = could not perform access	52 occurrences

Standard Benchmark Metrics Skipped or Failed

[github-actions]

Benchmark results, click to expand

Benchmark Results:

Metric	Value
Approved Decision Requests	5000
Denied Decision Requests	0
Total Time	464.27105ms

Bulk Benchmark Results

Metric	Value
Total Decrypts	100
Successful Decrypts	100
Failed Decrypts	0
Total Time	373.622957ms
Throughput	267.65 requests/second

TDF3 Benchmark Results:

Metric	Value
Total Requests	5000
Successful Requests	5000
Failed Requests	0
Concurrent Requests	50
Total Time	1m21.85720147s
Average Latency	816.165999ms
Throughput	61.08 requests/second

NANOTDF Benchmark Results:

Metric	Value
Total Requests	5000
Successful Requests	4947
Failed Requests	53
Concurrent Requests	50
Total Time	1m12.200883653s
Average Latency	717.589888ms
Throughput	68.52 requests/second

Error Summary:

Error Message	Occurrences
ReadNanoTDF error: getNanoRewrapKey: rewrapError: internal: internal error rpc error: code = Internal desc = could not perform access	53 occurrences

Standard Benchmark Metrics Skipped or Failed

[jrschumacher]

Backwards compatibility change with custom keycloak yaml files

[github-actions]

Benchmark results, click to expand

Benchmark Results:

Metric	Value
Approved Decision Requests	5000
Denied Decision Requests	0
Total Time	449.52195ms

Bulk Benchmark Results

Metric	Value
Total Decrypts	100
Successful Decrypts	100
Failed Decrypts	0
Total Time	363.529656ms
Throughput	275.08 requests/second

TDF3 Benchmark Results:

Metric	Value
Total Requests	5000
Successful Requests	5000
Failed Requests	0
Concurrent Requests	50
Total Time	1m17.385616493s
Average Latency	771.108156ms
Throughput	64.61 requests/second

NANOTDF Benchmark Results:

Metric	Value
Total Requests	5000
Successful Requests	4975
Failed Requests	25
Concurrent Requests	50
Total Time	1m8.886585135s
Average Latency	687.074798ms
Throughput	72.22 requests/second

Error Summary:

Error Message	Occurrences
ReadNanoTDF error: getNanoRewrapKey: rewrapError: internal: internal error rpc error: code = Internal desc = could not perform access	25 occurrences

Standard Benchmark Metrics Skipped or Failed