[Hitless Upgrades] Zero-server-side configuration with client-side opt-in #3380

ggivo · 2025-08-04T12:45:43Z

Client-Side Opt-in

Introduced CLIENT MAINT_NOTIFICATIONS command sent during handshake when maintenance events are enabled through MaintenanceEventsOptions:

CLIENT MAINT_NOTIFICATIONS <ON | OFF> [moving-endpoint-type <internal-ip | internal-fdqn | external-ip | external-fdqn | none>

Note: none option is not exposed currently since it is not yet supported by the client

Configuration Options:

Disabled (default): No maintenance notifications
Enabled - if Maintenance events are enabled, requested address type on rebind is configurable via
- Option 1: Auto-resolve: Automatic address type selection based on connection ( bound socket IP/isSSL)
  - if existing connection is SSL enabled request FQDN, otherwise request IP
  - if exisitng connection is bound to reserved private IP -> reaquest internal address, otherwise request external address
- Option 2: Fixed type: Enforced specific address type

Updated Maintenance Notification Format

Maintenance notifications updated to match latest Redis server message format with newly introduced sequence parameter:

MOVING - endpoint relocation notifications
MIGRATING / MIGRATED - shard migration events
FAILING_OVER / FAILED_OVER - failover events

Format Changes:

All messages now include a sequence number for request correlation
FAILED_OVER/MIGRATED extended with additional time parameter (temporary until removed from server)
Propagate shard_ids metadata to event handlers for debugging/logging for MIGRATING / MIGRATED and FAILING_OVER / FAILED_OVER

#CAE-1079

You have read the contribution guidelines.
You have created a feature request first to discuss your contribution intent. Please reference the feature request ticket number in the pull request.
You applied code formatting rules using the mvn formatter:format target. Don’t submit any formatting related changes.
You submit test cases (unit or integration tests) that back your changes.

Copilot

Pull Request Overview

This PR introduces zero-server-side configuration for hitless upgrades by implementing client-side opt-in for maintenance notifications. The client can now request maintenance events during the handshake process and configure the preferred address type for endpoint rebinding.

Replaced boolean maintenance events flag with comprehensive MaintenanceEventsOptions configuration
Updated maintenance notification message format to include sequence numbers and shard metadata
Added client-side address type resolution logic for automatic endpoint type selection

Reviewed Changes

Copilot reviewed 20 out of 20 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
MaintenanceEventsOptions.java	New configuration class providing maintenance events options with address type resolution
RedisHandshake.java	Enhanced handshake to send CLIENT MAINT_NOTIFICATIONS command when maintenance events are enabled
MaintenanceAwareConnectionWatchdog.java	Updated to parse new maintenance message format with sequence numbers and shard metadata
NetUtils.java	New utility class for detecting private IP addresses to support address type auto-resolution
ClientOptions.java	Replaced boolean maintenance events flag with MaintenanceEventsOptions configuration
Multiple test files	Updated tests to reflect new maintenance message format and configuration options

Comments suppressed due to low confidence (3)

src/main/java/io/lettuce/core/protocol/MaintenanceAwareConnectionWatchdog.java

A client can tell the server if it wants to receive maintenance push notifications via the following command: CLIENT MAINT_NOTIFICATIONS <ON | OFF> [parameter value parameter value ...]

- MIGRATING <seq_number> <time> <shard_id-s>: A shard migration is going to start within <time> seconds. - MIGRATED <seq_number> <shard_id-s>: A shard migration ended. - FAILING_OVER <seq_number> <time> <shard_id-s>: A shard failover of a healthy shard started. - FAILED_OVER <seq_number> <shard_id-s>: A shard failover of a healthy shard ended. - MOVING <seq_number> <time> <endpoint>: A specific endpoint is going to move to another node within <time> seconds

update is private reserver check

- MIGRATING <seq_number> <time> <shard_id-s>: A shard migration is going to start within <time> seconds. - MIGRATED <seq_number> <shard_id-s>: A shard migration ended. - FAILING_OVER <seq_number> <time> <shard_id-s>: A shard failover of a healthy shard started. - FAILED_OVER <seq_number> <shard_id-s>: A shard failover of a healthy shard ended. - MOVING <seq_number> <time> <endpoint>: A specific endpoint is going to move to another node within <time> seconds

tishun

Awesome, very elegant solution!

src/main/java/io/lettuce/core/internal/NetUtils.java

src/main/java/io/lettuce/core/MaintenanceEventsOptions.java

Co-authored-by: Tihomir Krasimirov Mateev <[email protected]>

kiryazovi-redis · 2025-08-07T08:53:57Z

Thanks for doing this so proactively.
I will take care to have my tests comply with it.

…t-in (#3380) * Support for Client-side opt-in A client can tell the server if it wants to receive maintenance push notifications via the following command: CLIENT MAINT_NOTIFICATIONS <ON | OFF> [parameter value parameter value ...] * update maintenance events to latest format - MIGRATING <seq_number> <time> <shard_id-s>: A shard migration is going to start within <time> seconds. - MIGRATED <seq_number> <shard_id-s>: A shard migration ended. - FAILING_OVER <seq_number> <time> <shard_id-s>: A shard failover of a healthy shard started. - FAILED_OVER <seq_number> <shard_id-s>: A shard failover of a healthy shard ended. - MOVING <seq_number> <time> <endpoint>: A specific endpoint is going to move to another node within <time> seconds * clean up * Update FAILED_OVER & MIGRATED to include additional time field * update is private reserver check & add unit tests update is private reserver check * add unit tests for handshake with enabled maintenance events * add missing copyrights/docs * format * address review comments * Revert address after rebind operation expires * Update event's validation spec - MIGRATING <seq_number> <time> <shard_id-s>: A shard migration is going to start within <time> seconds. - MIGRATED <seq_number> <shard_id-s>: A shard migration ended. - FAILING_OVER <seq_number> <time> <shard_id-s>: A shard failover of a healthy shard started. - FAILED_OVER <seq_number> <shard_id-s>: A shard failover of a healthy shard ended. - MOVING <seq_number> <time> <endpoint>: A specific endpoint is going to move to another node within <time> seconds * rebase * format after rebase * Apply suggestions from code review Co-authored-by: Tihomir Krasimirov Mateev <[email protected]> * javadoc updated * Update src/main/java/io/lettuce/core/internal/NetUtils.java Co-authored-by: Tihomir Krasimirov Mateev <[email protected]> --------- Co-authored-by: Tihomir Krasimirov Mateev <[email protected]>

* [Hitless Upgrades] React to maintenance events (#3345) * v0.1 * Simple reconnect now working * Bind address from message is now considered * Self-register the handler * Format code * Filter push messages in a more stable way * (very hacky) Relax comand expire timers globbaly * Configure if timeout relaxing should be applied * Proper way to close channel * Configure the timneout relaxing * Sequential handover implemented * Did not address formatting * Prolong the rebind windwow for relaxed tiemouts * PubSub no longer required; CommandExpiryWriter is now channel aware; Polishing * Use the new MOVING push message from the RE server * Unit test was not chaining delgates in the same way that the RedisClient/RedisClusterClient was * Fix REBIND message validation * Fixed the expiry mechanism * Polishing * Fix NPE. Seems like AttributeMap.attr is not accurate and actually return's null causing some unit test failures. * Add support for MIGRATING/MIGRATED message handling in command expiry This commit adds the ability to listen for MIGRATING and MIGRATED messages and trigger extended command expiry timeouts during Redis shard migration. Key changes: - Enhanced RebindAwareConnectionWatchdog to detect MIGRATING/MIGRATED messages - RebindAwareExpiryWriter to trigger timeout relaxation whenever MIGRATING message is received This feature allows commands to have relaxed timeouts during shard migration operations, preventing unnecessary timeouts when Redis is temporarily busy with migration tasks. * formating * Fix Disabling relaxTimeouts after upgrade can interfere with an ongoing one from re-bind * Additional fix for timeout relaxing disabled * Fix push message listener registered multiple times after rebind. * Fix: Report correct command timeout when relaxTimeout is configured * Disable relaxedTimeout after configured grace period - Introduce a delay before disabling relaxedTimeout - Grace period duration is provided via push notification * Code clean up - Remove reading from pub/sub chanel and relay only on push notifications * Add FAILING_OVER/FAILED_OVER * Polishing : Rename components to use the word 'maintenace' --------- Co-authored-by: Igor Malinovskiy <[email protected]> Co-authored-by: ggivo <[email protected]> # Conflicts: # src/main/java/io/lettuce/core/ClientOptions.java * [Hitless upgrades] Add unit tests for the newly introduced classes #3355 (#3356) * Unit tests for the maintanence aware classes * Did not format properly * Proper license * Cae 633 add functional tests notifications (#3357) - excluding JsonExample.java * Cae 1130 timeout tests (#3377) * initial WIP, with lots of debugging, and some non-working tests * debug * more attemtps at debugging * Refactor: Move cluster state management methods from MaintenanceNotificationTest to RedisEnterpriseConfig - Moved refreshClusterConfig, recordOriginalClusterState, and restoreOriginalClusterState methods - Updated call sites in MaintenanceNotificationTest to use RedisEnterpriseConfig methods - Added required imports and static variables to RedisEnterpriseConfig - Maintained existing functionality while improving code organization Improvements to RelaxedTimeoutConfigurationTest: - Simplified traffic generation logic by removing complex multi-phase testing - Streamlined BLPOP command execution with better timeout detection - Added relaxed timeout detection and recording during maintenance events - Improved logging and error handling for timeout analysis - Enhanced test assertions to focus on relaxed timeout detection rather than success counts - Added MOVING operation duration tracking for better test analysis * Improve test reliability and cleanup: Add @AfterEach cleanup, enhance endpoint tracking, and improve logging * add un-relaxed tests. will investigate further why they got broken at some point via diff * CAE-1130: Update timeout configuration test and watchdog implementation * Reset MaintenanceAwareConnectionWatchdog.java and log4j2-test.xml to upstream versions * Clean up debug info and outdated comments from timeout tests - Remove large debug block with reflection-based debugging in RelaxedTimeoutConfigurationTest - Simplify excessive debug logging and verbose markers (*** and ===) - Clean up maintenance notification test logging - Improve push notification monitor message formatting - Maintain all test functionality while improving code readability * Refactor: Move inline comments above code and fix string comparisons - Move all inline comments to be above the code they reference in: * RelaxedTimeoutConfigurationTest.java * RedisEnterpriseConfig.java * MaintenanceNotificationTest.java - Replace string != comparisons with .equals() for proper string comparison - Apply code formatting via Maven formatter This improves code readability and follows Java best practices for string comparison. * [Hitless Upgrades] Zero-server-side configuration with client-side opt-in (#3380) * Support for Client-side opt-in A client can tell the server if it wants to receive maintenance push notifications via the following command: CLIENT MAINT_NOTIFICATIONS <ON | OFF> [parameter value parameter value ...] * update maintenance events to latest format - MIGRATING <seq_number> <time> <shard_id-s>: A shard migration is going to start within <time> seconds. - MIGRATED <seq_number> <shard_id-s>: A shard migration ended. - FAILING_OVER <seq_number> <time> <shard_id-s>: A shard failover of a healthy shard started. - FAILED_OVER <seq_number> <shard_id-s>: A shard failover of a healthy shard ended. - MOVING <seq_number> <time> <endpoint>: A specific endpoint is going to move to another node within <time> seconds * clean up * Update FAILED_OVER & MIGRATED to include additional time field * update is private reserver check & add unit tests update is private reserver check * add unit tests for handshake with enabled maintenance events * add missing copyrights/docs * format * address review comments * Revert address after rebind operation expires * Update event's validation spec - MIGRATING <seq_number> <time> <shard_id-s>: A shard migration is going to start within <time> seconds. - MIGRATED <seq_number> <shard_id-s>: A shard migration ended. - FAILING_OVER <seq_number> <time> <shard_id-s>: A shard failover of a healthy shard started. - FAILED_OVER <seq_number> <shard_id-s>: A shard failover of a healthy shard ended. - MOVING <seq_number> <time> <endpoint>: A specific endpoint is going to move to another node within <time> seconds * rebase * format after rebase * Apply suggestions from code review Co-authored-by: Tihomir Krasimirov Mateev <[email protected]> * javadoc updated * Update src/main/java/io/lettuce/core/internal/NetUtils.java Co-authored-by: Tihomir Krasimirov Mateev <[email protected]> --------- Co-authored-by: Tihomir Krasimirov Mateev <[email protected]> * [hitless upgrade] Support for none moving-endpoint-type in maintenance event notifications (CAE-1285) (#3396) * support MOVING with none none indicates that the MOVING message doesn’t need to contain an endpoint. In such a case, the client is expected to schedule a graceful reconnect to its currently configured endpoint after half of the grace period that was communicated by the server is over. * formatting * Apply suggestions from code review Co-authored-by: Copilot <[email protected]> * Fix NPE * Add test to cover null rebindAddress null for rebind adress can be returned as part of MOVING notification if client is connected using 'moving-endpoint-type=none' * Add java docs to RebindAwareAddressSupplier --------- Co-authored-by: Copilot <[email protected]> * [Hitless Upgrades] Enable maintenance events support by default (CAE-1303) (#3415) * set default relaxed timeout to 10s * Enable maintenance events support by default * Enable maintenance events support by default * fix tests - ensure MaintenanceAwareExpiryWriter is registered for events when wathcdog is created - command timeout was not applied * fix tests - sporadic failure because of timeout of 50ms RedisHandshakeUnitTests.handshakeDelayedCredentialProvider:153 » ConditionTimeout - new command introduced during handshake, increase the timeout to 100ms * resolve errors after rebase on main - reset() - removed with issue#3328 - remove deprecated code from issue#907 (#3395) * Remove LettuceMaintenanceEventsDemo.java - no longer needed. --------- Co-authored-by: Igor Malinovskiy <[email protected]> Co-authored-by: ggivo <[email protected]> Co-authored-by: kiryazovi-redis <[email protected]> Co-authored-by: Copilot <[email protected]>

ggivo requested a review from Copilot August 4, 2025 12:45

This comment was marked as outdated.

Sign in to view

ggivo changed the title ~~[Maintenance events] Zero-server-side configuration with client-side opt-in~~ [[Hitless Upgrades] Zero-server-side configuration with client-side opt-in Aug 4, 2025

ggivo changed the title ~~[[Hitless Upgrades] Zero-server-side configuration with client-side opt-in~~ [Hitless Upgrades] Zero-server-side configuration with client-side opt-in Aug 4, 2025

ggivo requested a review from Copilot August 4, 2025 13:25

Copilot AI reviewed Aug 4, 2025

View reviewed changes

src/main/java/io/lettuce/core/protocol/MaintenanceAwareConnectionWatchdog.java Outdated Show resolved Hide resolved

src/main/java/io/lettuce/core/protocol/MaintenanceAwareConnectionWatchdog.java Outdated Show resolved Hide resolved

ggivo added 12 commits August 6, 2025 13:10

Support for Client-side opt-in

e11b986

A client can tell the server if it wants to receive maintenance push notifications via the following command: CLIENT MAINT_NOTIFICATIONS <ON | OFF> [parameter value parameter value ...]

clean up

7da08b1

Update FAILED_OVER & MIGRATED to include additional time field

e1e1b6b

update is private reserver check & add unit tests

9ea84d2

update is private reserver check

add unit tests for handshake with enabled maintenance events

d9c498e

add missing copyrights/docs

34a1087

format

1d78830

address review comments

c3d6305

Revert address after rebind operation expires

8e52f56

rebase

9dac6e8

ggivo force-pushed the feature/maintenance-events-maintenance-on branch from c2ab81d to 9dac6e8 Compare August 6, 2025 10:15

format after rebase

adc3f70

tishun approved these changes Aug 6, 2025

View reviewed changes

ggivo and others added 3 commits August 7, 2025 08:39

Apply suggestions from code review

dafaea8

Co-authored-by: Tihomir Krasimirov Mateev <[email protected]>

javadoc updated

db3fcf0

Update src/main/java/io/lettuce/core/internal/NetUtils.java

1062298

Co-authored-by: Tihomir Krasimirov Mateev <[email protected]>

ggivo merged commit bd408cf into redis:feature/maintenance-events Aug 7, 2025
2 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[Hitless Upgrades] Zero-server-side configuration with client-side opt-in #3380

[Hitless Upgrades] Zero-server-side configuration with client-side opt-in #3380

Uh oh!

ggivo commented Aug 4, 2025 •

edited

Loading

Uh oh!

This comment was marked as outdated.

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

tishun left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kiryazovi-redis commented Aug 7, 2025

Uh oh!

Uh oh!

[Hitless Upgrades] Zero-server-side configuration with client-side opt-in #3380

[Hitless Upgrades] Zero-server-side configuration with client-side opt-in #3380

Uh oh!

Conversation

ggivo commented Aug 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Client-Side Opt-in

Configuration Options:

Updated Maintenance Notification Format

Format Changes:

Uh oh!

This comment was marked as outdated.

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Uh oh!

tishun left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kiryazovi-redis commented Aug 7, 2025

Uh oh!

Uh oh!

ggivo commented Aug 4, 2025 •

edited

Loading