Skip to content

K8SPSMDB-723 Documented hidden nodes #254

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 6 commits into
base: main
Choose a base branch
from
+284 −77
Open

K8SPSMDB-723 Documented hidden nodes #254

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
020155d
K8SPSMDB-1247
nastena1606 Sep 12, 2025
9ce465c
K8SPSMDB-723 Documented the hidden nodes and their CR options
nastena1606 Sep 15, 2025
6898dcb
Added info how the Operator manages voting members
nastena1606 Sep 23, 2025
08ac46b
Update arbiter.md
nastena1606 Sep 29, 2025
7460343
Update arbiter.md
nastena1606 Sep 29, 2025
96a5dec
Updated after the review
nastena1606 Sep 30, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
125 changes: 63 additions & 62 deletions docs/arbiter.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,38 +1,28 @@
# Using Replica Set Arbiter nodes and non-voting nodes
# Replica set members and their usage

Percona Server for MongoDB [replication model :octicons-link-external-16:](https://www.percona.com/blog/2018/05/17/mongodb-replica-set-transport-encryption-part-1/)
is based on elections, when nodes of the Replica Set [choose which node :octicons-link-external-16:](https://docs.mongodb.com/manual/core/replica-set-elections/#replica-set-elections)
becomes the primary node.
Percona Server for MongoDB replica set is a number of `mongod` instances working together to ensure data durability and system resilience. Such configuration enhances fault tolerance and keeps your database accessible even during failures.

The need for elections influences the choice of the number of nodes in the cluster.
Elections are the reason to avoid even number of nodes, and to have at least
three and not more than seven participating nodes.
A replica set consists of one **primary** node and several **secondary** nodes. The primary node accepts all write operations, while secondary nodes replicate the data set to maintain redundancy. Secondary nodes can serve read queries, which helps distribute the read load. Secondary nodes can also have additional configuration, like be non-voting or hidden.

Still, sometimes there is a contradiction between the number of nodes suitable for
elections and the number of nodes needed to store data. You can solve this
contradiction in two ways:
Percona Server for MongoDB replication mechanism is based on elections, when replica set nodes [choose which node :octicons-link-external-16:](https://docs.mongodb.com/manual/core/replica-set-elections/#replica-set-elections) becomes the primary. For elections to be successful, the number fo voting members must be odd.

* Add *Arbiter* nodes, which participate in elections, but do not store data,
* Add *non-voting* nodes, which store data but do not participate in elections.
By default, the Operator creates Percona Server for MongoDB replica set with three members, one primary and the remaining secondaries. This is the minimal recommended configuration. A replica set can have up to 50 members with the maximum of 7 voting members.

## Adding Arbiter nodes
## Replica set member types

Normally, each node stores a complete copy of the data,
but there is also a possibility, to reduce disk IO and space used by the
database, to add an [arbiter node :octicons-link-external-16:](https://docs.mongodb.com/manual/core/replica-set-arbiter/).
An arbiter cannot become a primary and does not have a complete copy of the
data. The arbiter does have one election vote and can be the odd number for
elections. The arbiter does not demand a persistent volume.
Besides the primary and regular secondaries in a MongoDB replica set, you can have special member configurations like hidden, arbiter, and non-voting members.

Percona Operator for MongoDB has the ability to create Replica Set Arbiter nodes
if needed. This feature can be configured in the Replica Set section of the
[deploy/cr.yaml :octicons-link-external-16:](https://github.com/percona/percona-server-mongodb-operator/blob/main/deploy/cr.yaml)
file:
* **Arbiter**: An arbiter node participates in elections but does not store data. You may want to add arbiter nodes if cost constraints prevent you from adding another secondary node.
* **Non-voting**: This type of node stores a full copy of the data but does not participate in elections. This is useful for scaling read capacity beyond the seven-member voting limit of a replica set.
* **Hidden**: A hidden node is a secondary member that holds data but is invisible to client applications. It is added as a voting member and can participate in elections. It is useful for tasks like backups or running batch jobs that might otherwise interfere with primary operations.

* set `arbiter.enabled` option to `true` to allow Arbiter instances,
* use `arbiter.size` option to set the desired amount of Arbiter instances.
### Arbiter nodes

For example, the following keys in `deploy/cr.yaml` will create a cluster
An Arbiter node participates in the replica set elections but does not store any data. Its primary role is to act as a tiebreaker in a replica set with an even number of data-bearing nodes, ensuring that a primary can always be elected. By not storing data, Arbiter nodes require minimal resources, which can help reduce your overall costs. An does not demand a persistent volume.

To add an Arbiter node, you can update your `deploy/cr.yaml` file by adding an `arbiter` section under `replsets` and setting the `enabled` and `size` options to your desired values.

The following example configuration will create a cluster
with 4 data instances and 1 Arbiter:

```yaml
Expand All @@ -47,20 +37,14 @@ replsets:
....
```

!!! note
Find the description of other available options in the [replsets.arbiter section](operator.md#replsetsarbiterenabled) of the [Custom Resource options reference](operator.md).

You can find description of other possible options in the
[replsets.arbiter section](operator.md#replsetsarbiterenabled) of the
[Custom Resource options reference](operator.md).
### Prevent Arbiter nodes on the same Kubernetes hosts with data-bearing replica set members

### Preventing Arbiter instances to share Kubernetes Nodes with Replica Set

By default Arbiter instances are allowed to run on the same host as regular
Replica Set instances. This may be reasonable in terms of the number of
By default, Arbiter nodes are allowed to run on the same Kubernetes hosts as your data nodes. This may be reasonable in terms of the number of
Kubernetes Nodes required for the cluster. But as a result it increases
possibility to have 50/50 votes division in case of network partitioning.
You can use [anti-affinity constraints](constraints.md#affinity-and-anti-affinity)
to avoid such Pod alocation as follows:
possibility to have 50/50 votes division in case of network partitioning.
To prevent this, you can apply an [anti-affinity](constraints.md) constraint, which forces arbiter nodes to be scheduled on separate nodes:

```yaml
....
Expand All @@ -83,48 +67,65 @@ arbiter:
topologyKey: kubernetes.io/hostname
```

## Adding non-voting nodes
## Non-voting nodes

A non-voting node is a secondary member that stores a full copy of the data but does not participate in elections for the primary node. Non-voting nodes enable you to deploy a replica set with more than seven data-bearing nodes. You can also add a non-voting node to a remote location where network latency might make it unsuitable for voting.

You can add non-voting nodes by setting the `replsets.nonvoting.enabled` and `replsets.nonvoting.size` options in your `deploy/cr.yaml` file.

In this example, the Operator will create a cluster with
3 data instances and 1 non-voting instance:

[Non-voting member :octicons-link-external-16:](https://docs.mongodb.com/manual/tutorial/configure-a-non-voting-replica-set-member/)
is a Replica Set node which does not participate in the primary election
process. This feature is required to have more than 7 nodes, or if there is a
[node in the edge location :octicons-link-external-16:](https://en.wikipedia.org/wiki/Edge_computing), which
obviously should not participate in the voting process.
```yaml
....
replsets:
....
size: 3
....
nonvoting:
enabled: true
size: 1
....
```

!!! note
Find the description of other available options in the [replsets.nonvoting section](operator.md#replsetsnonvotingenabled) of the [Custom Resource options reference](operator.md).

Non-voting nodes support has technical preview status and is not recommended
for production environments.
Note that you can add a non-voting node in the edge location through the `externalNodes` option. Please see [cross-site replication documentation](replication.md) for details.

!!! note
## Hidden nodes

It is possible to add a non-voting node in the edge location through the
`externalNodes` option. Please see [cross-site replication documentation](replication.md)
for details.
Hidden nodes are secondary members that hold a full copy of the data but are not visible to client applications. Hidden nodes always have a 0 priority and therefore, cannot become a primary. But hidden members are added as voting members and may, therefore, vote in primary elections. Read more how the Operator [manages voting members in replica set](#manage-voting-members-in-replica-set).

Percona Operator for MongoDB has the ability to configure non-voting nodes in
the Replica Set section of the [deploy/cr.yaml :octicons-link-external-16:](https://github.com/percona/percona-server-mongodb-operator/blob/main/deploy/cr.yaml)
file:
Hidden nodes are useful for tasks like backups or reporting, as they do not affect primary operations. Client applications will not connect to hidden nodes because they are not listed in the replica set's SRV record.

* set `nonvoting.enabled` option to `true` to allow non-voting instances,
* use `nonvoting.size` option to set the desired amount of non-voting instances.
To add a hidden node with the Operator, set the setting the `replsets.hidden.enabled` and `replsets.hidden.size` options in the `deploy/cr.yaml` file:

For example, the following keys in `deploy/cr.yaml` will create a cluster with
3 data instances and 1 non-voting instance:
This configuration example creates a cluster with 3 data instances and 2 hidden nodes:

```yaml
....
replsets:
....
size: 3
....
nonvoting:
hidden:
enabled: true
size: 1
size: 2
....
```

!!! note
Find the description of other available options in the [replsets.hidden section](operator.md#replsetshidden) of the [Custom Resource options reference](operator.md).

## Manage voting members in replica set

Since [hidden nodes](#hidden-nodes) can participate in elections, the Operator enforces rules to ensure the odd number of voting members and maintain a stable and compliant replica set configuration:

You can find description of other possible options in the [replsets.nonvoting section](operator.md#replsetsnonvotingenabled)
of the [Custom Resource options reference](operator.md).
* If the total number of voting members is even, the Operator converts one node to non-voting to maintain an odd number of voters. The node to convert is typically the last Pod in the list
* If the number of voting members is odd and not more than 7, all nodes participate in elections.
* If the number of voting members exceeds 7, the Operator automatically converts some nodes to non-voting to stay within MongoDB’s limit of 7 voting members.

To inspect the current configuration, connect to the cluster with clusterAdmin privileges and run:

```javascript
rs.config() command
```
26 changes: 12 additions & 14 deletions docs/encryption.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,15 +1,16 @@
# Data at rest encryption

[Data at rest encryption in Percona Server for MongoDB :octicons-link-external-16:](https://docs.percona.com/percona-server-for-mongodb/latest/data-at-rest-encryption.html)
is supported by the Operator since version 1.1.0.
!!! admonition "Version added: [1.1.0](RN/Kubernetes-Operator-for-PSMONGODB-RN1.1.0.md)

!!! note

[Data at rest :octicons-link-external-16:](https://en.wikipedia.org/wiki/Data_at_rest) means inactive data stored as files, database records, etc.
Data-at-rest encryption ensures that data stored on disk remains protected even if the underlying storage is compromised. This process is transparent to your applications, meaning you don't need to change the application's code. If an unauthorized user gains access to the storage, they can't read the data files.

Data at rest encryption is turned on by default. The Operator implements it by
either using encryption key stored in a Secret, or obtaining encryption key
from the HashiCorp Vault key storage.
To learn more about data-at-rest-encryption in Percona Server for MongoDB, see the [Data-at-rest encryption :octicons-link-external-16:](https://docs.percona.com/percona-server-for-mongodb/latest/data-at-rest-encryption.html) documentation.

Data-at-rest encryption is turned on by default. The Operator implements it in one of the following ways:

* [uses an encryption key stored in a Secret](#using-encryption-key-secret)
* [gets encryption key from the HashiCorp Vault key storage](#using-hashicorp-vault-storage-for-encryption-keys)

## Using encryption key Secret

Expand All @@ -23,7 +24,7 @@ from the HashiCorp Vault key storage.
```

Encryption key Secret will be created automatically by the Operator if it
doesn’t exist. If you would like to create it yourself, take into account
doesn’t exist. If you would like to create it yourself, ensure
that [the key must be a 32 character string encoded in base64 :octicons-link-external-16:](https://docs.mongodb.com/manual/tutorial/configure-encryption/#local-key-management).

2. The `replsets.configuration`, `replsets.nonvoting.configuration`, and
Expand Down Expand Up @@ -51,14 +52,11 @@ Don't forget to apply the modified `cr.yaml` configuration file as usual:
$ kubectl deploy -f deploy/cr.yaml
```

## <a name="using-vault"></a>Using HashiCorp Vault storage for encryption keys

Starting from the version 1.13, the Operator supports using [HashiCorp Vault :octicons-link-external-16:](https://www.vaultproject.io/) storage for encryption keys - a universal, secure and reliable way to store and distribute secrets without depending on the operating system, platform or cloud provider.
## Using HashiCorp Vault storage for encryption keys

!!! warning
!!! admonition "Version added: [1.13.0](RN/Kubernetes-Operator-for-PSMONGODB-RN1.13.0.md)

Vault integration has technical preview status and is not yet recommended
for production environments.
The Operator supports using [HashiCorp Vault :octicons-link-external-16:](https://www.vaultproject.io/) storage for encryption keys - a universal, secure and reliable way to store and distribute secrets without depending on the operating system, platform or cloud provider.

The Operator will use Vault if the `deploy/cr.yaml` configuration file contains
the following items:
Expand Down
Loading