From e82f5a746f69d743653539e53c8177517827a897 Mon Sep 17 00:00:00 2001 From: jessica Date: Tue, 29 Jul 2025 11:06:03 -0400 Subject: [PATCH 1/8] guide-creation-1 --- .../raid_soft_uefi/guide.en-gb.md | 479 ++++++++++++++++++ 1 file changed, 479 insertions(+) create mode 100644 pages/bare_metal_cloud/dedicated_servers/raid_soft_uefi/guide.en-gb.md diff --git a/pages/bare_metal_cloud/dedicated_servers/raid_soft_uefi/guide.en-gb.md b/pages/bare_metal_cloud/dedicated_servers/raid_soft_uefi/guide.en-gb.md new file mode 100644 index 00000000000..57e2b6d0712 --- /dev/null +++ b/pages/bare_metal_cloud/dedicated_servers/raid_soft_uefi/guide.en-gb.md @@ -0,0 +1,479 @@ +--- +title: How to configure and rebuild software RAID on a Dedicated server with UEFI boot mode +excerpt: Find out how to verify the state of your software RAID and rebuild it after a disk replacement +updated: 2023-08-21 +--- + +## Objective + +Redundant Array of Independent Disks (RAID) is a technology that mitigates data loss on a server by replicating data across two or more disks. + +The default RAID level for OVHcloud server installations is RAID 1, which doubles the space taken up by your data, effectively halving the useable disk space. + +**This guide explains how to configure your server’s RAID array in the event that it needs to be rebuilt due to corruption or disk failure.** + +Before we begin, please note that this guide focuses on Dedicated servers that use UEFI as the boot mode. This is the case with modern motherboards. If your server uses the BIOS mode (old motherboards), refer to this guide [How to configure and rebuild software RAID on a Dedicated server with BIOS boot mode](). + +For more information on UEFI, consult the following [guide](https://uefi.org/about){.external} + +## Requirements + +- A [Dedicated server](/links/bare-metal/bare-metal) with a software RAID configuration +- Administrative (sudo) access to the server via SSH +- Understanding of RAID, partitions and GRUB + +## Instructions + +Over the course of this guide, we will explore three scenarios: + +- Disk failure simulation (preventive measure) +- Dealing with a faulty disk that is still present in RAID. +- Dealing with a faulty disk that is no longer present in RAID. + +### System verification + +To check whether a server runs on BIOS mode or UEFI mode, run the following command: + +```sh +[user@server_ip ~]# [ -d /sys/firmware/efi ] && echo UEFI || echo BIOS +``` + +#### Basic information + +To begin, type the following code in a command line session to determine the current RAID status: + +```sh +cat /proc/mdstat +Personalities : [raid1] [linear] [multipath] [raid0] [raid6] [raid5] [raid4] [raid10] +md3 : active raid1 sda3[0] sdb3[1] + 3904786432 blocks super 1.2 [2/2] [UU] + bitmap: 2/30 pages [8KB], 65536KB chunk + +md2 : active raid1 sda2[0] sdb2[1] + 1046528 blocks super 1.2 [2/2] [UU] + +unused devices: +``` + +This command shows us that we have two software RAID devices currently set up, with **md3** being the largest one. This array consists of two partitions, which are known as sda3 and sdb3. The [UU] means that all the disks are working normally. A `_` would indicate a failed disk. + +Although this command returns our RAID volumes, it doesn't tell us the size of the partitions themselves. We can find this information with the following command: + +```sh +sudo fdisk -l + +Disk /dev/sdb: 3.64 TiB, 4000787030016 bytes, 7814037168 sectors +Disk model: HGST HUS726T4TAL +Units: sectors of 1 * 512 = 512 bytes +Sector size (logical/physical): 512 bytes / 512 bytes +I/O size (minimum/optimal): 512 bytes / 512 bytes +Disklabel type: gpt +Disk identifier: 3253EEB9-DC20-4887-9E64-69655C758811 + +Device Start End Sectors Size Type +/dev/sdb1 2048 1048575 1046528 511M EFI System +/dev/sdb2 1048576 3145727 2097152 1G Linux RAID +/dev/sdb3 3145728 7812982783 7809837056 3.6T Linux RAID +/dev/sdb4 7812982784 7814031359 1048576 512M Linux filesystem + + +Disk /dev/sda: 3.64 TiB, 4000787030016 bytes, 7814037168 sectors +Disk model: HGST HUS726T4TAL +Units: sectors of 1 * 512 = 512 bytes +Sector size (logical/physical): 512 bytes / 512 bytes +I/O size (minimum/optimal): 512 bytes / 512 bytes +Disklabel type: gpt +Disk identifier: 5D8AF9B0-E363-40AF-A3CA-FADB7B3C6A10 + +Device Start End Sectors Size Type +/dev/sda1 2048 1048575 1046528 511M EFI System +/dev/sda2 1048576 3145727 2097152 1G Linux RAID +/dev/sda3 3145728 7812982783 7809837056 3.6T Linux RAID +/dev/sda4 7812982784 7814031359 1048576 512M Linux filesystem +/dev/sda5 7814033072 7814037134 4063 2M Linux filesystem + + +Disk /dev/md2: 1022 MiB, 1071644672 bytes, 2093056 sectors +Units: sectors of 1 * 512 = 512 bytes +Sector size (logical/physical): 512 bytes / 512 bytes +I/O size (minimum/optimal): 512 bytes / 512 bytes + + +Disk /dev/md3: 3.64 TiB, 3998501306368 bytes, 7809572864 sectors +Units: sectors of 1 * 512 = 512 bytes +Sector size (logical/physical): 512 bytes / 512 bytes +I/O size (minimum/optimal): 512 bytes / 512 bytes +``` + +The `fdisk -l` command also allows you to identify your partition type. This is an important information when it comes to rebuilding your RAID in case of a disk failure. + +For **GPT** partitions, the command will return: `Disklabel type: gpt`. + +```sh +Disk /dev/sdb: 3.64 TiB, 4000787030016 bytes, 7814037168 sectors +Disk model: HGST HUS726T4TAL +Units: sectors of 1 * 512 = 512 bytes +Sector size (logical/physical): 512 bytes / 512 bytes +I/O size (minimum/optimal): 512 bytes / 512 bytes +Disklabel type: gpt +Disk identifier: 3253EEB9-DC20-4887-9E64-69655C758811 +``` + +For **MBR** partitions, the command will return: `Disklabel type: dos`. + +```sh +Disk /dev/sda: 2.5 GiB, 2621440000 bytes, 5120000 sectors +Disk model: QEMU HARDDISK +Units: sectors of 1 * 512 = 512 bytes +Sector size (logical/physical): 512 bytes / 512 bytes +I/O size (minimum/optimal): 512 bytes / 512 bytes +'Disklabel type: dos' +Disk identifier: 0x150f6797 +``` + +Still going by the results of `fdisk -l`, we can see that `/dev/md2` consists of 1022 MiB and `/dev/md3` contains 3.64 TiB. If we were to run the mount command we can also find out the layout of the disk. + +```sh +mount + +sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime) +proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) +udev on /dev type devtmpfs (rw,nosuid,relatime,size=16288028k,nr_inodes=4072007,mode=755,inode64) +devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000) +tmpfs on /run type tmpfs (rw,nosuid,nodev,noexec,relatime,size=3263252k,mode=755,inode64) +/dev/md3 on / type ext4 (rw,relatime) +securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime) +tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,inode64) +tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k,inode64) +cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot) +pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime) +efivarfs on /sys/firmware/efi/efivars type efivarfs (rw,nosuid,nodev,noexec,relatime) +bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700) +systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=926) +hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M) +mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime) +debugfs on /sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,relatime) +tracefs on /sys/kernel/tracing type tracefs (rw,nosuid,nodev,noexec,relatime) +fusectl on /sys/fs/fuse/connections type fusectl (rw,nosuid,nodev,noexec,relatime) +configfs on /sys/kernel/config type configfs (rw,nosuid,nodev,noexec,relatime) +ramfs on /run/credentials/systemd-sysctl.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700) +ramfs on /run/credentials/systemd-sysusers.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700) +ramfs on /run/credentials/systemd-tmpfiles-setup-dev.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700) +/dev/md2 on /boot type ext4 (rw,relatime) +/dev/sda1 on /boot/efi type vfat (rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=ascii,shortname=mixed,utf8,errors=remount-ro) +ramfs on /run/credentials/systemd-tmpfiles-setup.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700) +binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,nosuid,nodev,noexec,relatime) +tmpfs on /run/user/1000 type tmpfs (rw,nosuid,nodev,relatime,size=3263248k,nr_inodes=815812,mode=700,uid=1000,gid=1000,inode64) +``` + +Alternatively, the `lsblk` command offers a different view of the partitions: + +```sh +lsblk + +NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS +sda 8:0 0 3.6T 0 disk +├─sda1 8:1 0 511M 0 part /boot/efi +├─sda2 8:2 0 1G 0 part +│ └─md2 9:2 0 1022M 0 raid1 /boot +├─sda3 8:3 0 3.6T 0 part +│ └─md3 9:3 0 3.6T 0 raid1 / +├─sda4 8:4 0 512M 0 part [SWAP] +└─sda5 8:5 0 2M 0 part +sdb 8:16 0 3.6T 0 disk +├─sdb1 8:17 0 511M 0 part +├─sdb2 8:18 0 1G 0 part +│ └─md2 9:2 0 1022M 0 raid1 /boot +├─sdb3 8:19 0 3.6T 0 part +│ └─md3 9:3 0 3.6T 0 raid1 / +└─sdb4 8:20 0 512M 0 part [SWAP] +sdc 8:32 1 0B 0 disk +sr0 11:0 1 1024M 0 rom +``` + +With UEFI mode, ESPs (lablled EFI_SYSPART by OVHcloud) partitions are created on the disks used for RAID, however, only one partition is automatically mounted in `/boot/efi` by the system after install. It is important to understand the role of this partition and how to manage it. + +Unfortunately, at this time, `/boot/efi` is not mirrored in our RAID installations, therefore, if you need to have the content of this partition up dated to the other disk(s), you will need sync these partitions. + +The command `lsblk -f` offers a more detailed output: + +```sh +lsblk -f + +NAME FSTYPE FSVER LABEL UUID FSAVAIL FSUSE% MOUNTPOINTS +sda +├─sda1 vfat FAT16 EFI_SYSPART 7A52-33BE 504.8M 1% /boot/efi +├─sda2 linux_raid_member 1.2 md2 aa1265ff-d8a7-9b64-0109-1086e78a451c +│ └─md2 ext4 1.0 boot 857b98e6-a5d8-4126-95c8-1451cea76a5b 850.3M 7% /boot +├─sda3 linux_raid_member 1.2 md3 7eb971ad-6f31-0998-50f2-50f330c34361 +│ └─md3 ext4 1.0 root 4b8782be-3bd4-4b96-8129-491806a54b46 3.4T 0% / +├─sda4 swap 1 swap-sda4 da546332-ea7c-4f0d-969c-5b8818c8dc81 [SWAP] +└─sda5 iso9660 Joliet Extension config-2 2025-07-23-17-11-22-00 +sdb +├─sdb1 vfat FAT16 EFI_SYSPART 7A68-19BC +├─sdb2 linux_raid_member 1.2 md2 aa1265ff-d8a7-9b64-0109-1086e78a451c +│ └─md2 ext4 1.0 boot 857b98e6-a5d8-4126-95c8-1451cea76a5b 850.3M 7% /boot +├─sdb3 linux_raid_member 1.2 md3 7eb971ad-6f31-0998-50f2-50f330c34361 +│ └─md3 ext4 1.0 root 4b8782be-3bd4-4b96-8129-491806a54b46 3.4T 0% / +└─sdb4 swap 1 swap-sdb4 37dda653-4074-4c28-953c-ae627fe53816 [SWAP] +sdc +sr0 +nvme0n1 +nvme1n1 +``` + +From the above output, we can see that two partitions (sda1 and sdb1) with the label `EFI_SYSPART` were created on both disks. However, the OS selected **sda1** to mount the partition in `/boot/efi`. + +This partition contains the boot loaders (and sometimes kernel images and drivers), which helps the server to boot but it is not protected through the software RAID. In general, the content of this partition remains the same **unless** there is a kernel update, in this case you must sync both partitions to keep them up to date. + +In case of the failure of the main disk containing this partition, the server will be rebooted in rescue mode for you to recreate the partition on the new disk and copy the files from the healthy disk to the new disk (More information on this later in the guide.). It is therefore important keep the duplicated partition up to date when necessary, since it is not present in the RAID. + +Alternatively, you can explore another premptive option such as keeping these partitions synchronised by either running a manual script daily (or when needed) or running an automatic script to sync both partitions on boot. You must be the user **root** to run this script. + +Scrip example: + +```bash +#!/bin/bash +set -euo pipefail + +MAIN_PARTITION="$(findmnt -n -o SOURCE /boot/efi)" +MOUNTPOINT="/var/lib/grub/esp" + +mkdir -p "${MOUNTPOINT}" + +while read -r partition; do + if [[ "${partition}" == "${MAIN_PARTITION}" ]]; then + continue + fi + echo "Working on ${partition}" + mount "${partition}" "${MOUNTPOINT}" + rsync -ax "/boot/efi/" "${MOUNTPOINT}/" + umount "${MOUNTPOINT}" +done < <(blkid -o device -t LABEL=EFI_SYSPART) +``` + +The script above can be run manually to keep them synchronized, the difficulty being knowing when to execute it. The ideal solution would probably be to use dpkg triggers to only do this when updating the `grub-efi-amd64` package. + + +> [!warning] +> If you are on an Ubuntu system, you can easily keep these partitions synchronised with a grub installation, but this ONLY works for Ubuntu system. For other linux systems, this must been done manually or when using a sync. For more information consult this article. +> + +### Simulating a Disk failure + +When you purchase a new server, you might feel the need to perform a series of tests and actions. One of those actions could be simulating a disk failure in order to understand the process of rebuilding the raid and prepare yourself in case this happens. + +#### Removing the Dis + +As the disks are currently mounted by default, to remove a disk from the RAID, we first need to unmount the disk, then simulate a failure, and finally remove it. We will remove `/dev/sda4` from the RAID with the following command: + +```sh +umount /dev/md4 +``` + +> [!warning] +> Please note that if you are connected as the user `root`, you may get the following message when you try to unmount the partition (in our case, where our md4 partition is mounted in /home): +> +> 
umount: /home: target is busy
+> +> In this case, you must log out as the user root and connect as a local user (in our case `debian`), and use the following command: +> +> 
debian@ns000000:/$ sudo umount /dev/md4
+> +> If you do not have a local user, you need to [create one](/pages/bare_metal_cloud/dedicated_servers/changing_root_password_linux_ds). + +This will provide us with the following output: + +```sh +sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime) +proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) +udev on /dev type devtmpfs (rw,nosuid,relatime,size=16315920k,nr_inodes=4078980,mode=755) +devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000) +tmpfs on /run type tmpfs (rw,nosuid,noexec,relatime,size=3266556k,mode=755) +/dev/md2 on / type ext4 (rw,relatime) +securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime) +tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev) +tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k) +tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755) +cgroup2 on /sys/fs/cgroup/unified type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate) +cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,name=systemd) +pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime) +bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700) +cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids) +cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory) +cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event) +cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma) +cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio) +cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct) +cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer) +cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio) +cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices) +cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset) +debugfs on /sys/kernel/debug type debugfs (rw,relatime) +hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M) +mqueue on /dev/mqueue type mqueue (rw,relatime) +systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=45,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=10340) +tmpfs on /run/user/1000 type tmpfs (rw,nosuid,nodev,relatime,size=3266552k,mode=700,uid=1000,gid=1000) +``` + +As we can see the, entry of `/dev/md4` is no longer mounted. However, the RAID is still active, so we need to simulate a failure to remove the disk. We can do this with the following command: + +```sh +sudo mdadm --fail /dev/md4 /dev/sda4 +``` + +We have now simulated a failure of the RAID. The next step is to remove the partition from the RAID array with the following command: + +```sh +sudo mdadm --remove /dev/md4 /dev/sda4 +``` + +You can verify that the partition has been removed with the following command: + +```sh +cat /proc/mdstat + +Personalities : [raid1] [linear] [multipath] [raid0] [raid6] [raid5] [raid4] [raid10] +md2 : active raid1 sda2[1] sdb2[0] + 931954688 blocks super 1.2 [2/2] [UU] + bitmap: 4/7 pages [16KB], 65536KB chunk + +md4 : active raid1 sdb4[1] + 1020767232 blocks super 1.2 [2/1] [_U] + bitmap: 0/8 pages [0KB], 65536KB chunk + +unused devices: +``` + +The following command will verify that the partition has been removed: + +```sh +mdadm --detail /dev/md4 + +/dev/md4: + Version : 1.2 + Creation Time : Tue Jan 24 15:35:02 2023 + Raid Level : raid1 + Array Size : 1020767232 (973.48 GiB 1045.27 GB) + Used Dev Size : 1020767232 (973.48 GiB 1045.27 GB) + Raid Devices : 2 + Total Devices : 1 + Persistence : Superblock is persistent + + Intent Bitmap : Internal + + Update Time : Tue Jan 24 16:28:03 2023 + State : clean, degraded + Active Devices : 1 + Working Devices : 1 + Failed Devices : 0 + Spare Devices : 0 + +Consistency Policy : bitmap + + Name : md4 + UUID : 7b5c1d80:0a7ab4c2:e769b5e5:9c6eaa0f + Events : 21 + + Number Major Minor RaidDevice State + - 0 0 0 removed + 1 8 20 1 active sync /dev/sdb4 +``` + +### Rebuilding the RAID + +Once the disk has been replaced, we need to copy the partition table from a healthy disk (in this example, sdb) to the new one (sda) with the following command: + +**For GPT partitions** + +```sh +sgdisk -R /dev/sda /dev/sdb +``` + +The command should be in this format: `sgdisk -R /dev/newdisk /dev/healthydisk` + +Once this is done, the next step is to randomize the GUID of the new disk to prevent GUID conflicts with other disks: + +```sh +sgdisk -G /dev/sda +``` + +**For MBR partitions** + +Once the disk has been replaced, we need to copy the partition table from a healthy disk (in this example, sdb) to the new one (sda) with the following command: + +```sh +sfdisk -d /dev/sdb | sfdisk /dev/sda +``` + +The command should be in this format: `sfdisk -d /dev/healthydisk | sfdisk /dev/newdisk` + +We can now rebuild the RAID array. The following code snippet shows how we can rebulid the `/dev/md4` partition layout with the recently-copied sda partition table: + +```sh +mdadm --add /dev/md4 /dev/sda4 +cat /proc/mdstat + +Personalities : [raid1] [linear] [multipath] [raid0] [raid6] [raid5] [raid4] [raid10] +md2 : active raid1 sda2[1] sdb2[0] + 931954688 blocks super 1.2 [2/2] [UU] + bitmap: 1/7 pages [4KB], 65536KB chunk + +md4 : active raid1 sda4[0] sdb4[1] + 1020767232 blocks super 1.2 [2/2] [UU] + bitmap: 0/8 pages [0KB], 65536KB chunk + +unused devices: +``` + +We can verify the RAID details with the following command: + +```sh +mdadm --detail /dev/md4 + +/dev/md4: + Version : 1.2 + Creation Time : Tue Jan 24 15:35:02 2023 + Raid Level : raid1 + Array Size : 1020767232 (973.48 GiB 1045.27 GB) + Used Dev Size : 1020767232 (973.48 GiB 1045.27 GB) + Raid Devices : 2 + Total Devices : 2 + Persistence : Superblock is persistent + + Intent Bitmap : Internal + + Update Time : Tue Jan 24 17:02:55 2023 + State : clean + Active Devices : 2 + Working Devices : 2 + Failed Devices : 0 + Spare Devices : 0 + + Rebuild Status : 21% complete + + UUID : 7f39d062:9f16a016:a4d2adc2:26fd5302 + Events : 0.95 + + Number Major Minor RaidDevice State + 0 8 2 0 spare rebuilding /dev/sda4 + 1 8 18 1 active sync /dev/sdb4 +``` + +The RAID has now been rebuilt, but we still need to mount the partition (`/dev/md4` in this example) with the following command: + +```sh +mount /dev/md4 /home +``` + +## Go Further + +[Hot Swap - Software RAID](/pages/bare_metal_cloud/dedicated_servers/hotswap_raid_soft) + +[OVHcloud API and Storage](/pages/bare_metal_cloud/dedicated_servers/partitioning_ovh) + +[Managing hardware RAID](/pages/bare_metal_cloud/dedicated_servers/raid_hard) + +[Hot Swap - Hardware RAID](/pages/bare_metal_cloud/dedicated_servers/hotswap_raid_hard) + +Join our [community of users](/links/community). From a920302a469a6f5d8047e48dcfabcaec1a80cbe0 Mon Sep 17 00:00:00 2001 From: jessica Date: Thu, 31 Jul 2025 12:39:52 -0400 Subject: [PATCH 2/8] update --- .../dedicated_servers/raid_soft_uefi/guide.en-gb.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/pages/bare_metal_cloud/dedicated_servers/raid_soft_uefi/guide.en-gb.md b/pages/bare_metal_cloud/dedicated_servers/raid_soft_uefi/guide.en-gb.md index 57e2b6d0712..2ad2e116095 100644 --- a/pages/bare_metal_cloud/dedicated_servers/raid_soft_uefi/guide.en-gb.md +++ b/pages/bare_metal_cloud/dedicated_servers/raid_soft_uefi/guide.en-gb.md @@ -384,10 +384,10 @@ Consistency Policy : bitmap Once the disk has been replaced, we need to copy the partition table from a healthy disk (in this example, sdb) to the new one (sda) with the following command: -**For GPT partitions** +**For GPT partitions**n n ```sh -sgdisk -R /dev/sda /dev/sdb +sgdisk -R /dev/sda /dev/sdb nmnn ``` The command should be in this format: `sgdisk -R /dev/newdisk /dev/healthydisk` From 89ff10f85537e75c78e229ffa7aae3d05cd8a3ce Mon Sep 17 00:00:00 2001 From: jessica Date: Mon, 4 Aug 2025 11:14:50 -0400 Subject: [PATCH 3/8] update --- .../raid_soft/guide.en-gb.md | 682 +++++++++++++----- 1 file changed, 491 insertions(+), 191 deletions(-) diff --git a/pages/bare_metal_cloud/dedicated_servers/raid_soft/guide.en-gb.md b/pages/bare_metal_cloud/dedicated_servers/raid_soft/guide.en-gb.md index 89e4dac2e54..b0e32354a5b 100644 --- a/pages/bare_metal_cloud/dedicated_servers/raid_soft/guide.en-gb.md +++ b/pages/bare_metal_cloud/dedicated_servers/raid_soft/guide.en-gb.md @@ -16,136 +16,130 @@ The default RAID level for OVHcloud server installations is RAID 1, which double - A [dedicated server](/links/bare-metal/bare-metal) with a software RAID configuration - Administrative (sudo) access to the server via SSH +- Understanding of RAID, partitions and GRUB ## Instructions -### Removing the disk +When you purchase a new server, you might feel the need to perform a series of tests and actions. One of those actions could be simulating a disk failure in order to understand the process of rebuilding the raid and prepare yourself in case this happens. + +#### Basic Information In a command line session, type the following code to determine the current RAID status: ```sh cat /proc/mdstat - Personalities : [raid1] [linear] [multipath] [raid0] [raid6] [raid5] [raid4] [raid10] -md2 : active raid1 sda2[1] sdb2[0] - 931954688 blocks super 1.2 [2/2] [UU] - bitmap: 2/7 pages [8KB], 65536KB chunk - -md4 : active raid1 sda4[0] sdb4[1] - 1020767232 blocks super 1.2 [2/2] [UU] - bitmap: 0/8 pages [0KB], 65536KB chunk - +md3 : active raid1 nvme1n1p3[1] nvme0n1p3[0] + 497875968 blocks super 1.2 [2/2] [UU] + bitmap: 2/4 pages [8KB], 65536KB chunk + +md2 : active raid1 nvme1n1p2[1] nvme0n1p2[0] + 1046528 blocks super 1.2 [2/2] [UU] + unused devices: ``` -This command shows us that we have two RAID arrays currently set up, with md4 being the largest partition. The partition consists of two disks, which are known as sda4 and sdb4. The [UU] means that all the disks are working normally. A `_` would indicate a failed disk. +This command shows us that we have two software RAID devices currently set up, with **md3** being the largest one. This array consists of two partitions, which are known as **nvme1n1p3** and **nvme0n1p3**. + +The [UU] means that all the disks are working normally. A `_` would indicate a failed disk. + +If you have a server with SATA disks, you would get the following results: + +```sh +cat /proc/mdstat +Personalities : [raid1] [linear] [multipath] [raid0] [raid6] [raid5] [raid4] [raid10] +md3 : active raid1 sda3[0] sdb3[1] + 3904786432 blocks super 1.2 [2/2] [UU] + bitmap: 2/30 pages [8KB], 65536KB chunk + +md2 : active raid1 sda2[0] sdb2[1] + 1046528 blocks super 1.2 [2/2] [UU] + +unused devices: +``` Although this command returns our RAID volumes, it doesn't tell us the size of the partitions themselves. We can find this information with the following command: ```sh -fdisk -l +sudo fdisk -l -Disk /dev/sdb: 1.8 TiB, 2000398934016 bytes, 3907029168 sectors -Disk model: HGST HUS724020AL +Disk /dev/nvme0n1: 476.94 GiB, 512110190592 bytes, 1000215216 sectors +Disk model: WDC CL SN720 SDAQNTW-512G-2000 Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: gpt -Disk identifier: F92B6C5B-2518-4B2D-8FF9-A311DED5845F +Disk identifier: 18EFC16E-F711-4858-8298-9FAA9218E309 + +Device Start End Sectors Size Type +/dev/nvme0n1p1 2048 1048575 1046528 511M EFI System +/dev/nvme0n1p2 1048576 3145727 2097152 1G Linux RAID +/dev/nvme0n1p3 3145728 999161855 996016128 474.9G Linux RAID +/dev/nvme0n1p4 999161856 1000210431 1048576 512M Linux filesystem +/dev/nvme0n1p5 1000211120 1000215182 4063 2M Linux filesystem -Device Start End Sectors Size Type -/dev/sdb1 2048 4095 2048 1M BIOS boot -/dev/sdb2 4096 1864177663 1864173568 888.9G Linux RAID -/dev/sdb3 1864177664 1865226239 1048576 512M Linux filesystem -/dev/sdb4 1865226240 3907024895 2041798656 973.6G Linux RAID -Disk /dev/sda: 1.8 TiB, 2000398934016 bytes, 3907029168 sectors -Disk model: HGST HUS724020AL +Disk /dev/nvme1n1: 476.94 GiB, 512110190592 bytes, 1000215216 sectors +Disk model: WDC CL SN720 SDAQNTW-512G-2000 Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: gpt -Disk identifier: 2E1DCCBA-8808-4D2B-BA33-9FEC3B96ADA8 +Disk identifier: A99678B9-49B0-44E1-B411-52FA2879B617 -Device Start End Sectors Size Type -/dev/sda1 2048 4095 2048 1M BIOS boot -/dev/sda2 4096 1864177663 1864173568 888.9G Linux RAID -/dev/sda3 1864177664 1865226239 1048576 512M Linux filesystem -/dev/sda4 1865226240 3907024895 2041798656 973.6G Linux RAID -/dev/sda5 3907025072 3907029134 4063 2M Linux filesystem +Device Start End Sectors Size Type +/dev/nvme1n1p1 2048 1048575 1046528 511M EFI System +/dev/nvme1n1p2 1048576 3145727 2097152 1G Linux RAID +/dev/nvme1n1p3 3145728 999161855 996016128 474.9G Linux RAID +/dev/nvme1n1p4 999161856 1000210431 1048576 512M Linux filesystem -Disk /dev/md4: 973.5 GiB, 1045265645568 bytes, 2041534464 sectors -Units: sectors of 1 * 512 = 512 bytes -Sector size (logical/physical): 512 bytes / 512 bytes -I/O size (minimum/optimal): 512 bytes / 512 bytes -Disk /dev/md2: 888.8 GiB, 954321600512 bytes, 1863909376 sectors +Disk /dev/md2: 1022 MiB, 1071644672 bytes, 2093056 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes -``` -The `fdisk -l` command also allows you to identify your partition type. This is an important information when it comes to rebuilding your RAID in case of a disk failure. -For **GPT** partitions, the command will return: `Disklabel type: gpt`. - -```sh -Disk /dev/sdb: 1.8 TiB, 2000398934016 bytes, 3907029168 sectors -Disk model: HGST HUS724020AL +Disk /dev/md3: 474.81 GiB, 509824991232 bytes, 995751936 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes -'Disklabel type: gpt' -Disk identifier: F92B6C5B-2518-4B2D-8FF9-A311DED5845F ``` -For **MBR** partitions, the command will return: `Disklabel type: dos`. +The `fdisk -l` command also allows you to identify your partition type. This is an important information when it comes to rebuilding your RAID in case of a disk failure. -```sh -Disk /dev/sda: 2.5 GiB, 2621440000 bytes, 5120000 sectors -Disk model: QEMU HARDDISK -Units: sectors of 1 * 512 = 512 bytes -Sector size (logical/physical): 512 bytes / 512 bytes -I/O size (minimum/optimal): 512 bytes / 512 bytes -'Disklabel type: dos' -Disk identifier: 0x150f6797 -``` +For **GPT** partitions, line 6 will display: `Disklabel type: gpt`. -We can see that `/dev/md2` consists of 888.8GB and `/dev/md4` contains 973.5GB. If we were to run the mount command we can also find out the layout of the disk. +For **MBR** partitions, line 6 will display: `Disklabel type: dos`. + +Still going by the results of `fdisk -l`, we can see that `/dev/md2` consists of 1022 MiB and `/dev/md3` contains 474.81 GiB. If we were to run the mount command we can also find out the layout of the disk. ```sh mount sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime) proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) -udev on /dev type devtmpfs (rw,nosuid,relatime,size=16315920k,nr_inodes=4078980,mode=755) +udev on /dev type devtmpfs (rw,nosuid,relatime,size=16348308k,nr_inodes=4087077,mode=755) devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000) -tmpfs on /run type tmpfs (rw,nosuid,noexec,relatime,size=3266556k,mode=755) -/dev/md2 on / type ext4 (rw,relatime) +tmpfs on /run type tmpfs (rw,nosuid,nodev,noexec,relatime,size=3275192k,mode=755) +/dev/md3 on / type ext4 (rw,relatime) securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime) tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev) tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k) -tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755) -cgroup2 on /sys/fs/cgroup/unified type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate) -cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,name=systemd) +cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot) pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime) -bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700) -cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids) -cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory) -cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event) -cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma) -cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio) -cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct) -cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer) -cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio) -cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices) -cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset) -debugfs on /sys/kernel/debug type debugfs (rw,relatime) +efivarfs on /sys/firmware/efi/efivars type efivarfs (rw,nosuid,nodev,noexec,relatime) +none on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700) +systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=10895) hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M) -mqueue on /dev/mqueue type mqueue (rw,relatime) -systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=45,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=10340) -/dev/md4 on /home type ext3 (rw,relatime) -tmpfs on /run/user/1000 type tmpfs (rw,nosuid,nodev,relatime,size=3266552k,mode=700,uid=1000,gid=1000) +mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime) +debugfs on /sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,relatime) +tracefs on /sys/kernel/tracing type tracefs (rw,nosuid,nodev,noexec,relatime) +fusectl on /sys/fs/fuse/connections type fusectl (rw,nosuid,nodev,noexec,relatime) +configfs on /sys/kernel/config type configfs (rw,nosuid,nodev,noexec,relatime) +/dev/md2 on /boot type ext4 (rw,relatime) +/dev/nvme1n1p1 on /boot/efi type vfat (rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=ascii,shortname=mixed,utf8,errors=remount-ro) +tmpfs on /run/user/1000 type tmpfs (rw,nosuid,nodev,relatime,size=3275188k,nr_inodes=818797,mode=700,uid=1000,gid=1000) ``` Alternatively, the `lsblk` command offers a different view of the partitions: @@ -153,122 +147,170 @@ Alternatively, the `lsblk` command offers a different view of the partitions: ```sh lsblk -NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT -sda 8:0 0 1.8T 0 disk -├─sda1 8:1 0 1M 0 part -├─sda2 8:2 0 888.9G 0 part -│ └─md2 9:2 0 888.8G 0 raid1 / -├─sda3 8:3 0 512M 0 part [SWAP] -├─sda4 8:4 0 973.6G 0 part -│ └─md4 9:4 0 973.5G 0 raid1 /home -└─sda5 8:5 0 2M 0 part -sdb 8:16 0 1.8T 0 disk -├─sdb1 8:17 0 1M 0 part -├─sdb2 8:18 0 888.9G 0 part -│ └─md2 9:2 0 888.8G 0 raid1 / -├─sdb3 8:19 0 512M 0 part [SWAP] -└─sdb4 8:20 0 973.6G 0 part - └─md4 9:4 0 973.5G 0 raid1 /home -``` - -As the disks are currently mounted by default, to remove a disk from the RAID, we first need to unmount the disk, then simulate a failure, and finally remove it. We will remove `/dev/sda4` from the RAID with the following command: - -```sh -umount /dev/md4 +NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT +nvme0n1 259:0 0 476.9G 0 disk +├─nvme0n1p1 259:6 0 511M 0 part +├─nvme0n1p2 259:7 0 1G 0 part +│ └─md2 9:2 0 1022M 0 raid1 /boot +├─nvme0n1p3 259:8 0 474.9G 0 part +│ └─md3 9:3 0 474.8G 0 raid1 / +├─nvme0n1p4 259:9 0 512M 0 part [SWAP] +└─nvme0n1p5 259:10 0 2M 0 part +nvme1n1 259:1 0 476.9G 0 disk +├─nvme1n1p1 259:2 0 511M 0 part /boot/efi +├─nvme1n1p2 259:3 0 1G 0 part +│ └─md2 9:2 0 1022M 0 raid1 /boot +├─nvme1n1p3 259:4 0 474.9G 0 part +│ └─md3 9:3 0 474.8G 0 raid1 / +└─nvme1n1p4 259:5 0 512M 0 part [SWAP] ``` -> [!warning] -> Please note that if you are connected as the user `root`, you may get the following message when you try to unmount the partition (in our case, where our md4 partition is mounted in /home): -> -> 
umount: /home: target is busy
-> -> In this case, you must log out as the user root and connect as a local user (in our case `debian`), and use the following command: -> -> 
debian@ns000000:/$ sudo umount /dev/md4
+We take note of the devices, partitions and their mount points. + +From the above commands and results, we have: + +- Two RAID arrays: `/dev/md2` and `/dev/md3`. +- Partitions part of the RAID: `/boot` and `/`. +- Partitions not part of the RAID: `/boot/efi` and [SWAP]. + +> [!primary] +> Most modern motherboards use UEFI mode (EFI). When a server is intalled, two ESPs partitions (labelled EFI_SYSPART by OVHcloud) are created on the disks used for RAID, however, only one partition is automatically mounted in `/boot/efi` by the system after install. It is important to understand the role of this partition and how to manage it. > -> If you do not have a local user, you need to [create one](/pages/bare_metal_cloud/dedicated_servers/changing_root_password_linux_ds). +> For more information on UEFI, consult the following [guide](https://uefi.org/about){.external} -This will provide us with the following output: +To check whether a server runs on BIOS mode or UEFI mode, run the following command: ```sh -sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime) -proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) -udev on /dev type devtmpfs (rw,nosuid,relatime,size=16315920k,nr_inodes=4078980,mode=755) -devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000) -tmpfs on /run type tmpfs (rw,nosuid,noexec,relatime,size=3266556k,mode=755) -/dev/md2 on / type ext4 (rw,relatime) -securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime) -tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev) -tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k) -tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755) -cgroup2 on /sys/fs/cgroup/unified type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate) -cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,name=systemd) -pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime) -bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700) -cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids) -cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory) -cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event) -cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma) -cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio) -cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct) -cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer) -cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio) -cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices) -cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset) -debugfs on /sys/kernel/debug type debugfs (rw,relatime) -hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M) -mqueue on /dev/mqueue type mqueue (rw,relatime) -systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=45,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=10340) -tmpfs on /run/user/1000 type tmpfs (rw,nosuid,nodev,relatime,size=3266552k,mode=700,uid=1000,gid=1000) +[user@server_ip ~]# [ -d /sys/firmware/efi ] && echo UEFI || echo BIOS ``` -As we can see the, entry of `/dev/md4` is no longer mounted. However, the RAID is still active, so we need to simulate a failure to remove the disk. We can do this with the following command: +### Simulating a disk failure + +Now that we have all the necessary information, we can simulate a disk failure and proceed with the tests. In this example, we will fail the disk `nvme0n1`. + +The preferred to do this is via the OVHcloud rescue environment. + +First reboot in rescue menu and log in with the credentials provided. + +To remove a disk from the RAID, the first step is to mark it as **Failed** and remove the partitions from their respective RAID arrays. ```sh -sudo mdadm --fail /dev/md4 /dev/sda4 +root@rescue12-customer-ca (nsxxxxx.ip-xx-xx-xx.eu) ~ # cat /proc/mdstat +Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] [multipath] [faulty] +md3 : active raid1 nvme0n1p3[0] nvme1n1p3[1] + 497875968 blocks super 1.2 [2/2] [UU] + bitmap: 0/4 pages [0KB], 65536KB chunk + +md2 : active raid1 nvme0n1p2[2] nvme1n1p2[1] + 1046528 blocks super 1.2 [2/2] [UU] + +unused devices: ``` -We have now simulated a failure of the RAID. The next step is to remove the partition from the RAID array with the following command: +From the above output, nvme0n1 consists of two partitions in raid which are **nvme0n1p2** and **nvme0n1p3**. + +#### Removing the failed disk + +First we mark the partitions **nvme0n1p2** and **nvme0n1p3** as failed. ```sh -sudo mdadm --remove /dev/md4 /dev/sda4 +sudo mdadm --manage /dev/md2 --fail /dev/nvme0n1p2 + +# mdadm: set /dev/nvme0n1p2 faulty in /dev/md2 ``` -You can verify that the partition has been removed with the following command: +```sh +sudo mdadm --manage /dev/md3 --fail /dev/nvme0n1p3 + +# mdadm: set /dev/nvme0n1p3 faulty in /dev/md3 +``` + +We have now simulated a failure of the RAID, when we run the `cat /proc/mdstat` command, we have the following output ```sh -cat /proc/mdstat +root@rescue12-customer-ca (nsxxxxx.ip-xx-xx-xx.eu) ~ # cat /proc/mdstat +Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] [multipath] [faulty] +md3 : active raid1 nvme0n1p3[0](F) nvme1n1p3[1] + 497875968 blocks super 1.2 [2/1] [_U] + bitmap: 0/4 pages [0KB], 65536KB chunk + +md2 : active raid1 nvme0n1p2[2](F) nvme1n1p2[1] + 1046528 blocks super 1.2 [2/1] [_U] -Personalities : [raid1] [linear] [multipath] [raid0] [raid6] [raid5] [raid4] [raid10] -md2 : active raid1 sda2[1] sdb2[0] - 931954688 blocks super 1.2 [2/2] [UU] - bitmap: 4/7 pages [16KB], 65536KB chunk - -md4 : active raid1 sdb4[1] - 1020767232 blocks super 1.2 [2/1] [_U] - bitmap: 0/8 pages [0KB], 65536KB chunk - unused devices: ``` -The following command will verify that the partition has been removed: +As we can see above, the [F] next to the partitions indicates that the disk has failed or is faulty. Next, we remove these partitions from the RAID arrays to completely remove the disk from RAID. + +```sh +sudo mdadm --manage /dev/md2 --remove /dev/nvme0n1p2 + +# mdadm: hot removed /dev/nvme0n1p2 from /dev/md2 +``` + +```sh +sudo mdadm --manage /dev/md3 --remove /dev/nvme0n1p3 + +# mdadm: hot removed /dev/nvme0n1p3 from /dev/md3 +``` + + +If we runnthe following command: ```sh -mdadm --detail /dev/md4 +parted /dev/nvme0n1 +GNU Parted 3.5 +Using /dev/nvme0n1 +Welcome to GNU Parted! Type 'help' to view a list of commands. +(parted) p +Error: /dev/nvme0n1: unrecognised disk label +Model: WDC CL SN720 SDAQNTW-512G-2000 (nvme) +Disk /dev/nvme0n1: 512GB +Sector size (logical/physical): 512B/512B +Partition Table: unknown +Disk Flags: +``` + +We see that our disk has been successfully wiped. + +Our RAID status should now look like this: + +```sh +root@rescue12-customer-ca (ns5009452.ip-51-222-254.net) ~ # cat /proc/mdstat +Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] [multipath] [faulty] +md3 : active raid1 nvme1n1p3[1] + 497875968 blocks super 1.2 [2/1] [_U] + bitmap: 0/4 pages [0KB], 65536KB chunk + +md2 : active raid1 nvme1n1p2[1] + 1046528 blocks super 1.2 [2/1] [_U] -/dev/md4: +unused devices: n +``` + +From the results above, we can see that only two partitions now appear in the RAID arrays. We have successfully failed the disk nvme0n1 and we can now proceed with the replacement. + +For more information on how to prepare and request a disk replacement, consult this [guide](/pages/bare_metal_cloud/dedicated_servers/disk_replacement) + +If you run the following command, you can have more details on the raid array. + + +```sh +mdadm --detail /dev/md3 + +/dev/md3: Version : 1.2 - Creation Time : Tue Jan 24 15:35:02 2023 + Creation Time : Fri Aug 1 14:51:13 2025 Raid Level : raid1 - Array Size : 1020767232 (973.48 GiB 1045.27 GB) - Used Dev Size : 1020767232 (973.48 GiB 1045.27 GB) + Array Size : 497875968 (474.81 GiB 509.82 GB) + Used Dev Size : 497875968 (474.81 GiB 509.82 GB) Raid Devices : 2 Total Devices : 1 Persistence : Superblock is persistent Intent Bitmap : Internal - Update Time : Tue Jan 24 16:28:03 2023 + Update Time : Fri Aug 1 15:56:17 2025 State : clean, degraded Active Devices : 1 Working Devices : 1 @@ -277,57 +319,272 @@ mdadm --detail /dev/md4 Consistency Policy : bitmap - Name : md4 - UUID : 7b5c1d80:0a7ab4c2:e769b5e5:9c6eaa0f - Events : 21 + Name : md3 + UUID : b383c3d5:7fb1bb5e:6b7c4d96:6ea817ff + Events : 215 Number Major Minor RaidDevice State - 0 0 0 removed - 1 8 20 1 active sync /dev/sdb4 + 1 259 4 1 active sync /dev/nvme1n1p3 ``` ### Rebuilding the RAID -Once the disk has been replaced, we need to copy the partition table from a healthy disk (in this example, sdb) to the new one (sda) with the following command: +> [!primary] +> This process might be different depending on the operating system you have installed on your server. We recommend that you consult the official documentation of your operating system to have access to the proper commands. +> + +In this situation, we will explore two cases + +- Dedicated servers with UEFI +- Dedicated servers with BIOS + + +> [!warning] +> +> With most cases, after a disk replacement, the server is able to reboot in normal mode (on the healthy disk) and these steps can be done in normal mode. However, if the server is not able to reboot after a disk replacement, it will be rebooted in rescue mode to proceed with the raid rebuild. These steps apply as well. +> + +#### Dedicated server with UEFI + +The steps described in this section apply to servers whose main disk containing the EFI partition mounted on /boot/efi failed, was replaced, and the server no longer boots in normal mode. If the main disk containing this partition was not replaced and the server boots in normal mode, please follow the steps for [Dedicated servers with BIOS]() and recreate an EFI partition in the secondary disk as described below. + +**Understanding the EFI partition** + +An EFI partition, is a partition which can contain the boot loaders, boot managers, or kernel images of an installed operating system. It also contains system utility programs designed to be run before the operating system boots, as well as data files such as error logs. + +By default, this partition is duplicated when one of our installation template is installed in RAID, but this partition is not included in the RAID. + +During installation, this partition is mounted on `/boot/efi` and the disk on which it is mounted is selected randomly by the operating system. Thus, your system has two EFI partitions, but only one is mounted in the RAID. + +Example: + +```sh +lsblk + +NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT +nvme0n1 259:0 0 476.9G 0 disk +├─nvme0n1p1 259:6 0 511M 0 part +├─nvme0n1p2 259:7 0 1G 0 part +│ └─md2 9:2 0 1022M 0 raid1 /boot +├─nvme0n1p3 259:8 0 474.9G 0 part +│ └─md3 9:3 0 474.8G 0 raid1 / +├─nvme0n1p4 259:9 0 512M 0 part [SWAP] +└─nvme0n1p5 259:10 0 2M 0 part +nvme1n1 259:1 0 476.9G 0 disk +├─nvme1n1p1 259:2 0 511M 0 part /boot/efi +├─nvme1n1p2 259:3 0 1G 0 part +│ └─md2 9:2 0 1022M 0 raid1 /boot +├─nvme1n1p3 259:4 0 474.9G 0 part +│ └─md3 9:3 0 474.8G 0 raid1 / +└─nvme1n1p4 259:5 0 512M 0 part [SWAP] +``` + +From the example above, we see that we have two identical partitions (nvme0n1p1 and nvme1n1p1) but only nvme1n1p1 is mounted on `/boot/efi`. + +We can also use the following command to confirm that two EFI partitions were actually created after install: + +```sh +lsblk -f + +NAME FSTYPE FSVER LABEL UUID FSAVAIL FSUSE% MOUNTPOINT +nvme0n1 +│ +├─nvme0n1p1 vfat FAT16 EFI_SYSPART 461C-DB56 +├─nvme0n1p2 +│ linux_ 1.2 md2 83719c5c-2a27-2a56-5268-7d49d8a1d84f +│ └─md2 +│ ext4 1.0 boot 4de80ae0-dd90-4256-9135-1735e7be4b4d 851.8M 7% /boot +├─nvme0n1p3 +│ linux_ 1.2 md3 b383c3d5-7fb1-bb5e-6b7c-4d966ea817ff +│ └─md3 +│ ext4 1.0 root 9bf386b6-9523-46bf-b8e5-4b8cc7c5786f 441.3G 0% / +├─nvme0n1p4 +│ swap 1 swap-nvme0n1p4 356439fe-0539-45ce-9eff-40b616689b0c [SWAP] +└─nvme0n1p5 + iso966 Jolie config-2 + 2025-08-01-14-51-51-00 +nvme1n1 +│ +├─nvme1n1p1 vfat FAT16 EFI_SYSPART 4629-D183 504.9M 1% /boot/efi +├─nvme1n1p2 +│ linux_ 1.2 md2 83719c5c-2a27-2a56-5268-7d49d8a1d84f +│ └─md2 +│ ext4 1.0 boot 4de80ae0-dd90-4256-9135-1735e7be4b4d 851.8M 7% /boot +├─nvme1n1p3 +│ linux_ 1.2 md3 b383c3d5-7fb1-bb5e-6b7c-4d966ea817ff +│ └─md3 +│ ext4 1.0 root 9bf386b6-9523-46bf-b8e5-4b8cc7c5786f 441.3G 0% / +└─nvme1n1p4 + swap 1 swap-nvme1n1p4 + 9bf292e8-0145-4d2f-b891-4cef93c0d209 [SWAP] +``` + +From the results above, we can see two partitions (**nvme0n1p1** and **nvme1n1p1**), with identical size (504.9M). Both partitions have the LABEL: `EFI_SYSPART` but only one is mounted on `/boot/efi`. + +In general, this is a partition whose contents do not change much, except when there is a GRUB or the kernel updated. In this case, we recommend running an automatic or manual script to synchronise both partitions often. + +The script below can be used: + +```sh +set -euo pipefail + +MAIN_PARTITION="$(findmnt -n -o SOURCE /boot/efi)" +MOUNTPOINT="/var/lib/grub/esp" + +mkdir -p "${MOUNTPOINT}" + +while read -r partition; do + if [[ "${partition}" == "${MAIN_PARTITION}" ]]; then + continue + fi + echo "Working on ${partition}" + mount "${partition}" "${MOUNTPOINT}" + rsync -ax "/boot/efi/" "${MOUNTPOINT}/" + umount "${MOUNTPOINT}" +done < <(blkid -o device -t LABEL=EFI_SYSPART) +``` + +Once the disk has been replaced, we need to copy the partition table from the healthy disk (in this example, nvme1n1) to the new one (nvme0n1). **For GPT partitions** ```sh -sgdisk -R /dev/sda /dev/sdb +sudo sgdisk -R /dev/sda /dev/sdb ``` The command should be in this format: `sgdisk -R /dev/newdisk /dev/healthydisk` +In our example: + +```sh +sudo sgdisk -R /dev/nvme0n1 /dev/nvme1n1 +``` + Once this is done, the next step is to randomize the GUID of the new disk to prevent GUID conflicts with other disks: ```sh -sgdisk -G /dev/sda +sudo sgdisk -G /dev/nvme0n1 ``` +If you receive a message like this: + +```console +Warning: The kernel is still using the old partition table. +The new table will be used at the next reboot or after you +run partprobe(8) or kpartx(8) +The operation has completed successfully. +``` + +You can simply run the `partprobe` command or reboot the server when the raid rebuilt is done. + **For MBR partitions** -Once the disk has been replaced, we need to copy the partition table from a healthy disk (in this example, sdb) to the new one (sda) with the following command: +```sh +sudo sfdisk -d /dev/sdb | sfdisk /dev/sda +``` + +The command should be in this format: `sfdisk -d /dev/healthydisk | sfdisk /dev/newdisk`. + +We can now rebuild the RAID array. The following code snippet shows how we can add the new partitions (nvme0n1p2 and nvme0n1p3) back in the RAID array. + + +```sh +sudo mdadm --add /dev/md2 /dev/nvme0n1p2 + +# mdadm: added /dev/nvme0n1p2 + +sudo mdadm --add /dev/md3 /dev/nvme0n1p3 + +# mdadm: re-added /dev/nvme0n1p3 + +cat /proc/mdstat +Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] [multipath] [faulty] +md3 : active raid1 nvme0n1p3[2] nvme1n1p3[1] + 497875968 blocks super 1.2 [2/1] [_U] + [>....................] recovery = 0.1% (801920/497875968) finish=41.3min speed=200480K/sec + bitmap: 0/4 pages [0KB], 65536KB chunk + +md2 : active raid1 nvme0n1p2[2] nvme1n1p2[1] + 1046528 blocks super 1.2 [2/2] [UU] +``` + +Once the raid rebuild is complete, run the following command to make sure that the partitions were properly added to the raid: + +```sh +``` + +The next step is to recreated the EFI partition on the newly added disk and format it. Please keep in mind that the content of this partition on the remaining drive (in our exmaple: nvme1n1) will be replicated to the new disk in order to enable our server to boot back into normal mode. This is possible because we kept the partition in sync while the server was running normally before the disk failure. + +It is possible that this process does not work if there was a kernel or grub update and both partitions were not synchronized. + + + +#### Dedicated servers with BIOS + +Use the appropriate commands: + +**For GPT partitions** ```sh -sfdisk -d /dev/sdb | sfdisk /dev/sda +sudo sgdisk -R /dev/sda /dev/sdb +``` + +The command should be in this format: `sgdisk -R /dev/newdisk /dev/healthydisk` + +In our example: + +```sh +sudo sgdisk -R /dev/nvme0n1 /dev/nvme1n1 +``` + +Once this is done, the next step is to randomize the GUID of the new disk to prevent GUID conflicts with other disks: + +```sh +sudo sgdisk -G /dev/nvme0n1 +``` + +If you receive a message like this: + +```console +Warning: The kernel is still using the old partition table. +The new table will be used at the next reboot or after you +run partprobe(8) or kpartx(8) +The operation has completed successfully. +``` + +You can simply run the `partprobe` command or reboot the server when the raid rebuilt is done. + +**For MBR partitions** + +```sh +sudo sfdisk -d /dev/sdb | sfdisk /dev/sda ``` The command should be in this format: `sfdisk -d /dev/healthydisk | sfdisk /dev/newdisk` -We can now rebuild the RAID array. The following code snippet shows how we can rebulid the `/dev/md4` partition layout with the recently-copied sda partition table: +We can now rebuild the RAID array. The following code snippet shows how we can add the new partitions (nvme0n1p2 and nvme0n1p3) back in the RAID array. + ```sh -mdadm --add /dev/md4 /dev/sda4 +sudo mdadm --add /dev/md2 /dev/nvme0n1p2 + +# mdadm: added /dev/nvme0n1p2 + +sudo mdadm --add /dev/md3 /dev/nvme0n1p3 + +# mdadm: re-added /dev/nvme0n1p3 + cat /proc/mdstat Personalities : [raid1] [linear] [multipath] [raid0] [raid6] [raid5] [raid4] [raid10] -md2 : active raid1 sda2[1] sdb2[0] - 931954688 blocks super 1.2 [2/2] [UU] - bitmap: 1/7 pages [4KB], 65536KB chunk +md3 : active raid1 nvme0n1p3[0] nvme1n1p3[1] + 497875968 blocks super 1.2 [2/1] [_U] + [============>........] recovery = 64.8% (322969856/497875968) finish=7.2min speed=401664K/sec + bitmap: 4/4 pages [16KB], 65536KB chunk -md4 : active raid1 sda4[0] sdb4[1] - 1020767232 blocks super 1.2 [2/2] [UU] - bitmap: 0/8 pages [0KB], 65536KB chunk +md2 : active raid1 nvme0n1p2[2] nvme1n1p2[1] + 1046528 blocks super 1.2 [2/2] [UU] unused devices: ``` @@ -335,43 +592,86 @@ unused devices: We can verify the RAID details with the following command: ```sh -mdadm --detail /dev/md4 - -/dev/md4: - Version : 1.2 - Creation Time : Tue Jan 24 15:35:02 2023 +sudo mdadm --detail /dev/md3 & sudo mdadm --detail /dev/md2 +[1] 2096 +/dev/md3: + Version : 1.2 + Creation Time : Fri Aug 1 14:51:13 2025 Raid Level : raid1 - Array Size : 1020767232 (973.48 GiB 1045.27 GB) - Used Dev Size : 1020767232 (973.48 GiB 1045.27 GB) + Array Size : 497875968 (474.81 GiB 509.82 GB) + Used Dev Size : 497875968 (474.81 GiB 509.82 GB) Raid Devices : 2 Total Devices : 2 Persistence : Superblock is persistent Intent Bitmap : Internal - Update Time : Tue Jan 24 17:02:55 2023 + Update Time : Fri Aug 1 16:48:03 2025 State : clean Active Devices : 2 Working Devices : 2 Failed Devices : 0 Spare Devices : 0 - Rebuild Status : 21% complete +Consistency Policy : bitmap - UUID : 7f39d062:9f16a016:a4d2adc2:26fd5302 - Events : 0.95 + Name : md3 + UUID : b383c3d5:7fb1bb5e:6b7c4d96:6ea817ff + Events : 538 Number Major Minor RaidDevice State - 0 8 2 0 spare rebuilding /dev/sda4 - 1 8 18 1 active sync /dev/sdb4 + 0 259 8 0 active sync /dev/nvme0n1p3 + 1 259 4 1 active sync /dev/nvme1n1p3 +/dev/md2: + Version : 1.2 + Creation Time : Fri Aug 1 14:51:13 2025 + Raid Level : raid1 + Array Size : 1046528 (1022.00 MiB 1071.64 MB) + Used Dev Size : 1046528 (1022.00 MiB 1071.64 MB) + Raid Devices : 2 + Total Devices : 2 + Persistence : Superblock is persistent + + Update Time : Fri Aug 1 16:45:59 2025 + State : clean + Active Devices : 2 + Working Devices : 2 + Failed Devices : 0 + Spare Devices : 0 + +Consistency Policy : resync + + Name : md2 + UUID : 83719c5c:2a272a56:52687d49:d8a1d84f + Events : 26 + + Number Major Minor RaidDevice State + 2 259 7 0 active sync /dev/nvme0n1p2 + 1 259 3 1 active sync /dev/nvme1n1p2 ``` -The RAID has now been rebuilt, but we still need to mount the partition (`/dev/md4` in this example) with the following command: +The RAID has now been rebuilt. + +Next, mount the partitions ```sh -mount /dev/md4 /home +sudo mount /dev/md2 /boot +sudo mount /dev/md3 / ``` + + + + +> [!primary] +> While this process helps understand what to do when a disk fails, it is a but simplistic. For further diagnostics, you can consult this section of the guide. +> + + + + + + ## Go Further [Hot Swap - Software RAID](/pages/bare_metal_cloud/dedicated_servers/hotswap_raid_soft) From 611ea4e37d3cd6732d249e080f01879be1a6daf9 Mon Sep 17 00:00:00 2001 From: jessica Date: Tue, 5 Aug 2025 08:59:44 -0400 Subject: [PATCH 4/8] update --- .../raid_soft/guide.en-gb.md | 227 +++++++++++++++++- 1 file changed, 219 insertions(+), 8 deletions(-) diff --git a/pages/bare_metal_cloud/dedicated_servers/raid_soft/guide.en-gb.md b/pages/bare_metal_cloud/dedicated_servers/raid_soft/guide.en-gb.md index b0e32354a5b..29eb737db02 100644 --- a/pages/bare_metal_cloud/dedicated_servers/raid_soft/guide.en-gb.md +++ b/pages/bare_metal_cloud/dedicated_servers/raid_soft/guide.en-gb.md @@ -1,7 +1,7 @@ --- title: How to configure and rebuild software RAID excerpt: Find out how to verify the state of the software RAID of your server and rebuild it after a disk replacement -updated: 2023-08-21 +updated: 2025-08-xx --- ## Objective @@ -255,7 +255,7 @@ sudo mdadm --manage /dev/md3 --remove /dev/nvme0n1p3 ``` -If we runnthe following command: +If we run the following command: ```sh parted /dev/nvme0n1 @@ -347,9 +347,14 @@ In this situation, we will explore two cases #### Dedicated server with UEFI -The steps described in this section apply to servers whose main disk containing the EFI partition mounted on /boot/efi failed, was replaced, and the server no longer boots in normal mode. If the main disk containing this partition was not replaced and the server boots in normal mode, please follow the steps for [Dedicated servers with BIOS]() and recreate an EFI partition in the secondary disk as described below. +In this section, we will focus on the steps to follow after the replacement of the main disk with the following scenario: -**Understanding the EFI partition** +- The EFI partition was mounted on this disk +- The content of the EFI partition have not been updated/they have been updated and both partitions have been synchronized. + +If your secondary disk was replaced, please follow the steps for [Dedicated servers with BIOS]() and recreate an EFI partition in the secondary disk as described below. + +/// details | **Understanding the EFI partition** An EFI partition, is a partition which can contain the boot loaders, boot managers, or kernel images of an installed operating system. It also contains system utility programs designed to be run before the operating system boots, as well as data files such as error logs. @@ -380,7 +385,7 @@ nvme1n1 259:1 0 476.9G 0 disk └─nvme1n1p4 259:5 0 512M 0 part [SWAP] ``` -From the example above, we see that we have two identical partitions (nvme0n1p1 and nvme1n1p1) but only nvme1n1p1 is mounted on `/boot/efi`. +From the example above, we see that we have two identical partitions (nvme0n1p1 and nvme1n1p1) but only **nvme1n1p1** is mounted on `/boot/efi`. We can also use the following command to confirm that two EFI partitions were actually created after install: @@ -422,7 +427,7 @@ nvme1n1 From the results above, we can see two partitions (**nvme0n1p1** and **nvme1n1p1**), with identical size (504.9M). Both partitions have the LABEL: `EFI_SYSPART` but only one is mounted on `/boot/efi`. -In general, this is a partition whose contents do not change much, except when there is a GRUB or the kernel updated. In this case, we recommend running an automatic or manual script to synchronise both partitions often. +In general, this is a partition is not often updated, except when there are relevant updates. In this case, we recommend running an automatic or manual script to copy the primary EFI partition to the second one when needed. Additionally, a boot entry for the secondary ESP can then be added manually using [efibootmgr](https://wiki.archlinux.org/title/Unified_Extensible_Firmware_Interface#efibootmgr){.external} The script below can be used: @@ -445,6 +450,10 @@ while read -r partition; do done < <(blkid -o device -t LABEL=EFI_SYSPART) ``` +/// + +**Disk replacement** + Once the disk has been replaced, we need to copy the partition table from the healthy disk (in this example, nvme1n1) to the new one (nvme0n1). **For GPT partitions** @@ -512,12 +521,214 @@ md2 : active raid1 nvme0n1p2[2] nvme1n1p2[1] Once the raid rebuild is complete, run the following command to make sure that the partitions were properly added to the raid: ```sh +lsblk -fA +NAME FSTYPE FSVER LABEL UUID FSAVAIL FSUSE% MOUNTPOINTS +nvme1n1 +├─nvme1n1p1 vfat FAT16 EFI_SYSPART 4629-D183 +├─nvme1n1p2 linux_raid_member 1.2 md2 83719c5c-2a27-2a56-5268-7d49d8a1d84f +│ └─md2 ext4 1.0 boot 4de80ae0-dd90-4256-9135-1735e7be4b4d +├─nvme1n1p3 linux_raid_member 1.2 md3 b383c3d5-7fb1-bb5e-6b7c-4d966ea817ff +│ └─md3 ext4 1.0 root 9bf386b6-9523-46bf-b8e5-4b8cc7c5786f +└─nvme1n1p4 swap 1 swap-nvme1n1p4 9bf292e8-0145-4d2f-b891-4cef93c0d209 +nvme0n1 +├─nvme0n1p1 +├─nvme0n1p2 linux_raid_member 1.2 md2 83719c5c-2a27-2a56-5268-7d49d8a1d84f +│ └─md2 ext4 1.0 boot 4de80ae0-dd90-4256-9135-1735e7be4b4d +├─nvme0n1p3 linux_raid_member 1.2 md3 b383c3d5-7fb1-bb5e-6b7c-4d966ea817ff +│ └─md3 ext4 1.0 root 9bf386b6-9523-46bf-b8e5-4b8cc7c5786f +└─nvme0n1p4 +``` + +From the above results, we can see that the partitions of our newly added disk have been properly added to the raid, however, the EFI partition was not duplicated, which is normal since it is not included in the raid. We also see that the [SWAP] partition **nvme0n1p4** does not have the label anymore. We also need reattribute its properties. + +> [!warning] +> The examples above are merely illustrating the necessary steps based on a typical server configuration. The information in the output table depends on your server's hardware and its partition scheme. When in doubt, consult the documentation of your operating system. +> +> If you require professional assistance with server administration, consider the details in the Go further section of this guide. +> + +The next step is to create an EFI partition (**nvme0n1p1**) on the newly added disk and format it. Please keep in mind that the content of this partition on the remaining drive (in our exmaple: nvme1n1) will be replicated to the new disk in order to enable our server to boot back into normal mode. This is possible because we kept the partition in sync while the server was running normally before the disk failure. + +It is possible that this process does not work if there was a kernel or grub update and both partitions were not synchronized. In this case, please consult the following guide []. + +We create a vfat partition + +```sh +mkfs.vfat /dev/nvme0n1p1 +``` + +Next, we lable the partition as `EFI_SYSPART` (this naming is proper to OVHcloud) + +```sh +fatlabel /dev/nvme0n1p1 EFI_SYSPART +``` + +Now we have to duplicate the partition from nvme1n1p1 (healthy partition), to nvme0n1p1 (new partition). To do so, we start by creating two folders named `old` and `new`. + +```sh +mkdir old new +``` + +Next, we mount mount **nvme1n1p1** in the `old` folder and **nvme0n1p1** in the `new` folder: + +```sh +mount /dev/nvme1n1p1 old +mount /dev/nvme0n1p1 new +``` + +Next, we copy the files from `old` to `new`. Depending on your operating system, you will have a similar output. Here we are using debian: + +```sh +rsync -axv old/ new/ +sending incremental file list +EFI/ +EFI/debian/ +EFI/debian/BOOTX64.CSV +EFI/debian/fbx64.efi +EFI/debian/grub.cfg +EFI/debian/grubx64.efi +EFI/debian/mmx64.efi +EFI/debian/shimx64.efi +``` + +Once this is done, we umount the **nvme0n1p1** partition. + +```sh +umount nvme0n1p1 +``` + +Next, we mount the partition containing our files on `/mnt`. In our example, that partition is `md3`. + +```sh +mount /dev/md3 /mnt +``` + +Next, we mount the following directories: + +```sh +mount --types proc /proc /mnt/proc +mount --rbind /sys /mnt/sys +mount --make-rslave /mnt/sys +mount --rbind /dev /mnt/dev +mount --make-rslave /mnt/dev +mount --bind /run /mnt/run +mount --make-slave /mnt/run +``` + +Next, we use the `chroot` command to access the mountpoint and make sure the new EFI partition has been properly created and the system recongnises both EFI partitions: + +```sh +chroot /mnt +``` + +```sh +blkid -t LABEL=EFI_SYSPART +/dev/nvme1n1p1: SEC_TYPE="msdos" LABEL_FATBOOT="EFI_SYSPART" LABEL="EFI_SYSPART" UUID="4629-D183" BLOCK_SIZE="512" TYPE="vfat" PARTLABEL="primary" PARTUUID="889f241b-49c3-4031-b5c9-60df0746f98f" +/dev/nvme0n1p1: SEC_TYPE="msdos" LABEL_FATBOOT="EFI_SYSPART" LABEL="EFI_SYSPART" UUID="521F-300B" BLOCK_SIZE="512" TYPE="vfat" PARTLABEL="primary" PARTUUID="02bf2b2d-7ada-4461-ba50-07683519f65d" +``` + +Still in the `chroot` environment, we create the `/boot/efi` folder in order to mount the EFI partition **nvme0n1p1** in it: + +```sh +cd /mnt +mkdir /boot/efi +mount /dev/nvme0n1p1 /boot/efi +``` + +Next, we install the grub bootloader to make sure the server can reboot in normal mode on the new disk (you won't have to do this if the disk replaced is the secondary disk. Simply duplicate the EFI partition and proceed to the RAID rebuild, then enable the [SWAP] partition (if applicable)): + +```sh +grub-install --efi-directory=/boot/efi /dev/nvme0n1p1 ``` -The next step is to recreated the EFI partition on the newly added disk and format it. Please keep in mind that the content of this partition on the remaining drive (in our exmaple: nvme1n1) will be replicated to the new disk in order to enable our server to boot back into normal mode. This is possible because we kept the partition in sync while the server was running normally before the disk failure. +Next, we exit the `chroot` environment, then we recreate our [SWAP] partition **nvme0n1p4** and add the label `swap-nvmenxxx`: + +```sh +mkswap /dev/nvme0n1p4 -L swap-nvme0n1p4 + +Setting up swapspace version 1, size = 512 MiB (536866816 bytes) +LABEL=swap-nvme0n1p4, UUID=256215f9-7694-4a88-aa47-335558fc6cd8 +``` + +We verify that the lable has been properly applied: + +```sh +NAME FSTYPE FSVER LABEL UUID FSAVAIL FSUSE% MOUNTPOINTS +nvme1n1 +├─nvme1n1p1 vfat FAT16 EFI_SYSPART 4629-D183 +├─nvme1n1p2 linux_raid_member 1.2 md2 83719c5c-2a27-2a56-5268-7d49d8a1d84f +│ └─md2 ext4 1.0 boot 4de80ae0-dd90-4256-9135-1735e7be4b4d +├─nvme1n1p3 linux_raid_member 1.2 md3 b383c3d5-7fb1-bb5e-6b7c-4d966ea817ff +│ └─md3 ext4 1.0 root 9bf386b6-9523-46bf-b8e5-4b8cc7c5786f 441.1G 0% /mnt +└─nvme1n1p4 swap 1 swap-nvme1n1p4 9bf292e8-0145-4d2f-b891-4cef93c0d209 +nvme0n1 +├─nvme0n1p1 vfat FAT16 EFI_SYSPART 521F-300B +├─nvme0n1p2 linux_raid_member 1.2 md2 83719c5c-2a27-2a56-5268-7d49d8a1d84f +│ └─md2 ext4 1.0 boot 4de80ae0-dd90-4256-9135-1735e7be4b4d +├─nvme0n1p3 linux_raid_member 1.2 md3 b383c3d5-7fb1-bb5e-6b7c-4d966ea817ff +│ └─md3 ext4 1.0 root 9bf386b6-9523-46bf-b8e5-4b8cc7c5786f 441.1G 0% /mnt +└─nvme0n1p4 swap 1 swap-nvme0n1p4 256215f9-7694-4a88-aa47-335558fc6cd8 +``` + +Take note of the UUID of both swap partitions **nvme0n1p4** and **nvme1n1p4**. + +Next, we need to add the new UUID of the swap partition in `/etc/fstab` using the `chroot` environment: -It is possible that this process does not work if there was a kernel or grub update and both partitions were not synchronized. +```sh +chroot /mnt +``` + +```sh +cat /etc/fstab + +UUID=9bf386b6-9523-46bf-b8e5-4b8cc7c5786f / ext4 defaults 0 1 +UUID=4de80ae0-dd90-4256-9135-1735e7be4b4d /boot ext4 defaults 0 0 +LABEL=EFI_SYSPART /boot/efi vfat defaults 0 1 +UUID=356439fe-0539-45ce-9eff-40b616689b0c swap swap defaults 0 0 +UUID=9bf292e8-0145-4d2f-b891-4cef93c0d209 swap swap defaults 0 0 +``` +We need to replace UUID `356439fe-0539-45ce-9eff-40b616689b0c` with `256215f9-7694-4a88-aa47-335558fc6cd8` since `9bf292e8-0145-4d2f-b891-4cef93c0d209` is the UUID of our second swap partition **nvme1n1p4** and we are not replacing it. You can use a text editor such as `vi` or `nano` to edit the file: + +```sh +nano etc/fstab +``` + +```sh +cat /etc/fstab + +UUID=9bf386b6-9523-46bf-b8e5-4b8cc7c5786f / ext4 defaults 0 1 +UUID=4de80ae0-dd90-4256-9135-1735e7be4b4d /boot ext4 defaults 0 0 +LABEL=EFI_SYSPART /boot/efi vfat defaults 0 1 +UUID=256215f9-7694-4a88-aa47-335558fc6cd8 swap swap defaults 0 0 +UUID=9bf292e8-0145-4d2f-b891-4cef93c0d209 swap swap defaults 0 0 +``` + +Next, we make sure everything is properly mounted: + +```sh +mount -av + +/ : ignored +/boot : successfully mounted +/boot/efi : successfully mounted +swap : ignored +swap : ignored +``` + +We enable the swap partition: + +```sh +swapon -av + +swapon: /dev/nvme0n1p4: found signature [pagesize=4096, signature=swap] +swapon: /dev/nvme0n1p4: pagesize=4096, swapsize=536870912, devsize=536870912 +swapon /dev/nvme0n1p4 +swapon: /dev/nvme1n1p4: found signature [pagesize=4096, signature=swap] +swapon: /dev/nvme1n1p4: pagesize=4096, swapsize=536870912, devsize=536870912 +swapon /dev/nvme1n1p4 +``` +We have now succesffuly completed the RAID rebuild on the server and we can now reboot the server in normal mode: #### Dedicated servers with BIOS From 15f5d8dee27a128c5f5104546a95d11fda8aab16 Mon Sep 17 00:00:00 2001 From: jessica Date: Tue, 12 Aug 2025 14:44:28 -0400 Subject: [PATCH 5/8] more updates --- .../raid_soft/guide.en-gb.md | 897 +++----------- .../raid_soft_bios/guide.en-gb.md | 604 ++++++++++ .../raid_soft_uefi/guide.en-gb.md | 1038 +++++++++++++---- 3 files changed, 1583 insertions(+), 956 deletions(-) create mode 100644 pages/bare_metal_cloud/dedicated_servers/raid_soft_bios/guide.en-gb.md diff --git a/pages/bare_metal_cloud/dedicated_servers/raid_soft/guide.en-gb.md b/pages/bare_metal_cloud/dedicated_servers/raid_soft/guide.en-gb.md index 29eb737db02..d68c723719b 100644 --- a/pages/bare_metal_cloud/dedicated_servers/raid_soft/guide.en-gb.md +++ b/pages/bare_metal_cloud/dedicated_servers/raid_soft/guide.en-gb.md @@ -1,7 +1,7 @@ --- title: How to configure and rebuild software RAID -excerpt: Find out how to verify the state of the software RAID of your server and rebuild it after a disk replacement -updated: 2025-08-xx +excerpt: Find out how to verify the state of your software RAID and rebuild it after a disk replacement +updated: 2023-08-21 --- ## Objective @@ -14,93 +14,72 @@ The default RAID level for OVHcloud server installations is RAID 1, which double ## Requirements -- A [dedicated server](/links/bare-metal/bare-metal) with a software RAID configuration +- A [dedicated server](/links/bare-metal/bare-metal){.external} with a software RAID configuration - Administrative (sudo) access to the server via SSH -- Understanding of RAID, partitions and GRUB ## Instructions -When you purchase a new server, you might feel the need to perform a series of tests and actions. One of those actions could be simulating a disk failure in order to understand the process of rebuilding the raid and prepare yourself in case this happens. - -#### Basic Information +### Removing the disk In a command line session, type the following code to determine the current RAID status: ```sh cat /proc/mdstat -Personalities : [raid1] [linear] [multipath] [raid0] [raid6] [raid5] [raid4] [raid10] -md3 : active raid1 nvme1n1p3[1] nvme0n1p3[0] - 497875968 blocks super 1.2 [2/2] [UU] - bitmap: 2/4 pages [8KB], 65536KB chunk - -md2 : active raid1 nvme1n1p2[1] nvme0n1p2[0] - 1046528 blocks super 1.2 [2/2] [UU] - -unused devices: -``` - -This command shows us that we have two software RAID devices currently set up, with **md3** being the largest one. This array consists of two partitions, which are known as **nvme1n1p3** and **nvme0n1p3**. - -The [UU] means that all the disks are working normally. A `_` would indicate a failed disk. -If you have a server with SATA disks, you would get the following results: - -```sh -cat /proc/mdstat Personalities : [raid1] [linear] [multipath] [raid0] [raid6] [raid5] [raid4] [raid10] -md3 : active raid1 sda3[0] sdb3[1] - 3904786432 blocks super 1.2 [2/2] [UU] - bitmap: 2/30 pages [8KB], 65536KB chunk - -md2 : active raid1 sda2[0] sdb2[1] - 1046528 blocks super 1.2 [2/2] [UU] - +md2 : active raid1 sda2[1] sdb2[0] + 931954688 blocks super 1.2 [2/2] [UU] + bitmap: 2/7 pages [8KB], 65536KB chunk + +md4 : active raid1 sda4[0] sdb4[1] + 1020767232 blocks super 1.2 [2/2] [UU] + bitmap: 0/8 pages [0KB], 65536KB chunk + unused devices: ``` +This command shows us that we have two RAID arrays currently set up, with md4 being the largest partition. The partition consists of two disks, which are known as sda4 and sdb4. The [UU] means that all the disks are working normally. A `_` would indicate a failed disk. + Although this command returns our RAID volumes, it doesn't tell us the size of the partitions themselves. We can find this information with the following command: ```sh -sudo fdisk -l +fdisk -l -Disk /dev/nvme0n1: 476.94 GiB, 512110190592 bytes, 1000215216 sectors -Disk model: WDC CL SN720 SDAQNTW-512G-2000 +Disk /dev/sdb: 1.8 TiB, 2000398934016 bytes, 3907029168 sectors +Disk model: HGST HUS724020AL Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: gpt -Disk identifier: 18EFC16E-F711-4858-8298-9FAA9218E309 - -Device Start End Sectors Size Type -/dev/nvme0n1p1 2048 1048575 1046528 511M EFI System -/dev/nvme0n1p2 1048576 3145727 2097152 1G Linux RAID -/dev/nvme0n1p3 3145728 999161855 996016128 474.9G Linux RAID -/dev/nvme0n1p4 999161856 1000210431 1048576 512M Linux filesystem -/dev/nvme0n1p5 1000211120 1000215182 4063 2M Linux filesystem +Disk identifier: F92B6C5B-2518-4B2D-8FF9-A311DED5845F +Device Start End Sectors Size Type +/dev/sdb1 2048 4095 2048 1M BIOS boot +/dev/sdb2 4096 1864177663 1864173568 888.9G Linux RAID +/dev/sdb3 1864177664 1865226239 1048576 512M Linux filesystem +/dev/sdb4 1865226240 3907024895 2041798656 973.6G Linux RAID -Disk /dev/nvme1n1: 476.94 GiB, 512110190592 bytes, 1000215216 sectors -Disk model: WDC CL SN720 SDAQNTW-512G-2000 +Disk /dev/sda: 1.8 TiB, 2000398934016 bytes, 3907029168 sectors +Disk model: HGST HUS724020AL Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: gpt -Disk identifier: A99678B9-49B0-44E1-B411-52FA2879B617 +Disk identifier: 2E1DCCBA-8808-4D2B-BA33-9FEC3B96ADA8 -Device Start End Sectors Size Type -/dev/nvme1n1p1 2048 1048575 1046528 511M EFI System -/dev/nvme1n1p2 1048576 3145727 2097152 1G Linux RAID -/dev/nvme1n1p3 3145728 999161855 996016128 474.9G Linux RAID -/dev/nvme1n1p4 999161856 1000210431 1048576 512M Linux filesystem +Device Start End Sectors Size Type +/dev/sda1 2048 4095 2048 1M BIOS boot +/dev/sda2 4096 1864177663 1864173568 888.9G Linux RAID +/dev/sda3 1864177664 1865226239 1048576 512M Linux filesystem +/dev/sda4 1865226240 3907024895 2041798656 973.6G Linux RAID +/dev/sda5 3907025072 3907029134 4063 2M Linux filesystem - -Disk /dev/md2: 1022 MiB, 1071644672 bytes, 2093056 sectors +Disk /dev/md4: 973.5 GiB, 1045265645568 bytes, 2041534464 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes - -Disk /dev/md3: 474.81 GiB, 509824991232 bytes, 995751936 sectors +Disk /dev/md2: 888.8 GiB, 954321600512 bytes, 1863909376 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes @@ -108,38 +87,65 @@ I/O size (minimum/optimal): 512 bytes / 512 bytes The `fdisk -l` command also allows you to identify your partition type. This is an important information when it comes to rebuilding your RAID in case of a disk failure. -For **GPT** partitions, line 6 will display: `Disklabel type: gpt`. +For **GPT** partitions, the command will return: `Disklabel type: gpt`. -For **MBR** partitions, line 6 will display: `Disklabel type: dos`. +```sh +Disk /dev/sdb: 1.8 TiB, 2000398934016 bytes, 3907029168 sectors +Disk model: HGST HUS724020AL +Units: sectors of 1 * 512 = 512 bytes +Sector size (logical/physical): 512 bytes / 512 bytes +I/O size (minimum/optimal): 512 bytes / 512 bytes +'Disklabel type: gpt' +Disk identifier: F92B6C5B-2518-4B2D-8FF9-A311DED5845F +``` + +For **MBR** partitions, the command will return: `Disklabel type: dos`. + +```sh +Disk /dev/sda: 2.5 GiB, 2621440000 bytes, 5120000 sectors +Disk model: QEMU HARDDISK +Units: sectors of 1 * 512 = 512 bytes +Sector size (logical/physical): 512 bytes / 512 bytes +I/O size (minimum/optimal): 512 bytes / 512 bytes +'Disklabel type: dos' +Disk identifier: 0x150f6797 +``` -Still going by the results of `fdisk -l`, we can see that `/dev/md2` consists of 1022 MiB and `/dev/md3` contains 474.81 GiB. If we were to run the mount command we can also find out the layout of the disk. +We can see that `/dev/md2` consists of 888.8GB and `/dev/md4` contains 973.5GB. If we were to run the mount command we can also find out the layout of the disk. ```sh mount sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime) proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) -udev on /dev type devtmpfs (rw,nosuid,relatime,size=16348308k,nr_inodes=4087077,mode=755) +udev on /dev type devtmpfs (rw,nosuid,relatime,size=16315920k,nr_inodes=4078980,mode=755) devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000) -tmpfs on /run type tmpfs (rw,nosuid,nodev,noexec,relatime,size=3275192k,mode=755) -/dev/md3 on / type ext4 (rw,relatime) +tmpfs on /run type tmpfs (rw,nosuid,noexec,relatime,size=3266556k,mode=755) +/dev/md2 on / type ext4 (rw,relatime) securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime) tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev) tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k) -cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot) +tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755) +cgroup2 on /sys/fs/cgroup/unified type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate) +cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,name=systemd) pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime) -efivarfs on /sys/firmware/efi/efivars type efivarfs (rw,nosuid,nodev,noexec,relatime) -none on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700) -systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=10895) +bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700) +cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids) +cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory) +cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event) +cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma) +cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio) +cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct) +cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer) +cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio) +cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices) +cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset) +debugfs on /sys/kernel/debug type debugfs (rw,relatime) hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M) -mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime) -debugfs on /sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,relatime) -tracefs on /sys/kernel/tracing type tracefs (rw,nosuid,nodev,noexec,relatime) -fusectl on /sys/fs/fuse/connections type fusectl (rw,nosuid,nodev,noexec,relatime) -configfs on /sys/kernel/config type configfs (rw,nosuid,nodev,noexec,relatime) -/dev/md2 on /boot type ext4 (rw,relatime) -/dev/nvme1n1p1 on /boot/efi type vfat (rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=ascii,shortname=mixed,utf8,errors=remount-ro) -tmpfs on /run/user/1000 type tmpfs (rw,nosuid,nodev,relatime,size=3275188k,nr_inodes=818797,mode=700,uid=1000,gid=1000) +mqueue on /dev/mqueue type mqueue (rw,relatime) +systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=45,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=10340) +/dev/md4 on /home type ext3 (rw,relatime) +tmpfs on /run/user/1000 type tmpfs (rw,nosuid,nodev,relatime,size=3266552k,mode=700,uid=1000,gid=1000) ``` Alternatively, the `lsblk` command offers a different view of the partitions: @@ -147,170 +153,122 @@ Alternatively, the `lsblk` command offers a different view of the partitions: ```sh lsblk -NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT -nvme0n1 259:0 0 476.9G 0 disk -├─nvme0n1p1 259:6 0 511M 0 part -├─nvme0n1p2 259:7 0 1G 0 part -│ └─md2 9:2 0 1022M 0 raid1 /boot -├─nvme0n1p3 259:8 0 474.9G 0 part -│ └─md3 9:3 0 474.8G 0 raid1 / -├─nvme0n1p4 259:9 0 512M 0 part [SWAP] -└─nvme0n1p5 259:10 0 2M 0 part -nvme1n1 259:1 0 476.9G 0 disk -├─nvme1n1p1 259:2 0 511M 0 part /boot/efi -├─nvme1n1p2 259:3 0 1G 0 part -│ └─md2 9:2 0 1022M 0 raid1 /boot -├─nvme1n1p3 259:4 0 474.9G 0 part -│ └─md3 9:3 0 474.8G 0 raid1 / -└─nvme1n1p4 259:5 0 512M 0 part [SWAP] -``` - -We take note of the devices, partitions and their mount points. - -From the above commands and results, we have: - -- Two RAID arrays: `/dev/md2` and `/dev/md3`. -- Partitions part of the RAID: `/boot` and `/`. -- Partitions not part of the RAID: `/boot/efi` and [SWAP]. - -> [!primary] -> Most modern motherboards use UEFI mode (EFI). When a server is intalled, two ESPs partitions (labelled EFI_SYSPART by OVHcloud) are created on the disks used for RAID, however, only one partition is automatically mounted in `/boot/efi` by the system after install. It is important to understand the role of this partition and how to manage it. -> -> For more information on UEFI, consult the following [guide](https://uefi.org/about){.external} - -To check whether a server runs on BIOS mode or UEFI mode, run the following command: - -```sh -[user@server_ip ~]# [ -d /sys/firmware/efi ] && echo UEFI || echo BIOS +NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT +sda 8:0 0 1.8T 0 disk +├─sda1 8:1 0 1M 0 part +├─sda2 8:2 0 888.9G 0 part +│ └─md2 9:2 0 888.8G 0 raid1 / +├─sda3 8:3 0 512M 0 part [SWAP] +├─sda4 8:4 0 973.6G 0 part +│ └─md4 9:4 0 973.5G 0 raid1 /home +└─sda5 8:5 0 2M 0 part +sdb 8:16 0 1.8T 0 disk +├─sdb1 8:17 0 1M 0 part +├─sdb2 8:18 0 888.9G 0 part +│ └─md2 9:2 0 888.8G 0 raid1 / +├─sdb3 8:19 0 512M 0 part [SWAP] +└─sdb4 8:20 0 973.6G 0 part + └─md4 9:4 0 973.5G 0 raid1 /home ``` -### Simulating a disk failure - -Now that we have all the necessary information, we can simulate a disk failure and proceed with the tests. In this example, we will fail the disk `nvme0n1`. - -The preferred to do this is via the OVHcloud rescue environment. - -First reboot in rescue menu and log in with the credentials provided. - -To remove a disk from the RAID, the first step is to mark it as **Failed** and remove the partitions from their respective RAID arrays. +As the disks are currently mounted by default, to remove a disk from the RAID, we first need to unmount the disk, then simulate a failure, and finally remove it. We will remove `/dev/sda4` from the RAID with the following command: ```sh -root@rescue12-customer-ca (nsxxxxx.ip-xx-xx-xx.eu) ~ # cat /proc/mdstat -Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] [multipath] [faulty] -md3 : active raid1 nvme0n1p3[0] nvme1n1p3[1] - 497875968 blocks super 1.2 [2/2] [UU] - bitmap: 0/4 pages [0KB], 65536KB chunk - -md2 : active raid1 nvme0n1p2[2] nvme1n1p2[1] - 1046528 blocks super 1.2 [2/2] [UU] - -unused devices: +umount /dev/md4 ``` -From the above output, nvme0n1 consists of two partitions in raid which are **nvme0n1p2** and **nvme0n1p3**. - -#### Removing the failed disk - -First we mark the partitions **nvme0n1p2** and **nvme0n1p3** as failed. - -```sh -sudo mdadm --manage /dev/md2 --fail /dev/nvme0n1p2 +> [!warning] +> Please note that if you are connected as the user `root`, you may get the following message when you try to unmount the partition (in our case, where our md4 partition is mounted in /home): +> +> 
umount: /home: target is busy
+> +> In this case, you must log out as the user root and connect as a local user (in our case `debian`), and use the following command: +> +> 
debian@ns000000:/$ sudo umount /dev/md4
+> +> If you do not have a local user, you need to [create one](/pages/bare_metal_cloud/dedicated_servers/changing_root_password_linux_ds). -# mdadm: set /dev/nvme0n1p2 faulty in /dev/md2 -``` +This will provide us with the following output: ```sh -sudo mdadm --manage /dev/md3 --fail /dev/nvme0n1p3 - -# mdadm: set /dev/nvme0n1p3 faulty in /dev/md3 +sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime) +proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) +udev on /dev type devtmpfs (rw,nosuid,relatime,size=16315920k,nr_inodes=4078980,mode=755) +devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000) +tmpfs on /run type tmpfs (rw,nosuid,noexec,relatime,size=3266556k,mode=755) +/dev/md2 on / type ext4 (rw,relatime) +securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime) +tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev) +tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k) +tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755) +cgroup2 on /sys/fs/cgroup/unified type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate) +cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,name=systemd) +pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime) +bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700) +cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids) +cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory) +cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event) +cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma) +cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio) +cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct) +cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer) +cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio) +cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices) +cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset) +debugfs on /sys/kernel/debug type debugfs (rw,relatime) +hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M) +mqueue on /dev/mqueue type mqueue (rw,relatime) +systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=45,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=10340) +tmpfs on /run/user/1000 type tmpfs (rw,nosuid,nodev,relatime,size=3266552k,mode=700,uid=1000,gid=1000) ``` -We have now simulated a failure of the RAID, when we run the `cat /proc/mdstat` command, we have the following output +As we can see the, entry of `/dev/md4` is no longer mounted. However, the RAID is still active, so we need to simulate a failure to remove the disk. We can do this with the following command: ```sh -root@rescue12-customer-ca (nsxxxxx.ip-xx-xx-xx.eu) ~ # cat /proc/mdstat -Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] [multipath] [faulty] -md3 : active raid1 nvme0n1p3[0](F) nvme1n1p3[1] - 497875968 blocks super 1.2 [2/1] [_U] - bitmap: 0/4 pages [0KB], 65536KB chunk - -md2 : active raid1 nvme0n1p2[2](F) nvme1n1p2[1] - 1046528 blocks super 1.2 [2/1] [_U] - -unused devices: +sudo mdadm --fail /dev/md4 /dev/sda4 ``` -As we can see above, the [F] next to the partitions indicates that the disk has failed or is faulty. Next, we remove these partitions from the RAID arrays to completely remove the disk from RAID. +We have now simulated a failure of the RAID. The next step is to remove the partition from the RAID array with the following command: ```sh -sudo mdadm --manage /dev/md2 --remove /dev/nvme0n1p2 - -# mdadm: hot removed /dev/nvme0n1p2 from /dev/md2 +sudo mdadm --remove /dev/md4 /dev/sda4 ``` -```sh -sudo mdadm --manage /dev/md3 --remove /dev/nvme0n1p3 - -# mdadm: hot removed /dev/nvme0n1p3 from /dev/md3 -``` - - -If we run the following command: +You can verify that the partition has been removed with the following command: ```sh -parted /dev/nvme0n1 -GNU Parted 3.5 -Using /dev/nvme0n1 -Welcome to GNU Parted! Type 'help' to view a list of commands. -(parted) p -Error: /dev/nvme0n1: unrecognised disk label -Model: WDC CL SN720 SDAQNTW-512G-2000 (nvme) -Disk /dev/nvme0n1: 512GB -Sector size (logical/physical): 512B/512B -Partition Table: unknown -Disk Flags: -``` +cat /proc/mdstat -We see that our disk has been successfully wiped. - -Our RAID status should now look like this: - -```sh -root@rescue12-customer-ca (ns5009452.ip-51-222-254.net) ~ # cat /proc/mdstat -Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] [multipath] [faulty] -md3 : active raid1 nvme1n1p3[1] - 497875968 blocks super 1.2 [2/1] [_U] - bitmap: 0/4 pages [0KB], 65536KB chunk - -md2 : active raid1 nvme1n1p2[1] - 1046528 blocks super 1.2 [2/1] [_U] - -unused devices: n +Personalities : [raid1] [linear] [multipath] [raid0] [raid6] [raid5] [raid4] [raid10] +md2 : active raid1 sda2[1] sdb2[0] + 931954688 blocks super 1.2 [2/2] [UU] + bitmap: 4/7 pages [16KB], 65536KB chunk + +md4 : active raid1 sdb4[1] + 1020767232 blocks super 1.2 [2/1] [_U] + bitmap: 0/8 pages [0KB], 65536KB chunk + +unused devices: ``` -From the results above, we can see that only two partitions now appear in the RAID arrays. We have successfully failed the disk nvme0n1 and we can now proceed with the replacement. - -For more information on how to prepare and request a disk replacement, consult this [guide](/pages/bare_metal_cloud/dedicated_servers/disk_replacement) - -If you run the following command, you can have more details on the raid array. - +The following command will verify that the partition has been removed: ```sh -mdadm --detail /dev/md3 +mdadm --detail /dev/md4 -/dev/md3: +/dev/md4: Version : 1.2 - Creation Time : Fri Aug 1 14:51:13 2025 + Creation Time : Tue Jan 24 15:35:02 2023 Raid Level : raid1 - Array Size : 497875968 (474.81 GiB 509.82 GB) - Used Dev Size : 497875968 (474.81 GiB 509.82 GB) + Array Size : 1020767232 (973.48 GiB 1045.27 GB) + Used Dev Size : 1020767232 (973.48 GiB 1045.27 GB) Raid Devices : 2 Total Devices : 1 Persistence : Superblock is persistent Intent Bitmap : Internal - Update Time : Fri Aug 1 15:56:17 2025 + Update Time : Tue Jan 24 16:28:03 2023 State : clean, degraded Active Devices : 1 Working Devices : 1 @@ -319,483 +277,57 @@ mdadm --detail /dev/md3 Consistency Policy : bitmap - Name : md3 - UUID : b383c3d5:7fb1bb5e:6b7c4d96:6ea817ff - Events : 215 + Name : md4 + UUID : 7b5c1d80:0a7ab4c2:e769b5e5:9c6eaa0f + Events : 21 Number Major Minor RaidDevice State - 0 0 0 removed - 1 259 4 1 active sync /dev/nvme1n1p3 + 1 8 20 1 active sync /dev/sdb4 ``` ### Rebuilding the RAID -> [!primary] -> This process might be different depending on the operating system you have installed on your server. We recommend that you consult the official documentation of your operating system to have access to the proper commands. -> - -In this situation, we will explore two cases - -- Dedicated servers with UEFI -- Dedicated servers with BIOS - - -> [!warning] -> -> With most cases, after a disk replacement, the server is able to reboot in normal mode (on the healthy disk) and these steps can be done in normal mode. However, if the server is not able to reboot after a disk replacement, it will be rebooted in rescue mode to proceed with the raid rebuild. These steps apply as well. -> - -#### Dedicated server with UEFI - -In this section, we will focus on the steps to follow after the replacement of the main disk with the following scenario: - -- The EFI partition was mounted on this disk -- The content of the EFI partition have not been updated/they have been updated and both partitions have been synchronized. - -If your secondary disk was replaced, please follow the steps for [Dedicated servers with BIOS]() and recreate an EFI partition in the secondary disk as described below. - -/// details | **Understanding the EFI partition** - -An EFI partition, is a partition which can contain the boot loaders, boot managers, or kernel images of an installed operating system. It also contains system utility programs designed to be run before the operating system boots, as well as data files such as error logs. - -By default, this partition is duplicated when one of our installation template is installed in RAID, but this partition is not included in the RAID. - -During installation, this partition is mounted on `/boot/efi` and the disk on which it is mounted is selected randomly by the operating system. Thus, your system has two EFI partitions, but only one is mounted in the RAID. - -Example: - -```sh -lsblk - -NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT -nvme0n1 259:0 0 476.9G 0 disk -├─nvme0n1p1 259:6 0 511M 0 part -├─nvme0n1p2 259:7 0 1G 0 part -│ └─md2 9:2 0 1022M 0 raid1 /boot -├─nvme0n1p3 259:8 0 474.9G 0 part -│ └─md3 9:3 0 474.8G 0 raid1 / -├─nvme0n1p4 259:9 0 512M 0 part [SWAP] -└─nvme0n1p5 259:10 0 2M 0 part -nvme1n1 259:1 0 476.9G 0 disk -├─nvme1n1p1 259:2 0 511M 0 part /boot/efi -├─nvme1n1p2 259:3 0 1G 0 part -│ └─md2 9:2 0 1022M 0 raid1 /boot -├─nvme1n1p3 259:4 0 474.9G 0 part -│ └─md3 9:3 0 474.8G 0 raid1 / -└─nvme1n1p4 259:5 0 512M 0 part [SWAP] -``` - -From the example above, we see that we have two identical partitions (nvme0n1p1 and nvme1n1p1) but only **nvme1n1p1** is mounted on `/boot/efi`. - -We can also use the following command to confirm that two EFI partitions were actually created after install: - -```sh -lsblk -f - -NAME FSTYPE FSVER LABEL UUID FSAVAIL FSUSE% MOUNTPOINT -nvme0n1 -│ -├─nvme0n1p1 vfat FAT16 EFI_SYSPART 461C-DB56 -├─nvme0n1p2 -│ linux_ 1.2 md2 83719c5c-2a27-2a56-5268-7d49d8a1d84f -│ └─md2 -│ ext4 1.0 boot 4de80ae0-dd90-4256-9135-1735e7be4b4d 851.8M 7% /boot -├─nvme0n1p3 -│ linux_ 1.2 md3 b383c3d5-7fb1-bb5e-6b7c-4d966ea817ff -│ └─md3 -│ ext4 1.0 root 9bf386b6-9523-46bf-b8e5-4b8cc7c5786f 441.3G 0% / -├─nvme0n1p4 -│ swap 1 swap-nvme0n1p4 356439fe-0539-45ce-9eff-40b616689b0c [SWAP] -└─nvme0n1p5 - iso966 Jolie config-2 - 2025-08-01-14-51-51-00 -nvme1n1 -│ -├─nvme1n1p1 vfat FAT16 EFI_SYSPART 4629-D183 504.9M 1% /boot/efi -├─nvme1n1p2 -│ linux_ 1.2 md2 83719c5c-2a27-2a56-5268-7d49d8a1d84f -│ └─md2 -│ ext4 1.0 boot 4de80ae0-dd90-4256-9135-1735e7be4b4d 851.8M 7% /boot -├─nvme1n1p3 -│ linux_ 1.2 md3 b383c3d5-7fb1-bb5e-6b7c-4d966ea817ff -│ └─md3 -│ ext4 1.0 root 9bf386b6-9523-46bf-b8e5-4b8cc7c5786f 441.3G 0% / -└─nvme1n1p4 - swap 1 swap-nvme1n1p4 - 9bf292e8-0145-4d2f-b891-4cef93c0d209 [SWAP] -``` - -From the results above, we can see two partitions (**nvme0n1p1** and **nvme1n1p1**), with identical size (504.9M). Both partitions have the LABEL: `EFI_SYSPART` but only one is mounted on `/boot/efi`. - -In general, this is a partition is not often updated, except when there are relevant updates. In this case, we recommend running an automatic or manual script to copy the primary EFI partition to the second one when needed. Additionally, a boot entry for the secondary ESP can then be added manually using [efibootmgr](https://wiki.archlinux.org/title/Unified_Extensible_Firmware_Interface#efibootmgr){.external} - -The script below can be used: - -```sh -set -euo pipefail - -MAIN_PARTITION="$(findmnt -n -o SOURCE /boot/efi)" -MOUNTPOINT="/var/lib/grub/esp" - -mkdir -p "${MOUNTPOINT}" - -while read -r partition; do - if [[ "${partition}" == "${MAIN_PARTITION}" ]]; then - continue - fi - echo "Working on ${partition}" - mount "${partition}" "${MOUNTPOINT}" - rsync -ax "/boot/efi/" "${MOUNTPOINT}/" - umount "${MOUNTPOINT}" -done < <(blkid -o device -t LABEL=EFI_SYSPART) -``` - -/// - -**Disk replacement** - -Once the disk has been replaced, we need to copy the partition table from the healthy disk (in this example, nvme1n1) to the new one (nvme0n1). +Once the disk has been replaced, we need to copy the partition table from a healthy disk (in this example, sdb) to the new one (sda) with the following command: **For GPT partitions** ```sh -sudo sgdisk -R /dev/sda /dev/sdb +sgdisk -R /dev/sda /dev/sdb ``` The command should be in this format: `sgdisk -R /dev/newdisk /dev/healthydisk` -In our example: - -```sh -sudo sgdisk -R /dev/nvme0n1 /dev/nvme1n1 -``` - Once this is done, the next step is to randomize the GUID of the new disk to prevent GUID conflicts with other disks: ```sh -sudo sgdisk -G /dev/nvme0n1 +sgdisk -G /dev/sda ``` -If you receive a message like this: - -```console -Warning: The kernel is still using the old partition table. -The new table will be used at the next reboot or after you -run partprobe(8) or kpartx(8) -The operation has completed successfully. -``` - -You can simply run the `partprobe` command or reboot the server when the raid rebuilt is done. - **For MBR partitions** -```sh -sudo sfdisk -d /dev/sdb | sfdisk /dev/sda -``` - -The command should be in this format: `sfdisk -d /dev/healthydisk | sfdisk /dev/newdisk`. - -We can now rebuild the RAID array. The following code snippet shows how we can add the new partitions (nvme0n1p2 and nvme0n1p3) back in the RAID array. - - -```sh -sudo mdadm --add /dev/md2 /dev/nvme0n1p2 - -# mdadm: added /dev/nvme0n1p2 - -sudo mdadm --add /dev/md3 /dev/nvme0n1p3 - -# mdadm: re-added /dev/nvme0n1p3 - -cat /proc/mdstat -Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] [multipath] [faulty] -md3 : active raid1 nvme0n1p3[2] nvme1n1p3[1] - 497875968 blocks super 1.2 [2/1] [_U] - [>....................] recovery = 0.1% (801920/497875968) finish=41.3min speed=200480K/sec - bitmap: 0/4 pages [0KB], 65536KB chunk - -md2 : active raid1 nvme0n1p2[2] nvme1n1p2[1] - 1046528 blocks super 1.2 [2/2] [UU] -``` - -Once the raid rebuild is complete, run the following command to make sure that the partitions were properly added to the raid: +Once the disk has been replaced, we need to copy the partition table from a healthy disk (in this example, sdb) to the new one (sda) with the following command: ```sh -lsblk -fA -NAME FSTYPE FSVER LABEL UUID FSAVAIL FSUSE% MOUNTPOINTS -nvme1n1 -├─nvme1n1p1 vfat FAT16 EFI_SYSPART 4629-D183 -├─nvme1n1p2 linux_raid_member 1.2 md2 83719c5c-2a27-2a56-5268-7d49d8a1d84f -│ └─md2 ext4 1.0 boot 4de80ae0-dd90-4256-9135-1735e7be4b4d -├─nvme1n1p3 linux_raid_member 1.2 md3 b383c3d5-7fb1-bb5e-6b7c-4d966ea817ff -│ └─md3 ext4 1.0 root 9bf386b6-9523-46bf-b8e5-4b8cc7c5786f -└─nvme1n1p4 swap 1 swap-nvme1n1p4 9bf292e8-0145-4d2f-b891-4cef93c0d209 -nvme0n1 -├─nvme0n1p1 -├─nvme0n1p2 linux_raid_member 1.2 md2 83719c5c-2a27-2a56-5268-7d49d8a1d84f -│ └─md2 ext4 1.0 boot 4de80ae0-dd90-4256-9135-1735e7be4b4d -├─nvme0n1p3 linux_raid_member 1.2 md3 b383c3d5-7fb1-bb5e-6b7c-4d966ea817ff -│ └─md3 ext4 1.0 root 9bf386b6-9523-46bf-b8e5-4b8cc7c5786f -└─nvme0n1p4 -``` - -From the above results, we can see that the partitions of our newly added disk have been properly added to the raid, however, the EFI partition was not duplicated, which is normal since it is not included in the raid. We also see that the [SWAP] partition **nvme0n1p4** does not have the label anymore. We also need reattribute its properties. - -> [!warning] -> The examples above are merely illustrating the necessary steps based on a typical server configuration. The information in the output table depends on your server's hardware and its partition scheme. When in doubt, consult the documentation of your operating system. -> -> If you require professional assistance with server administration, consider the details in the Go further section of this guide. -> - -The next step is to create an EFI partition (**nvme0n1p1**) on the newly added disk and format it. Please keep in mind that the content of this partition on the remaining drive (in our exmaple: nvme1n1) will be replicated to the new disk in order to enable our server to boot back into normal mode. This is possible because we kept the partition in sync while the server was running normally before the disk failure. - -It is possible that this process does not work if there was a kernel or grub update and both partitions were not synchronized. In this case, please consult the following guide []. - -We create a vfat partition - -```sh -mkfs.vfat /dev/nvme0n1p1 -``` - -Next, we lable the partition as `EFI_SYSPART` (this naming is proper to OVHcloud) - -```sh -fatlabel /dev/nvme0n1p1 EFI_SYSPART -``` - -Now we have to duplicate the partition from nvme1n1p1 (healthy partition), to nvme0n1p1 (new partition). To do so, we start by creating two folders named `old` and `new`. - -```sh -mkdir old new -``` - -Next, we mount mount **nvme1n1p1** in the `old` folder and **nvme0n1p1** in the `new` folder: - -```sh -mount /dev/nvme1n1p1 old -mount /dev/nvme0n1p1 new -``` - -Next, we copy the files from `old` to `new`. Depending on your operating system, you will have a similar output. Here we are using debian: - -```sh -rsync -axv old/ new/ -sending incremental file list -EFI/ -EFI/debian/ -EFI/debian/BOOTX64.CSV -EFI/debian/fbx64.efi -EFI/debian/grub.cfg -EFI/debian/grubx64.efi -EFI/debian/mmx64.efi -EFI/debian/shimx64.efi -``` - -Once this is done, we umount the **nvme0n1p1** partition. - -```sh -umount nvme0n1p1 -``` - -Next, we mount the partition containing our files on `/mnt`. In our example, that partition is `md3`. - -```sh -mount /dev/md3 /mnt -``` - -Next, we mount the following directories: - -```sh -mount --types proc /proc /mnt/proc -mount --rbind /sys /mnt/sys -mount --make-rslave /mnt/sys -mount --rbind /dev /mnt/dev -mount --make-rslave /mnt/dev -mount --bind /run /mnt/run -mount --make-slave /mnt/run -``` - -Next, we use the `chroot` command to access the mountpoint and make sure the new EFI partition has been properly created and the system recongnises both EFI partitions: - -```sh -chroot /mnt -``` - -```sh -blkid -t LABEL=EFI_SYSPART -/dev/nvme1n1p1: SEC_TYPE="msdos" LABEL_FATBOOT="EFI_SYSPART" LABEL="EFI_SYSPART" UUID="4629-D183" BLOCK_SIZE="512" TYPE="vfat" PARTLABEL="primary" PARTUUID="889f241b-49c3-4031-b5c9-60df0746f98f" -/dev/nvme0n1p1: SEC_TYPE="msdos" LABEL_FATBOOT="EFI_SYSPART" LABEL="EFI_SYSPART" UUID="521F-300B" BLOCK_SIZE="512" TYPE="vfat" PARTLABEL="primary" PARTUUID="02bf2b2d-7ada-4461-ba50-07683519f65d" -``` - -Still in the `chroot` environment, we create the `/boot/efi` folder in order to mount the EFI partition **nvme0n1p1** in it: - -```sh -cd /mnt -mkdir /boot/efi -mount /dev/nvme0n1p1 /boot/efi -``` - -Next, we install the grub bootloader to make sure the server can reboot in normal mode on the new disk (you won't have to do this if the disk replaced is the secondary disk. Simply duplicate the EFI partition and proceed to the RAID rebuild, then enable the [SWAP] partition (if applicable)): - -```sh -grub-install --efi-directory=/boot/efi /dev/nvme0n1p1 -``` - -Next, we exit the `chroot` environment, then we recreate our [SWAP] partition **nvme0n1p4** and add the label `swap-nvmenxxx`: - -```sh -mkswap /dev/nvme0n1p4 -L swap-nvme0n1p4 - -Setting up swapspace version 1, size = 512 MiB (536866816 bytes) -LABEL=swap-nvme0n1p4, UUID=256215f9-7694-4a88-aa47-335558fc6cd8 -``` - -We verify that the lable has been properly applied: - -```sh -NAME FSTYPE FSVER LABEL UUID FSAVAIL FSUSE% MOUNTPOINTS -nvme1n1 -├─nvme1n1p1 vfat FAT16 EFI_SYSPART 4629-D183 -├─nvme1n1p2 linux_raid_member 1.2 md2 83719c5c-2a27-2a56-5268-7d49d8a1d84f -│ └─md2 ext4 1.0 boot 4de80ae0-dd90-4256-9135-1735e7be4b4d -├─nvme1n1p3 linux_raid_member 1.2 md3 b383c3d5-7fb1-bb5e-6b7c-4d966ea817ff -│ └─md3 ext4 1.0 root 9bf386b6-9523-46bf-b8e5-4b8cc7c5786f 441.1G 0% /mnt -└─nvme1n1p4 swap 1 swap-nvme1n1p4 9bf292e8-0145-4d2f-b891-4cef93c0d209 -nvme0n1 -├─nvme0n1p1 vfat FAT16 EFI_SYSPART 521F-300B -├─nvme0n1p2 linux_raid_member 1.2 md2 83719c5c-2a27-2a56-5268-7d49d8a1d84f -│ └─md2 ext4 1.0 boot 4de80ae0-dd90-4256-9135-1735e7be4b4d -├─nvme0n1p3 linux_raid_member 1.2 md3 b383c3d5-7fb1-bb5e-6b7c-4d966ea817ff -│ └─md3 ext4 1.0 root 9bf386b6-9523-46bf-b8e5-4b8cc7c5786f 441.1G 0% /mnt -└─nvme0n1p4 swap 1 swap-nvme0n1p4 256215f9-7694-4a88-aa47-335558fc6cd8 -``` - -Take note of the UUID of both swap partitions **nvme0n1p4** and **nvme1n1p4**. - -Next, we need to add the new UUID of the swap partition in `/etc/fstab` using the `chroot` environment: - -```sh -chroot /mnt -``` - -```sh -cat /etc/fstab - -UUID=9bf386b6-9523-46bf-b8e5-4b8cc7c5786f / ext4 defaults 0 1 -UUID=4de80ae0-dd90-4256-9135-1735e7be4b4d /boot ext4 defaults 0 0 -LABEL=EFI_SYSPART /boot/efi vfat defaults 0 1 -UUID=356439fe-0539-45ce-9eff-40b616689b0c swap swap defaults 0 0 -UUID=9bf292e8-0145-4d2f-b891-4cef93c0d209 swap swap defaults 0 0 -``` -We need to replace UUID `356439fe-0539-45ce-9eff-40b616689b0c` with `256215f9-7694-4a88-aa47-335558fc6cd8` since `9bf292e8-0145-4d2f-b891-4cef93c0d209` is the UUID of our second swap partition **nvme1n1p4** and we are not replacing it. You can use a text editor such as `vi` or `nano` to edit the file: - -```sh -nano etc/fstab -``` - -```sh -cat /etc/fstab - -UUID=9bf386b6-9523-46bf-b8e5-4b8cc7c5786f / ext4 defaults 0 1 -UUID=4de80ae0-dd90-4256-9135-1735e7be4b4d /boot ext4 defaults 0 0 -LABEL=EFI_SYSPART /boot/efi vfat defaults 0 1 -UUID=256215f9-7694-4a88-aa47-335558fc6cd8 swap swap defaults 0 0 -UUID=9bf292e8-0145-4d2f-b891-4cef93c0d209 swap swap defaults 0 0 -``` - -Next, we make sure everything is properly mounted: - -```sh -mount -av - -/ : ignored -/boot : successfully mounted -/boot/efi : successfully mounted -swap : ignored -swap : ignored -``` - -We enable the swap partition: - -```sh -swapon -av - -swapon: /dev/nvme0n1p4: found signature [pagesize=4096, signature=swap] -swapon: /dev/nvme0n1p4: pagesize=4096, swapsize=536870912, devsize=536870912 -swapon /dev/nvme0n1p4 -swapon: /dev/nvme1n1p4: found signature [pagesize=4096, signature=swap] -swapon: /dev/nvme1n1p4: pagesize=4096, swapsize=536870912, devsize=536870912 -swapon /dev/nvme1n1p4 -``` - -We have now succesffuly completed the RAID rebuild on the server and we can now reboot the server in normal mode: - - -#### Dedicated servers with BIOS - -Use the appropriate commands: - -**For GPT partitions** - -```sh -sudo sgdisk -R /dev/sda /dev/sdb -``` - -The command should be in this format: `sgdisk -R /dev/newdisk /dev/healthydisk` - -In our example: - -```sh -sudo sgdisk -R /dev/nvme0n1 /dev/nvme1n1 -``` - -Once this is done, the next step is to randomize the GUID of the new disk to prevent GUID conflicts with other disks: - -```sh -sudo sgdisk -G /dev/nvme0n1 -``` - -If you receive a message like this: - -```console -Warning: The kernel is still using the old partition table. -The new table will be used at the next reboot or after you -run partprobe(8) or kpartx(8) -The operation has completed successfully. -``` - -You can simply run the `partprobe` command or reboot the server when the raid rebuilt is done. - -**For MBR partitions** - -```sh -sudo sfdisk -d /dev/sdb | sfdisk /dev/sda +sfdisk -d /dev/sdb | sfdisk /dev/sda ``` The command should be in this format: `sfdisk -d /dev/healthydisk | sfdisk /dev/newdisk` -We can now rebuild the RAID array. The following code snippet shows how we can add the new partitions (nvme0n1p2 and nvme0n1p3) back in the RAID array. - +We can now rebuild the RAID array. The following code snippet shows how we can rebulid the `/dev/md4` partition layout with the recently-copied sda partition table: ```sh -sudo mdadm --add /dev/md2 /dev/nvme0n1p2 - -# mdadm: added /dev/nvme0n1p2 - -sudo mdadm --add /dev/md3 /dev/nvme0n1p3 - -# mdadm: re-added /dev/nvme0n1p3 - +mdadm --add /dev/md4 /dev/sda4 cat /proc/mdstat Personalities : [raid1] [linear] [multipath] [raid0] [raid6] [raid5] [raid4] [raid10] -md3 : active raid1 nvme0n1p3[0] nvme1n1p3[1] - 497875968 blocks super 1.2 [2/1] [_U] - [============>........] recovery = 64.8% (322969856/497875968) finish=7.2min speed=401664K/sec - bitmap: 4/4 pages [16KB], 65536KB chunk +md2 : active raid1 sda2[1] sdb2[0] + 931954688 blocks super 1.2 [2/2] [UU] + bitmap: 1/7 pages [4KB], 65536KB chunk -md2 : active raid1 nvme0n1p2[2] nvme1n1p2[1] - 1046528 blocks super 1.2 [2/2] [UU] +md4 : active raid1 sda4[0] sdb4[1] + 1020767232 blocks super 1.2 [2/2] [UU] + bitmap: 0/8 pages [0KB], 65536KB chunk unused devices: ``` @@ -803,86 +335,43 @@ unused devices: We can verify the RAID details with the following command: ```sh -sudo mdadm --detail /dev/md3 & sudo mdadm --detail /dev/md2 -[1] 2096 -/dev/md3: - Version : 1.2 - Creation Time : Fri Aug 1 14:51:13 2025 +mdadm --detail /dev/md4 + +/dev/md4: + Version : 1.2 + Creation Time : Tue Jan 24 15:35:02 2023 Raid Level : raid1 - Array Size : 497875968 (474.81 GiB 509.82 GB) - Used Dev Size : 497875968 (474.81 GiB 509.82 GB) + Array Size : 1020767232 (973.48 GiB 1045.27 GB) + Used Dev Size : 1020767232 (973.48 GiB 1045.27 GB) Raid Devices : 2 Total Devices : 2 Persistence : Superblock is persistent Intent Bitmap : Internal - Update Time : Fri Aug 1 16:48:03 2025 - State : clean - Active Devices : 2 - Working Devices : 2 - Failed Devices : 0 - Spare Devices : 0 - -Consistency Policy : bitmap - - Name : md3 - UUID : b383c3d5:7fb1bb5e:6b7c4d96:6ea817ff - Events : 538 - - Number Major Minor RaidDevice State - 0 259 8 0 active sync /dev/nvme0n1p3 - 1 259 4 1 active sync /dev/nvme1n1p3 -/dev/md2: - Version : 1.2 - Creation Time : Fri Aug 1 14:51:13 2025 - Raid Level : raid1 - Array Size : 1046528 (1022.00 MiB 1071.64 MB) - Used Dev Size : 1046528 (1022.00 MiB 1071.64 MB) - Raid Devices : 2 - Total Devices : 2 - Persistence : Superblock is persistent - - Update Time : Fri Aug 1 16:45:59 2025 + Update Time : Tue Jan 24 17:02:55 2023 State : clean Active Devices : 2 Working Devices : 2 Failed Devices : 0 Spare Devices : 0 -Consistency Policy : resync + Rebuild Status : 21% complete - Name : md2 - UUID : 83719c5c:2a272a56:52687d49:d8a1d84f - Events : 26 + UUID : 7f39d062:9f16a016:a4d2adc2:26fd5302 + Events : 0.95 Number Major Minor RaidDevice State - 2 259 7 0 active sync /dev/nvme0n1p2 - 1 259 3 1 active sync /dev/nvme1n1p2 + 0 8 2 0 spare rebuilding /dev/sda4 + 1 8 18 1 active sync /dev/sdb4 ``` -The RAID has now been rebuilt. - -Next, mount the partitions +The RAID has now been rebuilt, but we still need to mount the partition (`/dev/md4` in this example) with the following command: ```sh -sudo mount /dev/md2 /boot -sudo mount /dev/md3 / +mount /dev/md4 /home ``` - - - - -> [!primary] -> While this process helps understand what to do when a disk fails, it is a but simplistic. For further diagnostics, you can consult this section of the guide. -> - - - - - - ## Go Further [Hot Swap - Software RAID](/pages/bare_metal_cloud/dedicated_servers/hotswap_raid_soft) @@ -893,4 +382,4 @@ sudo mount /dev/md3 / [Hot Swap - Hardware RAID](/pages/bare_metal_cloud/dedicated_servers/hotswap_raid_hard) -Join our [community of users](/links/community). +Join our [community of users](/links/community). \ No newline at end of file diff --git a/pages/bare_metal_cloud/dedicated_servers/raid_soft_bios/guide.en-gb.md b/pages/bare_metal_cloud/dedicated_servers/raid_soft_bios/guide.en-gb.md new file mode 100644 index 00000000000..fdfdd7514c2 --- /dev/null +++ b/pages/bare_metal_cloud/dedicated_servers/raid_soft_bios/guide.en-gb.md @@ -0,0 +1,604 @@ +--- +title: Managing and rebuilding software RAID on servers in BIOS boot mode +excerpt: Find out how to manage and rebuild software RAID after a disk replacement on your server in BIOS mode +updated: 2025-08-xx +--- + +## Objective + +Redundant Array of Independent Disks (RAID) is a technology that mitigates data loss on a server by replicating data across two or more disks. + +The default RAID level for OVHcloud server installations is RAID 1, which doubles the space taken up by your data, effectively halving the useable disk space. + +**This guide explains how to manage and rebuild software RAID after a disk replacement on your server in BIOS mode** + +Before we begin, please note that this guide focuses on Dedicated servers that use BIOS as the boot mode. If your server uses the UEFI mode (newer motherboards), refer to this guide [Managing and rebuilding software RAID on servers in UEFI boot mode](). + +## Requirements + +- A [Dedicated server](/links/bare-metal/bare-metal) with a software RAID configuration +- Administrative (sudo) access to the server via SSH +- Understanding of RAID, partitions and GRUB + +## Instructions + +### Basic Information + +In a command line session, type the following code to determine the current RAID status: + +```sh +cat /proc/mdstat +Personalities : [raid1] [linear] [multipath] [raid0] [raid6] [raid5] [raid4] [raid10] +md3 : active raid1 nvme1n1p3[1] nvme0n1p3[0] + 497875968 blocks super 1.2 [2/2] [UU] + bitmap: 2/4 pages [8KB], 65536KB chunk + +md2 : active raid1 nvme1n1p2[1] nvme0n1p2[0] + 1046528 blocks super 1.2 [2/2] [UU] + +unused devices: +``` + +This command shows us that we have two software RAID devices currently set up, with **md3** being the largest one. This array consists of two partitions, which are known as **nvme1n1p3** and **nvme0n1p3**. + +The [UU] means that all the disks are working normally. A `_` would indicate a failed disk. + +If you have a server with SATA disks, you would get the following results: + +```sh +cat /proc/mdstat +Personalities : [raid1] [linear] [multipath] [raid0] [raid6] [raid5] [raid4] [raid10] +md3 : active raid1 sda3[0] sdb3[1] + 3904786432 blocks super 1.2 [2/2] [UU] + bitmap: 2/30 pages [8KB], 65536KB chunk + +md2 : active raid1 sda2[0] sdb2[1] + 1046528 blocks super 1.2 [2/2] [UU] + +unused devices: +``` + +Although this command returns our RAID volumes, it doesn't tell us the size of the partitions themselves. We can find this information with the following command: + +```sh +sudo fdisk -l + +Disk /dev/sdb: 1.8 TiB, 2000398934016 bytes, 3907029168 sectors +Disk model: HGST HUS724020AL +Units: sectors of 1 * 512 = 512 bytes +Sector size (logical/physical): 512 bytes / 512 bytes +I/O size (minimum/optimal): 512 bytes / 512 bytes +Disklabel type: gpt +Disk identifier: F92B6C5B-2518-4B2D-8FF9-A311DED5845F + +Device Start End Sectors Size Type +/dev/sdb1 2048 4095 2048 1M BIOS boot +/dev/sdb2 4096 1864177663 1864173568 888.9G Linux RAID +/dev/sdb3 1864177664 1865226239 1048576 512M Linux filesystem +/dev/sdb4 1865226240 3907024895 2041798656 973.6G Linux RAID + +Disk /dev/sda: 1.8 TiB, 2000398934016 bytes, 3907029168 sectors +Disk model: HGST HUS724020AL +Units: sectors of 1 * 512 = 512 bytes +Sector size (logical/physical): 512 bytes / 512 bytes +I/O size (minimum/optimal): 512 bytes / 512 bytes +Disklabel type: gpt +Disk identifier: 2E1DCCBA-8808-4D2B-BA33-9FEC3B96ADA8 + +Device Start End Sectors Size Type +/dev/sda1 2048 4095 2048 1M BIOS boot +/dev/sda2 4096 1864177663 1864173568 888.9G Linux RAID +/dev/sda3 1864177664 1865226239 1048576 512M Linux filesystem +/dev/sda4 1865226240 3907024895 2041798656 973.6G Linux RAID +/dev/sda5 3907025072 3907029134 4063 2M Linux filesystem + +Disk /dev/md4: 973.5 GiB, 1045265645568 bytes, 2041534464 sectors +Units: sectors of 1 * 512 = 512 bytes +Sector size (logical/physical): 512 bytes / 512 bytes +I/O size (minimum/optimal): 512 bytes / 512 bytes + +Disk /dev/md2: 888.8 GiB, 954321600512 bytes, 1863909376 sectors +Units: sectors of 1 * 512 = 512 bytes +Sector size (logical/physical): 512 bytes / 512 bytes +I/O size (minimum/optimal): 512 bytes / 512 bytes +``` + +The `fdisk -l` command also allows you to identify your partition type. This is an important information when it comes to rebuilding your RAID in case of a disk failure. + +For **GPT** partitions, line 6 will display: `Disklabel type: gpt`. + +For **MBR** partitions, line 6 will display: `Disklabel type: dos`. + +Still going by the results of `fdisk -l`, we can see that `/dev/md2` consists of 888.8GB and `/dev/md4` contains 973.5GB. If we were to run the mount command we can also find out the layout of the disk. + +```sh +mount + +sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime) +proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) +udev on /dev type devtmpfs (rw,nosuid,relatime,size=16315920k,nr_inodes=4078980,mode=755) +devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000) +tmpfs on /run type tmpfs (rw,nosuid,noexec,relatime,size=3266556k,mode=755) +/dev/md2 on / type ext4 (rw,relatime) +securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime) +tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev) +tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k) +tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755) +cgroup2 on /sys/fs/cgroup/unified type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate) +cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,name=systemd) +pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime) +bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700) +cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids) +cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory) +cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event) +cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma) +cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio) +cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct) +cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer) +cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio) +cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices) +cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset) +debugfs on /sys/kernel/debug type debugfs (rw,relatime) +hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M) +mqueue on /dev/mqueue type mqueue (rw,relatime) +systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=45,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=10340) +/dev/md4 on /home type ext3 (rw,relatime) +tmpfs on /run/user/1000 type tmpfs (rw,nosuid,nodev,relatime,size=3266552k,mode=700,uid=1000,gid=1000) +``` + +Alternatively, the `lsblk` command offers a different view of the partitions: + +```sh +lsblk + +NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT +sda 8:0 0 1.8T 0 disk +├─sda1 8:1 0 1M 0 part +├─sda2 8:2 0 888.9G 0 part +│ └─md2 9:2 0 888.8G 0 raid1 / +├─sda3 8:3 0 512M 0 part [SWAP] +├─sda4 8:4 0 973.6G 0 part +│ └─md4 9:4 0 973.5G 0 raid1 /home +└─sda5 8:5 0 2M 0 part +sdb 8:16 0 1.8T 0 disk +├─sdb1 8:17 0 1M 0 part +├─sdb2 8:18 0 888.9G 0 part +│ └─md2 9:2 0 888.8G 0 raid1 / +├─sdb3 8:19 0 512M 0 part [SWAP] +└─sdb4 8:20 0 973.6G 0 part + └─md4 9:4 0 973.5G 0 raid1 /home +``` + +We take note of the devices, partitions and their mount points. From the above commands and results, we have: + +- Two RAID arrays: `/dev/md2` and `/dev/md4`. +- Partitions part of the RAID: `/` and `/home`. +- Partitions not part of the RAID: [SWAP]. + +### System verification + +To check whether a server runs on BIOS mode or BIOS mode, run the following command: + +```sh +[user@server_ip ~]# [ -d /sys/firmware/efi ] && echo BIOS || echo BIOS +``` + +### Simulating a disk failure + +Now that we have all the necessary information, we can simulate a disk failure and proceed with the tests. In this example, we will fail the main disk `sda`. + +The preferred way to do this is via the OVHcloud rescue mode environment. + +First reboot in rescue menu and log in with the credentials provided. + +To remove a disk from the RAID, the first step is to mark it as **Failed** and remove the partitions from their respective RAID arrays. + +```sh +cat /proc/mdstat + +Personalities : [raid1] [linear] [multipath] [raid0] [raid6] [raid5] [raid4] [raid10] +md2 : active raid1 sda2[1] sdb2[0] + 931954688 blocks super 1.2 [2/2] [UU] + bitmap: 2/7 pages [8KB], 65536KB chunk + +md4 : active raid1 sda4[0] sdb4[1] + 1020767232 blocks super 1.2 [2/2] [UU] + bitmap: 0/8 pages [0KB], 65536KB chunk + +unused devices: +``` +From the above output, sda consists of two partitions in RAID which are **sda2** and **sda4**. + +#### Removing the failed disk + +First we mark the partitions **sda2** and **sda4** as failed. + +```sh +sudo mdadm --manage /dev/md2 --fail /dev/sda2 + +# mdadm: set /dev/sda2 faulty in /dev/md2 +``` + +```sh +sudo mdadm --manage /dev/md4 --fail /dev/sda4 + +# mdadm: set /dev/sda4 faulty in /dev/md4 +``` + +We have now simulated a failure of the RAID, when we run the `cat /proc/mdstat` command, we have the following output + +```sh +cat /proc/mdstat + +Personalities : [raid1] [linear] [multipath] [raid0] [raid6] [raid5] [raid4] [raid10] +md2 : active raid1 sda2[1](F) sdb2[0] + 931954688 blocks super 1.2 [2/2] [_U] + bitmap: 2/7 pages [8KB], 65536KB chunk + +md4 : active raid1 sda4[0](F) sdb4[1] + 1020767232 blocks super 1.2 [2/2] [_U] + bitmap: 0/8 pages [0KB], 65536KB chunk +unused devices: +``` + +As we can see above, the [F] next to the partitions indicates that the disk has failed or is faulty. + +Next, we remove these partitions from the RAID arrays to completely remove the disk from RAID. + +```sh +sudo mdadm --manage /dev/md2 --remove /dev/sda2 + +# mdadm: hot removed /dev/sda2 from /dev/md2 +``` + +```sh +sudo mdadm --manage /dev/md3 --remove /dev/sda3 + +# mdadm: hot removed /dev/nvme0n1p3 from /dev/md3 +``` + +To make sure that we get a disk that is similar to an empty disk, we use the following command. Replace **sda** with your own values: + +```sh +sudo shred -s10M -n1 /dev/sda +``` + +The disk now appears as a new one: + +```sh +lsblk +NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT +sda 8:0 0 1.8T 0 disk +sdb 8:16 0 1.8T 0 disk +├─sdb1 8:17 0 1M 0 part +├─sdb2 8:18 0 888.9G 0 part +│ └─md2 9:2 0 888.8G 0 raid1 / +├─sdb3 8:19 0 512M 0 part [SWAP] +└─sdb4 8:20 0 973.6G 0 part + └─md4 9:4 0 973.5G 0 raid1 /home +``` + +### Rebuilding the RAID + +The following steps document the RAID rebuild in rescue mode. + +> [!primary] +> This process might be different depending on the operating system you have installed on your server. We recommend that you consult the official documentation of your operating system to have access to the proper commands. +> + +> [!warning] +> +> For most servers in software RAID, after a disk replacement, the server is able to reboot in normal mode (on the healthy disk) and the rebuild can be done in normal mode. However, if the server is not able to reboot in normal mode after a disk replacement, it will be rebooted in rescue mode to proceed with the RAID rebuild. +> + +#### Rebuilding the RAID after the main disk is replaced + +Once the disk has been replaced, we need to copy the partition table from the healthy disk (in this example, sdb ) to the new one (sda). + +> [!tabs] +> **For GPT partitions** +>> +>> ```sh +>> sudo sgdisk -R /dev/sdX /dev/sdX +>> ``` +>> +>> The command should be in this format: `sgdisk -R /dev/newdisk /dev/healthydisk` +>> +>> Example: +>> +>> ```sh +>> sudo sgdisk -R /dev/sda /dev/sdb +>> ``` +>> +>> Once this is done, the next step is to randomize the GUID of the new disk to prevent GUID conflicts with other disks: +>> +>> ```sh +>> sudo sgdisk -G /dev/sda +>> ``` +>> +>> If you receive a message like this: +>> +>> ```console +>> Warning: The kernel is still using the old partition table. +>> The new table will be used at the next reboot or after you +>> run partprobe(8) or kpartx(8) +>> The operation has completed successfully. +>> ``` +>> +>> You can simply run the `partprobe` command or reboot the server when the raid rebuilt is done. +>> +> **For MBR partitions** +>> +>> ```sh +>> sudo sfdisk -d /dev/sdb | sfdisk /dev/sda +>> ``` +>> +>> The command should be in this format: `sfdisk -d /dev/healthydisk | sfdisk /dev/newdisk` +>> + +We can now rebuild the RAID array. The following code snippet shows how we can add the new partitions (sda2 and sda4) back in the RAID array. + + +```sh +sudo mdadm --add /dev/md2 /dev/sda2 + +# mdadm: added /dev/sda2 + +sudo mdadm --add /dev/md4 /dev/sda4 + +# mdadm: re-added /dev/sda4 + +cat /proc/mdstat + +Personalities : [raid1] [linear] [multipath] [raid0] [raid6] [raid5] [raid4] [raid10] +md2 : active raid1 sda2[0] sdb2[1] + 931954688 blocks super 1.2 [2/2] [UU] + bitmap: 4/4 pages [16KB], 65536KB chunk + +md4 : active raid1 sda4[0](F) sdb4[1] + 1020767232 blocks super 1.2 [2/1] [UU] + [============>........] recovery = 64.8% (822969856/1020767232) finish=7.2min speed=401664K/sec + bitmap: 0/8 pages [0KB], 65536KB chunk +unused devices: +``` + +We can verify the RAID details with the following command: + +```sh +mdadm --detail /dev/md4 + +/dev/md4: + Version : 1.2 + Creation Time : Tue Jan 24 15:35:02 2023 + Raid Level : raid1 + Array Size : 1020767232 (973.48 GiB 1045.27 GB) + Used Dev Size : 1020767232 (973.48 GiB 1045.27 GB) + Raid Devices : 2 + Total Devices : 2 + Persistence : Superblock is persistent + + Intent Bitmap : Internal + + Update Time : Tue Jan 24 17:02:55 2023 + State : clean + Active Devices : 2 + Working Devices : 2 + Failed Devices : 0 + Spare Devices : 0 + + Rebuild Status : 21% complete + + UUID : 7f39d062:9f16a016:a4d2adc2:26fd5302 + Events : 0.95 + + Number Major Minor RaidDevice State + 0 8 2 0 spare rebuilding /dev/sda4 + 1 8 18 1 active sync /dev/sdb4 +``` + +The RAID has now been rebuilt. + +#### Adding the label to the SWAP partition (if applicable) + +First, we mount the partition containing our files on `/mnt`. In our example, that partition is `md4`. + +```sh +mount /dev/md4 /mnt +``` + +We add the label back to our swap partition: + +```sh +mkswap /dev/sda4 -L swap-sda4 +mkswap: /dev/sda4: warning: wiping old swap signature. +Setting up swapspace version 1, size = 512 MiB (536866816 bytes) +LABEL=swap-nvme0n1p4, UUID=b3c9e03a-52f5-4683-81b6-cc10091fcd +``` + +Next, we mount the following directories to make sure any manipulation we make in the chroot environment works properly: + +```sh +mount --types proc /proc /mnt/proc +mount --rbind /sys /mnt/sys +mount --make-rslave /mnt/sys +mount --rbind /dev /mnt/dev +mount --make-rslave /mnt/dev +mount --bind /run /mnt/run +mount --make-slave /mnt/run +``` + +Next, we access the `chroot` environment again: + +```sh +chroot /mnt +``` + +We retrieve the UUIDs of both swap partitions: + +```sh +blkid /dev/sda4 +blkid /dev/sdb4 +``` + +Example: + +```sh +blkid /dev/sda4 +/dev/sda4: LABEL="swap-sda4" UUID="b3c9e03a-52f5-4683-81b6-cc10091fcd15" TYPE="swap" PARTLABEL="logical" PARTUUID="cb6022d9-7631-449c-956a-116bcd0e2119" +``` + +```sh +blkid /dev/sdb4 +/dev/sdb4: LABEL="swap-sdb4" UUID="d6af33cf-fc15-4060-a43c-cb3b5537f58a" TYPE="swap" PARTLABEL="logical" PARTUUID="d037c35f-2ddb-40d5-be33-31cc496fe54b" +``` + +Next, we replace the old UUID of the swap partition (**nvme0n1p4)** with the new one in `/etc/fstab`: + +```sh +nano etc/fstab +``` + +Example: + +```sh +UUID=6abfaa3b-e630-457a-bbe0-e00e5b4b59e5 / ext4 defaults 0 1 +UUID=f925a033-0087-40ec-817e-44efab0351ac /home ext4 defaults 0 0 +UUID=b7b5dd38-9b51-4282-8f2d-26c65e8d58ec swap swap defaults 0 0 +UUID=d6af33cf-fc15-4060-a43c-cb3b5537f58a swap swap defaults 0 0 +``` + +Make sure you replace the proper UUID. In our example above, the UUID to replace is `b7b5dd38-9b51-4282-8f2d-26c65e8d58ec` with the new one `b3c9e03a-52f5-4683-81b6-cc10091fcd15` since the other one belongs to **sdb4**. + +Next, we make sure everything is properly mounted: + +```sh + mount -av +/ : ignored +mount: (hint) your fstab has been modified, but systemd still uses + the old version; use 'systemctl daemon-reload' to reload. +/boot : successfully mounted +/boot/efi : successfully mounted +swap : ignored +swap : ignored +``` + +We enable the swap partition: + +```sh +swapon -av + +swapon: /dev/sda4: found signature [pagesize=4096, signature=swap] +swapon: /dev/sda4: pagesize=4096, swapsize=536870912, devsize=536870912 +swapon /dev/sda4 +swapon: /dev/sdb4: found signature [pagesize=4096, signature=swap] +swapon: /dev/sdb4: pagesize=4096, swapsize=536870912, devsize=536870912 +swapon /dev/sdb4 +``` + +We exit the Chroot environment with `Exit` and unmount all the disks: + +```sh +umount -a +``` + +We have now successfully completed the RAID rebuild on the server and we can now reboot the server in normal mode. + +/// details | **Rebuilding the RAID after the secondary disk is replaced** + +The following steps are done in normal mode since it is the secondary disk that is was replaced. In our example, our secondary disk is named **sdb**. + +Once the disk has been replaced, we need to copy the partition table from the healthy disk (in this example, sda) to the new one (sdb). + +> [!tabs] +> **For GPT partitions** +>> +>> ```sh +>> sudo sgdisk -R /dev/sdX /dev/sdX +>> ``` +>> +>> The command should be in this format: `sgdisk -R /dev/newdisk /dev/healthydisk`. +>> +>> Once this is done, the next step is to randomize the GUID of the new disk to prevent GUID conflicts with other disks: +>> +>> ```sh +>> sudo sgdisk -G /dev/sdX +>> ``` +>> +>> If you receive a message like this: +>> +>> ```console +>> Warning: The kernel is still using the old partition table. +>> The new table will be used at the next reboot or after you +>> run partprobe(8) or kpartx(8) +>> The operation has completed successfully. +>> ``` +>> +>> You can simply run the `partprobe` command or reboot the server when the raid rebuilt is done. +>> +> **For MBR partitions** +>> +>> ```sh +>> sudo sfdisk -d /dev/sdX | sfdisk /dev/sdX +>> ``` +>> +>> The command should be in this format: `sfdisk -d /dev/healthydisk | sfdisk /dev/newdisk`. +>> + +Next, we add the partitions to the RAID: + +```sh +sudo mdadm --add /dev/md2 /dev/sdb2 + +# mdadm: added /dev/sdb2 + +sudo mdadm --add /dev/md4 /dev/sdb4 + +# mdadm: re-added /dev/sdb4 +``` + +Use the following command to follow the RAID rebuild `cat /proc/mdstat`. + +Lastly, we add a label and mount the [SWAP] partition (if applicable): + +```sh +sudo mkswap /dev/sdb4 -L swap-sdb4 +``` + +We retrieve the UUIDs of both swap partitions: + +```sh +sudo blkid /dev/sda4 +sudo blkid /dev/sdb4 +``` + +We replace the old UUID of the swap partition (**sdb4)** with the new one in `/etc/fstab`: + +```sh +sudo nano etc/fstab +``` + +Make sure you replace the correct UUID. Reload the system with the following command: + +```sh +sudo systemctl daemon-reload +``` + +Next, run the following command to enable it: + +```sh +sudo swapon -av +``` + +We have now successfully completed the RAID rebuild. + +## Go Further + +[Hot Swap - Software RAID](/pages/bare_metal_cloud/dedicated_servers/hotswap_raid_soft) + +[OVHcloud API and Storage](/pages/bare_metal_cloud/dedicated_servers/partitioning_ovh) + +[Managing hardware RAID](/pages/bare_metal_cloud/dedicated_servers/raid_hard) + +[Hot Swap - Hardware RAID](/pages/bare_metal_cloud/dedicated_servers/hotswap_raid_hard) + +Join our [community of users](/links/community). \ No newline at end of file diff --git a/pages/bare_metal_cloud/dedicated_servers/raid_soft_uefi/guide.en-gb.md b/pages/bare_metal_cloud/dedicated_servers/raid_soft_uefi/guide.en-gb.md index 2ad2e116095..7d55cfeee5f 100644 --- a/pages/bare_metal_cloud/dedicated_servers/raid_soft_uefi/guide.en-gb.md +++ b/pages/bare_metal_cloud/dedicated_servers/raid_soft_uefi/guide.en-gb.md @@ -1,7 +1,7 @@ --- -title: How to configure and rebuild software RAID on a Dedicated server with UEFI boot mode -excerpt: Find out how to verify the state of your software RAID and rebuild it after a disk replacement -updated: 2023-08-21 +title: Managing and rebuilding software RAID on servers in UEFI boot mode +excerpt: Find out how to manage and rebuild software RAID after a disk replacement on your server in UEFI mode +updated: 2025-08-xx --- ## Objective @@ -10,7 +10,7 @@ Redundant Array of Independent Disks (RAID) is a technology that mitigates data The default RAID level for OVHcloud server installations is RAID 1, which doubles the space taken up by your data, effectively halving the useable disk space. -**This guide explains how to configure your server’s RAID array in the event that it needs to be rebuilt due to corruption or disk failure.** +**This guide explains how to manage and rebuild software RAID after a disk replacement on your server in UEFI mode** Before we begin, please note that this guide focuses on Dedicated servers that use UEFI as the boot mode. This is the case with modern motherboards. If your server uses the BIOS mode (old motherboards), refer to this guide [How to configure and rebuild software RAID on a Dedicated server with BIOS boot mode](). @@ -18,29 +18,36 @@ For more information on UEFI, consult the following [guide](https://uefi.org/abo ## Requirements -- A [Dedicated server](/links/bare-metal/bare-metal) with a software RAID configuration +- A [dedicated server](/links/bare-metal/bare-metal) with a software RAID configuration - Administrative (sudo) access to the server via SSH - Understanding of RAID, partitions and GRUB ## Instructions -Over the course of this guide, we will explore three scenarios: +When you purchase a new server, you might feel the need to perform a series of tests and actions. One of those actions could be simulating a disk failure in order to understand the process of rebuilding the raid and prepare yourself in case this happens. -- Disk failure simulation (preventive measure) -- Dealing with a faulty disk that is still present in RAID. -- Dealing with a faulty disk that is no longer present in RAID. +### Basic Information -### System verification - -To check whether a server runs on BIOS mode or UEFI mode, run the following command: +In a command line session, type the following code to determine the current RAID status: ```sh -[user@server_ip ~]# [ -d /sys/firmware/efi ] && echo UEFI || echo BIOS +cat /proc/mdstat +Personalities : [raid1] [linear] [multipath] [raid0] [raid6] [raid5] [raid4] [raid10] +md3 : active raid1 nvme1n1p3[1] nvme0n1p3[0] + 497875968 blocks super 1.2 [2/2] [UU] + bitmap: 2/4 pages [8KB], 65536KB chunk + +md2 : active raid1 nvme1n1p2[1] nvme0n1p2[0] + 1046528 blocks super 1.2 [2/2] [UU] + +unused devices: ``` -#### Basic information +This command shows us that we have two software RAID devices currently set up, with **md3** being the largest one. This array consists of two partitions, which are known as **nvme1n1p3** and **nvme0n1p3**. -To begin, type the following code in a command line session to determine the current RAID status: +The [UU] means that all the disks are working normally. A `_` would indicate a failed disk. + +If you have a server with SATA disks, you would get the following results: ```sh cat /proc/mdstat @@ -55,42 +62,40 @@ md2 : active raid1 sda2[0] sdb2[1] unused devices: ``` -This command shows us that we have two software RAID devices currently set up, with **md3** being the largest one. This array consists of two partitions, which are known as sda3 and sdb3. The [UU] means that all the disks are working normally. A `_` would indicate a failed disk. - Although this command returns our RAID volumes, it doesn't tell us the size of the partitions themselves. We can find this information with the following command: ```sh sudo fdisk -l -Disk /dev/sdb: 3.64 TiB, 4000787030016 bytes, 7814037168 sectors -Disk model: HGST HUS726T4TAL +Disk /dev/nvme1n1: 476.94 GiB, 512110190592 bytes, 1000215216 sectors +Disk model: WDC CL SN720 SDAQNTW-512G-2000 Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: gpt -Disk identifier: 3253EEB9-DC20-4887-9E64-69655C758811 +Disk identifier: A11EDAA3-A984-424B-A6FE-386550A92435 -Device Start End Sectors Size Type -/dev/sdb1 2048 1048575 1046528 511M EFI System -/dev/sdb2 1048576 3145727 2097152 1G Linux RAID -/dev/sdb3 3145728 7812982783 7809837056 3.6T Linux RAID -/dev/sdb4 7812982784 7814031359 1048576 512M Linux filesystem +Device Start End Sectors Size Type +/dev/nvme1n1p1 2048 1048575 1046528 511M EFI System +/dev/nvme1n1p2 1048576 3145727 2097152 1G Linux RAID +/dev/nvme1n1p3 3145728 999161855 996016128 474.9G Linux RAID +/dev/nvme1n1p4 999161856 1000210431 1048576 512M Linux files -Disk /dev/sda: 3.64 TiB, 4000787030016 bytes, 7814037168 sectors -Disk model: HGST HUS726T4TAL +Disk /dev/nvme0n1: 476.94 GiB, 512110190592 bytes, 1000215216 sectors +Disk model: WDC CL SN720 SDAQNTW-512G-2000 Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: gpt -Disk identifier: 5D8AF9B0-E363-40AF-A3CA-FADB7B3C6A10 +Disk identifier: F03AC3C3-D7B7-43F9-88DB-9F12D7281D94 -Device Start End Sectors Size Type -/dev/sda1 2048 1048575 1046528 511M EFI System -/dev/sda2 1048576 3145727 2097152 1G Linux RAID -/dev/sda3 3145728 7812982783 7809837056 3.6T Linux RAID -/dev/sda4 7812982784 7814031359 1048576 512M Linux filesystem -/dev/sda5 7814033072 7814037134 4063 2M Linux filesystem +Device Start End Sectors Size Type +/dev/nvme0n1p1 2048 1048575 1046528 511M EFI System +/dev/nvme0n1p2 1048576 3145727 2097152 1G Linux RAID +/dev/nvme0n1p3 3145728 999161855 996016128 474.9G Linux RAID +/dev/nvme0n1p4 999161856 1000210431 1048576 512M Linux file +/dev/nvme0n1p5 1000211120 1000215182 4063 2M Linux file Disk /dev/md2: 1022 MiB, 1071644672 bytes, 2093056 sectors @@ -99,7 +104,7 @@ Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes -Disk /dev/md3: 3.64 TiB, 3998501306368 bytes, 7809572864 sectors +Disk /dev/md3: 474.81 GiB, 509824991232 bytes, 995751936 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes @@ -107,136 +112,170 @@ I/O size (minimum/optimal): 512 bytes / 512 bytes The `fdisk -l` command also allows you to identify your partition type. This is an important information when it comes to rebuilding your RAID in case of a disk failure. -For **GPT** partitions, the command will return: `Disklabel type: gpt`. - -```sh -Disk /dev/sdb: 3.64 TiB, 4000787030016 bytes, 7814037168 sectors -Disk model: HGST HUS726T4TAL -Units: sectors of 1 * 512 = 512 bytes -Sector size (logical/physical): 512 bytes / 512 bytes -I/O size (minimum/optimal): 512 bytes / 512 bytes -Disklabel type: gpt -Disk identifier: 3253EEB9-DC20-4887-9E64-69655C758811 -``` - -For **MBR** partitions, the command will return: `Disklabel type: dos`. +For **GPT** partitions, line 6 will display: `Disklabel type: gpt`. -```sh -Disk /dev/sda: 2.5 GiB, 2621440000 bytes, 5120000 sectors -Disk model: QEMU HARDDISK -Units: sectors of 1 * 512 = 512 bytes -Sector size (logical/physical): 512 bytes / 512 bytes -I/O size (minimum/optimal): 512 bytes / 512 bytes -'Disklabel type: dos' -Disk identifier: 0x150f6797 -``` +For **MBR** partitions, line 6 will display: `Disklabel type: dos`. -Still going by the results of `fdisk -l`, we can see that `/dev/md2` consists of 1022 MiB and `/dev/md3` contains 3.64 TiB. If we were to run the mount command we can also find out the layout of the disk. +Still going by the results of `fdisk -l`, we can see that `/dev/md2` consists of 1022 MiB and `/dev/md3` contains 474.81 GiB. If we were to run the mount command we can also find out the layout of the disk. ```sh mount sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime) proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) -udev on /dev type devtmpfs (rw,nosuid,relatime,size=16288028k,nr_inodes=4072007,mode=755,inode64) +udev on /dev type devtmpfs (rw,nosuid,relatime,size=16348288k,nr_inodes=4087072,mode=755) devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000) -tmpfs on /run type tmpfs (rw,nosuid,nodev,noexec,relatime,size=3263252k,mode=755,inode64) +tmpfs on /run type tmpfs (rw,nosuid,nodev,noexec,relatime,size=3275188k,mode=755) /dev/md3 on / type ext4 (rw,relatime) securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime) -tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,inode64) -tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k,inode64) +tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev) +tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k) cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot) pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime) efivarfs on /sys/firmware/efi/efivars type efivarfs (rw,nosuid,nodev,noexec,relatime) -bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700) -systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=926) +none on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700) +systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1462) hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M) mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime) debugfs on /sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,relatime) tracefs on /sys/kernel/tracing type tracefs (rw,nosuid,nodev,noexec,relatime) fusectl on /sys/fs/fuse/connections type fusectl (rw,nosuid,nodev,noexec,relatime) configfs on /sys/kernel/config type configfs (rw,nosuid,nodev,noexec,relatime) -ramfs on /run/credentials/systemd-sysctl.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700) -ramfs on /run/credentials/systemd-sysusers.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700) -ramfs on /run/credentials/systemd-tmpfiles-setup-dev.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700) /dev/md2 on /boot type ext4 (rw,relatime) -/dev/sda1 on /boot/efi type vfat (rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=ascii,shortname=mixed,utf8,errors=remount-ro) -ramfs on /run/credentials/systemd-tmpfiles-setup.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700) -binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,nosuid,nodev,noexec,relatime) -tmpfs on /run/user/1000 type tmpfs (rw,nosuid,nodev,relatime,size=3263248k,nr_inodes=815812,mode=700,uid=1000,gid=1000,inode64) +/dev/nvme0n1p1 on /boot/efi type vfat (rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=ascii,shortname=mixed,utf8,errors=remount-ro) +tmpfs on /run/user/1000 type tmpfs (rw,nosuid,nodev,relatime,size=3275184k,nr_inodes=818796,mode=700,uid=1000,gid=1000) ``` Alternatively, the `lsblk` command offers a different view of the partitions: ```sh lsblk +NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT +nvme1n1 259:0 0 476.9G 0 disk +├─nvme1n1p1 259:7 0 511M 0 part +├─nvme1n1p2 259:8 0 1G 0 part +│ └─md2 9:2 0 1022M 0 raid1 /boot +├─nvme1n1p3 259:9 0 474.9G 0 part +│ └─md3 9:3 0 474.8G 0 raid1 / +└─nvme1n1p4 259:10 0 512M 0 part [SWAP] +nvme0n1 259:1 0 476.9G 0 disk +├─nvme0n1p1 259:2 0 511M 0 part /boot/efi +├─nvme0n1p2 259:3 0 1G 0 part +│ └─md2 9:2 0 1022M 0 raid1 /boot +├─nvme0n1p3 259:4 0 474.9G 0 part +│ └─md3 9:3 0 474.8G 0 raid1 / +├─nvme0n1p4 259:5 0 512M 0 part [SWAP] +└─nvme0n1p5 259:6 0 2M 0 part +``` -NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS -sda 8:0 0 3.6T 0 disk -├─sda1 8:1 0 511M 0 part /boot/efi -├─sda2 8:2 0 1G 0 part -│ └─md2 9:2 0 1022M 0 raid1 /boot -├─sda3 8:3 0 3.6T 0 part -│ └─md3 9:3 0 3.6T 0 raid1 / -├─sda4 8:4 0 512M 0 part [SWAP] -└─sda5 8:5 0 2M 0 part -sdb 8:16 0 3.6T 0 disk -├─sdb1 8:17 0 511M 0 part -├─sdb2 8:18 0 1G 0 part -│ └─md2 9:2 0 1022M 0 raid1 /boot -├─sdb3 8:19 0 3.6T 0 part -│ └─md3 9:3 0 3.6T 0 raid1 / -└─sdb4 8:20 0 512M 0 part [SWAP] -sdc 8:32 1 0B 0 disk -sr0 11:0 1 1024M 0 rom -``` - -With UEFI mode, ESPs (lablled EFI_SYSPART by OVHcloud) partitions are created on the disks used for RAID, however, only one partition is automatically mounted in `/boot/efi` by the system after install. It is important to understand the role of this partition and how to manage it. - -Unfortunately, at this time, `/boot/efi` is not mirrored in our RAID installations, therefore, if you need to have the content of this partition up dated to the other disk(s), you will need sync these partitions. - -The command `lsblk -f` offers a more detailed output: - -```sh -lsblk -f - -NAME FSTYPE FSVER LABEL UUID FSAVAIL FSUSE% MOUNTPOINTS -sda -├─sda1 vfat FAT16 EFI_SYSPART 7A52-33BE 504.8M 1% /boot/efi -├─sda2 linux_raid_member 1.2 md2 aa1265ff-d8a7-9b64-0109-1086e78a451c -│ └─md2 ext4 1.0 boot 857b98e6-a5d8-4126-95c8-1451cea76a5b 850.3M 7% /boot -├─sda3 linux_raid_member 1.2 md3 7eb971ad-6f31-0998-50f2-50f330c34361 -│ └─md3 ext4 1.0 root 4b8782be-3bd4-4b96-8129-491806a54b46 3.4T 0% / -├─sda4 swap 1 swap-sda4 da546332-ea7c-4f0d-969c-5b8818c8dc81 [SWAP] -└─sda5 iso9660 Joliet Extension config-2 2025-07-23-17-11-22-00 -sdb -├─sdb1 vfat FAT16 EFI_SYSPART 7A68-19BC -├─sdb2 linux_raid_member 1.2 md2 aa1265ff-d8a7-9b64-0109-1086e78a451c -│ └─md2 ext4 1.0 boot 857b98e6-a5d8-4126-95c8-1451cea76a5b 850.3M 7% /boot -├─sdb3 linux_raid_member 1.2 md3 7eb971ad-6f31-0998-50f2-50f330c34361 -│ └─md3 ext4 1.0 root 4b8782be-3bd4-4b96-8129-491806a54b46 3.4T 0% / -└─sdb4 swap 1 swap-sdb4 37dda653-4074-4c28-953c-ae627fe53816 [SWAP] -sdc -sr0 -nvme0n1 +We take note of the devices, partitions and their mount points. + +From the above commands and results, we have: + +- Two RAID arrays: `/dev/md2` and `/dev/md3`. +- Partitions part of the RAID: `/boot` and `/`. +- Partitions not part of the RAID: `/boot/efi` and [SWAP]. + +### System verification + +To check whether a server runs on BIOS mode or UEFI mode, run the following command: + +```sh +[user@server_ip ~]# [ -d /sys/firmware/efi ] && echo UEFI || echo BIOS +``` + +### Understanding the EFI partition + +To begin let's understand what is UEFI mode. + +> [!primary] +> Please note that depending on your operating system, the process might be different. Ubuntu for example is able to keep both partitions synchronized at every GRUB update. However, it is the only operating system doing so. We recommend that you consult the official documentation of your operating system to understand how to manage this partition. +> +> In the guide, the operating system used is Debian. +> + +***What is an EFI partition?*** + +An EFI partition, is a partition which can contain the boot loaders, boot managers, or kernel images of an installed operating system. It also contains system utility programs designed to be run before the operating system boots, as well as data files such as error logs. + +***Is the EFI partition mirrored in RAID?*** + +No, by default, the EFI partition is a partition which cannot be included in the RAID. When you use one of our default installation templates to install your server in software RAID, two EFI partitions are created, one on each disk. However, only one EFI partition is used for the installation of the operating system. + +The partition is mounted on `/boot/efi` and the disk on which it is mounted is selected randomly by the operating system during installation. + +Example: + +```sh +lsblk + +NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT +nvme1n1 259:0 0 476.9G 0 disk +├─nvme1n1p1 259:7 0 511M 0 part +├─nvme1n1p2 259:8 0 1G 0 part +│ └─md2 9:2 0 1022M 0 raid1 /boot +├─nvme1n1p3 259:9 0 474.9G 0 part +│ └─md3 9:3 0 474.8G 0 raid1 / +└─nvme1n1p4 259:10 0 512M 0 part [SWAP] +nvme0n1 259:1 0 476.9G 0 disk +├─nvme0n1p1 259:2 0 511M 0 part /boot/efi +├─nvme0n1p2 259:3 0 1G 0 part +│ └─md2 9:2 0 1022M 0 raid1 /boot +├─nvme0n1p3 259:4 0 474.9G 0 part +│ └─md3 9:3 0 474.8G 0 raid1 / +├─nvme0n1p4 259:5 0 512M 0 part [SWAP] +└─nvme0n1p5 259:6 0 2M 0 part +``` + +From the example above, we see that we have two identical EFI partitions (nvme0n1p1 and nvme1n1p1) but only **nvme0n1p1** is mounted on `/boot/efi`. + +We can also use the following command to confirm that two EFI partitions were actually created after install: + +```sh +sudo lsblk -f +NAME FSTYPE FSVER LABEL UUID FSAVAIL FSUSE% MOUNTPOINT nvme1n1 +├─nvme1n1p1 vfat FAT16 EFI_SYSPART B493-9DFA +├─nvme1n1p2 linux_raid_member 1.2 md2 baae988b-bef3-fc07-615f-6f9043cfd5ea +│ └─md2 ext4 1.0 boot 96850c4e-e2b5-4048-8c39-525194e441aa 851.8M 7% /boot +├─nvme1n1p3 linux_raid_member 1.2 md3 ce0c7fac-0032-054c-eef7-7463b2245519 +│ └─md3 ext4 1.0 root 6fea39e9-6297-4ea3-82f1-bf1a3e88106a 441.3G 0% / +└─nvme1n1p4 swap 1 swap-nvme1n1p4 483b9b41-ada3-4143-8cac-5bff7afb73c7 [SWAP] +nvme0n1 +├─nvme0n1p1 vfat FAT16 EFI_SYSPART B486-9781 504.9M 1% /boot/efi +├─nvme0n1p2 linux_raid_member 1.2 md2 baae988b-bef3-fc07-615f-6f9043cfd5ea +│ └─md2 ext4 1.0 boot 96850c4e-e2b5-4048-8c39-525194e441aa 851.8M 7% /boot +├─nvme0n1p3 linux_raid_member 1.2 md3 ce0c7fac-0032-054c-eef7-7463b2245519 +│ └─md3 ext4 1.0 root 6fea39e9-6297-4ea3-82f1-bf1a3e88106a 441.3G 0% / +├─nvme0n1p4 swap 1 swap-nvme0n1p4 51e7172b-adb0-4729-b0f8-613e5dede38b [SWAP] +└─nvme0n1p5 iso9660 Joliet Extension config-2 2025-08-05-14-55-41-00 ``` -From the above output, we can see that two partitions (sda1 and sdb1) with the label `EFI_SYSPART` were created on both disks. However, the OS selected **sda1** to mount the partition in `/boot/efi`. +From the results above, we can see two partitions (**nvme0n1p1** and **nvme1n1p1**), with identical size (504.9M). Both partitions have the LABEL: `EFI_SYSPART` but only one is mounted on `/boot/efi`. -This partition contains the boot loaders (and sometimes kernel images and drivers), which helps the server to boot but it is not protected through the software RAID. In general, the content of this partition remains the same **unless** there is a kernel update, in this case you must sync both partitions to keep them up to date. +***Does the content of EFI partition change regularly?*** -In case of the failure of the main disk containing this partition, the server will be rebooted in rescue mode for you to recreate the partition on the new disk and copy the files from the healthy disk to the new disk (More information on this later in the guide.). It is therefore important keep the duplicated partition up to date when necessary, since it is not present in the RAID. +In general, the contents of this partition do not change much, except when there are relevant updates such as grub/kernel updates or even simple `apt` or `yum` updates. -Alternatively, you can explore another premptive option such as keeping these partitions synchronised by either running a manual script daily (or when needed) or running an automatic script to sync both partitions on boot. You must be the user **root** to run this script. +In this case, we recommend running an automatic or manual script keep the partitions in sync. This is because in case of the failure of the main disk on which this partition in mounted, you can use the rescue mode to recover it. -Scrip example: +***What happens if my main disk fails and is replaced?*** -```bash -#!/bin/bash +If the main disk on your server fails and it is replaced. The server will be rebooted in rescue mode in other for you to rebuild the raid. Since the EFI partition is not mirrored, the server will not be able to boot on the secondary disk (in most cases). You can rebuild the raid in rescue mode, recreate the EFI partition and the disk should be able to boot. + +***What if the main disk fails and I did not synchronise my EFI partitions after a major update (kernel/grub)?*** + +If this is the case, you will have to rebuild the raid, create the EFI partition on the new disk and install the grub/kernel update in rescue mode. If you are not able to perform some manipulations, we recommend contacting a professional service. + +***How can I keep my EFI partitions synchronized or how often should I sychronize them?*** + +We recommend synchronizing both partitions daily. Below is a script you can use to manually synchronize your EFI partitions. You can also run an automated script to synchronize the partitions daily or when the service is booted. + +```sh set -euo pipefail -MAIN_PARTITION="$(findmnt -n -o SOURCE /boot/efi)" +findmnt -n -o SOURCE /boot/efi +MAIN_PARTITION=/dev/ # sdX or nvmeXnXpX MOUNTPOINT="/var/lib/grub/esp" mkdir -p "${MOUNTPOINT}" @@ -249,120 +288,201 @@ while read -r partition; do mount "${partition}" "${MOUNTPOINT}" rsync -ax "/boot/efi/" "${MOUNTPOINT}/" umount "${MOUNTPOINT}" -done < <(blkid -o device -t LABEL=EFI_SYSPART) +done < <(blkid -o device -t LABEL=EFI_SYSPART) ``` -The script above can be run manually to keep them synchronized, the difficulty being knowing when to execute it. The ideal solution would probably be to use dpkg triggers to only do this when updating the `grub-efi-amd64` package. +In the script above, the `MAIN_PARTITION` is the one mounted on /boot/efi and `MOUNTPOINT` is where you want to sync the contents. +Before you run the script, make sure `rsync` is installed: -> [!warning] -> If you are on an Ubuntu system, you can easily keep these partitions synchronised with a grub installation, but this ONLY works for Ubuntu system. For other linux systems, this must been done manually or when using a sync. For more information consult this article. -> +**Debian/Ubuntu** + +```sh +sudo apt install rsync +``` + +**CentOS, Red Hat and Fedora** + +```sh +sudo yum install rsync +``` + +Example: + +```sh +set -euo pipefail + +findmnt -n -o SOURCE /boot/efi +MAIN_PARTITION=/dev/nvme0n1p1 +MOUNTPOINT="/var/lib/grub/esp" + +mkdir -p "${MOUNTPOINT}" + +while read -r partition; do + if [[ "${partition}" == "${MAIN_PARTITION}" ]]; then + continue + fi + echo "Working on ${partition}" + mount "${partition}" "${MOUNTPOINT}" + rsync -ax "/boot/efi/" "${MOUNTPOINT}/" + umount "${MOUNTPOINT}" +done < <(blkid -o device -t LABEL=EFI_SYSPART) +``` + +In the example above, the main partition is `nvme0n1p1` and the mountpoint we use is `/var/lib/grub/esp`. + +### Simulating a disk failure -### Simulating a Disk failure +Now that we have all the necessary information, we can simulate a disk failure and proceed with the tests. In this example, we will fail the main disk `nvme0n1`. -When you purchase a new server, you might feel the need to perform a series of tests and actions. One of those actions could be simulating a disk failure in order to understand the process of rebuilding the raid and prepare yourself in case this happens. +The preferred way to do this is via the OVHcloud rescue mode environment. -#### Removing the Dis +First reboot in rescue menu and log in with the credentials provided. -As the disks are currently mounted by default, to remove a disk from the RAID, we first need to unmount the disk, then simulate a failure, and finally remove it. We will remove `/dev/sda4` from the RAID with the following command: +To remove a disk from the RAID, the first step is to mark it as **Failed** and remove the partitions from their respective RAID arrays. ```sh -umount /dev/md4 +root@rescue12-customer-ca (nsxxxxx.ip-xx-xx-xx.eu) ~ # cat /proc/mdstat +Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] [multipath] [faulty] +md3 : active raid1 nvme0n1p3[0] nvme1n1p3[1] + 497875968 blocks super 1.2 [2/2] [UU] + bitmap: 0/4 pages [0KB], 65536KB chunk + +md2 : active raid1 nvme0n1p2[2] nvme1n1p2[1] + 1046528 blocks super 1.2 [2/2] [UU] + +unused devices: ``` -> [!warning] -> Please note that if you are connected as the user `root`, you may get the following message when you try to unmount the partition (in our case, where our md4 partition is mounted in /home): -> -> 
umount: /home: target is busy
-> -> In this case, you must log out as the user root and connect as a local user (in our case `debian`), and use the following command: -> -> 
debian@ns000000:/$ sudo umount /dev/md4
-> -> If you do not have a local user, you need to [create one](/pages/bare_metal_cloud/dedicated_servers/changing_root_password_linux_ds). +From the above output, nvme0n1 consists of two partitions in RAID which are **nvme0n1p2** and **nvme0n1p3**. + +#### Removing the failed disk -This will provide us with the following output: +First we mark the partitions **nvme0n1p2** and **nvme0n1p3** as failed. ```sh -sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime) -proc on /proc type proc (rw,nosuid,nodev,noexec,relatime) -udev on /dev type devtmpfs (rw,nosuid,relatime,size=16315920k,nr_inodes=4078980,mode=755) -devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000) -tmpfs on /run type tmpfs (rw,nosuid,noexec,relatime,size=3266556k,mode=755) -/dev/md2 on / type ext4 (rw,relatime) -securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime) -tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev) -tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k) -tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755) -cgroup2 on /sys/fs/cgroup/unified type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate) -cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,name=systemd) -pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime) -bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700) -cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids) -cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory) -cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event) -cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma) -cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio) -cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct) -cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer) -cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio) -cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices) -cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset) -debugfs on /sys/kernel/debug type debugfs (rw,relatime) -hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M) -mqueue on /dev/mqueue type mqueue (rw,relatime) -systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=45,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=10340) -tmpfs on /run/user/1000 type tmpfs (rw,nosuid,nodev,relatime,size=3266552k,mode=700,uid=1000,gid=1000) +sudo mdadm --manage /dev/md2 --fail /dev/nvme0n1p2 + +# mdadm: set /dev/nvme0n1p2 faulty in /dev/md2 +``` + +```sh +sudo mdadm --manage /dev/md3 --fail /dev/nvme0n1p3 + +# mdadm: set /dev/nvme0n1p3 faulty in /dev/md3 +``` + +We have now simulated a failure of the RAID, when we run the `cat /proc/mdstat` command, we have the following output + +```sh +root@rescue12-customer-ca (nsxxxxx.ip-xx-xx-xx.eu) ~ # cat /proc/mdstat +Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] [multipath] [faulty] +md3 : active raid1 nvme0n1p3[0](F) nvme1n1p3[1] + 497875968 blocks super 1.2 [2/1] [_U] + bitmap: 0/4 pages [0KB], 65536KB chunk + +md2 : active raid1 nvme0n1p2[2](F) nvme1n1p2[1] + 1046528 blocks super 1.2 [2/1] [_U] + +unused devices: +``` + +As we can see above, the [F] next to the partitions indicates that the disk has failed or is faulty. + +Next, we remove these partitions from the RAID arrays to completely remove the disk from RAID. + +```sh +sudo mdadm --manage /dev/md2 --remove /dev/nvme0n1p2 + +# mdadm: hot removed /dev/nvme0n1p2 from /dev/md2 +``` + +```sh +sudo mdadm --manage /dev/md3 --remove /dev/nvme0n1p3 + +# mdadm: hot removed /dev/nvme0n1p3 from /dev/md3 ``` -As we can see the, entry of `/dev/md4` is no longer mounted. However, the RAID is still active, so we need to simulate a failure to remove the disk. We can do this with the following command: +To make sure that we get a disk that is similar to an empty disk, we use the following command on each partition, then on the disk. ```sh -sudo mdadm --fail /dev/md4 /dev/sda4 +shred -s10M -n1 /dev/nvme0n1p1 +shred -s10M -n1 /dev/nvme0n1p2 +shred -s10M -n1 /dev/nvme0n1p3 +shred -s10M -n1 /dev/nvme0n1p4 +shred -s10M -n1 /dev/nvme0n1 ``` -We have now simulated a failure of the RAID. The next step is to remove the partition from the RAID array with the following command: +The disk now appears as a new one: ```sh -sudo mdadm --remove /dev/md4 /dev/sda4 +lsblk -A + +NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS +nvme1n1 259:0 0 476.9G 0 disk +├─nvme1n1p1 259:1 0 511M 0 part +├─nvme1n1p2 259:2 0 1G 0 part +│ └─md2 9:2 0 1022M 0 raid1 +├─nvme1n1p3 259:3 0 474.9G 0 part +│ └─md3 9:3 0 474.8G 0 raid1 +└─nvme1n1p4 259:4 0 512M 0 part +nvme0n1 259:5 0 476.9G 0 disk ``` -You can verify that the partition has been removed with the following command: +If we run the following command, we see that our disk has been successfully "wiped": ```sh -cat /proc/mdstat +parted /dev/nvme0n1 +GNU Parted 3.5 +Using /dev/nvme0n1 +Welcome to GNU Parted! Type 'help' to view a list of commands. +(parted) p +Error: /dev/nvme0n1: unrecognised disk label +Model: WDC CL SN720 SDAQNTW-512G-2000 (nvme) +Disk /dev/nvme0n1: 512GB +Sector size (logical/physical): 512B/512B +Partition Table: unknown +Disk Flags: +``` -Personalities : [raid1] [linear] [multipath] [raid0] [raid6] [raid5] [raid4] [raid10] -md2 : active raid1 sda2[1] sdb2[0] - 931954688 blocks super 1.2 [2/2] [UU] - bitmap: 4/7 pages [16KB], 65536KB chunk - -md4 : active raid1 sdb4[1] - 1020767232 blocks super 1.2 [2/1] [_U] - bitmap: 0/8 pages [0KB], 65536KB chunk - -unused devices: +Our RAID status should now look like this: + +```sh +root@rescue12-customer-ca (ns5009452.ip-51-222-254.net) ~ # cat /proc/mdstat +Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] [multipath] [faulty] +md3 : active raid1 nvme1n1p3[1] + 497875968 blocks super 1.2 [2/1] [_U] + bitmap: 0/4 pages [0KB], 65536KB chunk + +md2 : active raid1 nvme1n1p2[1] + 1046528 blocks super 1.2 [2/1] [_U] + +unused devices: n ``` -The following command will verify that the partition has been removed: +From the results above, we can see that only two partitions now appear in the RAID arrays. We have successfully failed the disk **nvme0n1** and we can now proceed with the hypothetical disk replacement. + +For more information on how to prepare and request for a disk replacement, consult this [guide](/pages/bare_metal_cloud/dedicated_servers/disk_replacement) + +If you run the following command, you can have more details on the RAID array (s): + ```sh -mdadm --detail /dev/md4 +mdadm --detail /dev/md3 -/dev/md4: +/dev/md3: Version : 1.2 - Creation Time : Tue Jan 24 15:35:02 2023 + Creation Time : Fri Aug 1 14:51:13 2025 Raid Level : raid1 - Array Size : 1020767232 (973.48 GiB 1045.27 GB) - Used Dev Size : 1020767232 (973.48 GiB 1045.27 GB) + Array Size : 497875968 (474.81 GiB 509.82 GB) + Used Dev Size : 497875968 (474.81 GiB 509.82 GB) Raid Devices : 2 Total Devices : 1 Persistence : Superblock is persistent Intent Bitmap : Internal - Update Time : Tue Jan 24 16:28:03 2023 + Update Time : Fri Aug 1 15:56:17 2025 State : clean, degraded Active Devices : 1 Working Devices : 1 @@ -371,101 +491,515 @@ mdadm --detail /dev/md4 Consistency Policy : bitmap - Name : md4 - UUID : 7b5c1d80:0a7ab4c2:e769b5e5:9c6eaa0f - Events : 21 + Name : md3 + UUID : b383c3d5:7fb1bb5e:6b7c4d96:6ea817ff + Events : 215 Number Major Minor RaidDevice State - 0 0 0 removed - 1 8 20 1 active sync /dev/sdb4 + 1 259 4 1 active sync /dev/nvme1n1p3 ``` ### Rebuilding the RAID -Once the disk has been replaced, we need to copy the partition table from a healthy disk (in this example, sdb) to the new one (sda) with the following command: +The following steps document the RAID rebuild in rescue mode. -**For GPT partitions**n n +> [!primary] +> This process might be different depending on the operating system you have installed on your server. We recommend that you consult the official documentation of your operating system to have access to the proper commands. +> + +> [!warning] +> +> For most servers in software RAID, after a disk replacement, the server is able to reboot in normal mode (on the healthy disk) and the rebuild can be done in normal mode. However, if the server is not able to reboot in normal mode after a disk replacement, it will be rebooted in rescue mode to proceed with the raid rebuild. +> +> If your server is able to boot in normal mode after the RAID rebuilding, simply proceed with the steps from [this section](). + +#### Rebuilding the RAID after the main disk is replaced + +Here, we assume that the EFI partitions have been kept in sync (including after GRUB/kernel updates. If that is not your case, we have provided additional information on how to proceed). + +Once the disk has been replaced, we need to copy the partition table from the healthy disk (in this example, nvme1n1) to the new one (nvme0n1). + +> [!tabs] +> **For GPT partitions** +>> +>> ```sh +>> sgdisk -R /dev/nvmeXnX /dev/nvmeAnA +>> ``` +>> +>> The command should be in this format: `sgdisk -R /dev/newdisk /dev/healthydisk` +>> +>> In our example: +>> +>> ```sh +>> sgdisk -R /dev/nvme0n1 /dev/nvme1n1 +>> ``` +>> +>> We run `lsblk` to make sure the partition tables have been properly copied: +>> +>> ```sh +>> lsblk +>> +>> NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS +>> nvme1n1 259:0 0 476.9G 0 disk +>> ├─nvme1n1p1 259:1 0 511M 0 part +>> ├─nvme1n1p2 259:2 0 1G 0 part +>> │ └─md2 9:2 0 1022M 0 raid1 +>> ├─nvme1n1p3 259:3 0 474.9G 0 part +>> │ └─md3 9:3 0 474.8G 0 raid1 +>> └─nvme1n1p4 259:4 0 512M 0 part +>> nvme0n1 259:5 0 476.9G 0 disk +>> ├─nvme0n1p1 259:10 0 511M 0 part +>> ├─nvme0n1p2 259:11 0 1G 0 part +>> ├─nvme0n1p3 259:12 0 474.9G 0 part +>> └─nvme0n1p4 259:13 0 512M 0 part +>> ``` +>> +>> Once this is done, the next step is to randomize the GUID of the new disk to prevent GUID conflicts with other disks: +>> +>> ```sh +>> sgdisk -G /dev/nvme0n1 +>> ``` +>> +>> If you receive a message like this: +>> +>> ```console +>> Warning: The kernel is still using the old partition table. +>> The new table will be used at the next reboot or after you +>> run partprobe(8) or kpartx(8) +>> The operation has completed successfully. +>> ``` +>> +>> You can simply run the `partprobe` command or reboot the server when the raid rebuilt is done. +>> +> **For MBR partitions** +>> +>> ```sh +>> sudo sfdisk -d /dev/sdX | sfdisk /dev/sdX +>> ``` +>> +>> The command should be in this format: `sfdisk -d /dev/healthydisk | sfdisk /dev/newdisk`. +>> + +We can now rebuild the RAID array. The following code snippet shows how we can add the new partitions (nvme0n1p2 and nvme0n1p3) back in the RAID array. You need to replace the appropriate values with your own. ```sh -sgdisk -R /dev/sda /dev/sdb nmnn +sudo mdadm --add /dev/md2 /dev/nvme0n1p2 + +# mdadm: added /dev/nvme0n1p2 + +sudo mdadm --add /dev/md3 /dev/nvme0n1p3 + +# mdadm: re-added /dev/nvme0n1p3 ``` -The command should be in this format: `sgdisk -R /dev/newdisk /dev/healthydisk` +You can check the rebuild process + +```sh +cat /proc/mdstat +Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] [multipath] [faulty] +md3 : active raid1 nvme0n1p3[2] nvme1n1p3[1] + 497875968 blocks super 1.2 [2/1] [_U] + [>....................] recovery = 0.1% (801920/497875968) finish=41.3min speed=200480K/sec + bitmap: 0/4 pages [0KB], 65536KB chunk + +md2 : active raid1 nvme0n1p2[2] nvme1n1p2[1] + 1046528 blocks super 1.2 [2/2] [UU] +``` -Once this is done, the next step is to randomize the GUID of the new disk to prevent GUID conflicts with other disks: +Once the RAID rebuild is complete, run the following command to make sure that the partitions were properly added to the RAID: ```sh -sgdisk -G /dev/sda +lsblk -fA +NAME FSTYPE FSVER LABEL UUID FSAVAIL FSUSE% MOUNTPOINTS +nvme1n1 +├─nvme1n1p1 vfat FAT16 EFI_SYSPART 4629-D183 +├─nvme1n1p2 linux_raid_member 1.2 md2 83719c5c-2a27-2a56-5268-7d49d8a1d84f +│ └─md2 ext4 1.0 boot 4de80ae0-dd90-4256-9135-1735e7be4b4d +├─nvme1n1p3 linux_raid_member 1.2 md3 b383c3d5-7fb1-bb5e-6b7c-4d966ea817ff +│ └─md3 ext4 1.0 root 9bf386b6-9523-46bf-b8e5-4b8cc7c5786f +└─nvme1n1p4 swap 1 swap-nvme1n1p4 9bf292e8-0145-4d2f-b891-4cef93c0d209 +nvme0n1 +├─nvme0n1p1 +├─nvme0n1p2 linux_raid_member 1.2 md2 83719c5c-2a27-2a56-5268-7d49d8a1d84f +│ └─md2 ext4 1.0 boot 4de80ae0-dd90-4256-9135-1735e7be4b4d +├─nvme0n1p3 linux_raid_member 1.2 md3 b383c3d5-7fb1-bb5e-6b7c-4d966ea817ff +│ └─md3 ext4 1.0 root 9bf386b6-9523-46bf-b8e5-4b8cc7c5786f +└─nvme0n1p4 ``` -**For MBR partitions** +From the above results, we can see that the partitions of our newly added disk have been properly added to the RAID, however, the EFI partition was not duplicated, which is normal since it is not included in the RAID. We also see that the [SWAP] partition **nvme0n1p4** does not have a label anymore (this will not be the same for all cases, especially if you customise your partitions before your server's installation). + +> [!warning] +> The examples above are merely illustrating the necessary steps based on a default server configuration. The information in the output table depends on your server's hardware and its partition scheme. When in doubt, consult the documentation of your operating system. +> +> If you require professional assistance with server administration, consider the details in the Go further section of this guide. +> + +#### Recreating the EFI partition + +The next step is to format **nvme0n1p1** to create the EFI partition, then replicate the content of this partition from the healthy drive (in our example: nvme1n1) to the new one. + +This is possible because we kept the partitions in sync while the server was running normally before the disk failure. So we have both partitions up to date. + +> [!warning] +> If there was a kernel or grub update or a major update and both partitions were not synchronized, consult the following guide [Repairing the GRUB bootloader]() once you are done creating the new EFI partition. +> -Once the disk has been replaced, we need to copy the partition table from a healthy disk (in this example, sdb) to the new one (sda) with the following command: +First, we format the partition: ```sh -sfdisk -d /dev/sdb | sfdisk /dev/sda +mkfs.vfat /dev/nvme0n1p1 ``` -The command should be in this format: `sfdisk -d /dev/healthydisk | sfdisk /dev/newdisk` +Next, we label the partition as `EFI_SYSPART` (this naming is proper to OVHcloud) -We can now rebuild the RAID array. The following code snippet shows how we can rebulid the `/dev/md4` partition layout with the recently-copied sda partition table: +```sh +fatlabel /dev/nvme0n1p1 EFI_SYSPART +``` + +Now we have to duplicate the content from the partition nvme1n1p1 (healthy partition), to nvme0n1p1 (new partition). To do so, we start by creating two folders named `old` and `new`. ```sh -mdadm --add /dev/md4 /dev/sda4 -cat /proc/mdstat +mkdir old new +``` -Personalities : [raid1] [linear] [multipath] [raid0] [raid6] [raid5] [raid4] [raid10] -md2 : active raid1 sda2[1] sdb2[0] - 931954688 blocks super 1.2 [2/2] [UU] - bitmap: 1/7 pages [4KB], 65536KB chunk +Next, we mount **nvme1n1p1** in the `old` folder and **nvme0n1p1** in the `new` folder to make the distinction: -md4 : active raid1 sda4[0] sdb4[1] - 1020767232 blocks super 1.2 [2/2] [UU] - bitmap: 0/8 pages [0KB], 65536KB chunk +```sh +mount /dev/nvme1n1p1 old +mount /dev/nvme0n1p1 new +``` -unused devices: +Next, we copy the files from `old` to `new`. Depending on your operating system, you will have a similar output. Here we are using debian: + +```sh +rsync -axv old/ new/ ``` -We can verify the RAID details with the following command: +Once this is done, we unmount the **nvme0n1p1** partition. ```sh -mdadm --detail /dev/md4 +umount /dev/nvme0n1p1 +``` -/dev/md4: - Version : 1.2 - Creation Time : Tue Jan 24 15:35:02 2023 - Raid Level : raid1 - Array Size : 1020767232 (973.48 GiB 1045.27 GB) - Used Dev Size : 1020767232 (973.48 GiB 1045.27 GB) - Raid Devices : 2 - Total Devices : 2 - Persistence : Superblock is persistent +Next, we mount the partition containing our files on `/mnt`. In our example, that partition is `md3`. - Intent Bitmap : Internal +```sh +mount /dev/md3 /mnt +``` - Update Time : Tue Jan 24 17:02:55 2023 - State : clean - Active Devices : 2 - Working Devices : 2 - Failed Devices : 0 - Spare Devices : 0 +Next, we mount the following directories to make sure any manipulation we make in the chroot environment works properly: - Rebuild Status : 21% complete +```sh +mount --types proc /proc /mnt/proc +mount --rbind /sys /mnt/sys +mount --make-rslave /mnt/sys +mount --rbind /dev /mnt/dev +mount --make-rslave /mnt/dev +mount --bind /run /mnt/run +mount --make-slave /mnt/run +``` - UUID : 7f39d062:9f16a016:a4d2adc2:26fd5302 - Events : 0.95 +Next, we use the `chroot` command to access the mountpoint and make sure the new EFI partition has been properly created and the system recongnises both EFI partitions: - Number Major Minor RaidDevice State - 0 8 2 0 spare rebuilding /dev/sda4 - 1 8 18 1 active sync /dev/sdb4 +```sh +chroot /mnt +``` + +```sh +blkid -t LABEL=EFI_SYSPART +/dev/nvme1n1p1: SEC_TYPE="msdos" LABEL_FATBOOT="EFI_SYSPART" LABEL="EFI_SYSPART" UUID="4629-D183" BLOCK_SIZE="512" TYPE="vfat" PARTLABEL="primary" PARTUUID="889f241b-49c3-4031-b5c9-60df0746f98f" +/dev/nvme0n1p1: SEC_TYPE="msdos" LABEL_FATBOOT="EFI_SYSPART" LABEL="EFI_SYSPART" UUID="521F-300B" BLOCK_SIZE="512" TYPE="vfat" PARTLABEL="primary" PARTUUID="02bf2b2d-7ada-4461-ba50-07683519f65d" +``` + +/// details | **GRUB/kernel update with EFI partitions not synchronized** + +If GRUB/kernal updates were made and the EFI partitions were not kept in sync, this additional step could necessary. + +Still in the `chroot` environment, we create the `/boot/efi` folder in order to mount the EFI partition **nvme0n1p1** in it: + +```sh +cd /mnt +mkdir /boot/efi +mount /dev/nvme0n1p1 /boot/efi +``` + +Next, we install the grub bootloader to make sure the server can reboot in normal mode on the new disk (you won't have to do this if the disk replaced is the secondary disk. Simply duplicate the EFI partition and proceed to the RAID rebuild, then enable the [SWAP] partition (if applicable)): + +```sh +grub-install --efi-directory=/boot/efi /dev/nvme0n1p1 +``` +/// + + + +#### Adding the label to the SWAP partition + +We exit the `chroot` environment, then we recreate our [SWAP] partition **nvme0n1p4** and add the label `swap-nvmenxxx`: + +```sh +mkswap /dev/nvme0n1p4 -L swap-nvme0n1p4 +mkswap: /dev/nvme0n1p4: warning: wiping old swap signature. +Setting up swapspace version 1, size = 512 MiB (536866816 bytes) +LABEL=swap-nvme0n1p4, UUID=b3c9e03a-52f5-4683-81b6-cc10091fcd +``` + +We verify that the label has been properly applied: + +```sh +lsblk -fA +NAME FSTYPE FSVER LABEL UUID FSAVAIL FSUSE% MOUNTPOINTS +nvme1n1 + +├─nvme1n1p1 +│ vfat FAT16 EFI_SYSPART +│ BA77-E844 504.9M 1% /root/old +├─nvme1n1p2 +│ linux_ 1.2 md2 53409058-480a-bc65-4e1d-6acc848fe233 +│ └─md2 +│ ext4 1.0 boot f925a033-0087-40ec-817e-44efab0351ac +├─nvme1n1p3 +│ linux_ 1.2 md3 a3b8816c-a5c3-7f01-ee17-e1aa9685c35c +│ └─md3 +│ ext4 1.0 root 6abfaa3b-e630-457a-bbe0-e00e5b4b59e5 441.2G 0% /mnt +└─nvme1n1p4 + swap 1 swap-nvme1n1p4 + d6af33cf-fc15-4060-a43c-cb3b5537f58a +nvme0n1 + +├─nvme0n1p1 +│ vfat FAT16 EFI_SYSPART +│ 477D-6658 +├─nvme0n1p2 +│ linux_ 1.2 md2 53409058-480a-bc65-4e1d-6acc848fe233 +│ └─md2 +│ ext4 1.0 boot f925a033-0087-40ec-817e-44efab0351ac +├─nvme0n1p3 +│ linux_ 1.2 md3 a3b8816c-a5c3-7f01-ee17-e1aa9685c35c +│ └─md3 +│ ext4 1.0 root 6abfaa3b-e630-457a-bbe0-e00e5b4b59e5 441.2G 0% /mnt +└─nvme0n1p4 + swap 1 swap-nvme0n1p4 + b3c9e03a-52f5-4683-81b6-cc10091fcd15 +``` + +Next, we access the `chroot` environment again: + +```sh +chroot /mnt +``` + +We retrieve the UUIDs of both swap partitions: + +```sh +blkid /dev/nvme0n1p4 +blkid /dev/nvme1n1p4 ``` -The RAID has now been rebuilt, but we still need to mount the partition (`/dev/md4` in this example) with the following command: +Example: ```sh -mount /dev/md4 /home +blkid /dev/nvme0n1p4 +/dev/nvme0n1p4: LABEL="swap-nvme0n1p4" UUID="b3c9e03a-52f5-4683-81b6-cc10091fcd15" TYPE="swap" PARTLABEL="logical" PARTUUID="cb6022d9-7631-449c-956a-116bcd0e2119" ``` +```sh +blkid /dev/nvme1n1p4 +/dev/nvme1n1p4: LABEL="swap-nvme1n1p4" UUID="d6af33cf-fc15-4060-a43c-cb3b5537f58a" TYPE="swap" PARTLABEL="logical" PARTUUID="d037c35f-2ddb-40d5-be33-31cc496fe54b" +``` + +Next, we replace the old UUID of the swap partition (**nvme0n1p4)** with the new one in `/etc/fstab`: + +```sh +nano etc/fstab +``` + +Example: + +```sh +UUID=6abfaa3b-e630-457a-bbe0-e00e5b4b59e5 / ext4 defaults 0 1 +UUID=f925a033-0087-40ec-817e-44efab0351ac /boot ext4 defaults 0 0 +LABEL=EFI_SYSPART /boot/efi vfat defaults 0 1 +UUID=b7b5dd38-9b51-4282-8f2d-26c65e8d58ec swap swap defaults 0 0 +UUID=d6af33cf-fc15-4060-a43c-cb3b5537f58a swap swap defaults 0 0 +``` + +Make sure you replace the proper UUID. In our example above, the UUID to replace is `b7b5dd38-9b51-4282-8f2d-26c65e8d58ec` with the new one `b3c9e03a-52f5-4683-81b6-cc10091fcd15` since the other one belongs to **nvme1n1p4**. + +Next, we make sure everything is properly mounted: + +```sh +mount -av +/ : ignored +mount: (hint) your fstab has been modified, but systemd still uses + the old version; use 'systemctl daemon-reload' to reload. +/boot : successfully mounted +/boot/efi : successfully mounted +swap : ignored +swap : ignored +``` + +We enable the swap partition: + +```sh +swapon -av + +swapon: /dev/nvme0n1p4: found signature [pagesize=4096, signature=swap] +swapon: /dev/nvme0n1p4: pagesize=4096, swapsize=536870912, devsize=536870912 +swapon /dev/nvme0n1p4 +swapon: /dev/nvme1n1p4: found signature [pagesize=4096, signature=swap] +swapon: /dev/nvme1n1p4: pagesize=4096, swapsize=536870912, devsize=536870912 +swapon /dev/nvme1n1p4 +``` + +We exit the Chroot environment with `Exit` and unmount all the disks: + +```sh +umount -a +``` + +We have now successfully completed the RAID rebuild on the server and we can now reboot the server in normal mode. + +/// details | **Rebuilding the RAID after the secondary disk is replaced** + +The following are to be followed in normal mode since it is the secondary disk that is being replaced and the server will be booted in normal mode. Our secondary disk is **nvme1n1**. + +Once the disk has been replaced, we need to copy the partition table from the healthy disk (in this example, nvme0n1) to the new one (nvme1n1). + +> [!tabs] +> **For GPT partitions** +>> +>> ```sh +>> sgdisk -R /dev/nvmeXnX /dev/nvmeAnA +>> ``` +>> +>> The command should be in this format: `sgdisk -R /dev/newdisk /dev/healthydisk`. +>> +>> Once this is done, the next step is to randomize the GUID of the new disk to prevent GUID conflicts with other disks: +>> +>> ```sh +>> sgdisk -G /dev/nvmeXnX +>> ``` +>> +>> If you receive a message like this: +>> +>> ```console +>> Warning: The kernel is still using the old partition table. +>> The new table will be used at the next reboot or after you +>> run partprobe(8) or kpartx(8) +>> The operation has completed successfully. +>> ``` +>> +>> You can simply run the `partprobe` command or reboot the server when the raid rebuilt is done. +>> +> **For MBR partitions** +>> +>> ```sh +>> sudo sfdisk -d /dev/sdX | sfdisk /dev/sdX +>> ``` +>> +>> The command should be in this format: `sfdisk -d /dev/healthydisk | sfdisk /dev/newdisk`. +>> + +Next, we add the partitions to the RAID: + +```sh +sudo mdadm --add /dev/md2 /dev/nvme1n1p2 + +# mdadm: added /dev/nvme1n1p2 + +sudo mdadm --add /dev/md3 /dev/nvme1n1p3 + +# mdadm: re-added /dev/nvme1n1p3 +``` + +Use the following command to follow the RAID rebuild `cat /proc/mdstat`. + +**Recreating the EFI partition on the disk** + +First, we install the necessary tools: + +**Debian and Ubuntu** + +```sh +sudo apt install dosfstools +``` + +**CentOS** + +```sh +sudo yum install dosfstools +``` + +Next, we format the partition. In our example nvme1n1p1: + +```sh +sudo mkfs.vfat /dev/nvme1n1p1 +``` + +Next, we label the partition as `EFI_SYSPART` (this naming is proper to OVHcloud) + +```sh +sudo fatlabel /dev/nvme1n1p1 EFI_SYSPART +``` + +We sync both partitions using a script. + +Example: + +```sh +set -euo pipefail + +findmnt -n -o SOURCE /boot/efi +MAIN_PARTITION=/dev/nvme0n1p1 +MOUNTPOINT="/var/lib/grub/esp" + +mkdir -p "${MOUNTPOINT}" + +while read -r partition; do + if [[ "${partition}" == "${MAIN_PARTITION}" ]]; then + continue + fi + echo "Working on ${partition}" + mount "${partition}" "${MOUNTPOINT}" + rsync -ax "/boot/efi/" "${MOUNTPOINT}/" + umount "${MOUNTPOINT}" +done < <(blkid -o device -t LABEL=EFI_SYSPART) +``` + +Lastly, we enable the [SWAP] partition (if applicable): + +```sh +sudo mkswap /dev/nvme1n1p4 -L swap-nvme1n1p4 +``` + +We retrieve the UUIDs of both swap partitions: + +```sh +sudo blkid /dev/nvme0n1p4 +sudo blkid /dev/nvme1n1p4 +``` + +We replace the old UUID of the swap partition (**nvme1n1p4)** with the new one in `/etc/fstab`: + +```sh +sudo nano /etc/fstab +``` +Make sure you replace the correct UUID. Reload the system with the following command: + +```sh +sudo systemctl daemon-reload +``` + +Next, run the following command to enable it: + +```sh +sudo swapon -av +``` + +We have now successfully completed the RAID rebuild. + ## Go Further [Hot Swap - Software RAID](/pages/bare_metal_cloud/dedicated_servers/hotswap_raid_soft) @@ -476,4 +1010,4 @@ mount /dev/md4 /home [Hot Swap - Hardware RAID](/pages/bare_metal_cloud/dedicated_servers/hotswap_raid_hard) -Join our [community of users](/links/community). +Join our [community of users](/links/community). \ No newline at end of file From 757c1294ff110b4d1b9d2f40141c455649de5082 Mon Sep 17 00:00:00 2001 From: Montrealhub <89825661+Jessica41@users.noreply.github.com> Date: Mon, 18 Aug 2025 08:45:50 -0400 Subject: [PATCH 6/8] Update guide.en-gb.md --- .../raid_soft_bios/guide.en-gb.md | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/pages/bare_metal_cloud/dedicated_servers/raid_soft_bios/guide.en-gb.md b/pages/bare_metal_cloud/dedicated_servers/raid_soft_bios/guide.en-gb.md index fdfdd7514c2..fefdeaa4172 100644 --- a/pages/bare_metal_cloud/dedicated_servers/raid_soft_bios/guide.en-gb.md +++ b/pages/bare_metal_cloud/dedicated_servers/raid_soft_bios/guide.en-gb.md @@ -427,7 +427,7 @@ mount --bind /run /mnt/run mount --make-slave /mnt/run ``` -Next, we access the `chroot` environment again: +Next, we access the `chroot` environment: ```sh chroot /mnt @@ -452,7 +452,7 @@ blkid /dev/sdb4 /dev/sdb4: LABEL="swap-sdb4" UUID="d6af33cf-fc15-4060-a43c-cb3b5537f58a" TYPE="swap" PARTLABEL="logical" PARTUUID="d037c35f-2ddb-40d5-be33-31cc496fe54b" ``` -Next, we replace the old UUID of the swap partition (**nvme0n1p4)** with the new one in `/etc/fstab`: +Next, we replace the old UUID of the swap partition (**sda4**) with the new one in `/etc/fstab`: ```sh nano etc/fstab @@ -495,7 +495,7 @@ swapon: /dev/sdb4: pagesize=4096, swapsize=536870912, devsize=536870912 swapon /dev/sdb4 ``` -We exit the Chroot environment with `Exit` and unmount all the disks: +We exit the Chroot environment with `exit` and unmount all the disks: ```sh umount -a @@ -505,7 +505,7 @@ We have now successfully completed the RAID rebuild on the server and we can now /// details | **Rebuilding the RAID after the secondary disk is replaced** -The following steps are done in normal mode since it is the secondary disk that is was replaced. In our example, our secondary disk is named **sdb**. +The following steps are performed in normal mode, as it is the secondary disk that has been replaced. In our example, our secondary disk is named **sdb**. Once the disk has been replaced, we need to copy the partition table from the healthy disk (in this example, sda) to the new one (sdb). @@ -556,7 +556,7 @@ sudo mdadm --add /dev/md4 /dev/sdb4 # mdadm: re-added /dev/sdb4 ``` -Use the following command to follow the RAID rebuild `cat /proc/mdstat`. +Use the following command to monitor the RAID rebuild: `cat /proc/mdstat`. Lastly, we add a label and mount the [SWAP] partition (if applicable): @@ -571,7 +571,7 @@ sudo blkid /dev/sda4 sudo blkid /dev/sdb4 ``` -We replace the old UUID of the swap partition (**sdb4)** with the new one in `/etc/fstab`: +We replace the old UUID of the swap partition (**sdb4**) with the new one in `/etc/fstab`: ```sh sudo nano etc/fstab @@ -583,7 +583,7 @@ Make sure you replace the correct UUID. Reload the system with the following com sudo systemctl daemon-reload ``` -Next, run the following command to enable it: +Next, run the following command to enable the swap partition: ```sh sudo swapon -av @@ -601,4 +601,4 @@ We have now successfully completed the RAID rebuild. [Hot Swap - Hardware RAID](/pages/bare_metal_cloud/dedicated_servers/hotswap_raid_hard) -Join our [community of users](/links/community). \ No newline at end of file +Join our [community of users](/links/community). From 14f48cb69ae8e99145b9a3d9e4661cb6943f9fd7 Mon Sep 17 00:00:00 2001 From: Montrealhub <89825661+Jessica41@users.noreply.github.com> Date: Mon, 18 Aug 2025 11:35:10 -0400 Subject: [PATCH 7/8] Update guide.en-gb.md --- .../raid_soft_uefi/guide.en-gb.md | 63 +++++++++---------- 1 file changed, 31 insertions(+), 32 deletions(-) diff --git a/pages/bare_metal_cloud/dedicated_servers/raid_soft_uefi/guide.en-gb.md b/pages/bare_metal_cloud/dedicated_servers/raid_soft_uefi/guide.en-gb.md index 7d55cfeee5f..d52304a778c 100644 --- a/pages/bare_metal_cloud/dedicated_servers/raid_soft_uefi/guide.en-gb.md +++ b/pages/bare_metal_cloud/dedicated_servers/raid_soft_uefi/guide.en-gb.md @@ -12,7 +12,7 @@ The default RAID level for OVHcloud server installations is RAID 1, which double **This guide explains how to manage and rebuild software RAID after a disk replacement on your server in UEFI mode** -Before we begin, please note that this guide focuses on Dedicated servers that use UEFI as the boot mode. This is the case with modern motherboards. If your server uses the BIOS mode (old motherboards), refer to this guide [How to configure and rebuild software RAID on a Dedicated server with BIOS boot mode](). +Before we begin, please note that this guide focuses on Dedicated servers that use UEFI as the boot mode. This is the case with modern motherboards. If your server uses the BIOS mode (old motherboards), refer to this guide [How to configure and rebuild software RAID on a Dedicated server with BIOS boot mode](Managing and rebuilding software RAID on servers in BIOS boot mode). For more information on UEFI, consult the following [guide](https://uefi.org/about){.external} @@ -24,7 +24,7 @@ For more information on UEFI, consult the following [guide](https://uefi.org/abo ## Instructions -When you purchase a new server, you might feel the need to perform a series of tests and actions. One of those actions could be simulating a disk failure in order to understand the process of rebuilding the raid and prepare yourself in case this happens. +When you purchase a new server, you may feel the need to perform a series of tests and actions. One of these actions could be to simulate a disk failure in order to understand the RAID rebuild process and prepare yourself in case it ever happens. ### Basic Information @@ -200,9 +200,9 @@ An EFI partition, is a partition which can contain the boot loaders, boot manage ***Is the EFI partition mirrored in RAID?*** -No, by default, the EFI partition is a partition which cannot be included in the RAID. When you use one of our default installation templates to install your server in software RAID, two EFI partitions are created, one on each disk. However, only one EFI partition is used for the installation of the operating system. +No, by default, the EFI partition is a partition that cannot be included in the RAID. When you use one of our default installation templates to install your server in a software RAID, two EFI partitions are created, one on each disk. However, only one EFI partition is mounted and used for the operating system installation. -The partition is mounted on `/boot/efi` and the disk on which it is mounted is selected randomly by the operating system during installation. +The partition is mounted on the mount point: `/boot/efi` and the disk on which it is mounted is selected randomly by the operating system during installation. Example: @@ -229,7 +229,7 @@ nvme0n1 259:1 0 476.9G 0 disk From the example above, we see that we have two identical EFI partitions (nvme0n1p1 and nvme1n1p1) but only **nvme0n1p1** is mounted on `/boot/efi`. -We can also use the following command to confirm that two EFI partitions were actually created after install: +We can also use the following command to confirm that two EFI partitions have been created after installation: ```sh sudo lsblk -f @@ -251,25 +251,25 @@ nvme0n1 └─nvme0n1p5 iso9660 Joliet Extension config-2 2025-08-05-14-55-41-00 ``` -From the results above, we can see two partitions (**nvme0n1p1** and **nvme1n1p1**), with identical size (504.9M). Both partitions have the LABEL: `EFI_SYSPART` but only one is mounted on `/boot/efi`. +From the results above, we can see two partitions (**nvme0n1p1** and **nvme1n1p1**), with identical size (504.9M). Both partitions have the LABEL: `EFI_SYSPART` but only one is mounted. ***Does the content of EFI partition change regularly?*** In general, the contents of this partition do not change much, except when there are relevant updates such as grub/kernel updates or even simple `apt` or `yum` updates. -In this case, we recommend running an automatic or manual script keep the partitions in sync. This is because in case of the failure of the main disk on which this partition in mounted, you can use the rescue mode to recover it. +In this case, we recommend running an automatic or manual script to synchronise the partitions. This is because if the main disk on which this partition is mounted fails, you can use rescue mode to recover it by recreating a new EFI partition on the new disk and copying the contents of the second partition to it. ***What happens if my main disk fails and is replaced?*** -If the main disk on your server fails and it is replaced. The server will be rebooted in rescue mode in other for you to rebuild the raid. Since the EFI partition is not mirrored, the server will not be able to boot on the secondary disk (in most cases). You can rebuild the raid in rescue mode, recreate the EFI partition and the disk should be able to boot. +If your server's main disk fails and is replaced, the server will be restarted in rescue mode so that you can rebuild the RAID. Since the EFI partition is not mirrored, the server will not be able to boot from the secondary disk (in most cases). You can rebuild the RAID in rescue mode, recreate the EFI partition, and the drive should be able to boot. ***What if the main disk fails and I did not synchronise my EFI partitions after a major update (kernel/grub)?*** -If this is the case, you will have to rebuild the raid, create the EFI partition on the new disk and install the grub/kernel update in rescue mode. If you are not able to perform some manipulations, we recommend contacting a professional service. +If this is the case, you will have to rebuild the RAID in rescue mode, create an EFI partition on the new disk and install the grub/kernel update in rescue mode. If you are not able to perform some manipulations, we recommend contacting a professional service. ***How can I keep my EFI partitions synchronized or how often should I sychronize them?*** -We recommend synchronizing both partitions daily. Below is a script you can use to manually synchronize your EFI partitions. You can also run an automated script to synchronize the partitions daily or when the service is booted. +We recommend synchronizing both partitions daily or whenever there are major updates on your system. Below is a script you can use to manually synchronize your EFI partitions. You can also run an automated script to synchronize the partitions daily or when the service is booted. ```sh set -euo pipefail @@ -337,7 +337,7 @@ Now that we have all the necessary information, we can simulate a disk failure a The preferred way to do this is via the OVHcloud rescue mode environment. -First reboot in rescue menu and log in with the credentials provided. +First reboot the server in rescue menu and log in with the credentials provided. To remove a disk from the RAID, the first step is to mark it as **Failed** and remove the partitions from their respective RAID arrays. @@ -403,7 +403,7 @@ sudo mdadm --manage /dev/md3 --remove /dev/nvme0n1p3 # mdadm: hot removed /dev/nvme0n1p3 from /dev/md3 ``` -To make sure that we get a disk that is similar to an empty disk, we use the following command on each partition, then on the disk. +To make sure that we get a disk that is similar to an empty disk, we use the following command on each partition, then on the disk: ```sh shred -s10M -n1 /dev/nvme0n1p1 @@ -457,16 +457,15 @@ md3 : active raid1 nvme1n1p3[1] md2 : active raid1 nvme1n1p2[1] 1046528 blocks super 1.2 [2/1] [_U] -unused devices: n +unused devices: ``` -From the results above, we can see that only two partitions now appear in the RAID arrays. We have successfully failed the disk **nvme0n1** and we can now proceed with the hypothetical disk replacement. +Based on the above results, we can see that only two partitions now appear in the RAID arrays. We have successfully failed the **nvme0n1** drive and can now proceed with replacing the drive. -For more information on how to prepare and request for a disk replacement, consult this [guide](/pages/bare_metal_cloud/dedicated_servers/disk_replacement) +For more information on how to prepare and request for a disk replacement, consult this [guide](/pages/bare_metal_cloud/dedicated_servers/disk_replacement). If you run the following command, you can have more details on the RAID array (s): - ```sh mdadm --detail /dev/md3 @@ -593,7 +592,7 @@ sudo mdadm --add /dev/md3 /dev/nvme0n1p3 # mdadm: re-added /dev/nvme0n1p3 ``` -You can check the rebuild process +To check the rebuild process: ```sh cat /proc/mdstat @@ -607,7 +606,7 @@ md2 : active raid1 nvme0n1p2[2] nvme1n1p2[1] 1046528 blocks super 1.2 [2/2] [UU] ``` -Once the RAID rebuild is complete, run the following command to make sure that the partitions were properly added to the RAID: +Once the RAID rebuild is complete, run the following command to make sure that the partitions have been properly added to the RAID: ```sh lsblk -fA @@ -628,22 +627,22 @@ nvme0n1 └─nvme0n1p4 ``` -From the above results, we can see that the partitions of our newly added disk have been properly added to the RAID, however, the EFI partition was not duplicated, which is normal since it is not included in the RAID. We also see that the [SWAP] partition **nvme0n1p4** does not have a label anymore (this will not be the same for all cases, especially if you customise your partitions before your server's installation). +Based on the above results, we can see that the partitions on our newly added drive have been successfully added to the RAID. However, the EFI partition has not been duplicated, which is normal since it is not included in the RAID. We also note that the [SWAP] partition **nvme0n1p4** no longer has a label (this will not be the case in all situations, especially if you customise your partitions before installing your server). > [!warning] > The examples above are merely illustrating the necessary steps based on a default server configuration. The information in the output table depends on your server's hardware and its partition scheme. When in doubt, consult the documentation of your operating system. > -> If you require professional assistance with server administration, consider the details in the Go further section of this guide. +> If you require professional assistance with server administration, consider the details in the [Go further]() section of this guide. > #### Recreating the EFI partition -The next step is to format **nvme0n1p1** to create the EFI partition, then replicate the content of this partition from the healthy drive (in our example: nvme1n1) to the new one. +The next step is to format **nvme0n1p1** to recreate the EFI partition, then copy the content of the secondary EFI partition (in our example: nvme1n1) to this one. -This is possible because we kept the partitions in sync while the server was running normally before the disk failure. So we have both partitions up to date. +We have an updated version of the partition, as we assume that both partitions have been synchronised. If the two partitions have not been synchronised after major updates (particularly GRUB), you are copying an older version of the partition to the new one. In this case, it is imperative to install these updates before restarting the server in normal mode. > [!warning] -> If there was a kernel or grub update or a major update and both partitions were not synchronized, consult the following guide [Repairing the GRUB bootloader]() once you are done creating the new EFI partition. +> If there was a kernel or grub update or a major update and both partitions were not synchronized, consult the following guide [Repairing the GRUB bootloader](/pages/public_cloud/compute/repairing_the_grub_bootloader) once you are done creating the new EFI partition. > First, we format the partition: @@ -658,7 +657,7 @@ Next, we label the partition as `EFI_SYSPART` (this naming is proper to OVHcloud fatlabel /dev/nvme0n1p1 EFI_SYSPART ``` -Now we have to duplicate the content from the partition nvme1n1p1 (healthy partition), to nvme0n1p1 (new partition). To do so, we start by creating two folders named `old` and `new`. +We now need to duplicate the contents of the nvme1n1p1 partition (the partition in good condition) to nvme0n1p1 (the new partition). To do this, we start by creating two folders named ‘old’ and ‘new’. ```sh mkdir old new @@ -671,7 +670,7 @@ mount /dev/nvme1n1p1 old mount /dev/nvme0n1p1 new ``` -Next, we copy the files from `old` to `new`. Depending on your operating system, you will have a similar output. Here we are using debian: +Next, we copy the files from the `old` folder to the `new` folder. Depending on your operating system, you will have a similar output. Here we are using debian: ```sh rsync -axv old/ new/ @@ -689,7 +688,7 @@ Next, we mount the partition containing our files on `/mnt`. In our example, tha mount /dev/md3 /mnt ``` -Next, we mount the following directories to make sure any manipulation we make in the chroot environment works properly: +We mount the following directories to make sure any manipulation we make in the chroot environment works properly: ```sh mount --types proc /proc /mnt/proc @@ -701,7 +700,7 @@ mount --bind /run /mnt/run mount --make-slave /mnt/run ``` -Next, we use the `chroot` command to access the mountpoint and make sure the new EFI partition has been properly created and the system recongnises both EFI partitions: +Next, we use the `chroot` command to access the mount point and ensure that the new EFI partition has been created correctly and that the system recognises both EFI partitions: ```sh chroot /mnt @@ -715,9 +714,9 @@ blkid -t LABEL=EFI_SYSPART /// details | **GRUB/kernel update with EFI partitions not synchronized** -If GRUB/kernal updates were made and the EFI partitions were not kept in sync, this additional step could necessary. +If GRUB/kernel updates have been performed and the EFI partitions were not synchronised, this additional step may be necessary. -Still in the `chroot` environment, we create the `/boot/efi` folder in order to mount the EFI partition **nvme0n1p1** in it: +Still in the `chroot` environment, we create the `/boot/efi` folder in order to mount the EFI partition **nvme0n1p1**: ```sh cd /mnt @@ -863,7 +862,7 @@ We have now successfully completed the RAID rebuild on the server and we can now /// details | **Rebuilding the RAID after the secondary disk is replaced** -The following are to be followed in normal mode since it is the secondary disk that is being replaced and the server will be booted in normal mode. Our secondary disk is **nvme1n1**. +The following steps must be followed in normal mode, as it is the secondary disk that is being replaced and the server will be started in normal mode. Our secondary disk is called **nvme1n1**. Once the disk has been replaced, we need to copy the partition table from the healthy disk (in this example, nvme0n1) to the new one (nvme1n1). @@ -914,7 +913,7 @@ sudo mdadm --add /dev/md3 /dev/nvme1n1p3 # mdadm: re-added /dev/nvme1n1p3 ``` -Use the following command to follow the RAID rebuild `cat /proc/mdstat`. +Use the following command to follow the RAID rebuild: `cat /proc/mdstat`. **Recreating the EFI partition on the disk** @@ -1010,4 +1009,4 @@ We have now successfully completed the RAID rebuild. [Hot Swap - Hardware RAID](/pages/bare_metal_cloud/dedicated_servers/hotswap_raid_hard) -Join our [community of users](/links/community). \ No newline at end of file +Join our [community of users](/links/community). From d2589107bdbc9a1762e85fb2f65c929e418ce9f3 Mon Sep 17 00:00:00 2001 From: Montrealhub <89825661+Jessica41@users.noreply.github.com> Date: Mon, 18 Aug 2025 11:35:41 -0400 Subject: [PATCH 8/8] Update guide.en-gb.md --- .../dedicated_servers/raid_soft/guide.en-gb.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/pages/bare_metal_cloud/dedicated_servers/raid_soft/guide.en-gb.md b/pages/bare_metal_cloud/dedicated_servers/raid_soft/guide.en-gb.md index d68c723719b..0de4ca168ec 100644 --- a/pages/bare_metal_cloud/dedicated_servers/raid_soft/guide.en-gb.md +++ b/pages/bare_metal_cloud/dedicated_servers/raid_soft/guide.en-gb.md @@ -14,7 +14,7 @@ The default RAID level for OVHcloud server installations is RAID 1, which double ## Requirements -- A [dedicated server](/links/bare-metal/bare-metal){.external} with a software RAID configuration +- A [dedicated server](/links/bare-metal/bare-metal) with a software RAID configuration - Administrative (sudo) access to the server via SSH ## Instructions @@ -382,4 +382,4 @@ mount /dev/md4 /home [Hot Swap - Hardware RAID](/pages/bare_metal_cloud/dedicated_servers/hotswap_raid_hard) -Join our [community of users](/links/community). \ No newline at end of file +Join our [community of users](/links/community).