Linux: Fixing a lost drive in Ubuntu when using software Raid 1

Example:
Drives:
/dev/sda
/dev/sdb

Raid 1 with boot, swap, and root - created with Ubuntu installer

Problem:
Lost Drive 0 : /dev/sda

Note:  When using grub and if the drives are not hot swappable the server will need rebooted into the drive that has grub and the raid1 data.  In the case of this example on reboot the BIOS drive boot order needs changed from drive0 being first to drive1 being above drive0 to make it first.  After the repair has taken place this can be put back in the BIOS.

The servers stats pre drive loss:
root@ubuntumdtest:~# fdisk -l /dev/sda
Disk /dev/sda: 20 GiB, 21474836480 bytes, 41943040 sectors
Disk model: Virtual disk
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 5ECE8771-DEC1-444A-B370-880C670ABE8B

Device       Start      End  Sectors Size Type
/dev/sda1     2048     4095     2048   1M BIOS boot
/dev/sda2     4096  2101247  2097152   1G Linux filesystem
/dev/sda3  2101248  6295551  4194304   2G Linux filesystem
/dev/sda4  6295552 41940991 35645440  17G Linux filesystem

root@ubuntumdtest:~# fdisk -l /dev/sdb
Disk /dev/sdb: 20 GiB, 21474836480 bytes, 41943040 sectors
Disk model: Virtual disk
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 5ECE8771-DEC1-444A-B370-880C670ABE8B

Device       Start      End  Sectors Size Type
/dev/sdb1     2048     4095     2048   1M BIOS boot
/dev/sdb2     4096  2101247  2097152   1G Linux filesystem
/dev/sdb3  2101248  6295551  4194304   2G Linux filesystem
/dev/sdb4  6295552 41940991 35645440  17G Linux filesystem


root@ubuntumdtest:~# lsblk
NAME      MAJ:MIN RM  SIZE RO TYPE  MOUNTPOINT
loop0       7:0    0 55.5M  1 loop  /snap/core18/2284
loop1       7:1    0 55.4M  1 loop  /snap/core18/2128
loop2       7:2    0 43.6M  1 loop  /snap/snapd/14978
loop3       7:3    0 61.9M  1 loop  /snap/core20/1328
loop4       7:4    0 70.3M  1 loop  /snap/lxd/21029
loop5       7:5    0 67.2M  1 loop  /snap/lxd/21835
sda         8:0    0   20G  0 disk
├─sda1      8:1    0    1M  0 part
├─sda2      8:2    0    1G  0 part
│ └─md126   9:126  0 1022M  0 raid1 /boot
├─sda3      8:3    0    2G  0 part
│ └─md127   9:127  0    2G  0 raid1 [SWAP]
└─sda4      8:4    0   17G  0 part
  └─md125   9:125  0   17G  0 raid1 /
sdb         8:16   0   20G  0 disk
├─sdb1      8:17   0    1M  0 part
├─sdb2      8:18   0    1G  0 part
│ └─md126   9:126  0 1022M  0 raid1 /boot
├─sdb3      8:19   0    2G  0 part
│ └─md127   9:127  0    2G  0 raid1 [SWAP]
└─sdb4      8:20   0   17G  0 part
  └─md125   9:125  0   17G  0 raid1 /
sr0        11:0    1 1024M  0 rom


cat /proc/mdstat
Personalities : [raid1] [linear] [multipath] [raid0] [raid6] [raid5] [raid4] [raid10]
md125 : active raid1 sda4[2] sdb4[1]
      17805312 blocks super 1.2 [2/2] [UU]

md126 : active raid1 sda2[2] sdb2[1]
      1046528 blocks super 1.2 [2/2] [UU]

md127 : active raid1 sda3[2] sdb3[1]
      2094080 blocks super 1.2 [2/2] [UU]

unused devices: <none>


The Repair: Process:
Replace Drive0 which equates to /dev/sda

Boot into Drive1 by reordering the Bios boot order:

Login and sudo up:

cat /proc/mdstat
fdisk -l
fdisk -l /dev/sda
sfdisk -d /dev/sdb | sed 's/,\s*uuid=[^,]\+//gi' | sfdisk /dev/sda
fdisk -l /dev/sdb
fdisk -l /dev/sda
cat /proc/mdstat
mdadm --add /dev/md125 /dev/sda4
mdadm --add /dev/md126 /dev/sda3
mdadm --add /dev/md127 /dev/sda2
cat /proc/mdstat
grub-install /dev/sda
reboot

Change the Bios back to Drive0 being first in the boot order if desired.

By: tconrad