February 17

Linux: Fixing a lost drive in Ubuntu when using software Raid 1 

Example:
Drives:
/dev/sda
/dev/sdb

Raid 1 with boot, swap, and root - created with Ubuntu installer

Problem:
Lost Drive 0 : /dev/sda

Note:  When using grub and if the drives are not hot swappable the server will need rebooted into the drive that has grub and the raid1 data.  In the case of this example on reboot the BIOS drive boot order needs changed from drive0 being first to drive1 being above drive0 to make it first.  After the repair has taken place this can be put back in the BIOS.

The servers stats pre drive loss:
root@ubuntumdtest:~# fdisk -l /dev/sda
Disk /dev/sda: 20 GiB, 21474836480 bytes, 41943040 sectors
Disk model: Virtual disk
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 5ECE8771-DEC1-444A-B370-880C670ABE8B

Device       Start      End  Sectors Size Type
/dev/sda1     2048     4095     2048   1M BIOS boot
/dev/sda2     4096  2101247  2097152   1G Linux filesystem
/dev/sda3  2101248  6295551  4194304   2G Linux filesystem
/dev/sda4  6295552 41940991 35645440  17G Linux filesystem

root@ubuntumdtest:~# fdisk -l /dev/sdb
Disk /dev/sdb: 20 GiB, 21474836480 bytes, 41943040 sectors
Disk model: Virtual disk
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 5ECE8771-DEC1-444A-B370-880C670ABE8B

Device       Start      End  Sectors Size Type
/dev/sdb1     2048     4095     2048   1M BIOS boot
/dev/sdb2     4096  2101247  2097152   1G Linux filesystem
/dev/sdb3  2101248  6295551  4194304   2G Linux filesystem
/dev/sdb4  6295552 41940991 35645440  17G Linux filesystem


root@ubuntumdtest:~# lsblk
NAME      MAJ:MIN RM  SIZE RO TYPE  MOUNTPOINT
loop0       7:0    0 55.5M  1 loop  /snap/core18/2284
loop1       7:1    0 55.4M  1 loop  /snap/core18/2128
loop2       7:2    0 43.6M  1 loop  /snap/snapd/14978
loop3       7:3    0 61.9M  1 loop  /snap/core20/1328
loop4       7:4    0 70.3M  1 loop  /snap/lxd/21029
loop5       7:5    0 67.2M  1 loop  /snap/lxd/21835
sda         8:0    0   20G  0 disk
├─sda1      8:1    0    1M  0 part
├─sda2      8:2    0    1G  0 part
│ └─md126   9:126  0 1022M  0 raid1 /boot
├─sda3      8:3    0    2G  0 part
│ └─md127   9:127  0    2G  0 raid1 [SWAP]
└─sda4      8:4    0   17G  0 part
  └─md125   9:125  0   17G  0 raid1 /
sdb         8:16   0   20G  0 disk
├─sdb1      8:17   0    1M  0 part
├─sdb2      8:18   0    1G  0 part
│ └─md126   9:126  0 1022M  0 raid1 /boot
├─sdb3      8:19   0    2G  0 part
│ └─md127   9:127  0    2G  0 raid1 [SWAP]
└─sdb4      8:20   0   17G  0 part
  └─md125   9:125  0   17G  0 raid1 /
sr0        11:0    1 1024M  0 rom


cat /proc/mdstat
Personalities : [raid1] [linear] [multipath] [raid0] [raid6] [raid5] [raid4] [raid10]
md125 : active raid1 sda4[2] sdb4[1]
      17805312 blocks super 1.2 [2/2] [UU]

md126 : active raid1 sda2[2] sdb2[1]
      1046528 blocks super 1.2 [2/2] [UU]

md127 : active raid1 sda3[2] sdb3[1]
      2094080 blocks super 1.2 [2/2] [UU]

unused devices: <none>


The Repair: Process:
Replace Drive0 which equates to /dev/sda

Boot into Drive1 by reordering the Bios boot order:

Login and sudo up:

cat /proc/mdstat
fdisk -l
fdisk -l /dev/sda
sfdisk -d /dev/sdb | sed 's/,\s*uuid=[^,]\+//gi' | sfdisk /dev/sda
fdisk -l /dev/sdb
fdisk -l /dev/sda
cat /proc/mdstat
mdadm --add /dev/md125 /dev/sda4
mdadm --add /dev/md126 /dev/sda3
mdadm --add /dev/md127 /dev/sda2
cat /proc/mdstat
grub-install /dev/sda
reboot

Change the Bios back to Drive0 being first in the boot order if desired.

By: tconrad
Category: Linux | Comments Off on Linux: Fixing a lost drive in Ubuntu when using software Raid 1
January 28

Linux: Configuring Software RAID1 on a Running Ubuntu System 

This is an example of migrating a running Ubuntu system to a software RAID1.
In the process, you will need to perform two reboots.

The first step is to switch to the root user if not yet:
sudo -i

List the disk partitions - look to see if the drives appear as sd, vd, nvme, etc..
After seeing the drive type modify these instructions accordingly.
fdisk -l

Disk /dev/sda: 25 GiB, 26843545600 bytes, 52428800 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 318758B2-4D70-4BD5-A9FC-C4DE66E692F7

Device     Start      End  Sectors Size Type
/dev/sda1   2048     4095     2048   1M BIOS boot
/dev/sda2   4096 52428766 52424671  25G Linux RAID


fdisk -l | grep '/dev/sd'
Disk /dev/sda: 25 GiB, 26843545600 bytes, 52428800 sectors
/dev/sda1   2048     4095     2048   1M BIOS boot
/dev/sda2   4096 52428766 52424671  25G Linux RAID


lsblk -o NAME,UUID
NAME    UUID
sda     
├─sda1  
└─sda2  3cc51393-2a21-9158-6f63-87a3a85bb7ee
  └─md0 003fde66-cb4a-453e-9cc5-87f4ce46a722


Suppose that the system uses one disk, for example /dev/sda and has one main partition, /dev/sda1.
For the test, I installed a clean Ubuntu Server 20.04, the disk was parted by default, flat ext4 with no LVM, swap was the file on the same partition.

To create a raid, we connect another disk of the same size, it will be called /dev/sdb.

Install mdadm and necessary utilities (they are usually installed by default):
apt-get install initramfs-tools mdadm

In order to make sure that all necessary modules and components are installed, execute the following command:
cat /proc/mdstat

If the necessary modules are not loaded, then load them:
modprobe linear
modprobe multipath
modprobe raid1

Divide the new disk /dev/sdb in the same way as:
sfdisk -d /dev/sda | sfdisk --force /dev/sdb

Check:
fdisk -l

In the next step, change the partition type of the new hard disk /dev/sdb to “Linux raid autodetect” (since partition 1, then after “t” it will not be asked to specify the partition number):
fdisk /dev/sdb
t
29
w

Make sure that the partition type /dev/sdb is Linux RAID:
fdisk -l

Create an array md0 using the missing:
mdadm --create /dev/md0 --level=1 --metadata=1.0 --raid-disks=2 missing /dev/sdb1

Check:
cat /proc/mdstat

If something does not work, then you can remove the raid and try again:
mdadm --stop /dev/md0

Specify the file system of the array:
mkfs.ext4 /dev/md0

Make a backup copy of the configuration file mdadm and add information about the new array:
cp /etc/mdadm/mdadm.conf /etc/mdadm/mdadm_backup.conf
mdadm --examine --scan >> /etc/mdadm/mdadm.conf

Mount /dev/md0 into the system:
	
mkdir /mnt/md0
mount /dev/md0 /mnt/md0
mount

For me it was displayed at the bottom of the list:
/dev/md0 on /mnt/md0 type ext4 (rw,relatime,data=ordered)

In the /etc/fstab file comment the lines about /dev/sda and add about the array:
nano /etc/fstab
/dev/md0 /               ext4    errors=remount-ro 0       1

Let’s see the file /etc/mtab whether there is a record about the raid:
cat /etc/mtab

Let’s look at the exact names of the files.  In this case /boot/vmlinuz-5.4.0-96-generic, /boot/initrd.img-5.4.0-96-generic:
ls /boot

Create a file from the GRUB2 boot menu and open it in the editor:
cp /etc/grub.d/40_custom /etc/grub.d/09_raid1_test
nano /etc/grub.d/09_raid1_test

Add the contents (instead of /vmlinuz and /initrd.img, we’ll specify the correct names if they are different):
#!/bin/sh
exec tail -n +3 $0
# This file provides an easy way to add custom menu entries.  Simply type the
# menu entries you want to add after this comment.  Be careful not to change
# the 'exec tail' line above.
menuentry 'Debian GNU/Linux, with Linux' --class debian --class gnu-linux --class gnu --class os {
        insmod mdraid1x
        insmod part_msdos
        insmod ext2
        set root='(md/0)'
        echo    'Loading Linux'
        linux   /boot/vmlinuz-5.4.0-96-generic root=/dev/md0 ro  quiet
        echo    'Loading initial ramdisk ...'
        initrd  /boot/initrd.img-5.4.0-96-generic
}

Open the file /etc/default/grub in the text editor:
nano /etc/default/grub

Uncomment a couple of lines:
GRUB_TERMINAL=console
GRUB_DISABLE_LINUX_UUID=true

Update the loader:

update-grub

Prepare ramdisk:
update-initramfs -u

Install the bootloader on both disks:
grub-install /dev/sda
grub-install /dev/sdb

Copy all the data to the previously mounted md0 array:
cp -dpRx / /mnt/md0

After the copy has completed, restart the system:
reboot

Note:
When the system starts, in the boot menu it will be the first menu /etc/grub.d/09_raid1_test, if there are problems with the download, you can choose to boot from /dev/sda.  You can also try to remount the system in rw mode if needed:
mount -o remount,rw /partition/identifier /mount/point
I initially had an /etc/fstab issue so I did the followind: mount -o remount,rw /dev/md0 /


Make sure that the system is started with /dev/md0:
df -h

Again, switch to the root user if not under it:
sudo -i

Copy the working partition of /dev/sdb to /dev/sda
sfdisk -d /dev/sdb | sfdisk --force /dev/sda

Check:
fdisk -l

Add to the array the old disk:
mdadm --add /dev/md0 /dev/sda1

Wait until the synchronization is completed and make sure that the raid is in order – UU:
cat /proc/mdstat

Update the array information in the mdadm configuration file:
cp /etc/mdadm/mdadm_backup.conf /etc/mdadm/mdadm.conf
mdadm --examine --scan >> /etc/mdadm/mdadm.conf

Remove our temporary GRUB menu, it’s no longer necessary:
rm -f /etc/grub.d/09_raid1_test

Update and install GRUB again:
update-grub
update-initramfs -u
grub-install /dev/sda
grub-install /dev/sdb

Restart the system to make sure it runs successfully:
	
reboot

At this, the migration of the running Ubuntu system to the software RAID1 is complete.
If one of the disks, /dev/sda or /dev/sdb stops working, the system will run and boot.
For stability, you can add more disks of the same size to the array.

By: vyacheslav
Update for 20.04 by tconrad
Category: Linux | Comments Off on Linux: Configuring Software RAID1 on a Running Ubuntu System