Linux: Fixing a VDO Volume due to running out of space

If you fill the physical space of a VDO volume it will corrupt the data and you will see I/O errors
dmesg will probably look something like this:

[ 3150.967303] XFS (dm-2): writeback error on sector 10950872
[ 3153.384860] XFS (dm-2): metadata I/O error in "xlog_iodone" at daddr 0x1200068 len 64 error 28
[ 3153.384873] XFS (dm-2): xfs_do_force_shutdown(0x2) called from line 1261 of file fs/xfs/xfs_log.c.  Return address = 000000000d11b9e8
[ 3153.384926] XFS (dm-2): Log I/O Error Detected.  Shutting down filesystem
[ 3153.384928] XFS (dm-2): Please umount the filesystem and rectify the problem(s)

The only way to fix this as of 8/21/2018 is to expand the raid.  This should be kept in mind when building VDO volumes.  If you "land-lock" yourself out of the ability to add drive space then you be relying on any backups you might have to recover this data.

Process for recovery:
In this "mini" example we have 4 drives in a raid 5
2 TB - drives
8 TB - Raw
6 TB - Usable after the raid

VDO is setup for 18 TB --name=vdo_data

We artificially create files us in dd if=/dev/urandom and if=/dev/zero.
After the space is consumed we begin to get I/O errors on our mounted share.

Process for fixing:
Check for new vdo size:
vdostats --human-readable
[root@localhost mnt]# vdostats --human-readable
Device                    Size      Used Available Use% Space saving%
/dev/mapper/vdo_data      6.0T      6.0T      0.0B 100%            0%


Unmount your broken clock level mount:
umount /mnt/vdo_data

Add another 2 TB drive to the raid5

Use fdisk -l to make certain you know the device id
In this example our new and 5 drive is: /dev/sdf
Also, take note of the raid size for /dev/md0

Add drive:
mdadm --add /dev/md0 /dev/sdf

Watch process:
watch cat /proc/mdstat 

Grow raid group:
mdadm --grow /dev/md0 --raid-devices=5

Watch process:
watch cat /proc/mdstat

Check raid size:
fdisk -l
You should see /dev/md0 is larger than before:

Grow the Logical volume size of the vdo volume
vdo growLogical --name=vdo_data --vdoLogicalSize=24G

Grow the physical size of vdo
vdo growPhysical --name=vdo_data

Check for new vdo size:
vdostats --human-readable
You should now see available freespace

Repair your file system:
mount -t xfs /dev/mapper/vdo_data /mnt/vdo_data
ummount /mnt/vdo_data
xfs_repair /dev/mapper/vdo_data 
mount -t xfs /dev/mapper/vdo_data /mnt/vdo_data

The final file that filled the drive will be missing data.