Linux: Mounting Images Using Alternate Superblocks
Version 1
Mounting Unix file system images is a common investigative technique, because it allows examiners to use standard file system tools (e.g., find, grep) to look for evidence. However, sometimes you run into difficulties:
# mount -o loop,ro dev_sda2.dd /mnt
mount: wrong fs type, bad option, bad superblock on /dev/loop0,
missing codepage or helper program, or other error
In some cases useful info is found in syslog – try
dmesg | tail or so
So what’s going on here? Well, let’s try the dmesg command suggested in the above error message:
# dmesg | tail
[…]
[565633.964688] EXT3-fs: INFO: recovery required on readonly filesystem.
[565633.964698] EXT3-fs: write access unavailable, cannot proceed.
Actually, in this case the “file” command can provide us with more succinct information:
# file dev_sda2.dd
dev_sda2.dd: Linux rev 1.0 ext3 filesystem data (needs journal recovery)
See the “needs journal recovery” bit at the end? Looks like the image comes from a machine that was shut down ungracefully, and the ext3 file system driver wants to sync the operations in the file system journal before allowing the file system to be mounted.
Now we could mount the image read-write and allow the journal to be flushed, but the concern is that we’d be damaging the forensic integrity of the image– to say nothing of losing actual evidence. In fact, the file system in the image is probably in a decent enough state that we could actually mount it, if only there were some way to coerce the file system drivers into not requiring a journal recovery. Needless to say, I wouldn’t be writing this article unless there was a way to do this…
The Alternate Superblock Trick
As most of you are aware, Unix file system metadata is kept in the superblock at the beginning of the file system image. The superblock stores information like file system state (clean or dirty), last mounted time and location, and pointers to the root of the file system, free block and inode lists, and the file system journal (if any). For redundancy, the primary superblock is replicated throughout the file system. That way if you lose the primary superblock for some reason, you can often recover the file system using one of the alternates, depending on the nature of the corruption.
However, the data in the primary superblock is not perfectly replicated to the alternates. Most important for our purposes, the file system clean/dirty information that’s causing the ext3 file system drivers to require a journal recovery is not flagged in the alternate superblocks. The upshot is that if we tell the mount command to use an alternate superblock, the OS will allow us to mount the file system image without requiring a journal recovery. The hard part is figuring out where the alternate superblocks live and then setting options for the mount command appropriately.
Mounting With Alternate Superblocks
The first step is to discover the location of the alternate superblocks. The trick is to use mkfs with the “-n” option. As with many other Unix commands, “-n” means “show me what you would do, but don’t actually do it”:
# mkfs -n dev_sda2.dd
mke2fs 1.41.3 (12-Oct-2008)
dev_sda2.dd is not a block special device.
Proceed anyway? (y,n) y
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
58880 inodes, 235516 blocks
11775 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=243269632
8 block groups
32768 blocks per group, 32768 fragments per group
7360 inodes per group
Superblock backups stored on blocks:
32768, 98304, 163840, 229376
There are two important pieces of information in the above output. The first is obviously the list of alternate superblock locations at the end of the output. But we also need the block size information from earlier in the output.
Why is the block size relevant? It turns out that the mount command wants you to specify the alternate superblock location in terms of 1K offsets. But mkfs is reporting the alternate superblock addresses in terms of 4K blocks, so we’ll need to multiply the values reported by mkfs by four in order to produce addresses we can feed into the mount command.
You can use any of the alternate superblock addresses you want– let’s just use the first one. The offset we need to feed into the mount command then is 32768*4 = 131072. The mount option for specifying an alternate superblock is “sb=”:
# mount -o loop,ro,sb=131072 dev_sda2.dd /mnt
# ls /mnt
bin dev home lib mnt proc sbin usr
boot etc initrd lost+found opt root tmp var
Excellent! We can now view the contents of our file system image as a mounted file system.
Beyond Forensics
Of course, outside of the forensic context, being able to recover file systems using alternate superblocks is incredibly useful in case of file system corruption. While I hope none of you are ever in a situation where you need to do this, it can make you look godlike in the eyes of your co-workers and management. So keep this arrow in your quiver for emergencies.
Version 2
Note that the information in this article has beensupersededby a new technique that leverages the EXT4 file system drivers. If your analysis system supports EXT4 (and most Linux distros do at this point), then I recommend looking at this article for more details.
Several months ago, I blogged about using alternate superblocks to fake out the ext3 drivers so you could mount file system images read-only, even if they were needing journal recovery. However, due to recent changes in the ext file system driver the method I describe in my posting is no longer sufficient. Happily, there’s a quick work-around.
Let’s try the solution from the end of my previous posting under a more recent Linux kernel:
# mount -o loop,ro,sb=131072 dev_sda2.dd /mnt
mount: wrong fs type, bad option, bad superblock on /dev/loop0,
missing codepage or helper program, or other error
In some cases useful info is found in syslog – try
dmesg | tail or so
This looks like the original error output we got when using the primary superblock. Looking at the relevant dmesg output we see something different, however:
[163135.527484] JBD: Ignoring recovery information on journal
[163135.795917] Buffer I/O error on device loop0, logical block 0
[163135.795931] lost page write due to I/O error on loop0
[163135.795944] Buffer I/O error on device loop0, logical block 1
[163135.795949] lost page write due to I/O error on loop0
[163135.795958] Buffer I/O error on device loop0, logical block 2
[163135.795963] lost page write due to I/O error on loop0
[163135.795973] Buffer I/O error on device loop0, logical block 3
[163135.795977] lost page write due to I/O error on loop0
[163135.795986] Buffer I/O error on device loop0, logical block 18
[163135.795991] lost page write due to I/O error on loop0
[163135.795999] Buffer I/O error on device loop0, logical block 32
[163135.796034] lost page write due to I/O error on loop0
[163135.796232] Buffer I/O error on device loop0, logical block 73
[163135.796238] lost page write due to I/O error on loop0
[163135.796248] Buffer I/O error on device loop0, logical block 74
[163135.796253] lost page write due to I/O error on loop0
[163135.796261] Buffer I/O error on device loop0, logical block 94
[163135.796267] lost page write due to I/O error on loop0
[163135.796275] Buffer I/O error on device loop0, logical block 96
[163135.796280] lost page write due to I/O error on loop0
[163135.796516] JBD: recovery failed
[163135.796520] EXT3-fs: error loading journal.
It would appear that even though we’re using an alternate superblock that’s marked as not requiring journal recovery, the ext file system driver is still finding the uncompleted journal entries and trying to apply them. This is arguably “more correct” behavior than the old driver used, but it doesn’t help us very much.
The simple work-around is to tell the ext file system driver to ignore the journal by forcing the file system to be mounted as ext2:
# mount -t ext2 -o loop,ro,sb=131072 dev_sda2.dd /mnt
# ls /mnt
bin dev home lib mnt proc sbin usr
boot etc initrd lost+found opt root tmp var
Excellent! With this small modification our trick is working again. Hurrah!
You might well wonder what happens if you just try to mount our image as ext2 without using the alternate superblock. Unfortunately, simply mounting as ext2 is not sufficient because the primary superblock is still marked as needing journal recovery. Though I wonder why this flag should be relevant to an ext2 file system, it’s enough to prevent the mount from happening. So the result is that you need to both mount using an alternate superblock and (at least on modern Linux kernels) mount the file system as ext2 to stop the file system driver from looking at the journal.
Version 3
As some of you may remember, I’ve previously written about a technique for mounting EXT3 file system images with the read-only option, even when power was abruptly removed from the system– as is typical during forensic seizure– and the file system is still “dirty”. In these cases, my technique involves using an alternate superblock, which will not have the “needs journal recovery” flag set, and using the “-t ext2” option to ignore any entries in the EXT3 file system’s journal.
In the last year, however, I’ve been starting to see cases where I’ve had to analyze the newer EXT4 file system (hence my recent series of articles on EXT4 internals). It turns out that the technique I developed for mounting EXT3 file systems does not work with EXT4:
# mount -t ext2 -o loop,ro,noexec,sb=131072 ext4-test.img /mnt/test/
mount: wrong fs type, bad option, bad superblock on /dev/loop0,
missing codepage or helper program, or other error
In some cases useful info is found in syslog – try
dmesg | tail or so
# dmesg | tail -1
[124897.443002] EXT2-fs: loop0: couldn’t mount because of unsupported optional features (240).
So, unfortunately, our trick of using “-t ext2” to get the file system drivers to ignore the journal is not going to work for us here, because the EXT2 drivers don’t recognize many of the new file system options in EXT4.
So what can we do to mount our EXT4 file systems? When in doubt, refer to the man page:
# man mount
[…]
-r, –read-only
Mount the filesystem read-only. A synonym is -o ro.
Note that, depending on the filesystem type, state and kernel
behavior, the system may still write to the device. For example,
Ext3 or ext4 will replay its journal if the filesystem is dirty.
To prevent this kind of write access, you may want to mount ext3
or ext4 filesystem with “ro,noload” mount options or set the
block device to read-only mode, see command blockdev(8).
[…]
noload Do not load the ext3 filesystem’s journal on mounting.
[…]
The “noload” option looks promising. Let’s give it a try:
# file ext4-test.img
ext4-test.img: Linux rev 1.0 ext4 filesystem data, UUID=… (needs journal recovery) …
# mount -o loop,ro,noexec,noload ext4-test.img /mnt/test/
# ls /mnt/test
backups crash lib lock lost+found opt spool
cache games local log mail run tmp
You can see the “needs journal recovery” flag in the output of the file command, so our file system is definitely dirty. But happily the “noload” option does indeed allow us to mount the EXT4 file system.
But what about EXT3? The manual page suggests that “noload” will work there as well. Unfortunately, this doesn’t appear to be correct:
# file ext3-test.img
ext3-test.img: Linux rev 1.0 ext3 filesystem data, UUID=… (needs journal recovery)
# mount -o loop,ro,noexec,noload ext3-test.img /mnt/test/
mount: wrong fs type, bad option, bad superblock on /dev/loop0,
missing codepage or helper program, or other error
In some cases useful info is found in syslog – try
dmesg | tail or so
# dmesg | tail -1
[126955.823010] ext3: No journal on filesystem on loop0
It appears that the “noload” option does in fact cause the journal not to be loaded. But the EXT3 drivers apparently regard this as an error condition and refuse to do the mount. And before you ask, adding “-t ext2” to the command line above doesn’t work either.
So at this point I was stuck with having one method for mounting EXT4 and a different, painful method for mounting EXT3. But then I got an email from Gebhard Zocher, which pointed out a clever solution:
# mount -t ext4 -o loop,ro,noexec,noload ext3-test.img /mnt/test/
# ls /mnt/test
bin dev home lib mnt proc sbinusr
boot etc initrd lost+found opt root tmpvar
Since the EXT4 drivers are backwards compatible with EXT3 file systems, you can just specify “-t ext4” and then use “noload” to mount your EXT3 file systems without mucking around with alternate superblocks. And that means we now have a consistent solution for mounting both EXT3 and EXT4 file systems. Thanks Gebhard!
By: H. Pomeranz
