February 25

VMWARE: Recovering a lost partition table on a VMFS volume

 (1002281)

Details

  • A VMFS datastore is missing or inaccessible
  • The missing VMFS datastore’s LUN is still presented and detected under the host’s Configuration and Storage Adapters view in the VMware vSphere Client or VMware Infrastructure Client.
  • In the /var/log/vmkernel log file, you see one of these messages:
    • cpu0)WARNING: SCSI: 6011: SCSI IO to non-existent partition 1, np 5
    • cpu0)WARNING: SCSI: 5935: returns 0xbad0011 for vmhba0:0:68
    • cpu0:1031)LVM: 2294: Could not open device , vol [45e3ef74-706b3012-b730-000423cd, 45e3ef74-706b3012-b730-000423cd17ea, 1]: No such partition on target
    • cpu0:1031)FSS: 343: Failed with status 0xbad0012 for f530 28 1 45e3ef75 b000548 400d1b0 ea17cd23 0 0 0 0 0 0 0
    • cpu1:1039)WARNING: Vol3: 607: Couldn’t read volume header from 4816d0dd-804c9a8c-2c76-001e0beb5f56: Address temporarily unmapped

Solution

Warnings:
  • This article guides you through recreating the partition table of a LUN that has a VMFS datastore. Ensure that you are familiar with the way VMware ESX identifies disks and LUNs before proceeding. For more information, see Identifying disks when working with VMware ESX (1014953).
  • The steps outlined here are potentially hazardous for your environment if they are not followed exactly. If you are not comfortable performing these steps, contact VMware Technical Support to resolve the issue. For more information, see How to Submit a Support Request.
  • If a LUN does not have a valid partition table, it may not meet the criteria for VMware vCenter and VMware VirtualCenter’s storage filtering safeguards. Refer to your storage configuration or SAN administrator before making any changes, such as creating a VMFS datastore or Raw Device Mapping (RDM), adding an extent, or increasing a VMFS datastore.

ESX 4.x

To recreate the partition table:

  1. Log in to the ESX host service console. For ESXi, see Tech Support Mode for Emergency Support (1003677).
  2. Run the command:

    esxcfg-scsidevs -c

    The output appears similar to:

    Device UID                            Device Type      Console Device                                            Size      Plugin  Display Name
    mpx.vmhba32:C0:T0:L0                  CD-ROM           /dev/sr0                                                  0MB       NMP     Local USB CD-ROM (mpx.vmhba32:C0:T0:L0)
    mpx.vmhba3:C0:T0:L0                   CD-ROM           /dev/sr1                                                  0MB       NMP     Local PLDS CD-ROM (mpx.vmhba3:C0:T0:L0)
    naa.50060160bce0383c50060160bce0383c  Direct-Access    /dev/sdj                                                  0MB       NMP     DGC Fibre Channel Disk (naa.50060160bce0383c50060160bce0383c)
    naa.60024e8056ca2100119f56f605695091  Direct-Access    /dev/sdk                                                  139392MB  NMP     Local DELL Disk (naa.60024e8056ca2100119f56f605695091)
    naa.6006016045502500d0c748a4198bde11  Direct-Access    /dev/sdh                                                  512000MB  NMP     DGC Fibre Channel Disk (naa.6006016045502500d0c748a4198bde11)
    naa.6006016045502500ec34346d188bde11  Direct-Access    /dev/sdi                                                  512000MB  NMP     DGC Fibre Channel Disk (naa.6006016045502500ec34346d188bde11)
    naa.60060160d2ed18012eea2d1ce42ddf11  Direct-Access    /dev/sdn                                                  819200MB  NMP     DGC iSCSI Disk (naa.60060160d2ed18012eea2d1ce42ddf11)
    naa.60060160d2ed180176d68215e42ddf11  Direct-Access    /dev/sdo                                                  819200MB  NMP     abcDGC iSCSI Disk (naa.60060160d2ed180176d68215e42ddf11)
    naa.60060160d2ed18017ea10024e42ddf11  Direct-Access    /dev/sdm                                                  819200MB  NMP     DGC iSCSI Disk (naa.60060160d2ed18017ea10024e42ddf11)
    naa.60060160d2ed180190df110ee42ddf11  Direct-Access    /dev/sdp                                                  819200MB  NMP     logical
    naa.600a0b80002666d40000000000000000  Direct-Access    /vmfs/devices/disks/naa.600a0b80002666d40000000000000000  20MB      NMP     IBM Fibre Channel Disk (naa.600a0b80002666d40000000000000000)
    naa.600a0b80002666d400003a0a4b99f95c  Direct-Access    /dev/sdd                                                  102400MB  NMP     IBM Fibre Channel Disk (naa.600a0b80002666d400003a0a4b99f95c)
    naa.600a0b80002666d400003a114b99f9b4  Direct-Access    /dev/sdc                                                  102400MB  NMP     IBM Fibre Channel Disk (naa.600a0b80002666d400003a114b99f9b4)
    naa.600a0b80002666d400003a184b99f9ee  Direct-Access    /dev/sdb                                                  102400MB  NMP     IBM Fibre Channel Disk (naa.600a0b80002666d400003a184b99f9ee)
    naa.600a0b80002901aa0000000000000000  Direct-Access    /vmfs/devices/disks/naa.600a0b80002901aa0000000000000000  20MB      NMP     IBM Fibre Channel Disk (naa.600a0b80002901aa0000000000000000)
    naa.600a0b80002901aa0000a11b4a3b51ad  Direct-Access    /dev/sdg                                                  102400MB  NMP     IBM Fibre Channel Disk (naa.600a0b80002901aa0000a11b4a3b51ad)
    naa.600a0b80002901aa0000a1214a3b51ef  Direct-Access    /dev/sdf                                                  81920MB   NMP     IBM Fibre Channel Disk (naa.600a0b80002901aa0000a1214a3b51ef)
    naa.600a0b80002901aa0000a1274a3b5231  Direct-Access    /dev/sde                                                  71680MB   NMP     IBM Fibre Channel Disk (naa.600a0b80002901aa0000a1274a3b5231)
    naa.600a0b80002901aa0000a3774b99f77d  Direct-Access    /dev/sda                                                  102400MB  NMP     IBM Fibre Channel Disk (naa.600a0b80002901aa0000a3774b99f77d)

  3. Identify the disk in question, and note the value under the Console Device column.

    Note: There are no Linux console device names in ESXi. In ESXi you use the NAAs, for example: /vmfs/devices/disks/naa.600a0b80002666d40000000000000000.

    If you need to map an NAA ID to a VMFS datastore name use the following command:

    esxcfg-scsidevs -m

  4. Run the command:

    fdisk -l <dev>

    Where <dev> is the value recorded in step 2

    The output appears similar to:

    Disk /dev/sde: 268.4 GB, 268440698880 bytes
    255 heads, 63 sectors/track, 32636 cylinders
    Units = cylinders of 16065 * 512 = 8225280 bytes

    Device Boot Start End Blocks Id System

  5. Start fdisk with this command and press Enter.

    fdisk -u <dev>

    Where <dev> is the value recorded in step 2.

  6. Create the partition:
    1. Press n and press Enter to create a new partition.
    2. Press p and press Enter to select that this is a primary partition.
    3. Press 1 and press Enter to make the first partition.
    4. Press 128 and Enter to  align the partition to sector 128.
    5. Press Enter again to retain the default value.
    6. Change the partition to type fb (VMFS):
      1. Press t and press Enter. Partition 1 is automatically selected.
      2. Enter fb and press Enter.
  7. Press w and press Enter to save.
  8. Run vmkfstools -V and press Enter to discover the VMFS.

ESX 3.x

To recreate the partition table:
  1. Log in to the ESX host service console. For ESXi, see Tech Support Mode for Emergency Support (1003677).
  2. Type esxcfg-vmhbadevs -q and press Enter to get a list of Linux devices and their VMkernel name.
  3. Type fdisk -lu /dev/sdX and press Enter to confirm that the partition is no longer here.
  4. Recreate the partition table:
    1. Start fdisk with the command fdisk -u /dev/sdX and press Enter.
    2. Create the partition:
      1. Press n and press Enter to create a new partition.
      2. Press p and press Enter to select that this is a primary partition.
      3. Press 1 and press Enter to make the first partition.
      4. Press 128 and press Enter to align the partition to sector 128.

        Note        : If the volume was upgraded from an ESX 2.x , the partition should be aligned to 63, which is the default when using fdisk.

      5. Press Enter to retain the default value.

    3. Change the partition to type fb (VMFS):
      1. Press t and press Enter. Partition 1 is automatically selected.
      2. Enter fb and press Enter.

    4. Type w and press Enter to save. This command exits fdisk and returns you to the command prompt.

  5. Type vmkfstools -V and press Enter to discover the VMFS.

ESX 2.x

To recreate the partition table:
  1. Log in to the ESX host service console.
  2. Typevmkpcidivy -q vmhba_devsand press Enter to get a list of Linux devices and their VMkernel name.
  3. Typefdisk -l /dev/sdXand press Enter to confirm that the partition is no longer here.
  4. Recreate the partition table:
    1. Start fdisk with the commandfdisk /dev/sdXand press Enter.
    2. Create the partition:
      1. Type n and press Enter to create a new partition.
      2. Type p and press Enter to create a primary partition.
      3. Type 1 and press Enter to create the first partition.
      4. Press Enter to keep the default value.
      5. Press Enter to keep the default value.

    3. Change the partition to type fb (VMFS):
      1. Type t and press Enter to change the type of partition.
      2. Type fb and press Enter to set the partition as VMFS.

    4. Type w and press Enter to save.
      This command will exit fdisk and returns you to the command prompt.

  5. Typevmkfstools -Vand press Enter to discover the VMFS.
Note: This procedure may not work on VMFS volumes that are participating in a spanned set (extents). Contact VMware Technical Support and work with them to resolve the issue.
By: VMWARE


Copyright 2021. All rights reserved.

Posted February 25, 2015 by Timothy Conrad in category "Virtualization

About the Author

If I were to describe myself with one word it would be, creative. I am interested in almost everything which keeps me rather busy. Here you will find some of my technical musings. Securely email me using - PGP: 4CB8 91EB 0C0A A530 3BE9 6D76 B076 96F1 6135 0A1B