How to identify which block device in guest corresponds to the storage device attached

Discussions about using Linux guests in VirtualBox.
Post Reply
tolland
Posts: 5
Joined: 9. Aug 2012, 15:15

How to identify which block device in guest corresponds to the storage device attached

Post by tolland »

I am using VirtualBox to test building clusters of lustre nodes, and as part of that I am using ansible to configure the lustre on block devices that are attached as storage. With lustre I am looking to create a "1GB storage attachment" for the manager service, a "2 GB storage attachment" for the metadata store, and lots of 10GB storage attachments for the object stores. However it seems difficult to find the relevant storage attachment in the guest.

I am running into a couple of issues:

The first one is that the assignment of the naming of the block devices is not consistent between builds. So I am creating a vm and then adding more storage (using vagrant)

Code: Select all

VBoxManage createhd --filename disk-1.vdi --size 1000 --format VDI
VBoxManage storageattach server-1 --storagectl SATA Controller --port 1 --device 0 --type hdd --medium disk-1.vdi
VBoxManage createhd --filename disk-2.vdi --size 2000 --format VDI
VBoxManage storageattach server-1 --storagectl SATA Controller --port 2 --device 0 --type hdd --medium disk-2.vdi
and when the vm boots, sometimes the OS is on /dev/sda and sometimes those 2 attached storages are /dev/sda and /dev/sdb and the OS disk is on /dev/sdc, for example

Code: Select all

# lsblk
NAME                    MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda                       8:0    0   40G  0 disk
├─sda1                    8:1    0    1M  0 part
├─sda2                    8:2    0    1G  0 part /boot
└─sda3                    8:3    0   39G  0 part
  ├─VolGroup00-LogVol00 253:0    0 37.5G  0 lvm  /
  └─VolGroup00-LogVol01 253:1    0  1.5G  0 lvm  [SWAP]
sdb                       8:16   0    2G  0 disk
sdc                       8:32   0    2G  0 disk
and sometimes this:

Code: Select all

[root@oss-1 ~]# lsblk
NAME                    MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda                       8:0    0  3.9G  0 disk
sdb                       8:16   0  3.9G  0 disk
sdc                       8:32   0   40G  0 disk
├─sdc1                    8:33   0    1M  0 part
├─sdc2                    8:34   0    1G  0 part /boot
└─sdc3                    8:35   0   39G  0 part
  ├─VolGroup00-LogVol00 253:0    0 37.5G  0 lvm  /
  └─VolGroup00-LogVol01 253:1    0  1.5G  0 lvm  [SWAP]

The second problem is that I haven't found a good way to identify which block device corresponds to which storage attachment.

For example this storage attachment doesn't have any information that would allow me to know which block device it was in the guest (that I can find)...:

Code: Select all

$ VBoxManage showmediuminfo disk 8eeb4296-b12d-429f-aacf-7887109f006a
UUID:           8eeb4296-b12d-429f-aacf-7887109f006a
Parent UUID:    base
State:          locked write
Type:           normal (base)
Location:       /home/user/.cache/molecule/limepepper.lustre/default/disk-1.vdi
Storage format: VDI
Format variant: dynamic default
Capacity:       2000 MBytes
Size on disk:   2 MBytes
Encryption:     disabled
Property:       AllocationBlockSize=1048576
In use by VMs:  default_server-1_1632947020173_28144 (UUID: 84807671-cffa-4628-a703-3433b67735a9)

and from the vminfo

Code: Select all

Storage Controller Name (0):            IDE
Storage Controller Type (0):            PIIX4
Storage Controller Instance Number (0): 0
Storage Controller Max Port Count (0):  2
Storage Controller Port Count (0):      2
Storage Controller Bootable (0):        on
Storage Controller Name (1):            SATA Controller
Storage Controller Type (1):            IntelAhci
Storage Controller Instance Number (1): 0
Storage Controller Max Port Count (1):  30
Storage Controller Port Count (1):      30
Storage Controller Bootable (1):        on
IDE (0, 0): /home/user/VirtualBox VMs/default_server-1_1632947020173_28144/Snapshots/{17f378ba-d5d1-4c64-b6d9-1cdaecbdbf97}.vmdk (UUID: 17f378ba-d5d1-4c64-b6d9-1cdaecbdbf97)
SATA Controller (1, 0): /home/user/.cache/molecule/limepepper.lustre/default/disk-1.vdi (UUID: 8eeb4296-b12d-429f-aacf-7887109f006a)
SATA Controller (2, 0): /home/user/.cache/molecule/limepepper.lustre/default/disk-2.vdi (UUID: 62884296-97c0-4815-8c19-75b4013eb8db)



Those values don't seem to match up with the IDs used in the guest, for example

Code: Select all

[root@server-1 ~]# lshw -class disk -class storage
  *-ide
       description: IDE interface
       product: 82371AB/EB/MB PIIX4 IDE
       vendor: Intel Corporation
       physical id: 1.1
       bus info: pci@0000:00:01.1
       logical name: scsi0
       version: 01
       width: 32 bits
       clock: 33MHz
       capabilities: ide isa_compat_mode pci_native_mode bus_master emulated
       configuration: driver=ata_piix latency=64
       resources: irq:0 ioport:1f0(size=8) ioport:3f6 ioport:170(size=8) ioport:376 ioport:d000(size=16)
     *-disk
          description: ATA Disk
          product: VBOX HARDDISK
          vendor: VirtualBox
          physical id: 0.0.0
          bus info: scsi@0:0.0.0
          logical name: /dev/sda
          version: 1.0
          serial: VBadc40c44-d88452e2
          size: 40GiB (42GB)
          capabilities: partitioned partitioned:dos
          configuration: ansiversion=5 logicalsectorsize=512 sectorsize=512 signature=000a218d
  *-sata
       description: SATA controller
       product: 82801HM/HEM (ICH8M/ICH8M-E) SATA Controller [AHCI mode]
       vendor: Intel Corporation
       physical id: d
       bus info: pci@0000:00:0d.0
       logical name: scsi3
       logical name: scsi4
       version: 02
       width: 32 bits
       clock: 33MHz
       capabilities: sata pm ahci_1.0 bus_master cap_list emulated
       configuration: driver=ahci latency=64
       resources: irq:21 ioport:d248(size=8) ioport:d250(size=4) ioport:d258(size=8) ioport:d260(size=4) ioport:d270(size=16) memory:f0840000-f0841fff
     *-disk:0
          description: ATA Disk
          product: VBOX HARDDISK
          vendor: VirtualBox
          physical id: 0
          bus info: scsi@3:0.0.0
          logical name: /dev/sdb
          version: 1.0
          serial: VB8eeb4296-6a009f10
          size: 2000MiB (2097MB)
          configuration: ansiversion=5 logicalsectorsize=512 sectorsize=512
     *-disk:1
          description: ATA Disk
          product: VBOX HARDDISK
          vendor: VirtualBox
          physical id: 1
          bus info: scsi@4:0.0.0
          logical name: /dev/sdc
          version: 1.0
          serial: VB62884296-dbb83e01
          size: 2000MiB (2097MB)
          configuration: ansiversion=5 logicalsectorsize=512 sectorsize=512
tolland
Posts: 5
Joined: 9. Aug 2012, 15:15

Re: How to identify which block device in guest corresponds to the storage device attached

Post by tolland »

I was wondering if the port and device on the SATA controller can be used to identify the source storage attachment

Code: Select all

[root@server-1 ~]# ls -lah /dev/disk/by-path/
total 0
drwxr-xr-x. 2 root root 160 Sep 29 20:23 .
drwxr-xr-x. 5 root root 100 Sep 29 20:23 ..
lrwxrwxrwx. 1 root root   9 Sep 29 20:24 pci-0000:00:01.1-ata-1.0 -> ../../sda
lrwxrwxrwx. 1 root root  10 Sep 29 20:24 pci-0000:00:01.1-ata-1.0-part1 -> ../../sda1
lrwxrwxrwx. 1 root root  10 Sep 29 20:24 pci-0000:00:01.1-ata-1.0-part2 -> ../../sda2
lrwxrwxrwx. 1 root root  10 Sep 29 20:24 pci-0000:00:01.1-ata-1.0-part3 -> ../../sda3
lrwxrwxrwx. 1 root root   9 Sep 29 20:24 pci-0000:00:0d.0-ata-2.0 -> ../../sdb
lrwxrwxrwx. 1 root root   9 Sep 29 20:24 pci-0000:00:0d.0-ata-3.0 -> ../../sdc

The interesting thing, is that the serial number and disk id seem to partially derived from the VirtualBox storagemedium UUID

Code: Select all

$ VBoxManage showmediuminfo disk 8eeb4296-b12d-429f-aacf-7887109f006a | egrep ^UUID
UUID:           [b]8eeb4296[/b]-b12d-429f-aacf-7887109f006a
seems to match partially with the disk serial number

Code: Select all

[root@server-1 ~]# hdparm -I /dev/sdb | grep Serial
        Serial Number:      VB[b]8eeb4296[/b]-6a009f10
scottgus1
Site Moderator
Posts: 20965
Joined: 30. Dec 2009, 20:14
Primary OS: MS Windows 10
VBox Version: PUEL
Guest OSses: Windows, Linux

Re: How to identify which block device in guest corresponds to the storage device attached

Post by scottgus1 »

One thing that may undo any ability for us to troubleshoot this is:
tolland wrote:using vagrant
Vagrant-controlled VMs aren't supported here. You can try asking Vagrant, or use Virtualbox only to try your experiments.

One thing I remember reading is that the Linux enumeration of /dev/sd# can change from one boot to another. Got no idea why. There is another (maybe UUID-based?) disk enumeration method in Linux that always comes out the same each boot.

Make sure that the VM's disks always attach to the same disk controller ports each time (default behavior on Virtualbox-controlled Virtualbox :D ), and the 'always-stable' enumeration method should give you consistent output.
tolland
Posts: 5
Joined: 9. Aug 2012, 15:15

Re: How to identify which block device in guest corresponds to the storage device attached

Post by tolland »

scottgus1 wrote:One thing that may undo any ability for us to troubleshoot this is:
tolland wrote:using vagrant
Vagrant-controlled VMs aren't supported here. You can try asking Vagrant, or use Virtualbox only to try your experiments.
Hah, yes, I have removed vagrant from the equation for this troubleshooting, and am directly running the commands using VBoxManage. Obviously learning a lot along the way. ;-)
scottgus1 wrote: One thing I remember reading is that the Linux enumeration of /dev/sd# can change from one boot to another. Got no idea why. There is another (maybe UUID-based?) disk enumeration method in Linux that always comes out the same each boot.
Yep, I guess the switching is not really a problem if I can reliable track what they are... I have a couple of work-arounds which get me past the problem, but I am intriged to find out how this is supposed to be done.

The simple way I am using is to loop all the devices looking for an unmounted one that corresponds to the required size, and this does the trick
tolland
Posts: 5
Joined: 9. Aug 2012, 15:15

Re: How to identify which block device in guest corresponds to the storage device attached

Post by tolland »

Ah, so I found a couple of (IMO) better ways of doing this

So you can set the Disk Serial number to arbitrary values, so creating the storage, attaching it, and setting its Serial can be done like so:

Code: Select all

      VBoxManage createhd --filename disk-mgr.vdi --size 2000 --format "VDI"
      VBoxManage storageattach  server-1 --storagectl "SATA Controller" --port 1 --type "hdd" --medium "disk-mgr.vdi" --comment "This is the Mgr disk"
      VBoxManage setextradata server-1 "VBoxInternal/Devices/ahci/0/Config/Port1/SerialNumber", "mgr-1"
Once that machine is booted, all the disks have serial numbers that can be used to lookup the block device:

Code: Select all

[root@server-1 ~]# hwinfo --disk | egrep 'Serial ID|Device File:'
  Serial ID: "VBadc40c44-d88452e2"
  Device File: /dev/sda
  Serial ID: "mgr-1"
  Device File: /dev/sdb
  Serial ID: "mdt-1"
  Device File: /dev/sdc
  Serial ID: "other-1"
  Device File: /dev/sdd
and you can do something like the following to do a lookup in a one liner:

Code: Select all

[root@server-1 ~]# 
get_block_device(){
  for device in $(ls /dev/sd?) ; do 
    SERIAL=$(udevadm info --query=all --name=${device} | 
      egrep '^E: ID_SERIAL_SHORT' | 
        awk -F'=' '{ print $2 }')
    if [ "$SERIAL" = "$1" ] ; then
      echo ${device}
      return
    fi    
  done
  echo "not found device for serial $1" && return 1
}

[root@server-1 ~]# get_block_device other-1
/dev/sdd
[root@server-1 ~]# get_block_device mgr-1
/dev/sdb
[root@server-1 ~]# get_block_device other-1
/dev/sdd
[root@server-1 ~]# get_block_device mdt-1
/dev/sdc

scottgus1
Site Moderator
Posts: 20965
Joined: 30. Dec 2009, 20:14
Primary OS: MS Windows 10
VBox Version: PUEL
Guest OSses: Windows, Linux

Re: How to identify which block device in guest corresponds to the storage device attached

Post by scottgus1 »

Nice workaround! Glad you got something that works!
fth0
Volunteer
Posts: 5668
Joined: 14. Feb 2019, 03:06
Primary OS: Mac OS X other
VBox Version: PUEL
Guest OSses: Linux, Windows 10, ...
Location: Germany

Re: How to identify which block device in guest corresponds to the storage device attached

Post by fth0 »

tolland wrote:I was wondering if the port and device on the SATA controller can be used to identify the source storage attachment
VirtualBox defines the PCI bus layout of the VM, and the SATA controller is on PCI bus 0 device 13 function 0 (legacy BIOS VM). If you add a virtual hard disk to port 8, you'll find it inside the Linux guest as /dev/disk/by-path/pci-0000:00:0d.0-ata-9.0.
tolland wrote:The interesting thing, is that the serial number and disk id seem to partially derived from the VirtualBox storagemedium UUID
If you look at the UUID more closely, you'll also find the remainder (except the "VB", of course). ;)
Post Reply