NVMe support is broken (write)

Postings relating to old VirtualBox pre-releases

NVMe support is broken (write)

Postby jcadduono » 30. Aug 2017, 07:34

After upgrading to 5.2b2 I lost write support on my NVMe mode drives which work fine on 5.1.26+5.1.27 test builds.
Reverting to 5.1.26 fixes it.
Booting results in a confused kernel that eventually mounts everything on NVMe drivers as read-only and shows them as corrupt volumes.
fsck completes successfully but the changes are never actually written to the drives, so they only exist in kernel memory cache of filesystem changes. (wait is that a thing, I just kind of shot words out of my fingers there)
Apologies for not collecting kernel logs, I did not have any volumes to save them to at the time. This is on Linux 4.12.9 mainline compiled kernel running Debian GNU/Linux 9.1 amd64.
I assume it should be simple to reproduce.

Host: Windows 10 Pro 1703 64-bit
Guest: Debian GNU/Linux 9.1 with 4.12.9 mainline kernel (amd64)
Host disk: Samsung NVMe 960 Pro 512 GB

vbox file if it helps you to diagnose/reproduce:
Code: Select all   Expand viewCollapse view
<?xml version="1.0"?>
<!--
** DO NOT EDIT THIS FILE.
** If you make changes to this file while any VirtualBox related application
** is running, your changes will be overwritten later, without taking effect.
** Use VBoxManage or the VirtualBox Manager GUI to make changes.
-->
<VirtualBox xmlns="http://www.virtualbox.org/" version="1.16-windows">
  <Machine uuid="{b5ceec70-899c-4a7c-ad08-8716852813f7}" name="Arkyn" OSType="Debian_64" snapshotFolder="Snapshots" lastStateChange="2017-08-29T20:00:40Z">
    <MediaRegistry>
      <HardDisks>
        <HardDisk uuid="{a57bde44-e90f-4254-949a-8cf78684b30c}" location="Arkyn.vdi" format="VDI" type="Normal"/>
        <HardDisk uuid="{100cd783-9621-43d0-9ce6-b188d5c69fab}" location="Arkyn - Home.vdi" format="VDI" type="Normal"/>
      </HardDisks>
      <DVDImages>
        <Image uuid="{74c838c3-e8e6-4280-a4aa-068d195ad697}" location="C:/Users/Jc/Downloads/debian-testing-amd64-netinst.iso"/>
        <Image uuid="{e0402ffb-02a7-43e9-af6f-e4d626c15ca8}" location="C:/Users/Jc/Downloads/VBoxGuestAdditions_5.1.27-117460.iso"/>
        <Image uuid="{f2677837-62c2-4567-8a67-a0fd0d7de6de}" location="C:/Program Files/Oracle/VirtualBox/VBoxGuestAdditions.iso"/>
      </DVDImages>
    </MediaRegistry>
    <ExtraData>
      <ExtraDataItem name="GUI/LastCloseAction" value="PowerOff"/>
      <ExtraDataItem name="GUI/LastGuestSizeHint" value="2560,1346"/>
      <ExtraDataItem name="GUI/LastNormalWindowPosition" value="421,130,640,480,max"/>
      <ExtraDataItem name="GUI/RestrictedRuntimeDevicesMenuActions" value="HardDrives"/>
      <ExtraDataItem name="GUI/RestrictedRuntimeMachineMenuActions" value="SaveState,PowerOff"/>
      <ExtraDataItem name="GUI/ScaleFactor" value="1.5"/>
      <ExtraDataItem name="GUI/StatusBar/IndicatorOrder" value="HardDisks,OpticalDisks,FloppyDisks,Network,USB,SharedFolders,Display,VideoCapture,Features,Mouse,Keyboard"/>
    </ExtraData>
    <Hardware>
      <CPU count="4">
        <PAE enabled="true"/>
        <LongMode enabled="true"/>
        <X2APIC enabled="true"/>
        <HardwareVirtExLargePages enabled="true"/>
      </CPU>
      <Memory RAMSize="4096"/>
      <Firmware type="EFI"/>
      <HID Pointing="USBTablet"/>
      <Chipset type="ICH9"/>
      <Paravirt provider="KVM"/>
      <Boot>
        <Order position="1" device="HardDisk"/>
        <Order position="2" device="None"/>
        <Order position="3" device="None"/>
        <Order position="4" device="None"/>
      </Boot>
      <Display VRAMSize="64"/>
      <RemoteDisplay>
        <VRDEProperties>
          <Property name="TCP/Ports" value="3389"/>
        </VRDEProperties>
      </RemoteDisplay>
      <BIOS>
        <IOAPIC enabled="true"/>
      </BIOS>
      <USB>
        <Controllers>
          <Controller name="OHCI" type="OHCI"/>
          <Controller name="EHCI" type="EHCI"/>
        </Controllers>
        <DeviceFilters>
          <DeviceFilter name="ASMT ASM1351 [0100]" active="true" vendorId="174c" productId="55aa" revision="0100" manufacturer="ASMT" remote="0"/>
          <DeviceFilter name="ASMedia Technology Inc. ASM1051E SATA 6Gb/s bridge, ASM1053E SATA 6Gb/s bridge, ASM1153 SATA 3Gb/s bridge [0100]" active="true" vendorId="174c" productId="55aa" revision="0100" remote="0"/>
        </DeviceFilters>
      </USB>
      <Network>
        <Adapter slot="0" enabled="true" MACAddress="08002776B582" type="virtio">
          <DisabledModes>
            <InternalNetwork name="intnet"/>
            <NATNetwork name="NatNetwork"/>
          </DisabledModes>
          <BridgedInterface name="Intel(R) Gigabit CT Desktop Adapter"/>
        </Adapter>
        <Adapter slot="1" cable="false"/>
        <Adapter slot="2" cable="false"/>
        <Adapter slot="3" cable="false"/>
        <Adapter slot="4" cable="false"/>
        <Adapter slot="5" cable="false"/>
        <Adapter slot="6" cable="false"/>
        <Adapter slot="7" cable="false"/>
        <Adapter slot="8" cable="false"/>
        <Adapter slot="9" cable="false"/>
        <Adapter slot="10" cable="false"/>
        <Adapter slot="11" cable="false"/>
        <Adapter slot="12" cable="false"/>
        <Adapter slot="13" cable="false"/>
        <Adapter slot="14" cable="false"/>
        <Adapter slot="15" cable="false"/>
        <Adapter slot="16" cable="false"/>
        <Adapter slot="17" cable="false"/>
        <Adapter slot="18" cable="false"/>
        <Adapter slot="19" cable="false"/>
        <Adapter slot="20" cable="false"/>
        <Adapter slot="21" cable="false"/>
        <Adapter slot="22" cable="false"/>
        <Adapter slot="23" cable="false"/>
        <Adapter slot="24" cable="false"/>
        <Adapter slot="25" cable="false"/>
        <Adapter slot="26" cable="false"/>
        <Adapter slot="27" cable="false"/>
        <Adapter slot="28" cable="false"/>
        <Adapter slot="29" cable="false"/>
        <Adapter slot="30" cable="false"/>
        <Adapter slot="31" cable="false"/>
        <Adapter slot="32" cable="false"/>
        <Adapter slot="33" cable="false"/>
        <Adapter slot="34" cable="false"/>
        <Adapter slot="35" cable="false"/>
      </Network>
      <AudioAdapter controller="HDA" driver="DirectSound"/>
      <RTC localOrUTC="UTC"/>
      <SharedFolders>
        <SharedFolder name="shared" hostPath="C:\Users\Jc\Desktop\VM Shared" writable="true" autoMount="false"/>
      </SharedFolders>
      <Clipboard mode="Bidirectional"/>
      <DragAndDrop mode="Bidirectional"/>
      <GuestProperties>
        <GuestProperty name="/VirtualBox/GuestAdd/HostVerLastChecked" value="5.1.26" timestamp="1503877083524549600" flags=""/>
        <GuestProperty name="/VirtualBox/GuestAdd/Revision" value="117563" timestamp="1503877048088941103" flags=""/>
        <GuestProperty name="/VirtualBox/GuestAdd/Version" value="5.2.0" timestamp="1503877048088941101" flags=""/>
        <GuestProperty name="/VirtualBox/GuestAdd/VersionExt" value="5.2.0_BETA2" timestamp="1503877048088941102" flags=""/>
        <GuestProperty name="/VirtualBox/GuestInfo/Net/0/MAC" value="08002776B582" timestamp="1503877048091870905" flags=""/>
        <GuestProperty name="/VirtualBox/GuestInfo/Net/0/Name" value="enp0s3" timestamp="1503877048091870907" flags=""/>
        <GuestProperty name="/VirtualBox/GuestInfo/Net/0/Status" value="Up" timestamp="1503877048091870906" flags=""/>
        <GuestProperty name="/VirtualBox/GuestInfo/Net/0/V4/Broadcast" value="192.168.255.255" timestamp="1503877048091870903" flags=""/>
        <GuestProperty name="/VirtualBox/GuestInfo/Net/0/V4/IP" value="192.168.1.29" timestamp="1503877048091870902" flags=""/>
        <GuestProperty name="/VirtualBox/GuestInfo/Net/0/V4/Netmask" value="255.255.0.0" timestamp="1503877048091870904" flags=""/>
        <GuestProperty name="/VirtualBox/GuestInfo/Net/Count" value="1" timestamp="1503877093083903900" flags=""/>
        <GuestProperty name="/VirtualBox/GuestInfo/OS/Product" value="Linux" timestamp="1503877048087964200" flags=""/>
        <GuestProperty name="/VirtualBox/GuestInfo/OS/Release" value="4.12.9-arkyn" timestamp="1503877048087964201" flags=""/>
        <GuestProperty name="/VirtualBox/GuestInfo/OS/Version" value="#1 SMP Sat Aug 26 00:09:30 EDT 2017" timestamp="1503877048087964202" flags=""/>
        <GuestProperty name="/VirtualBox/HostInfo/GUI/LanguageID" value="en_US" timestamp="1503877096442569600" flags=""/>
      </GuestProperties>
    </Hardware>
    <StorageControllers>
      <StorageController name="NVMe" type="NVMe" PortCount="2" useHostIOCache="true" Bootable="true">
        <AttachedDevice nonrotational="true" type="HardDisk" hotpluggable="false" port="0" device="0">
          <Image uuid="{a57bde44-e90f-4254-949a-8cf78684b30c}"/>
        </AttachedDevice>
        <AttachedDevice nonrotational="true" type="HardDisk" hotpluggable="false" port="1" device="0">
          <Image uuid="{100cd783-9621-43d0-9ce6-b188d5c69fab}"/>
        </AttachedDevice>
      </StorageController>
      <StorageController name="SATA" type="AHCI" PortCount="1" useHostIOCache="true" Bootable="true" IDE0MasterEmulationPort="0" IDE0SlaveEmulationPort="1" IDE1MasterEmulationPort="2" IDE1SlaveEmulationPort="3">
        <AttachedDevice passthrough="false" type="DVD" hotpluggable="false" port="0" device="0">
          <Image uuid="{f2677837-62c2-4567-8a67-a0fd0d7de6de}"/>
        </AttachedDevice>
      </StorageController>
    </StorageControllers>
  </Machine>
</VirtualBox>
jcadduono
 
Posts: 5
Joined: 20. Mar 2015, 01:58

Re: NVMe support is broken (write)

Postby aeichner » 30. Aug 2017, 10:32

Please attach a VBox.log of the affected VM.
aeichner
Oracle Corporation
 
Posts: 154
Joined: 31. Aug 2007, 19:12

Re: NVMe support is broken (write)

Postby jcadduono » 31. Aug 2017, 00:20

Logs from latest development build. (117717)

https://adduono.com/vbox/logs/2017-08-30/

Attachment limit is 128 KiB (they are ~350 KiB each) so hosting them on my dedi if that is ok
jcadduono
 
Posts: 5
Joined: 20. Mar 2015, 01:58

Re: NVMe support is broken (write)

Postby aeichner » 31. Aug 2017, 10:16

Your log shows some weird VERR_NOT_IMPLEMENTED errors which I can't reproduce so far unfortunately. No idea what could be wrong at the moment, there were a lot of storage related changes for 5.2 but none of our manual and automated testing shows this behavior...
aeichner
Oracle Corporation
 
Posts: 154
Joined: 31. Aug 2007, 19:12

Re: NVMe support is broken (write)

Postby aeichner » 31. Aug 2017, 10:22

What you could try as a workaround for now is to change the "Use Host I/O cache" setting from enabled to diasbled (which is the default anyway) and check whether the issue goes away. Changing the setting results in different code paths used when the VM is running even though we test both settings.
aeichner
Oracle Corporation
 
Posts: 154
Joined: 31. Aug 2007, 19:12

Re: NVMe support is broken (write)

Postby jcadduono » 5. Sep 2017, 07:14

Confirming that the issue is gone when disabling "Use Host I/O cache"

Awaiting next dev builds for more testing, thanks!
jcadduono
 
Posts: 5
Joined: 20. Mar 2015, 01:58


Return to Old Beta Postings

Who is online

Users browsing this forum: No registered users and 2 guests

cron