NVMe support is broken (write)

[jcadduono]

After upgrading to 5.2b2 I lost write support on my NVMe mode drives which work fine on 5.1.26+5.1.27 test builds.
Reverting to 5.1.26 fixes it.
Booting results in a confused kernel that eventually mounts everything on NVMe drivers as read-only and shows them as corrupt volumes.
fsck completes successfully but the changes are never actually written to the drives, so they only exist in kernel memory cache of filesystem changes. (wait is that a thing, I just kind of shot words out of my fingers there)
Apologies for not collecting kernel logs, I did not have any volumes to save them to at the time. This is on Linux 4.12.9 mainline compiled kernel running Debian GNU/Linux 9.1 amd64.
I assume it should be simple to reproduce.

Host: Windows 10 Pro 1703 64-bit
Guest: Debian GNU/Linux 9.1 with 4.12.9 mainline kernel (amd64)
Host disk: Samsung NVMe 960 Pro 512 GB

vbox file if it helps you to diagnose/reproduce:

Code: Select all

<?xml version="1.0"?>
<!--
** DO NOT EDIT THIS FILE.
** If you make changes to this file while any VirtualBox related application
** is running, your changes will be overwritten later, without taking effect.
** Use VBoxManage or the VirtualBox Manager GUI to make changes.
-->
<VirtualBox xmlns="http://www.virtualbox.org/" version="1.16-windows">
  <Machine uuid="{b5ceec70-899c-4a7c-ad08-8716852813f7}" name="Arkyn" OSType="Debian_64" snapshotFolder="Snapshots" lastStateChange="2017-08-29T20:00:40Z">
    <MediaRegistry>
      <HardDisks>
        <HardDisk uuid="{a57bde44-e90f-4254-949a-8cf78684b30c}" location="Arkyn.vdi" format="VDI" type="Normal"/>
        <HardDisk uuid="{100cd783-9621-43d0-9ce6-b188d5c69fab}" location="Arkyn - Home.vdi" format="VDI" type="Normal"/>
      </HardDisks>
      <DVDImages>
        <Image uuid="{74c838c3-e8e6-4280-a4aa-068d195ad697}" location="C:/Users/Jc/Downloads/debian-testing-amd64-netinst.iso"/>
        <Image uuid="{e0402ffb-02a7-43e9-af6f-e4d626c15ca8}" location="C:/Users/Jc/Downloads/VBoxGuestAdditions_5.1.27-117460.iso"/>
        <Image uuid="{f2677837-62c2-4567-8a67-a0fd0d7de6de}" location="C:/Program Files/Oracle/VirtualBox/VBoxGuestAdditions.iso"/>
      </DVDImages>
    </MediaRegistry>
    <ExtraData>
      <ExtraDataItem name="GUI/LastCloseAction" value="PowerOff"/>
      <ExtraDataItem name="GUI/LastGuestSizeHint" value="2560,1346"/>
      <ExtraDataItem name="GUI/LastNormalWindowPosition" value="421,130,640,480,max"/>
      <ExtraDataItem name="GUI/RestrictedRuntimeDevicesMenuActions" value="HardDrives"/>
      <ExtraDataItem name="GUI/RestrictedRuntimeMachineMenuActions" value="SaveState,PowerOff"/>
      <ExtraDataItem name="GUI/ScaleFactor" value="1.5"/>
      <ExtraDataItem name="GUI/StatusBar/IndicatorOrder" value="HardDisks,OpticalDisks,FloppyDisks,Network,USB,SharedFolders,Display,VideoCapture,Features,Mouse,Keyboard"/>
    </ExtraData>
    <Hardware>
      <CPU count="4">
        <PAE enabled="true"/>
        <LongMode enabled="true"/>
        <X2APIC enabled="true"/>
        <HardwareVirtExLargePages enabled="true"/>
      </CPU>
      <Memory RAMSize="4096"/>
      <Firmware type="EFI"/>
      <HID Pointing="USBTablet"/>
      <Chipset type="ICH9"/>
      <Paravirt provider="KVM"/>
      <Boot>
        <Order position="1" device="HardDisk"/>
        <Order position="2" device="None"/>
        <Order position="3" device="None"/>
        <Order position="4" device="None"/>
      </Boot>
      <Display VRAMSize="64"/>
      <RemoteDisplay>
        <VRDEProperties>
          <Property name="TCP/Ports" value="3389"/>
        </VRDEProperties>
      </RemoteDisplay>
      <BIOS>
        <IOAPIC enabled="true"/>
      </BIOS>
      <USB>
        <Controllers>
          <Controller name="OHCI" type="OHCI"/>
          <Controller name="EHCI" type="EHCI"/>
        </Controllers>
        <DeviceFilters>
          <DeviceFilter name="ASMT ASM1351 [0100]" active="true" vendorId="174c" productId="55aa" revision="0100" manufacturer="ASMT" remote="0"/>
          <DeviceFilter name="ASMedia Technology Inc. ASM1051E SATA 6Gb/s bridge, ASM1053E SATA 6Gb/s bridge, ASM1153 SATA 3Gb/s bridge [0100]" active="true" vendorId="174c" productId="55aa" revision="0100" remote="0"/>
        </DeviceFilters>
      </USB>
      <Network>
        <Adapter slot="0" enabled="true" MACAddress="08002776B582" type="virtio">
          <DisabledModes>
            <InternalNetwork name="intnet"/>
            <NATNetwork name="NatNetwork"/>
          </DisabledModes>
          <BridgedInterface name="Intel(R) Gigabit CT Desktop Adapter"/>
        </Adapter>
        <Adapter slot="1" cable="false"/>
        <Adapter slot="2" cable="false"/>
        <Adapter slot="3" cable="false"/>
        <Adapter slot="4" cable="false"/>
        <Adapter slot="5" cable="false"/>
        <Adapter slot="6" cable="false"/>
        <Adapter slot="7" cable="false"/>
        <Adapter slot="8" cable="false"/>
        <Adapter slot="9" cable="false"/>
        <Adapter slot="10" cable="false"/>
        <Adapter slot="11" cable="false"/>
        <Adapter slot="12" cable="false"/>
        <Adapter slot="13" cable="false"/>
        <Adapter slot="14" cable="false"/>
        <Adapter slot="15" cable="false"/>
        <Adapter slot="16" cable="false"/>
        <Adapter slot="17" cable="false"/>
        <Adapter slot="18" cable="false"/>
        <Adapter slot="19" cable="false"/>
        <Adapter slot="20" cable="false"/>
        <Adapter slot="21" cable="false"/>
        <Adapter slot="22" cable="false"/>
        <Adapter slot="23" cable="false"/>
        <Adapter slot="24" cable="false"/>
        <Adapter slot="25" cable="false"/>
        <Adapter slot="26" cable="false"/>
        <Adapter slot="27" cable="false"/>
        <Adapter slot="28" cable="false"/>
        <Adapter slot="29" cable="false"/>
        <Adapter slot="30" cable="false"/>
        <Adapter slot="31" cable="false"/>
        <Adapter slot="32" cable="false"/>
        <Adapter slot="33" cable="false"/>
        <Adapter slot="34" cable="false"/>
        <Adapter slot="35" cable="false"/>
      </Network>
      <AudioAdapter controller="HDA" driver="DirectSound"/>
      <RTC localOrUTC="UTC"/>
      <SharedFolders>
        <SharedFolder name="shared" hostPath="C:\Users\Jc\Desktop\VM Shared" writable="true" autoMount="false"/>
      </SharedFolders>
      <Clipboard mode="Bidirectional"/>
      <DragAndDrop mode="Bidirectional"/>
      <GuestProperties>
        <GuestProperty name="/VirtualBox/GuestAdd/HostVerLastChecked" value="5.1.26" timestamp="1503877083524549600" flags=""/>
        <GuestProperty name="/VirtualBox/GuestAdd/Revision" value="117563" timestamp="1503877048088941103" flags=""/>
        <GuestProperty name="/VirtualBox/GuestAdd/Version" value="5.2.0" timestamp="1503877048088941101" flags=""/>
        <GuestProperty name="/VirtualBox/GuestAdd/VersionExt" value="5.2.0_BETA2" timestamp="1503877048088941102" flags=""/>
        <GuestProperty name="/VirtualBox/GuestInfo/Net/0/MAC" value="08002776B582" timestamp="1503877048091870905" flags=""/>
        <GuestProperty name="/VirtualBox/GuestInfo/Net/0/Name" value="enp0s3" timestamp="1503877048091870907" flags=""/>
        <GuestProperty name="/VirtualBox/GuestInfo/Net/0/Status" value="Up" timestamp="1503877048091870906" flags=""/>
        <GuestProperty name="/VirtualBox/GuestInfo/Net/0/V4/Broadcast" value="192.168.255.255" timestamp="1503877048091870903" flags=""/>
        <GuestProperty name="/VirtualBox/GuestInfo/Net/0/V4/IP" value="192.168.1.29" timestamp="1503877048091870902" flags=""/>
        <GuestProperty name="/VirtualBox/GuestInfo/Net/0/V4/Netmask" value="255.255.0.0" timestamp="1503877048091870904" flags=""/>
        <GuestProperty name="/VirtualBox/GuestInfo/Net/Count" value="1" timestamp="1503877093083903900" flags=""/>
        <GuestProperty name="/VirtualBox/GuestInfo/OS/Product" value="Linux" timestamp="1503877048087964200" flags=""/>
        <GuestProperty name="/VirtualBox/GuestInfo/OS/Release" value="4.12.9-arkyn" timestamp="1503877048087964201" flags=""/>
        <GuestProperty name="/VirtualBox/GuestInfo/OS/Version" value="#1 SMP Sat Aug 26 00:09:30 EDT 2017" timestamp="1503877048087964202" flags=""/>
        <GuestProperty name="/VirtualBox/HostInfo/GUI/LanguageID" value="en_US" timestamp="1503877096442569600" flags=""/>
      </GuestProperties>
    </Hardware>
    <StorageControllers>
      <StorageController name="NVMe" type="NVMe" PortCount="2" useHostIOCache="true" Bootable="true">
        <AttachedDevice nonrotational="true" type="HardDisk" hotpluggable="false" port="0" device="0">
          <Image uuid="{a57bde44-e90f-4254-949a-8cf78684b30c}"/>
        </AttachedDevice>
        <AttachedDevice nonrotational="true" type="HardDisk" hotpluggable="false" port="1" device="0">
          <Image uuid="{100cd783-9621-43d0-9ce6-b188d5c69fab}"/>
        </AttachedDevice>
      </StorageController>
      <StorageController name="SATA" type="AHCI" PortCount="1" useHostIOCache="true" Bootable="true" IDE0MasterEmulationPort="0" IDE0SlaveEmulationPort="1" IDE1MasterEmulationPort="2" IDE1SlaveEmulationPort="3">
        <AttachedDevice passthrough="false" type="DVD" hotpluggable="false" port="0" device="0">
          <Image uuid="{f2677837-62c2-4567-8a67-a0fd0d7de6de}"/>
        </AttachedDevice>
      </StorageController>
    </StorageControllers>
  </Machine>
</VirtualBox>

[aeichner]

Please attach a VBox.log of the affected VM.

[jcadduono]

Logs from latest development build. (117717)

https://adduono.com/vbox/logs/2017-08-30/

Attachment limit is 128 KiB (they are ~350 KiB each) so hosting them on my dedi if that is ok

[aeichner]

Your log shows some weird VERR_NOT_IMPLEMENTED errors which I can't reproduce so far unfortunately. No idea what could be wrong at the moment, there were a lot of storage related changes for 5.2 but none of our manual and automated testing shows this behavior...

[aeichner]

What you could try as a workaround for now is to change the "Use Host I/O cache" setting from enabled to diasbled (which is the default anyway) and check whether the issue goes away. Changing the setting results in different code paths used when the VM is running even though we test both settings.

[jcadduono]

Confirming that the issue is gone when disabling "Use Host I/O cache"

Awaiting next dev builds for more testing, thanks!