Comment Re:This is neat! (Score 1) 22
I want this feature like now. But to be really effective you need to be able to migrate between hosts as well. So the host needs to be able to set the device IDs, not the NVMe device.
That is not a missing piece. That is pretty much the point of the exercise. The proposal lays it all out.
I managed Citrix and VMware pools backed by NetApp filers for decades. In that world, migration was comparatively straightforward because both hosts could already see the same backing storage. Direct-assigned local NVMe buys you much better performance, but it also exposes hardware-specific identity, capabilities, topology, and internal controller state to the guest. That is not so good when the guest has to wake up on a different host attached to a different physical device.
So they present a hybrid abstraction: give the VM something that looks and smells like its own NVMe SSD, but make the identity and migratable state of that SSD virtual objects created inside the hardware. Hypervisors, good ones anyway, can trap and emulate portions of that interface to disguise differences between source and destination hardware, but that is expensive and undermines the reason for direct assignment in the first place.
This is a good compromise.
The host-side orchestrates an exported controller and namespace with guest-visible IDs that can be reproduced on another host. You are dead-on about the host setting the IDs -- under this spec, the hypervisor defines the NQN and GUIDs, not the physical silicon. The SSD implements that identity in hardware. The VM is not being handed the physical drive; it is getting a reproducible object. At the destination, the orchestration layer recreates a compatible exported subsystem, writes the extracted controller state blob into the new hardware, and resumes. The guest OS NVMe driver never even realizes the underlying silicon changed. This is a real win -- you've just turned a dock-side warehouse into a curated pret-a-porter rack at Nordstrom's.
NB: For standard local flash drives, this doesn't magically eliminate the need to move the backing data via storage vMotion or block-mirroring. this is where NVMe over TCP actually hits the nail on the head. In our hyperscaler world, a "local PCIe NVMe drive" becomes a DPU/SmartNIC that translates local PCIe commands into NVMe over TCP attached to the shared storage backend. The new spec lets us extract the PCIe controller state from one host's DPU, inject it into a different host's DPU, and viola: because the backing data is already on the network, the migration happens instantly without copying a single block of storage. This is a real win.
What this adds is a standard way to do that while preserving the device identity and operational state presented to the VM, without forcing the hypervisor to intercept and emulate the NVMe administrative path, checking it off against a vendor-specific clipboard. If you read past the executive summary, that is what most of the update is there to accomplish.
I'm assuming that they've thought of that, though. Seems like they also introduced QoS. And what about using this for RAID? NVMe over TCP seems to be in there as well.
You assume correctly.
The QoS piece is separate but obviously useful here: if every VM gets an exported controller, per-controller IOPS and bandwidth limits become a very clean way to keep one noisy tenant from eating the whole device. Also, RAID can exist above the exported namespace, below it, or behind it, but this proposal is not defining RAID semantics. It is defining how an exported NVMe device can retain its identity and operational state while the VM attached to it moves elsewhere. You are right that NVMe over TCP is highly relevant to making this work at scale, but it is actually a separate, established standard that has been around for years. NVMe over TCP handles the back-end transport across the data center fabric to the shared storage array. The proposal fixes the front-end PCIe problem. It defines how an exported PCIe NVMe device can retain its identity, queues, and operational state while the VM attached to it moves to new physical hardware. It bridges the gap, allowing a VM to get bare-metal local PCIe performance while still being able to live-migrate across hosts pointing to the back-end NVMe/TCP storage