IOMMU PCI Express USB3 Controller Passthrough

One of the best things about Virtual Machines for me is I can bake in support for legacy systems like Windows XP. I needed to support the Renesas SH2 microcontroller and needed the Windows XP version of their PG-FP5 programmer. I currently use a Dell Inspiron 1420, and it is getting a little frail - the battery is failing and it is a security risk to any LAN. I airgapped it, and the constant copying of program SH2 files via thumbdrive is getting tiresome.

One way to link up with the PG-FP5 was an old-fashioned windows COM which do not exist anymore in the new laptops. The PG-FP5 version I have does not work with USB serial port dongles, but they have a USB port. The idea is to passthrough an entire USB controller to my Windows XP qemu Virtual Machine.

I picked up some generic USB3 PCI Express controller(Top: Via VL805. Bottom: NEC uPD720200):

With the NEC uPD720200 adapter installed, just do the usual (see previous blog post):

root@crosshair:/home/heong$lspci | grep USB

00:12.0 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0

 USB OHCI0 Controller

00:12.2 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0

 USB EHCI Controller

00:13.0 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0

 USB OHCI0 Controller

00:13.2 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0

 USB EHCI Controller

00:14.5 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0

 USB OHCI2 Controller

00:16.0 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0

 USB OHCI0 Controller

00:16.2 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0

 USB EHCI Controller

04:00.0 USB controller: NEC Corporation uPD720200 USB 3.0 Host Controller (rev 03)

05:00.0 USB controller: NEC Corporation uPD720200 USB 3.0 Host Controller (rev 03)

There are 2 identical NEC controllers here.

root@crosshair:/home/heong$lspci -nn | grep 05:00

05:00.0 USB controller [0c03]: NEC Corporation uPD720200 USB 3.0 Host Controller

 [1033:0194] (rev 03)

root@crosshair:/home/heong$lspci -nn | grep 04:00

04:00.0 USB controller [0c03]: NEC Corporation uPD720200 USB 3.0 Host Controller

 [1033:0194] (rev 03)

Indeed, the Crosshair IV Formula has a builtin USB3 controller. This complicates things a little. We need to passthrough the correct controller. I happened to have a Sandisk thumbdrive handy and with it plugged into the new PCI card, we do:

dmesg -T | tail

And we get something like:

[Mon Jun 26 15:28:08 2017] usb 10-2: new high-speed USB device number 2 using xh

ci_hcd

[Mon Jun 26 15:28:08 2017] usb 10-2: New USB device found, idVendor=0781, idProd

uct=5572

[Mon Jun 26 15:28:08 2017] usb 10-2: New USB device strings: Mfr=1, Product=2, S

erialNumber=3

[Mon Jun 26 15:28:08 2017] usb 10-2: Product: Cruzer Switch

[Mon Jun 26 15:28:08 2017] usb 10-2: Manufacturer: SanDisk

[Mon Jun 26 15:28:08 2017] usb 10-2: SerialNumber: 4C532000020924119064

[Mon Jun 26 15:28:08 2017] usb-storage 10-2:1.0: USB Mass Storage device detected

[Mon Jun 26 15:28:08 2017] scsi host11: usb-storage 10-2:1.0

[Mon Jun 26 15:28:09 2017] scsi 11:0:0:0: Direct-Access     SanDisk  Cruzer Switch    1.20 PQ: 0 ANSI: 5

[Mon Jun 26 15:28:09 2017] sd 11:0:0:0: [sdg] 15633408 512-byte logical blocks:

(8.00 GB/7.45 GiB)

[Mon Jun 26 15:28:09 2017] sd 11:0:0:0: [sdg] Write Protect is off

[Mon Jun 26 15:28:09 2017] sd 11:0:0:0: [sdg] Mode Sense: 43 00 00 00

[Mon Jun 26 15:28:09 2017] sd 11:0:0:0: [sdg] Write cache: disabled, read cache:

 enabled, doesn't support DPO or FUA

[Mon Jun 26 15:28:09 2017]  sdg: sdg1

[Mon Jun 26 15:28:09 2017] sd 11:0:0:0: [sdg] Attached SCSI removable disk

We now know that it is in USB device 10-2. But which PCI device it it in?

root@crosshair:/home/heong$lsusb -v 2>/dev/null | grep '^Bus\|iSerial'

Bus 003 Device 002: ID 058f:6362 Alcor Micro Corp. Flash Card Reader/Writer

  iSerial                 3 058F312D81B

Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

  iSerial                 1 0000:00:16.2

Bus 007 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub

  iSerial                 1 0000:00:16.0

Bus 006 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub

  iSerial                 1 0000:00:14.5

Bus 002 Device 002: ID 0781:5572 SanDisk Corp.

  iSerial                 3 4C532000020924119064

Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

  iSerial                 1 0000:00:13.2

Bus 005 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub

  iSerial                 1 0000:00:13.0

Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

  iSerial                 1 0000:00:12.2

Bus 004 Device 002: ID 046d:c247 Logitech, Inc.

  iSerial                 0

Bus 004 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub

  iSerial                 1 0000:00:12.0

Bus 011 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub

  iSerial                 1 0000:04:00.0

Bus 010 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

  iSerial                 1 0000:04:00.0

Bus 009 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub

  iSerial                 1 0000:05:00.0

Bus 008 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

  iSerial                 1 0000:05:00.0

That would be Bus 010 0000:04:00.0. Now we do the usual

root@crosshair:/home/heong$lspci -nn | grep 04:00

04:00.0 USB controller [0c03]: NEC Corporation uPD720200 USB 3.0 Host Controller

 [1033:0194] (rev 03)

root@crosshair:/home/heong$modprobe vfio

root@crosshair:/home/heong$modprobe -v vfio_iommu_type1

root@crosshair:/home/heong$modprobe -v vfio_pci

insmod /lib/modules/4.11.2/kernel/drivers/vfio/vfio_virqfd.ko

insmod /lib/modules/4.11.2/kernel/drivers/vfio/pci/vfio-pci.ko

root@crosshair:/home/heong$echo '0000:04:00.0' >  /sys/bus/pci/devices/0000:04:0

0.0/driver/unbind

root@crosshair:/home/heong$echo '0x1033 0x0194' > /sys/bus/pci/drivers/vfio-pci/

new_id

The half-size DVD-ROM was a problem for my vertically mounted drive, so I ripped it in my laptop's drive thus:

root@aspireF15:/home/heong/ECI$dd if=/dev/dvd of=usb3pcie_nec.iso

And copied the resulting ISO file into the VM host, the Crosshair Formula IV.

Next you make a fake DVD for the qemu VM:

root@crosshair:/home/heong/ECI$losetup /dev/loop0 ./usb3pcie_nec.iso

Your qemu command should be something like:

qemu-system-x86_64 -cpu host,kvm=off -enable-kvm -m 4G -balloon none -bios ./seabios

/out/bios.bin -hda XP-12GB.img -rtc clock=host,base=localtime -device vfio-pci,host=04:00.0 -cdrom /dev/loop0

You install the Windows XP USB3 driver from the DVD-ROM and it should work with no trouble. Indeed I installed the USB drivers for the Renesas PG-FP5 and that worked first time too.

That is the sanitized version, of course. In the real world, rather than buying the NEC PCI Express card, I tried passing through the onboard NEC USB3 controller, 05:00.0. The Windows XP driver installed correctly and the VM image rebooted successfully, but it would not recognize devices plugged into it.

Next I ran to the nearest computer store and bought the one PCI Express USB3 card they have on hand, the Via VL805. This went a lot worse- the Windows XP driver locked up the VM during installation, and the VM locked up on all subsequent reboots.

With some care the next PCI Express USB3 controller I chose had a different chipset, the NEC  uPD720200 with the happy results detailed earlier.

So if at first you don't succeed, try again ... Happy Trails.