Monday, 26 June 2017

IOMMU PCI Express USB3 Controller Passthrough


One of the best things about Virtual Machines for me is I can bake in support for legacy systems like Windows XP. I needed to support the Renesas SH2 microcontroller and needed the Windows XP version of their PG-FP5 programmer. I currently use a Dell Inspiron 1420, and it is getting a little frail - the battery is failing and it is a security risk to any LAN. I airgapped it, and the constant copying of program SH2 files via thumbdrive is getting tiresome.

One way to link up with the PG-FP5 was an old-fashioned windows COM which do not exist anymore in the new laptops. The PG-FP5 version I have does not work with USB serial port dongles, but they have a USB port. The idea is to passthrough an entire USB controller to my Windows XP qemu Virtual Machine.

I picked up some generic USB3 PCI Express controller(Top: Via VL805. Bottom: NEC uPD720200):


With the NEC uPD720200 adapter installed, just do the usual (see previous blog post):

root@crosshair:/home/heong$lspci | grep USB
00:12.0 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0
 USB OHCI0 Controller
00:12.2 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0
 USB EHCI Controller
00:13.0 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0
 USB OHCI0 Controller
00:13.2 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0
 USB EHCI Controller
00:14.5 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0
 USB OHCI2 Controller
00:16.0 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0
 USB OHCI0 Controller
00:16.2 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0
 USB EHCI Controller
04:00.0 USB controller: NEC Corporation uPD720200 USB 3.0 Host Controller (rev 03)
05:00.0 USB controller: NEC Corporation uPD720200 USB 3.0 Host Controller (rev 03)

There are 2 identical NEC controllers here.

root@crosshair:/home/heong$lspci -nn | grep 05:00
05:00.0 USB controller [0c03]: NEC Corporation uPD720200 USB 3.0 Host Controller
 [1033:0194] (rev 03)
root@crosshair:/home/heong$lspci -nn | grep 04:00
04:00.0 USB controller [0c03]: NEC Corporation uPD720200 USB 3.0 Host Controller
 [1033:0194] (rev 03)

Indeed, the Crosshair IV Formula has a builtin USB3 controller. This complicates things a little. We need to passthrough the correct controller. I happened to have a Sandisk thumbdrive handy and with it plugged into the new PCI card, we do:

dmesg -T | tail

And we get something like:

[Mon Jun 26 15:28:08 2017] usb 10-2: new high-speed USB device number 2 using xh
ci_hcd
[Mon Jun 26 15:28:08 2017] usb 10-2: New USB device found, idVendor=0781, idProd
uct=5572
[Mon Jun 26 15:28:08 2017] usb 10-2: New USB device strings: Mfr=1, Product=2, S
erialNumber=3
[Mon Jun 26 15:28:08 2017] usb 10-2: Product: Cruzer Switch
[Mon Jun 26 15:28:08 2017] usb 10-2: Manufacturer: SanDisk
[Mon Jun 26 15:28:08 2017] usb 10-2: SerialNumber: 4C532000020924119064
[Mon Jun 26 15:28:08 2017] usb-storage 10-2:1.0: USB Mass Storage device detected
[Mon Jun 26 15:28:08 2017] scsi host11: usb-storage 10-2:1.0
[Mon Jun 26 15:28:09 2017] scsi 11:0:0:0: Direct-Access     SanDisk  Cruzer Switch    1.20 PQ: 0 ANSI: 5
[Mon Jun 26 15:28:09 2017] sd 11:0:0:0: [sdg] 15633408 512-byte logical blocks:
(8.00 GB/7.45 GiB)
[Mon Jun 26 15:28:09 2017] sd 11:0:0:0: [sdg] Write Protect is off
[Mon Jun 26 15:28:09 2017] sd 11:0:0:0: [sdg] Mode Sense: 43 00 00 00
[Mon Jun 26 15:28:09 2017] sd 11:0:0:0: [sdg] Write cache: disabled, read cache:
 enabled, doesn't support DPO or FUA
[Mon Jun 26 15:28:09 2017]  sdg: sdg1
[Mon Jun 26 15:28:09 2017] sd 11:0:0:0: [sdg] Attached SCSI removable disk

We now know that it is in USB device 10-2. But which PCI device it it in?

root@crosshair:/home/heong$lsusb -v 2>/dev/null | grep '^Bus\|iSerial'
Bus 003 Device 002: ID 058f:6362 Alcor Micro Corp. Flash Card Reader/Writer
  iSerial                 3 058F312D81B
Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
  iSerial                 1 0000:00:16.2
Bus 007 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
  iSerial                 1 0000:00:16.0
Bus 006 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
  iSerial                 1 0000:00:14.5
Bus 002 Device 002: ID 0781:5572 SanDisk Corp.
  iSerial                 3 4C532000020924119064
Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
  iSerial                 1 0000:00:13.2
Bus 005 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
  iSerial                 1 0000:00:13.0
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
  iSerial                 1 0000:00:12.2
Bus 004 Device 002: ID 046d:c247 Logitech, Inc.
  iSerial                 0
Bus 004 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
  iSerial                 1 0000:00:12.0
Bus 011 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
  iSerial                 1 0000:04:00.0
Bus 010 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
  iSerial                 1 0000:04:00.0
Bus 009 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
  iSerial                 1 0000:05:00.0
Bus 008 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
  iSerial                 1 0000:05:00.0


That would be Bus 010 0000:04:00.0. Now we do the usual

root@crosshair:/home/heong$lspci -nn | grep 04:00
04:00.0 USB controller [0c03]: NEC Corporation uPD720200 USB 3.0 Host Controller
 [1033:0194] (rev 03)

root@crosshair:/home/heong$modprobe vfio
root@crosshair:/home/heong$modprobe -v vfio_iommu_type1
root@crosshair:/home/heong$modprobe -v vfio_pci
insmod /lib/modules/4.11.2/kernel/drivers/vfio/vfio_virqfd.ko
insmod /lib/modules/4.11.2/kernel/drivers/vfio/pci/vfio-pci.ko
root@crosshair:/home/heong$echo '0000:04:00.0' >  /sys/bus/pci/devices/0000:04:0
0.0/driver/unbind
root@crosshair:/home/heong$echo '0x1033 0x0194' > /sys/bus/pci/drivers/vfio-pci/
new_id

The half-size DVD-ROM was a problem for my vertically mounted drive, so I ripped it in my laptop's drive thus:
root@aspireF15:/home/heong/ECI$dd if=/dev/dvd of=usb3pcie_nec.iso

And copied the resulting ISO file into the VM host, the Crosshair Formula IV.

Next you make a fake DVD for the qemu VM:
root@crosshair:/home/heong/ECI$losetup /dev/loop0 ./usb3pcie_nec.iso

Your qemu command should be something like:
qemu-system-x86_64 -cpu host,kvm=off -enable-kvm -m 4G -balloon none -bios ./seabios
/out/bios.bin -hda XP-12GB.img -rtc clock=host,base=localtime -device vfio-pci,host=04:00.0 -cdrom /dev/loop0

You install the Windows XP USB3 driver from the DVD-ROM and it should work with no trouble. Indeed I installed the USB drivers for the Renesas PG-FP5 and that worked first time too.



That is the sanitized version, of course. In the real world, rather than buying the NEC PCI Express card, I tried passing through the onboard NEC USB3 controller, 05:00.0. The Windows XP driver installed correctly and the VM image rebooted successfully, but it would not recognize devices plugged into it.

Next I ran to the nearest computer store and bought the one PCI Express USB3 card they have on hand, the Via VL805. This went a lot worse- the Windows XP driver locked up the VM during installation, and the VM locked up on all subsequent reboots.

With some care the next PCI Express USB3 controller I chose had a different chipset, the NEC  uPD720200 with the happy results detailed earlier.

So if at first you don't succeed, try again ... Happy Trails.


2 comments:

  1. Hi i know it's a year since you posted this but i am having big problems trying to pass through a usb3 controller to Windows guest VM's with KVM/QEMU. The controller dies after so long and i have to reboot for it to detect devices again. I tried a Renesas, Via and ASMEDIA USB3 controller. All have similar issues. When i turn off IOMMU completely, the problem goes away but i obviously lose the ability to pass through the controller. Did you have any problems like this?

    ReplyDelete
  2. Sorry for my late reply. No I did not encounter a problem with the USB3 controller, and I did run the VM continously for a few weeks as I was testing an industrial-grade SCADA program on it. On and off, I used the USB3 setup to program my Renesas SuperH PCB. I have been told IOMMU is very finicky about its hardware, hence I used that ancient Crosshair IV. The NEC USB3 controller was the 4th one I tried before I got a good stable result, I blogged about it here; good luck: https://cmheong.blogspot.com/2017/06/internet-of-things-legacy-industrial.html

    ReplyDelete