IOMMU PCI Express USB3 Controller Passthrough

One of the best things about Virtual Machines for me is I can bake in support for legacy systems like Windows XP. I needed to support the Renesas SH2 microcontroller and needed the Windows XP version of their PG-FP5 programmer. I currently use a Dell Inspiron 1420, and it is getting a little frail - the battery is failing and it is a security risk to any LAN. I airgapped it, and the constant copying of program SH2 files via thumbdrive is getting tiresome.

One way to link up with the PG-FP5 was an old-fashioned windows COM which do not exist anymore in the new laptops. The PG-FP5 version I have does not work with USB serial port dongles, but they have a USB port. The idea is to passthrough an entire USB controller to my Windows XP qemu Virtual Machine.

I picked up some generic USB3 PCI Express controller(Top: Via VL805. Bottom: NEC uPD720200):

With the NEC uPD720200 adapter installed, just do the usual (see previous blog post):

root@crosshair:/home/heong$lspci | grep USB

00:12.0 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0

 USB OHCI0 Controller

00:12.2 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0

 USB EHCI Controller

00:13.0 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0

 USB OHCI0 Controller

00:13.2 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0

 USB EHCI Controller

00:14.5 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0

 USB OHCI2 Controller

00:16.0 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0

 USB OHCI0 Controller

00:16.2 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0

 USB EHCI Controller

04:00.0 USB controller: NEC Corporation uPD720200 USB 3.0 Host Controller (rev 03)

05:00.0 USB controller: NEC Corporation uPD720200 USB 3.0 Host Controller (rev 03)

There are 2 identical NEC controllers here.

root@crosshair:/home/heong$lspci -nn | grep 05:00

05:00.0 USB controller [0c03]: NEC Corporation uPD720200 USB 3.0 Host Controller

 [1033:0194] (rev 03)

root@crosshair:/home/heong$lspci -nn | grep 04:00

04:00.0 USB controller [0c03]: NEC Corporation uPD720200 USB 3.0 Host Controller

 [1033:0194] (rev 03)

Indeed, the Crosshair IV Formula has a builtin USB3 controller. This complicates things a little. We need to passthrough the correct controller. I happened to have a Sandisk thumbdrive handy and with it plugged into the new PCI card, we do:

dmesg -T | tail

And we get something like:

[Mon Jun 26 15:28:08 2017] usb 10-2: new high-speed USB device number 2 using xh

ci_hcd

[Mon Jun 26 15:28:08 2017] usb 10-2: New USB device found, idVendor=0781, idProd

uct=5572

[Mon Jun 26 15:28:08 2017] usb 10-2: New USB device strings: Mfr=1, Product=2, S

erialNumber=3

[Mon Jun 26 15:28:08 2017] usb 10-2: Product: Cruzer Switch

[Mon Jun 26 15:28:08 2017] usb 10-2: Manufacturer: SanDisk

[Mon Jun 26 15:28:08 2017] usb 10-2: SerialNumber: 4C532000020924119064

[Mon Jun 26 15:28:08 2017] usb-storage 10-2:1.0: USB Mass Storage device detected

[Mon Jun 26 15:28:08 2017] scsi host11: usb-storage 10-2:1.0

[Mon Jun 26 15:28:09 2017] scsi 11:0:0:0: Direct-Access     SanDisk  Cruzer Switch    1.20 PQ: 0 ANSI: 5

[Mon Jun 26 15:28:09 2017] sd 11:0:0:0: [sdg] 15633408 512-byte logical blocks:

(8.00 GB/7.45 GiB)

[Mon Jun 26 15:28:09 2017] sd 11:0:0:0: [sdg] Write Protect is off

[Mon Jun 26 15:28:09 2017] sd 11:0:0:0: [sdg] Mode Sense: 43 00 00 00

[Mon Jun 26 15:28:09 2017] sd 11:0:0:0: [sdg] Write cache: disabled, read cache:

 enabled, doesn't support DPO or FUA

[Mon Jun 26 15:28:09 2017]  sdg: sdg1

[Mon Jun 26 15:28:09 2017] sd 11:0:0:0: [sdg] Attached SCSI removable disk

We now know that it is in USB device 10-2. But which PCI device it it in?

root@crosshair:/home/heong$lsusb -v 2>/dev/null | grep '^Bus\|iSerial'

Bus 003 Device 002: ID 058f:6362 Alcor Micro Corp. Flash Card Reader/Writer

  iSerial                 3 058F312D81B

Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

  iSerial                 1 0000:00:16.2

Bus 007 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub

  iSerial                 1 0000:00:16.0

Bus 006 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub

  iSerial                 1 0000:00:14.5

Bus 002 Device 002: ID 0781:5572 SanDisk Corp.

  iSerial                 3 4C532000020924119064

Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

  iSerial                 1 0000:00:13.2

Bus 005 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub

  iSerial                 1 0000:00:13.0

Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

  iSerial                 1 0000:00:12.2

Bus 004 Device 002: ID 046d:c247 Logitech, Inc.

  iSerial                 0

Bus 004 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub

  iSerial                 1 0000:00:12.0

Bus 011 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub

  iSerial                 1 0000:04:00.0

Bus 010 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

  iSerial                 1 0000:04:00.0

Bus 009 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub

  iSerial                 1 0000:05:00.0

Bus 008 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

  iSerial                 1 0000:05:00.0

That would be Bus 010 0000:04:00.0. Now we do the usual

root@crosshair:/home/heong$lspci -nn | grep 04:00

04:00.0 USB controller [0c03]: NEC Corporation uPD720200 USB 3.0 Host Controller

 [1033:0194] (rev 03)

root@crosshair:/home/heong$modprobe vfio

root@crosshair:/home/heong$modprobe -v vfio_iommu_type1

root@crosshair:/home/heong$modprobe -v vfio_pci

insmod /lib/modules/4.11.2/kernel/drivers/vfio/vfio_virqfd.ko

insmod /lib/modules/4.11.2/kernel/drivers/vfio/pci/vfio-pci.ko

root@crosshair:/home/heong$echo '0000:04:00.0' >  /sys/bus/pci/devices/0000:04:0

0.0/driver/unbind

root@crosshair:/home/heong$echo '0x1033 0x0194' > /sys/bus/pci/drivers/vfio-pci/

new_id

The half-size DVD-ROM was a problem for my vertically mounted drive, so I ripped it in my laptop's drive thus:

root@aspireF15:/home/heong/ECI$dd if=/dev/dvd of=usb3pcie_nec.iso

And copied the resulting ISO file into the VM host, the Crosshair Formula IV.

Next you make a fake DVD for the qemu VM:

root@crosshair:/home/heong/ECI$losetup /dev/loop0 ./usb3pcie_nec.iso

Your qemu command should be something like:

qemu-system-x86_64 -cpu host,kvm=off -enable-kvm -m 4G -balloon none -bios ./seabios

/out/bios.bin -hda XP-12GB.img -rtc clock=host,base=localtime -device vfio-pci,host=04:00.0 -cdrom /dev/loop0

You install the Windows XP USB3 driver from the DVD-ROM and it should work with no trouble. Indeed I installed the USB drivers for the Renesas PG-FP5 and that worked first time too.

That is the sanitized version, of course. In the real world, rather than buying the NEC PCI Express card, I tried passing through the onboard NEC USB3 controller, 05:00.0. The Windows XP driver installed correctly and the VM image rebooted successfully, but it would not recognize devices plugged into it.

Next I ran to the nearest computer store and bought the one PCI Express USB3 card they have on hand, the Via VL805. This went a lot worse- the Windows XP driver locked up the VM during installation, and the VM locked up on all subsequent reboots.

With some care the next PCI Express USB3 controller I chose had a different chipset, the NEC  uPD720200 with the happy results detailed earlier.

So if at first you don't succeed, try again ... Happy Trails.

Internet of Things, Legacy Industrial Applications, and PCI Passthrough

It's been a while since the last blog- been busy with a huge order of over 200 escalators for the above-ground stations of the Klang Valley Mass Rapid Transit (KV-MRT).

Building Management System

Elevator & Escalator Monitoring & Control System in Visual Basic

BMS, EMS, Modbus (and other acronyms)

The new twist to an otherwise standard escalator is the requirement for Modbus input and output. The intention is to connect  them to a Building Management System via BACNet. Because all 31 stations are linked, an operator can start and stop the escalators remotely, generally not a good idea from a safety point of view. Still, those are the customer's requirements, and with a proper standard operating procedure this allows for very quick mass evacuations.

Our existing Escalator Management System runs on the 16-year old Windows XP and worse, on Visual Basic. BMS support was via a proprietary MS Elevator protocol. At design stage back in 2011, given the delivery deadline, and the fact that all new designs and development systems run on Linux, I added a Modbus-MS protocol translator Windows program, written in python. This linked to the BMS via rs-485, as it was risky to connect Windows XP to a WAN.

It is now clear that this is still a security risk as nearly everyone now have a smartphone with a mobile dataline and it is straightforward to connect the Windows XP system to WAN with a USB cable. Indeed the rail operator staff have begun to do that, mainly to recharge their phones.

Both Windows XP and Visual Basic have been orphaned by their creator, Microsoft. To harden it, we can install an up-to-date Linux distribution and run Windows XP and the EMS application in a virtual machine. This hardens it to the extent a CAT6 Ethernet Modbus link is possible. The disadvantage is IO throughput now drops some 30%, as the virtual machine is an additional layer of code to run through.

Native Virtualization, IOMMU and PCI Passthrough

Back in 2011 AMD published a specification for IOMMU which a virtual machine like Qemu-KVM can exploit to improve IO throughput, together with hardware-assisted(native) virtualization. We use PCI-SIG Single Root I/O Virtualization, or PCI passthrough. It did not work out for us back then, but from 2016 or so PC Gamers have been reporting success with VGA passthrough so it is time to revisit it.

The Rig

Native Virtualization, and IOMMU in particular require a very specific subset of hardware. I used my old 2011 rig, an Asus Crosshair Formula IV, with an AMD Phenom II X4 945 and 16GB RAM. The VGA cards are an old (2011) Radeon HD 5550 and a new Nvidia Gigabyte Geforce GT710. The rs485 link was through a Moxa CP114EL. Only the PCI Express slots were used. The Nvidia card was used by the Linux host while the old Radeon card was passed through to the Windows XP guest. This is because there were no XP drivers for the newer card.

The Code

I used Linux Slackware 14.2, with the kernel upgraded to 4.11.2. Qemu-KVM version is 2.9.0. You will need a pre-installed Qemu image of Windows. You will also need seabios. I used David Yates' excellent guide. If you do not have a Windows image, David Yates' guide has the procedure to generate one.

root@crosshair:/home/heong/ECI$lspci | grep VGA

03:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Redwood PRO [Radeon HD 5550/5570/5630/6510/6610/7570]

08:00.0 VGA compatible controller: NVIDIA Corporation Device 128b (rev a1)

root@crosshair:/home/heong/ECI$lspci -nn | grep 03:00

03:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Redwood PRO [Radeon HD 5550/5570/5630/6510/6610/7570] [1002:68d9]

03:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Redwood HDMI Audio [Radeon HD 5000 Series] [1002:aa60]

root@crosshair:/home/heong/ECI$lspci  | grep Moxa

02:00.0 Serial controller: Moxa Technologies Co Ltd CP-114EL (4-port RS-232/422/485 Smart PCI Express Serial Board)

root@crosshair:/home/heong/ECI$lspci  -nn | grep 02:00

02:00.0 Serial controller [0700]: Moxa Technologies Co Ltd CP-114EL (4-port RS-232/422/485 Smart PCI Express Serial Board) [1393:1144]

Unlike David, I use the lilo bootloader, and its configuration file /etc/lilo.conf needs to be changed to:

append=" vt.default_utf8=0 quiet splash modprobe.blacklist=radeon"

Update the boot track using liloconfig and re-use the existing /etc/lilo.conf.

For clarity, rather than use a bash script, I have the following set of manual commands:

modprobe -v vfio

modprobe -v vfio_iommu_type1

modprobe -v vfio_pci

And you follow that with:

echo '0000:03:00.0' >  /sys/bus/pci/devices/0000:03:00.0/driver/unbind

echo '0x1002 0x68d9' > /sys/bus/pci/drivers/vfio-pci/new_id

echo '0000:03:00.1' >  /sys/bus/pci/devices/0000:03:00.1/driver/unbind

echo '0x1002 0aa60' > /sys/bus/pci/drivers/vfio-pci/new_id

echo '0000:02:00.0' >  /sys/bus/pci/devices/0000:02:00.0/driver/unbind

echo '0x1393 0x1144' > /sys/bus/pci/drivers/vfio-pci/new_id

Note the numbers 1002:68d9,1002:aa60, and 1393:1144 are from the lspci commands earlier.

The Windows XP device driver for the Radeon card is on a DVD. The Moxa CP-114REL driver was on a different DVD, which needed to be swapped in and the following command repeated:

My Qemu-KVM command is:

qemu-system-x86_64 -cpu host,kvm=off -enable-kvm -m 4G -balloon none -bios /home

/heong/ECI/bios/seabios/out/bios.bin -hda XP-12GB.img -hdb scratch--4GB.img -cdrom /dev/dvd -device vfio-pci,host=03:00.0,multifunction=on -device vfio-pci,host=03:00.1 -rtc clock=host,base=localtime -device vfio-pci,host=02:00.0

And it worked first time, no fuss. The first thing to do is to update the wiki and check off the Asus Crosshair Formula IV :7).

PCI (VGA & Moxa Multiport) Passthrough

The IO throughput is very near bare metal and has been running 16 hours.

Internet of Things

The Linux Slackware host is also a LAMP Stack capable of supporting smartphone applications as a server. This would make the escalators part of the Internet of Things (IoT). Even the original Windows XP Modbus translator in python can be run with minimal modification from the Linux side.

Security

The Linux VM can be locked down for only https and ssh access. In particular, the virtual machine can be configured with a virtual network with to give Windows XP only ssh access and only from the Linux host. But that is the another blog post. Stay tuned!