Internet of Things, Legacy Industrial Applications, and PCI Passthrough

It's been a while since the last blog- been busy with a huge order of over 200 escalators for the above-ground stations of the Klang Valley Mass Rapid Transit (KV-MRT).

Building Management System

Elevator & Escalator Monitoring & Control System in Visual Basic

BMS, EMS, Modbus (and other acronyms)

The new twist to an otherwise standard escalator is the requirement for Modbus input and output. The intention is to connect  them to a Building Management System via BACNet. Because all 31 stations are linked, an operator can start and stop the escalators remotely, generally not a good idea from a safety point of view. Still, those are the customer's requirements, and with a proper standard operating procedure this allows for very quick mass evacuations.

Our existing Escalator Management System runs on the 16-year old Windows XP and worse, on Visual Basic. BMS support was via a proprietary MS Elevator protocol. At design stage back in 2011, given the delivery deadline, and the fact that all new designs and development systems run on Linux, I added a Modbus-MS protocol translator Windows program, written in python. This linked to the BMS via rs-485, as it was risky to connect Windows XP to a WAN.

It is now clear that this is still a security risk as nearly everyone now have a smartphone with a mobile dataline and it is straightforward to connect the Windows XP system to WAN with a USB cable. Indeed the rail operator staff have begun to do that, mainly to recharge their phones.

Both Windows XP and Visual Basic have been orphaned by their creator, Microsoft. To harden it, we can install an up-to-date Linux distribution and run Windows XP and the EMS application in a virtual machine. This hardens it to the extent a CAT6 Ethernet Modbus link is possible. The disadvantage is IO throughput now drops some 30%, as the virtual machine is an additional layer of code to run through.

Native Virtualization, IOMMU and PCI Passthrough

Back in 2011 AMD published a specification for IOMMU which a virtual machine like Qemu-KVM can exploit to improve IO throughput, together with hardware-assisted(native) virtualization. We use PCI-SIG Single Root I/O Virtualization, or PCI passthrough. It did not work out for us back then, but from 2016 or so PC Gamers have been reporting success with VGA passthrough so it is time to revisit it.

The Rig

Native Virtualization, and IOMMU in particular require a very specific subset of hardware. I used my old 2011 rig, an Asus Crosshair Formula IV, with an AMD Phenom II X4 945 and 16GB RAM. The VGA cards are an old (2011) Radeon HD 5550 and a new Nvidia Gigabyte Geforce GT710. The rs485 link was through a Moxa CP114EL. Only the PCI Express slots were used. The Nvidia card was used by the Linux host while the old Radeon card was passed through to the Windows XP guest. This is because there were no XP drivers for the newer card.

The Code

I used Linux Slackware 14.2, with the kernel upgraded to 4.11.2. Qemu-KVM version is 2.9.0. You will need a pre-installed Qemu image of Windows. You will also need seabios. I used David Yates' excellent guide. If you do not have a Windows image, David Yates' guide has the procedure to generate one.

root@crosshair:/home/heong/ECI$lspci | grep VGA

03:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Redwood PRO [Radeon HD 5550/5570/5630/6510/6610/7570]

08:00.0 VGA compatible controller: NVIDIA Corporation Device 128b (rev a1)

root@crosshair:/home/heong/ECI$lspci -nn | grep 03:00

03:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Redwood PRO [Radeon HD 5550/5570/5630/6510/6610/7570] [1002:68d9]

03:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Redwood HDMI Audio [Radeon HD 5000 Series] [1002:aa60]

root@crosshair:/home/heong/ECI$lspci  | grep Moxa

02:00.0 Serial controller: Moxa Technologies Co Ltd CP-114EL (4-port RS-232/422/485 Smart PCI Express Serial Board)

root@crosshair:/home/heong/ECI$lspci  -nn | grep 02:00

02:00.0 Serial controller [0700]: Moxa Technologies Co Ltd CP-114EL (4-port RS-232/422/485 Smart PCI Express Serial Board) [1393:1144]

Unlike David, I use the lilo bootloader, and its configuration file /etc/lilo.conf needs to be changed to:

append=" vt.default_utf8=0 quiet splash modprobe.blacklist=radeon"

Update the boot track using liloconfig and re-use the existing /etc/lilo.conf.

For clarity, rather than use a bash script, I have the following set of manual commands:

modprobe -v vfio

modprobe -v vfio_iommu_type1

modprobe -v vfio_pci

And you follow that with:

echo '0000:03:00.0' >  /sys/bus/pci/devices/0000:03:00.0/driver/unbind

echo '0x1002 0x68d9' > /sys/bus/pci/drivers/vfio-pci/new_id

echo '0000:03:00.1' >  /sys/bus/pci/devices/0000:03:00.1/driver/unbind

echo '0x1002 0aa60' > /sys/bus/pci/drivers/vfio-pci/new_id

echo '0000:02:00.0' >  /sys/bus/pci/devices/0000:02:00.0/driver/unbind

echo '0x1393 0x1144' > /sys/bus/pci/drivers/vfio-pci/new_id

Note the numbers 1002:68d9,1002:aa60, and 1393:1144 are from the lspci commands earlier.

The Windows XP device driver for the Radeon card is on a DVD. The Moxa CP-114REL driver was on a different DVD, which needed to be swapped in and the following command repeated:

My Qemu-KVM command is:

qemu-system-x86_64 -cpu host,kvm=off -enable-kvm -m 4G -balloon none -bios /home

/heong/ECI/bios/seabios/out/bios.bin -hda XP-12GB.img -hdb scratch--4GB.img -cdrom /dev/dvd -device vfio-pci,host=03:00.0,multifunction=on -device vfio-pci,host=03:00.1 -rtc clock=host,base=localtime -device vfio-pci,host=02:00.0

And it worked first time, no fuss. The first thing to do is to update the wiki and check off the Asus Crosshair Formula IV :7).

PCI (VGA & Moxa Multiport) Passthrough

The IO throughput is very near bare metal and has been running 16 hours.

Internet of Things

The Linux Slackware host is also a LAMP Stack capable of supporting smartphone applications as a server. This would make the escalators part of the Internet of Things (IoT). Even the original Windows XP Modbus translator in python can be run with minimal modification from the Linux side.

Security

The Linux VM can be locked down for only https and ssh access. In particular, the virtual machine can be configured with a virtual network with to give Windows XP only ssh access and only from the Linux host. But that is the another blog post. Stay tuned!