USB over IP: How to remote-access your USB devices over the network

 

First appearing in 2005 in Takahiro Hirofuchi's kickass paper, there are much better guides on USB/IP, like Linux Magazine's, Ridgerun's, etc. From 2009 USB/IP was accepted into mainline Linux kernel and up-to-date documentation on it tends to get lost in the Linux haystack. The wrinkle here is USB/IP running on Slackware 14.2-current as of July 2019. Unlike Debian/Ubuntu you cannot usually seamlessly install USB/IP (or most other things) on Slackware. A little DIY is in order.

I know that is cold comfort compared to Debian's 'apt install'. Indeed, Slackware too has its (a little rickety) SlackBuild scripts. But in USB/IP case there is no SlackBuild as it was already included in the kernel. Well, after a fashion.

But take heart: in 30 years of Slackware I seldom fail to install the things I want. This is a chance to poke a little into the innards of Linux. You have the source code in all its glory. And it is built beautifully - I say this as one who has seen Microsoft Windows NT source code [shudder]. 

And the experience will prove useful in the rare instances Debian's installs fail, like the time I installed the Java development kit and could no longer log into my desktop. This means when things fail the same methods work: paste the error into google and look for the patches and workarounds. But I digress; back to USB/IP.  

I should test my Acer aspireM3's USB webcam before the install breaks things:

# mplayer tv:// -tv driver=v4l2:width=640:height=480 -vo xv -tv device=/dev/video1

Let's verify that my Slackware kernel actuall has USB/IP:

root@aspireM3:~# ls -lR /usr/src/linux/ | grep -i usbip
-rw-r--r-- 1 root root  1172 Jan 26  2019 sysfs-platform-usbip-vudc
-rw-r--r-- 1 root root 23365 Jan 26  2019 usbip_protocol.txt

drwxr-xr-x 2 root root  4096 Jan 26  2019 usbip/
/usr/src/linux/drivers/usb/usbip:
-rw-r--r-- 1 root root 18694 Jan 26  2019 usbip_common.c
-rw-r--r-- 1 root root 10117 Jan 26  2019 usbip_common.h
-rw-r--r-- 1 root root  3969 Jan 26  2019 usbip_event.c
drwxr-xr-x  3 root root   4096 Jan 27  2019 usbip/

Looks good. A look at the kernel's config file shows that the binary has not been left out at compile time:

root@aspireM3:~# grep -i usbip /usr/src/linux/.config

CONFIG_USBIP_CORE=m

CONFIG_USBIP_VHCI_HCD=m

CONFIG_USBIP_VHCI_HC_PORTS=8

CONFIG_USBIP_VHCI_NR_HCS=1

CONFIG_USBIP_HOST=m

# CONFIG_USBIP_DEBUG is not set

And finally, locate the kernel modules (ie device drivers) themselves:

# ls -l /lib/modules/4.19.18/kernel/drivers/usb/usbip

total 128

-rw-r--r-- 1 root root 22576 Jan 27  2019 usbip-core.ko

-rw-r--r-- 1 root root 42480 Jan 27  2019 usbip-host.ko

-rw-r--r-- 1 root root 58016 Jan 27  2019 vhci-hcd.ko

And it runs OK:

# modprobe -v usbip-core

insmod /lib/modules/4.19.18/kernel/drivers/usb/usbip/usbip-core.ko

Let's launch the USB/IP server daemon:

# usbipd -D

-su: usbipd: command not found

Oops. The userspace binaries are not installed. Let's see if my Slackware installation has the userspace source code:

# ls -l /usr/src/linux/tools/usb/usbip/src

total 96

-rw-r--r-- 1 root root   440 Jan 26  2019 Makefile.am

-rw-r--r-- 1 root root  4406 Jan 26  2019 usbip.c

-rw-r--r-- 1 root root  1292 Jan 26  2019 usbip.h

-rw-r--r-- 1 root root  5404 Jan 26  2019 usbip_attach.c

-rw-r--r-- 1 root root  5169 Jan 26  2019 usbip_bind.c

-rw-r--r-- 1 root root  2922 Jan 26  2019 usbip_detach.c

-rw-r--r-- 1 root root  9713 Jan 26  2019 usbip_list.c

-rw-r--r-- 1 root root  6179 Jan 26  2019 usbip_network.c

-rw-r--r-- 1 root root  5295 Jan 26  2019 usbip_network.h

-rw-r--r-- 1 root root  1536 Jan 26  2019 usbip_port.c

-rw-r--r-- 1 root root  3494 Jan 26  2019 usbip_unbind.c

-rw-r--r-- 1 root root 15083 Jan 26  2019 usbipd.c

-rw-r--r-- 1 root root  1630 Jan 26  2019 utils.c

-rw-r--r-- 1 root root   863 Jan 26  2019 utils.h

Yes, I do. I just need to compile it but it fails:

/usr/src/linux/tools/usb/usbip# ./autogen.sh

/usr/src/linux/tools/usb/usbip# ./configure

/usr/src/linux/tools/usb/usbip# make

make  all-recursive

make[1]: Entering directory '/usr/src/linux-4.19.18/tools/usb/usbip'

Making all in libsrc

make[2]: Entering directory '/usr/src/linux-4.19.18/tools/usb/usbip/libsrc'

  CC       libusbip_la-usbip_device_driver.lo

usbip_device_driver.c: In function �..read_usb_vudc_device�..:

usbip_device_driver.c:106:2: error: �..strncpy�.. specified bound 256 equals destination size [Werror=stringop-truncation]

  strncpy(dev->path, path, SYSFS_PATH_MAX);

  ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

usbip_device_driver.c:125:2: error: �..strncpy�.. specified bound 32 equals dest

ination size [-Werror=stringop-truncation]

  strncpy(dev->busid, name, SYSFS_BUS_ID_SIZE);

  ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

cc1: all warnings being treated as errors

make[2]: *** [Makefile:471: libusbip_la-usbip_device_driver.lo] Error 1

make[2]: Leaving directory '/usr/src/linux-4.19.18/tools/usb/usbip/libsrc'

make[1]: *** [Makefile:497: all-recursive] Error 1

make[1]: Leaving directory '/usr/src/linux-4.19.18/tools/usb/usbip'

make: *** [Makefile:365: all] Error 2

It does not look serious. What used to be a compiler warning has now been classified as an error so the compiler obligingly stops. A quick google comes up with a patch:

https://patchwork.kernel.org/project/linux-usb/patch/20180721021232.GR14131@deca

dent.org.uk/

> +++ b/tools/usb/usbip/libsrc/usbip_common.c

> @@ -226,8 +226,8 @@ int read_usb_device(struct udev_device *

>       path = udev_device_get_syspath(sdev);

>       name = udev_device_get_sysname(sdev);

>

> -     strncpy(udev->path,  path,  SYSFS_PATH_MAX);

> -     strncpy(udev->busid, name, SYSFS_BUS_ID_SIZE);

> +     snprintf(udev->path, SYSFS_PATH_MAX, "%s", path);

> +     snprintf(udev->busid, SYSFS_BUS_ID_SIZE, "%s", name);

So I just edit usbip_device_driver.c:

# vi libsrc/usbip_device_driver.c

 And at line 106

        strncpy(dev->path, path, SYSFS_PATH_MAX);

 Is changed to

        // strncpy(dev->path, path, SYSFS_PATH_MAX); cmheong 2021-03-27

        snprintf(dev->path, SYSFS_PATH_MAX, "%s", path);

At line 125:

strncpy(dev->busid, name, SYSFS_BUS_ID_SIZE);

           Became

snprintf(dev->busid, SYSFS_BUS_ID_SIZE, "%s", name);

Anow for the file usbip_common.c line 230:

        // strncpy(udev->path,  path,  SYSFS_PATH_MAX); // cmheong 2021-03-27

        // strncpy(udev->busid, name, SYSFS_BUS_ID_SIZE);

Becomes

        snprintf(udev->path, SYSFS_PATH_MAX, "%s", path);

        snprintf(udev->busid, SYSFS_BUS_ID_SIZE, "%s", name);

Now some Slackware versions, depending on the date of your install, you might get an additional error:

  CC       usbip_network.o

usbip_network.c: In function âusbip_net_pack_usb_deviceâ:

usbip_network.c:91:32: error: taking address of packed member of âstruct usbip_

usb_deviceâ may result in an unaligned pointer value [-Werror=address-of-packed

-member]

   91 |  usbip_net_pack_uint32_t(pack, &udev->busnum);

      |                                ^~~~~~~~~~~~~

We just need to tell the compiler not to freak out and treat the warning as an error. This can be done by changing line 12174 of the 'configure' file:

/usr/src/linux/tools/usb/usbip$vi configure

EXTRA_CFLAGS="-Wall -Wno-error=address-of-packed-member -Werror -Wextra -std=gnu

99"

After which the compile completes successfully:

# make

make  all-recursive

make[1]: Entering directory '/usr/src/linux-4.19.18/tools/usb/usbip'

Making all in libsrc

make[2]: Entering directory '/usr/src/linux-4.19.18/tools/usb/usbip/libsrc'

  CC       libusbip_la-usbip_common.lo

  CC       libusbip_la-usbip_host_common.lo

  CC       libusbip_la-vhci_driver.lo

  CC       libusbip_la-sysfs_utils.lo

  CCLD     libusbip.la

make[2]: Leaving directory '/usr/src/linux-4.19.18/tools/usb/usbip/libsrc'

Making all in src

make[2]: Entering directory '/usr/src/linux-4.19.18/tools/usb/usbip/src'

  CC       usbip.o

  CC       utils.o

  CC       usbip_network.o

  CC       usbip_attach.o

  CC       usbip_detach.o

  CC       usbip_list.o

  CC       usbip_bind.o

  CC       usbip_unbind.o

  CC       usbip_port.o

  CCLD     usbip

  CC       usbipd.o

  CCLD     usbipd

make[2]: Leaving directory '/usr/src/linux-4.19.18/tools/usb/usbip/src'

make[2]: Entering directory '/usr/src/linux-4.19.18/tools/usb/usbip'

make[2]: Leaving directory '/usr/src/linux-4.19.18/tools/usb/usbip'

make[1]: Leaving directory '/usr/src/linux-4.19.18/tools/usb/usbip'

Now the server daemon runs:

# /usr/src/linux/tools/usb/usbip/src/usbipd -D

A quick check:

/usr/src/linux/tools/usb/usbip# ps -ef | grep -e usbip

root     23597     2  0 14:57 ?        00:00:00 [usbip_event]

root     23608     1  0 19:53 ?        00:00:00 /usr/src/linux/tools/usb/usbip/src/.libs/lt-usbipd -D

root     23835  1533  0 19:54 pts/0    00:00:00 grep -e usbip

Next we list the USB devices available to the server:

/usr/src/linux/tools/usb/usbip# /usr/src/linux/tools/usb/usbip/src/usbip list --local

 - busid 1-3.1 (0835:8501)

   Action Star Enterprise Co., Ltd : unknown product (0835:8501)

 - busid 1-3.2 (046d:c077)

   Logitech, Inc. : M105 Optical Mouse (046d:c077)

 - busid 1-3.3 (13ba:0017)

   PCPlay : PS/2 Keyboard+Mouse Adapter (13ba:0017)

 - busid 1-3.4 (10c4:ea60)

   Cygnal Integrated Products, Inc. : CP2102/CP2109 UART Bridge Controller [CP21

0x family] (10c4:ea60)

 - busid 1-3.5.1 (0835:8502)

   Action Star Enterprise Co., Ltd : unknown product (0835:8502)

 - busid 1-3.5.2 (046d:0829)

   Logitech, Inc. : unknown product (046d:0829)

 - busid 1-3.5.4 (0c45:62f1)

   Microdia : unknown product (0c45:62f1)

 - busid 2-1.3 (04f2:b300)

   Chicony Electronics Co., Ltd : unknown product (04f2:b300)

 - busid 2-1.4 (04ca:3006)

   Lite-On Technology Corp. : unknown product (04ca:3006)

Using the displayed busis, we use 'lsusb -v' to zero in on the webcam:

Bus 002 Device 003: ID 04f2:b300 Chicony Electronics Co., Ltd

  iManufacturer           1 Chicony Electronics Co., Ltd.

  iProduct                2 HD WebCam

  iSerial                 3 SN0001

 - busid 2-1.3 (04f2:b300)

Now I need to launch another kernel module:

# modprobe -v usbip-host

insmod /lib/modules/4.19.18/kernel/drivers/usb/usbip/usbip-host.ko

We now have the USB/IP server grab the webcam:

/usr/src/linux/tools/usb/usbip# /usr/src/linux/tools/usb/usbip/src/usbip bind --busid=2-1.3

usbip: info: bind device on busid 2-1.3: complete

Now we move over to the other (ie client) computer, and repeat the process of installing USB/IP. We then launch the USB/IP kernel client modules:

/usr/src/linux/tools/usb/usbip$modprobe -v vhci-hcd

insmod /lib/modules/4.19.62/kernel/drivers/usb/usbip/usbip-core.ko

insmod /lib/modules/4.19.62/kernel/drivers/usb/usbip/vhci-hcd.ko

Let's say the server's IP address is 12.34.56.78. We can see what USB device on the server is available over IP:

/usr/src/linux/tools/usb/usbip$src/usbip list --remote=12.34.56.78

Exportable USB devices

======================

 - 12.34.56.78

      2-1.3: Chicony Electronics Co., Ltd : unknown product (04f2:b300)

           : /sys/devices/pci0000:00/0000:00:1a.0/usb2/2-1/2-1.3

           : Miscellaneous Device / ? / Interface Association (ef/02/01). 

 

Ah it is a webcam ;7) We attach to it:

/usr/src/linux/tools/usb/usbip# /usr/src/linux/tools/usb/usbip/src/usbip attach -debug --remote=12.34.56.78 --busid=2-1.3

And we play the remote (ie server) webcam by using mplayer:

/usr/src/linux/tools/usb/usbip$ mplayer -cache 128 -tv device=/dev/video2:driver=v4l2:width=640:height=480:outfmt=i420 -vo xv tv://

Note that here I access /dev/video2, since my client laptop also has a webcam at /dev/video1, and the USB/IP attach made a new one at /dev/video2.  Also the server webcam resolution is 1080 x 720 but that seemed to mave maxed out my old 100Mbps fiber link. Dropping the resolution down to 640 x 480 worked out well.

You can see your attached USB device using:

/usr/src/linux/tools/usb/usbip$src/usbip port --remote=12.34.56.78

Imported USB devices

====================

Port 00: <Port in Use> at High Speed(480Mbps)

       Chicony Electronics Co., Ltd : unknown product (04f2:b300)

       3-1 -> usbip://192.168.1.4:3240/3-1.3

           -> remote bus/dev 003/003

Notice the port number 00. We will be needing it. Lastly once you are through with the webcam you release it so another computer can use it:

/usr/src/linux/tools/usb/usbip$src/usbip detach --port=00

usbip: info: Port 0 is now detached!

There you have it. USB/IP comes standard with Linux, and lets you access USB devices remotely. I had great fun spending an afternoon doing this to the boogie blues. Here's a video of Daisy tapping her feet to it:

Click on picture for the video

Happy Trails.

Fiber optics for the Home Network

 

“We cannot live only for ourselves. A thousand fibres connect us with our fellow men; and among those fibres, as sympathetic threads, our actions run as causes, and they come back to us as effects. ” – Henry Melville

Fiber optics networking, is normally expensive and fragile. Telecoms-grade equipment come to mind. Maybe we even have a broadband fiber to the house (helpfully called FTTH). This usually ends in a telecoms-supplied box, Passive Optical Network (PON) into which we plug out usual copper (ie RJ45 UTP) LAN cable.

Fiber broadband usually ends in copper LAN connection

But why would I even want fiber for home LAN? Regular readers will know my house is on a hill which regularly gets struck by lightning. We get used to being off-grid for the duration of the storm, which happily is not usually long. But it would be nice not to have damaged electronics. Even better if we can cut over to UPS and keep watching IP TV or youtube. If my copper LAN cable runs are too long (maybe 30m) lightning often damages the network switches, or even fuse the UTP connectors together.

A Huawei ONT 'fiber modem' commonly supplied with Unifi fiber broadband

Or you might want more reliable and secure links for your security cameras/CCTVs which are often outdoors and at the end of long cable runs. Maybe you want to share your neighbor's broadband connection. 

Or maybe you simply want to speed up/secure that wireless WiFi repeater for when you are at one end of the garden. High speed WiFi is well and good, but once your neighbors have theirs installed the airways can get pretty crowded. 

Fiber LAN often means using telecoms equipment which are not only expensive but often not available to the general public. First, the fiber optic cable. Your best chance would be to use the type that your local telecoms monopoly/behemoth uses. High manufacturing volumes usually mean lower prices. Here in Malaysia it is  G.657 Class A single mode fiber.  A 1000m roll of outdoor cable costs less than RM200 (USD50) and even RM100 (USD25) if you are willing to order from mainland China. That is comparable to a 300m roll of copper Cat 5 UTP LAN cable. 

1km roll of G.657 Class A single mode fiber

But fiber cable is more fragile? Yes, if you used the equivalent indoor drop cable. The outdoor cable is often extremely strong. Mine consisted of not one but 3 steel cables reinforcing the fiber cable. I have had tree branches pulling it almost to the ground and the fiber core remained unbroken. They are often stronger than the copper UTP cables.

Outdoor fiber cables are often extremely strong

The outdoor fiber cable is far more heavy but they are still smaller than the Cat 5 or 6 copper cable. It is surprisingly bendable, considering fiber optics is a glass. And since there is no ohmic contact, you can run the cable parallel to the mains cables, in the same cable trays or conduits. This greatly reduces cabling costs.

Most indoor fiber is limp and frail

OK, but what about the fiber optic interface to the computer? A single mode single core fiber cable specifications are something like this:

The last line reads 850/1300nm: it carries just 2 light frequencies. If you need more channels you need to run another cable. The traditional multi-mode cables carries lots of frequencies, but with a price tag to match.

Poor cousin: single-mode versus multi-mode fiber

We will be needing something to convert between UTP and fiber: two fiber modems, one at each end. As usual the Chinese have something cheap and cheerful (only RM25/USD5) called a media converter: the HTB-3100.

The HTB-3100A and HTB-3100B are sometimes sold as a matched  pair

Now this being a Chinese no-name box, be careful to look for a media converter that has only one fiber SC UPC port. The picture above shows two (marked TX and RX) but only the TX port can be uncovered. If you, like me, happen to buy the dual-port box by mistake you will need to lay 2 fiber cables for every copper UTP connection. You want to look for WDM (Wavelength Division Multiplexing). To carry 2 channels in one fiber, it transmits in one wavelength, 1550nm and receives at another, 1310nm. That is Type A. Type B is just the reverse, transmitting at 1310nm and receiving at 1550nm. You will be needing one unit of each type. They are often sold in matched pairs.

Both the single fiber and dual fiber media converters might be labelled as Half/Full Duplex. My guess is this refers to the UTP (ie RJ45 or copper) end. These days most CAT5 or CAT6 copper LAN cables come with both TX and RX pairs, and Half Duplex might be when Ethernet is negotiated down to CSMA/CD. 

Last but not least there is the complex matter of cutting, splicing and terminating your fiber optic cable. Years ago, it took expensive equipment, highly-trained operators and extremely clean conditions, which are sometimes difficult to do on-site. But for now there is and end-run, a workaround. Again from the Chinese. You can buy the cables already terminated for very low prices. Like RM36 for 50m. That is just USD9.

Pre-terminated outdoor single mode fiber cable.

Some suppliers will do it to a custom length. Just make sure the connector is SC UPC (it is easy to specify the incompatible SC APC or LC connectors). Now I had coax 10Base2 concealed LAN wiring installed in my house (yes, yes I am a dinosaur), so it was an easy matter to rip it out of the conduits and install the smaller fiber cable in it place.

The aim is to replace your long copper cable runs with fiber. One benefit is if the cable run is over 100m you do not need to install the repeaters (ie LAN switches) that UTP Ethernet needs. If you add in the cost of the power wiring, enclosures, the savings quickly pile up.

The aim is to replace long copper LAN cable runs with fiber (in red)

It worked so well I ran another 100m fiber cable outdoors to my home office so I can share the broadband. The system has been in place through several violent thunderstorms and one fallen tree and did not miss a beat.

If you live in Malaysia, you are in luck, for Talikom Malaysia, the telecoms monopoly mostly uses sub-contractors to install your FTTH fiber cable. For a very reasonable fee, not exceeding the cost of a roll of fiber cable, they can be persuaded to do your internal house fiber cabling. In my case it was money well spent for much of the work involved climbing onto the roof. A huge advantage is they will splice and terminate your fiber cable using proper equipment, resulting in a very good connection.

The only thing left would be an inexpensive way to cut, join and terminate a fiber cable myself. But that is for another post.

Happy Trails. 

 
 

RESTful IoT with privacy & security: How to set up a Debian HTTPS Server

 

"It is quiet here and restful, and the air is delicious. There are gardens everywhere and police spies lie in the bushes ... " - Maxim Gorky

It is very tempting to use the ubiquitous HTTP web protocol for IoT. It is easy to test, and lets you operate your IoT from smartphones, tablets, desktops and even a computer program. Such a setup is called RESTful. It simply means your IoT device speaks the language of the web browser and web server. 

So we rush ahead with our RESTful API, and the IoT device is soon working and indispensable. Pretty soon we realize we need security and privacy: it won't do to have a hacker open the voice-controlled garage door ...

Our first line of defense is our WiFi password. It is reasonable to assume those living in the house should have access to WiFi and IoT. But what if we had guests or lodgers? Changing WiFi passwords can be a real bear, especially if you have 20-odd IoT devices. In fact it makes sense to localize the changes to a dedicated IoT server. RESTful, naturally.  

We will be needing some form of authentication: account names and passwords should do for now. Next we will need a reasonable amount of privacy, i.e., encryption so that someone else should not be able to lift the IoT password off the WiFi. That means HTTPS, or HTTP with SSL.

We start by implementing HTTPS server on a Linux system. The ESP8266 is known to be a little wobbly running HTTPS. ESP32 is better, but we can do without the complication for now. The traditional way is to use Apache. There are other, easier ways (like nginx, nodejs and even python) but Apache lets you run multiple servers right off the bat. This means you can keep your bad old HTTP server, add another HTTPS server on top of that and lets you support both your HTTP and HTTPS IoT devices.

From a bog-standard Debian (mine is a Beaglebone on eMMC), do the usual:

# apt-get update

# apt-get upgrade

Next, get Apache:

# apt install apache2

And while you are at it, you might as well make sure you have ssh. I got my DNS from duckdns.

Apache should come up complete with the stock webpage at http://localhost. Put your webserver files at /var/www/html/

To access the webserver from outside your WiFi access point, you will need a DNS server, but once you get it organized, a bog standard browser will display a warning before it will display your home page:

That means you need SSL, which usually costs money. You can opt for a self-signed certificate but this will produce a warning with most browsers. You then elect to disregard the warning and proceed, but this is a real problem if you are trying to sell the IoT device.

One way out is to get a 90-day certificate free from sslforfree. You just have to register, input your domain name and prove that you have access to the webserver, usually by uploading an sslforfree file to it. After it checks out the certificates can be downloaded. sslforfree links to a youtube video describing the process.

Do check out the video. I will simply list the differences relevant to a Debian installation. In Debian it is a simple:

# a2enmod ssl

Considering dependency setenvif for ssl:

Module setenvif already enabled

Considering dependency mime for ssl:

Module mime already enabled

Considering dependency socache_shmcb for ssl:

Enabling module socache_shmcb.

Enabling module ssl.

See /usr/share/doc/apache2/README.Debian.gz on how to configure SSL and create s

elf-signed certificates.

To activate the new configuration, you need to run:

  systemctl restart apache2

I now need a configuration file for my HTTPS (or SSL) webserver. There is a template in Debian:

# cp /etc/apache2/sites-available/default-ssl.conf /etc/apache2/sites-available/secure.cmheong.duckdns.org.conf

secure.cmheong.duckdns.org being the domain name of my new HTTPS server. The parameters for the new server are put in:

# cat /etc/apache2/sites-available/secure.cmheong.duckdns.org.conf | head -n 16

<IfModule mod_ssl.c>

        <VirtualHost _default_:443>

                ServerName secure.cmheong.duckdns.org

                ServerAlias www.secure.cmheong.duckdns.org

                ServerAdmin webmaster@secure.cmheong.duckdns.org

                DocumentRoot /var/www/html

                # Available loglevels: trace8, ..., trace1, debug, info, notice, warn,

                # error, crit, alert, emerg.

                # It is also possible to configure the loglevel for particular

                # modules, e.g.

                #LogLevel info ssl:warn

                ErrorLog ${APACHE_LOG_DIR}/secure.cmheong.error.log

                CustomLog ${APACHE_LOG_DIR}/secure.cmheong.access.log combined

You then check your configuration and do not proceed further until this passes:

# apachectl configtest

Syntax OK

Make sure your webserver is now accessible from the Internet. Usually this means setting up Port Forwarding in your gateway to forward all port 443 traffic to your server IP address.

# systemctl restart apache2

You will then need to prepare for the sslforfree test of webserver and domain name ownership:

# mkdir /var/www/html/.well-known

# mkdir /var/www/html/.well-known/pki-validation

Register with sslforfree, download the challenge file they provided and put it in the new directory. This is where the Debian ssh installation comes in handy. 

If the sslforfree challenge succeeds, then the certificates and private key will be generated as a zip file.

# unzip secure.cmheong.duckdns.org.zip

Archive:  secure.cmheong.duckdns.org.zip

 extracting: certificate.crt

 extracting: ca_bundle.crt

 extracting: private.key

You then move them to their final secure directories:

# cp -v ./sslforfree/*.crt  /etc/ssl/certs

'./sslforfree/ca_bundle.crt' -> 'certs/ca_bundle.crt'

'./sslforfree/certificate.crt' -> 'certs/certificate.crt'

# cp  ./sslforfree/private.key  /etc/ssl/private/private.key

Remember to delete the ./sslforfree directory. If you want to put the certificates in a different place you will need to update the site config file accordingly:

# cat /etc/apache2/sites-available/secure.cmheong.duckdns.org.conf | grep -i SSLCerti

                #   SSLCertificateFile directive is needed.

                #SSLCertificateFile     /etc/ssl/certs/ssl-cert-snakeoil.pem

                #SSLCertificateKeyFile /etc/ssl/private/ssl-cert-snakeoil.key

                SSLCertificateFile      /etc/ssl/certs/certificate.crt

                SSLCertificateKeyFile /etc/ssl/private/private.key

                SSLCertificateChainFile /etc/ssl/certs/ca_bundle.crt

As usual test the Apache configuration:

# apachectl configtest

And then restart Apache:

# systemctl restart apache2

The just aim your Chrome browser at https://www.yoursecureserver.com. If it worked you get something like this:

 

Now the traffic to and from the IoT RESTful server is encrypted. Note the sslforfree certificates expire in 90 days, but you are free to generate a new set. They will even email you a reminder. 

There you have it: a secure Internet-facing RESTful IoT server.

Happy Trails.

AS3935: The Next Generation Episode 3

 

"The drama's done. Why then here does any one step forth? — Because one did survive the wreck" - Herman Melville, 'Moby Dick'

It's been a strange dry season - there wasn't one. Instead of the choking haze from the fires of Indonesia, August and September brought only rain and thunderstorms. The AS3935 survived them all, until now. As usual the AS3935 detector flagged the oncoming storm. Since the lightning appears to come from the phone copper lines, esp8266 relay modules were used to disconnect the phone line from the ADSL modem. The modem is powered from a float-charged NS60 12V automotive battery. The battery charger, too was disconnected from the mains, both Live and Neutral, so as to deny the strike a path to Earth.

In order to reconnect after the storm an esp8266 4-channel module itself is powered from the same battery, but a separate 2-channel esp8266 relay module is used to disconnect the ADSL phone lines. The ADSL relay module itself is also disconnected from its 5V power supply, as previous strikes tended to jump from the relay contacts to the 5V/GND lines powering the relay coils.

The esp8266 and as3935 modules are wirelessly linked via MQTT server. 

On Thursday Sep 24, a nearby strike came in via the ADSL line, and flashed over the open relay contacts. This destroyed the 2-channel relay module (including the attached esp8266) and propagated along the 5V power line over to the 4-channel relay contacts. There it flashed over the open relay contacts to the 12V-5V buck converter and damaged it. This resulted in a short-circuit across the 12V battery and also damaged the 13.8V CC CV module used to float-charge the battery. There is no direct connection to Live, Neutral or Earth (the 4-channel relays were off).

The 4-channel relay is the only component along the strike path which survived.

From right: lightning strike along the ADSL line flashed over the relay contacts to the 5V power lines. Insulating tape was used to reduce severity of flashover

Note the burn mark along the 5V power PCB trace 

Burned area: relay coil 5V pin near relay contact connected the surge to the 5V power system and spread via the LM1117 to the ESP8266 via the 3.3V output. It also propagated to the  12V-5V buck converter

12V-5V buck converter had shorted output schottky diode SS34

13.8V CC CV float charge module takes unregulated 18Vdc and puts out 13V8 constant current constant voltage. Its output schottky diode SS54 was shorted, effectively shorting the battery

Sole survivor: 4-channel relay module used to disconnect power to battery charger and to ADSL relay module

The lightning strike must have been really close because it also took out my autogate controller and damaged lights a few houses down the street. Unlike Captain Ahab, this is where I concede defeat. After 25 years, the lightning won and I disconnected the ADSL system. 

Henceforth the future is fiber-optic. In fact it was operating concurrently with the ADSL modem when the lightning struck and it survived unscathed. At least I had Netflix.

Happy Trails.

"And I only am escaped alone to tell thee." -- Job

AS3935: The Next Generation Episode 2

Hors de combat: AMS1117-3.3 LDO with crater and ejecta

In Episode 1, I isolated the modem ADSL disconnect relay module, and this setup survived about three months of storms, until this week. The system worked pretty well: the remote AS3935 disconnected the ADSL line from the modem a good 20 minutes earlier, so the problem area was limited to just the relay board. Instead of blowing everything up like my first rig.

Setup at time of strikes

Relay board with ESP-01S dismounted. No obvious burn marks this time.

However the ADSL line is still connected to the relay board pins, a lightning strike first took out the 12V to 5V DC-DC buck converter that served as the power module for the relay board. This caused the output to short-circuit to the input: 12V from the battery now appeared at the relay board power input.

Relay board after the second strike. Note the insulation tape over the relay pins

Now the AMS1117-3.3 LDO power regulator in the relay board is rated for 15V and while it crashed the ESP8266 CPU it probably  did not kill it. It did cause the 5V relays to run really hot. This set it up for the second close strike, just minutes later, which blew a neat little hole in the AMS1117-3.3.

It is very impressive what damage lightning can do. But there seems to be progress. The AS3935 lightning detector allowed an early modem disconnect, which probably saved the modem. And this time round there seems to be a lot less damage, probably because the lightning could not find an easy path to earth. The earlier 'disconnect' signal from the AS3935 module caused a second relay board to disconnect the battery charger from the mains. At the time of the second strike the system had been running off the 12V battery.

Since the strike also killed the 12V-5V DC buck converter powering the relay board from the battery, perhaps disconnecting the ADSL relay board from the buck connector might improve things. It does not really need to be powered up: I can use the relay 'Normally Open' pins to ensure the ADSL stayed disconnected. The relay board's 5V is really close to the ADSL pins as it needed to power the relay coil.

And I can make things a little more robust by using a 12V relay board to disconnect the ADSL relay board and to monitor for the storm's passing so that the modem can be reconnected when it is safe.

Diymore 4-channel WiFi Relay board with serial interface

And as a bonus the module PCB designer has helpfully cut a slot around the vulnerable relay output COMMON pin.

Note the 'U'-shaped slots cut around the relay COMMON pin

The power regulator is still an AMS1117-3.3 but now there is an added 78M05 linear regulator in series. This raises the maximum input voltage (ie the tolerance to surges) to 35V, from the AMS1117-3.5's 15V. Hopefully this is enough, but we are never really sure until the next lightning strike demolishes it. The relay coils are still exposed to the full weight of the surges but they are relatively tougher devices.

The setup is now thus:

System block diagram: the other 2 relays in the 4-channel module are used to disconnect the battery charger from the mains

Next Generation Episode 2 build: from top: 2-channel ADSL relay module, ADSL modem and 4-channel power disconnect module

Thus fortified, we await the next thunderstorm. Indeed I welcome the strikes, for it is looking like my 30-year struggle with lightning strikes may be coming to a close: fiber-optic broadband network has reached my front gate, and the service provider salesman will not be long after. Just when I felt like I am winning.

Fiber at the gates: the end for copper-based ADSL broadband is nigh

What is Ahab without his Moby Dick? Come, lightning and welcome. 

Happy Trails.

Ahab and his whale

Raspberry Pi 4 Voice Assistant Part 2 of 4: Google Text to Speech

gTTS: The Empire Strikes Back

In Part 1, one of my goals was to have my laptop issue voice commands to to my Google Home smart speaker. After trying out Mycroft, Jasper seems like the logical next step. The other text to speech systems voice quality were something like the Texas Instruments Speak & Spell products: especially ESP32Talkie. Even Mycroft sounded a bit sad next to my Home Mini speaker.

And then I stumbled upon gTTS, Google Text to Speech.  This python interface to Google's text to speech can be installed using:

pip install gTTS

And requires a program, of only ten lines:

from gtts import gTTS

def text_to_speech(input_name, output_name, language):

    file = open(input_name, 'r')

    content = file.read()

    file.close()

    sound = gTTS(text=content, lang=language)

    sound.save(output_name + '.mp3')

    #https://pypi.org/project/gTTS/

text_to_speech('input_en.txt', 'sound_en', 'en')

#text_to_speech('input_tr.txt', 'sound_tr', 'tr')

print('Done!')

You put your text in a file, input_en.txt:

$cat ./input_en.txt

Hey, Google

$python tts.py

And you get back an mp3 file, sound_en.mp3

And all you need to do now to trigger the Google Home smart speaker is:

$mplayer sound_en.mp3

A typical complete command would be something like:

$mplayer HeyGoogle.mp3; sleep 1; mplayer OfficeLampOn.mp3

For some reason, the other trigger phrase 'OK Google' did not work, but for very little effort I can now integrate disparate IoT devices, be they home brewed, Alexa, or Google Home into one Voice Assistant that rules them all.

Here's what it sounds like:

Happy Trails.

Luke: Vader... Is the dark side stronger?

Yoda: No, no, no. Quicker, easier, more seductive.

Geekworm Raspberry Pi UPS Hat

I did not really have much faith in the Geekworm Raspberry Pi UPS Hat. It cost just RM48 and change and it came with a 3.7V 2700mAh lithium battery. The software links did not work (the correct link is here) and the driver looked out of date.

Plus there has been some rather harsh comments about it. Some like this one, seems reasonable. Now I am not saying they are wrong, but that Geekworm Raspberry Pi UPS Hat worked for me, much to my surprise.

It is quite well described here and I will not repeat the information. I originally had the idea of using a store-bought power bank for the very same purpose. I thought if I ran the power bank with the charger always connected and the raspberry Pi always drawing power, it should work like a cheap UPS. Not bad for an RM30 no-name 3000mAh power bank.

Power bank as UPS. Note the Qi receiver coil

I found one while walking my dogs. It's cover was cracked open and it looked like it had been thrown out of a car, but when it was dried out it worked. The battery looked intact, did not overheat and was not bulging.

But sadly it did not work consistently. When the mains charger was powered off it ran from battery well enough. The problem was when the charger was switched on, it sometimes rebooted. Looks like the output power was not quite stable enough during the switchover. A common enough problem with regular UPS.

Now I can still use it as a UPS for systems that can tolerate a reboot. In some cases I simply put a little restart code in /etc/rc.local. And I can't really complain about the price. Do be careful of discarded lithium batteries though. They have been known to explode, or burn white-hot.

But a real UPS would switch over and back without a glitch. And it would be nice to have an indication of the battery state. The Geekworm UPS actually looked like (I am not sure; I did not check) it was adapted from a power bank circuit  and it would be good so see where I fell short.

Having heard some comments about bad batteries, miswired batteries and faulty UPS boards, I checked both the battery output and the polarity before I hooked it up. It all checked out and powered on a Raspberry Pi 3 Model B+ quite nicely while still on charge.

Geekworm UPS Hat installed

It ran nicely on battery, and did not reset when the charger is reconnected. Even when running X Windows, Chrome and a youtube video at 1900x1080 resolution.

The extremely brief 'manual' (http://raspberrypiwiki.com/File:UserManual.pdf) called for installation of a driver:

User Guide:
1. Upgrade software:
sudo apt-get update
sudo apt-get upgrade
2. Enable the I2C function via raspi-config tool.
3. Install wiringPi .
git clone git://git.drogon.net/wiringPi
cd wiringPi
git pull origin
cd wiringPi
./build
4. Download the zip package;?rpi-ups-hat.zip?
unzip rpi-ups-hat.zip
cd rpi-ups-hat
5. Run the tested program
sudo python example.py

But there is also a C program, which I reproduce in full here:

$cat main.c

#include <unistd.h>                     // close read write

#include <stdio.h>                      // printf

#include <fcntl.h>                      // open

#include <linux/i2c-dev.h>

#include <sys/ioctl.h>

#include <getopt.h>

#define VREG 2

#define CREG 4

#define BUFSIZE 16

#define DEV "/dev/i2c-1"

#define ADRS 0x36

static int readReg(int busfd, __uint16_t reg, unsigned char *buf, int bufsize)

{

    unsigned char reg_buf[2];

    reg_buf[0] = (reg >> 0) & 0xFF;

    reg_buf[1] = (reg >> 8) & 0xFF;

    int ret = write(busfd, reg_buf, 2);

    if (ret < 0) {

        printf("Write failed trying to read reg: %04x (0x%02x 0x%02x)\n", reg, r

eg_buf[0], reg_buf[1], reg);

        return ret;

    }

    return read(busfd, buf, bufsize);

}

int main(int argc, char **argv)

{

    int vOpt = 0, cOpt = 0, o;

    while ((o = getopt (argc, argv, "vc")) != -1) {

        switch (o)

        {

        case 'v':

            vOpt = 1;

            break;

        case 'c':

            cOpt = 1;

            break;

        }

    }

    int bus = 1;

    unsigned char buf[BUFSIZE] = {0};

    int busfd;

    if ((busfd = open(DEV, O_RDWR)) < 0) {

        printf("can't open %s (running as root?)\n",DEV);

        return(-1);

    }

    int ret = ioctl(busfd, I2C_SLAVE, ADRS);

    if (ret < 0)

        printf("i2c device initialisation failed\n");

    if (ret < 0) return(-1);

    readReg(busfd, VREG, buf, 2);

    int hi,lo;

    hi = buf[0];

    lo = buf[1];

    int v = (hi << 8)+lo;

    if (vOpt) {

                printf("%fV ",(((float)v)* 78.125 / 1000000.0));

        }

    readReg(busfd, CREG, buf, 2);

    hi = buf[0];

    lo = buf[1];

    v = (hi << 8)+lo;

    if (!cOpt && !vOpt) {

                printf("%i",(int)(((float)v) / 256.0));

        }

        if (cOpt) {

                printf("%f%%",(((float)v) / 256.0));

        }

        printf("\n");

    close(busfd);

    return 0;

}

I started with the usual:

sudo apt-get update
sudo apt-get upgrade

And used raspi-config to turn on i2c. As soon as I did that an i2c device came up:

# ls -l /dev/i2*
crw-rw---- 1 root i2c 89, 1 Apr 30 20:28 /dev/i2c-1

A quick scan of the C program main.c showed that that was all it needed to run. So, I skipped all the other steps and went straight to:

# gcc main.c -o ups_read

And it runs:
# ./ups_read
97

When I disconnected the charger and ran on battery it actually reads a little higher:
# ./ups_read
98

But it quickly starts to show a correct, discharging trend:
# ./ups_read
97

I ran it for some 6 minutes at pretty much full power and it went to 83%:
# date;./ups_read
Thu Apr 30 20:46:49 +08 2020
83

When I plugged the charger in I get the wobble:
# date;./ups_read
Thu Apr 30 20:47:27 +08 2020
82
            And yes, it charges:
# date;./ups_read
Thu Apr 30 20:47:49 +08 2020
83

# ./ups_read -vc
4.161250V 96.550781%

And that was all it took. Raspberry Pi UPS Hat on the cheap.

Happy Trails.


[Update 2020-05-06]
Managed to reproduce some of the problems mentioned above by increasing the current drawn from the UPS. Decreasing the UPS input power did not affect it much, except it increased the charging time. But, if I increased the UPS load too much (by loading up the Pi USB ports with for example a WiFi dongle at full power), a switchover from mains to battery supply now caused the Pi 1 to reset. And on reboot, the Geekworm UPS often failed to charge unless the load is power-cycled.

600mA peak current draw from the Geekworm UPS Hat was enough to trigger the problems mentioned. A 510mA peak draw was more or less OK for trouble-free operation. Treat the numbers as a guide: I only used a USB Charge Doctor to measure the peak current consumption. This almost guarantees an incorrect reading. An oscilloscope would be a better choice.  

The 4 LED do not accurately reflect the battery state of charge. The I2C value read by ups_read is much more accurate.  

AS3935 Lightning Detector: The Next Generation

“Data: My positronic brain has several layers of shielding to protect me from power surges. It would be possible for you to remove my cranial unit and take it with you.

Riker: Let me get this straight--you want me to take off your head?

Data: Yes sir”

(2020-07-19 update: see also this blog post)

Now that all my new parts have arrived, the mantra with this next iteration is isolation. The first system came to a dramatic end at the very beginning of the monsoon season. Repeated lightning strikes in the space of just a few minutes damaged every single part of the last AS3935 lightning detector. The surge nearly always ran along the phone line running into my ADSL modem.

Next Generation System is all about MQTT

First we isolate the electronics in the direct line of fire: the relay module which disconnects the ADSL line from the modem. One encouraging result is my ADSL modem survived unscathed, which is the main aim of the AS3935. The lightning surge arrived at relay COM pin and flashed over to the 5Vdc line.

ADSL Switch. Note the TTL serial lines running to the charger relay board. Hopefully the electricians' tape wrapped over the PCB traces and the relay terminals  will help resist flashovers 

And if that should come up short, we separate the relay board from the AS3935 NodeMCU ESP-12E system. Henceforth they communicate over WiFi via an MQTT server.

And where it used to share the same 5V supply as the relay board, we will now run from its own; where previously it shared the AS3935's. This still means they still share the same mains 230Vac line. And since the lightning strikes coming along the ADSL line seeks out the AC mains Live or Neutral lines, we isolate that as well: the relay 5V supply shall be derived from a 12V lead-acid battery.

From left: N40 lead-acid battery, battery charger. The white mains power cord of the charger is switched by another 2-channel relay module. Note the original charger display has been replaced by a 13.8V CV-CC DC boost-buck module

AS3935 with NodeMCU ESP-12E ESP8266 board. Note the 5V power cable is filtered by a few loops round a ferrite doughnut

The battery needs to be float-charged from a mains charger, and as an extra precaution the charger will be disconnected from the mains using a separate relay module, again cued from the MQTT server. The server will need to be run from UPS as well.

Charger mains supply cutoff relay. Note the TTL serial link from the ADSL relay module.

The AS3935 system now only publishes to the MQTT server. It is powered directly from the mains; it normally detects the storm well beforehand, and if the stroke should take out the AC mains, it does not matter for its job is done. On restoration of power it gets the last state again from the MQTT server and resumes operation.

And for debugging instead of risking a laptop running Arduino IDE, debug messages are published to the same MQTT server. More details on setting up MQTT Server in Slackware here.

The MQTT server, ADSL modem as well as the Huawei 3G Failover router all run from a UPS so on detecting an imminent a lightning strike, the ADSL line is disconnected, and IoT remote services will get maybe 2 minutes interruption  while the Huawei router switches to 3G.

As usual, the source code is in my github repository. The esp8266 code will look for an MQTT server at mqttserver.local on mDNS. Once located, they will subscribe to the topic lightning/commands and publish on lightning/messages.

An unexpected advantage is the AS3935 sensor can now be put in an electrically quieter part of the house. It seemed happiest at the back porch window, equidistant from the computers in the study and the power inverter of the refrigerator. This allowed me to increase the sensitivity setting.

For now it is set a little over-sensitive. A storm vertically overhead but too high up to trigger a strike to ground will still set it off. As does the little 50-cc 2-stroke engine from my brush-cutter. The occasional false ADSL disconnects did not bother us too much: the Huawei 3G failover system cuts over within 2 minutes, and even the IoT nodejs server does not drop its ngrok connection.

False alarm: a nearby internal combustion engine will trigger a stream of false alarms

The stage is set. It is now the inter-monsoon, and thunderstorms are beginning to come overhead, this time from the south-west. And with the Covid-19 lockdown we will be at home, for the next bout of fireworks. And all I need now is the Big Strike.

After more than 20 years, I feel a little for Captain Ahab, as played by the same excellent Patrick Stewart:

"... To the last, I grapple with thee; From Hell's heart, I stab at thee; For hate's sake, I spit my last breath at thee." - Herman Melville, Moby Dick

Happy Trails.


P.S.: A promising new relay PCB from LC Tech has slots cut around the COM pins.

LC Tech 4-Channel Relay Module

Note the slots cut around the relays' COM pin

In the last setup, the lightning struck relay COM pin, and flashed over to the 5V relay coil power line. Now the relay coil drives are optically isolated, which means the 3.3V power for the ESP8266 is better shielded from the relay contact pins.