USB over IP: How to remote-access your USB devices over the network

 

First appearing in 2005 in Takahiro Hirofuchi's kickass paper, there are much better guides on USB/IP, like Linux Magazine's, Ridgerun's, etc. From 2009 USB/IP was accepted into mainline Linux kernel and up-to-date documentation on it tends to get lost in the Linux haystack. The wrinkle here is USB/IP running on Slackware 14.2-current as of July 2019. Unlike Debian/Ubuntu you cannot usually seamlessly install USB/IP (or most other things) on Slackware. A little DIY is in order.

I know that is cold comfort compared to Debian's 'apt install'. Indeed, Slackware too has its (a little rickety) SlackBuild scripts. But in USB/IP case there is no SlackBuild as it was already included in the kernel. Well, after a fashion.

But take heart: in 30 years of Slackware I seldom fail to install the things I want. This is a chance to poke a little into the innards of Linux. You have the source code in all its glory. And it is built beautifully - I say this as one who has seen Microsoft Windows NT source code [shudder]. 

And the experience will prove useful in the rare instances Debian's installs fail, like the time I installed the Java development kit and could no longer log into my desktop. This means when things fail the same methods work: paste the error into google and look for the patches and workarounds. But I digress; back to USB/IP.  

I should test my Acer aspireM3's USB webcam before the install breaks things:

# mplayer tv:// -tv driver=v4l2:width=640:height=480 -vo xv -tv device=/dev/video1

Let's verify that my Slackware kernel actuall has USB/IP:

root@aspireM3:~# ls -lR /usr/src/linux/ | grep -i usbip
-rw-r--r-- 1 root root  1172 Jan 26  2019 sysfs-platform-usbip-vudc
-rw-r--r-- 1 root root 23365 Jan 26  2019 usbip_protocol.txt

drwxr-xr-x 2 root root  4096 Jan 26  2019 usbip/
/usr/src/linux/drivers/usb/usbip:
-rw-r--r-- 1 root root 18694 Jan 26  2019 usbip_common.c
-rw-r--r-- 1 root root 10117 Jan 26  2019 usbip_common.h
-rw-r--r-- 1 root root  3969 Jan 26  2019 usbip_event.c
drwxr-xr-x  3 root root   4096 Jan 27  2019 usbip/

Looks good. A look at the kernel's config file shows that the binary has not been left out at compile time:

root@aspireM3:~# grep -i usbip /usr/src/linux/.config

CONFIG_USBIP_CORE=m

CONFIG_USBIP_VHCI_HCD=m

CONFIG_USBIP_VHCI_HC_PORTS=8

CONFIG_USBIP_VHCI_NR_HCS=1

CONFIG_USBIP_HOST=m

# CONFIG_USBIP_DEBUG is not set

And finally, locate the kernel modules (ie device drivers) themselves:

# ls -l /lib/modules/4.19.18/kernel/drivers/usb/usbip

total 128

-rw-r--r-- 1 root root 22576 Jan 27  2019 usbip-core.ko

-rw-r--r-- 1 root root 42480 Jan 27  2019 usbip-host.ko

-rw-r--r-- 1 root root 58016 Jan 27  2019 vhci-hcd.ko

And it runs OK:

# modprobe -v usbip-core

insmod /lib/modules/4.19.18/kernel/drivers/usb/usbip/usbip-core.ko

Let's launch the USB/IP server daemon:

# usbipd -D

-su: usbipd: command not found

Oops. The userspace binaries are not installed. Let's see if my Slackware installation has the userspace source code:

# ls -l /usr/src/linux/tools/usb/usbip/src

total 96

-rw-r--r-- 1 root root   440 Jan 26  2019 Makefile.am

-rw-r--r-- 1 root root  4406 Jan 26  2019 usbip.c

-rw-r--r-- 1 root root  1292 Jan 26  2019 usbip.h

-rw-r--r-- 1 root root  5404 Jan 26  2019 usbip_attach.c

-rw-r--r-- 1 root root  5169 Jan 26  2019 usbip_bind.c

-rw-r--r-- 1 root root  2922 Jan 26  2019 usbip_detach.c

-rw-r--r-- 1 root root  9713 Jan 26  2019 usbip_list.c

-rw-r--r-- 1 root root  6179 Jan 26  2019 usbip_network.c

-rw-r--r-- 1 root root  5295 Jan 26  2019 usbip_network.h

-rw-r--r-- 1 root root  1536 Jan 26  2019 usbip_port.c

-rw-r--r-- 1 root root  3494 Jan 26  2019 usbip_unbind.c

-rw-r--r-- 1 root root 15083 Jan 26  2019 usbipd.c

-rw-r--r-- 1 root root  1630 Jan 26  2019 utils.c

-rw-r--r-- 1 root root   863 Jan 26  2019 utils.h

Yes, I do. I just need to compile it but it fails:

/usr/src/linux/tools/usb/usbip# ./autogen.sh

/usr/src/linux/tools/usb/usbip# ./configure

/usr/src/linux/tools/usb/usbip# make

make  all-recursive

make[1]: Entering directory '/usr/src/linux-4.19.18/tools/usb/usbip'

Making all in libsrc

make[2]: Entering directory '/usr/src/linux-4.19.18/tools/usb/usbip/libsrc'

  CC       libusbip_la-usbip_device_driver.lo

usbip_device_driver.c: In function �..read_usb_vudc_device�..:

usbip_device_driver.c:106:2: error: �..strncpy�.. specified bound 256 equals destination size [Werror=stringop-truncation]

  strncpy(dev->path, path, SYSFS_PATH_MAX);

  ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

usbip_device_driver.c:125:2: error: �..strncpy�.. specified bound 32 equals dest

ination size [-Werror=stringop-truncation]

  strncpy(dev->busid, name, SYSFS_BUS_ID_SIZE);

  ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

cc1: all warnings being treated as errors

make[2]: *** [Makefile:471: libusbip_la-usbip_device_driver.lo] Error 1

make[2]: Leaving directory '/usr/src/linux-4.19.18/tools/usb/usbip/libsrc'

make[1]: *** [Makefile:497: all-recursive] Error 1

make[1]: Leaving directory '/usr/src/linux-4.19.18/tools/usb/usbip'

make: *** [Makefile:365: all] Error 2

It does not look serious. What used to be a compiler warning has now been classified as an error so the compiler obligingly stops. A quick google comes up with a patch:

https://patchwork.kernel.org/project/linux-usb/patch/20180721021232.GR14131@deca

dent.org.uk/

> +++ b/tools/usb/usbip/libsrc/usbip_common.c

> @@ -226,8 +226,8 @@ int read_usb_device(struct udev_device *

>       path = udev_device_get_syspath(sdev);

>       name = udev_device_get_sysname(sdev);

>

> -     strncpy(udev->path,  path,  SYSFS_PATH_MAX);

> -     strncpy(udev->busid, name, SYSFS_BUS_ID_SIZE);

> +     snprintf(udev->path, SYSFS_PATH_MAX, "%s", path);

> +     snprintf(udev->busid, SYSFS_BUS_ID_SIZE, "%s", name);

So I just edit usbip_device_driver.c:

# vi libsrc/usbip_device_driver.c

 And at line 106

        strncpy(dev->path, path, SYSFS_PATH_MAX);

 Is changed to

        // strncpy(dev->path, path, SYSFS_PATH_MAX); cmheong 2021-03-27

        snprintf(dev->path, SYSFS_PATH_MAX, "%s", path);

At line 125:

strncpy(dev->busid, name, SYSFS_BUS_ID_SIZE);

           Became

snprintf(dev->busid, SYSFS_BUS_ID_SIZE, "%s", name);

Anow for the file usbip_common.c line 230:

        // strncpy(udev->path,  path,  SYSFS_PATH_MAX); // cmheong 2021-03-27

        // strncpy(udev->busid, name, SYSFS_BUS_ID_SIZE);

Becomes

        snprintf(udev->path, SYSFS_PATH_MAX, "%s", path);

        snprintf(udev->busid, SYSFS_BUS_ID_SIZE, "%s", name);

Now some Slackware versions, depending on the date of your install, you might get an additional error:

  CC       usbip_network.o

usbip_network.c: In function âusbip_net_pack_usb_deviceâ:

usbip_network.c:91:32: error: taking address of packed member of âstruct usbip_

usb_deviceâ may result in an unaligned pointer value [-Werror=address-of-packed

-member]

   91 |  usbip_net_pack_uint32_t(pack, &udev->busnum);

      |                                ^~~~~~~~~~~~~

We just need to tell the compiler not to freak out and treat the warning as an error. This can be done by changing line 12174 of the 'configure' file:

/usr/src/linux/tools/usb/usbip$vi configure

EXTRA_CFLAGS="-Wall -Wno-error=address-of-packed-member -Werror -Wextra -std=gnu

99"

After which the compile completes successfully:

# make

make  all-recursive

make[1]: Entering directory '/usr/src/linux-4.19.18/tools/usb/usbip'

Making all in libsrc

make[2]: Entering directory '/usr/src/linux-4.19.18/tools/usb/usbip/libsrc'

  CC       libusbip_la-usbip_common.lo

  CC       libusbip_la-usbip_host_common.lo

  CC       libusbip_la-vhci_driver.lo

  CC       libusbip_la-sysfs_utils.lo

  CCLD     libusbip.la

make[2]: Leaving directory '/usr/src/linux-4.19.18/tools/usb/usbip/libsrc'

Making all in src

make[2]: Entering directory '/usr/src/linux-4.19.18/tools/usb/usbip/src'

  CC       usbip.o

  CC       utils.o

  CC       usbip_network.o

  CC       usbip_attach.o

  CC       usbip_detach.o

  CC       usbip_list.o

  CC       usbip_bind.o

  CC       usbip_unbind.o

  CC       usbip_port.o

  CCLD     usbip

  CC       usbipd.o

  CCLD     usbipd

make[2]: Leaving directory '/usr/src/linux-4.19.18/tools/usb/usbip/src'

make[2]: Entering directory '/usr/src/linux-4.19.18/tools/usb/usbip'

make[2]: Leaving directory '/usr/src/linux-4.19.18/tools/usb/usbip'

make[1]: Leaving directory '/usr/src/linux-4.19.18/tools/usb/usbip'

Now the server daemon runs:

# /usr/src/linux/tools/usb/usbip/src/usbipd -D

A quick check:

/usr/src/linux/tools/usb/usbip# ps -ef | grep -e usbip

root     23597     2  0 14:57 ?        00:00:00 [usbip_event]

root     23608     1  0 19:53 ?        00:00:00 /usr/src/linux/tools/usb/usbip/src/.libs/lt-usbipd -D

root     23835  1533  0 19:54 pts/0    00:00:00 grep -e usbip

Next we list the USB devices available to the server:

/usr/src/linux/tools/usb/usbip# /usr/src/linux/tools/usb/usbip/src/usbip list --local

 - busid 1-3.1 (0835:8501)

   Action Star Enterprise Co., Ltd : unknown product (0835:8501)

 - busid 1-3.2 (046d:c077)

   Logitech, Inc. : M105 Optical Mouse (046d:c077)

 - busid 1-3.3 (13ba:0017)

   PCPlay : PS/2 Keyboard+Mouse Adapter (13ba:0017)

 - busid 1-3.4 (10c4:ea60)

   Cygnal Integrated Products, Inc. : CP2102/CP2109 UART Bridge Controller [CP21

0x family] (10c4:ea60)

 - busid 1-3.5.1 (0835:8502)

   Action Star Enterprise Co., Ltd : unknown product (0835:8502)

 - busid 1-3.5.2 (046d:0829)

   Logitech, Inc. : unknown product (046d:0829)

 - busid 1-3.5.4 (0c45:62f1)

   Microdia : unknown product (0c45:62f1)

 - busid 2-1.3 (04f2:b300)

   Chicony Electronics Co., Ltd : unknown product (04f2:b300)

 - busid 2-1.4 (04ca:3006)

   Lite-On Technology Corp. : unknown product (04ca:3006)

Using the displayed busis, we use 'lsusb -v' to zero in on the webcam:

Bus 002 Device 003: ID 04f2:b300 Chicony Electronics Co., Ltd

  iManufacturer           1 Chicony Electronics Co., Ltd.

  iProduct                2 HD WebCam

  iSerial                 3 SN0001

 - busid 2-1.3 (04f2:b300)

Now I need to launch another kernel module:

# modprobe -v usbip-host

insmod /lib/modules/4.19.18/kernel/drivers/usb/usbip/usbip-host.ko

We now have the USB/IP server grab the webcam:

/usr/src/linux/tools/usb/usbip# /usr/src/linux/tools/usb/usbip/src/usbip bind --busid=2-1.3

usbip: info: bind device on busid 2-1.3: complete

Now we move over to the other (ie client) computer, and repeat the process of installing USB/IP. We then launch the USB/IP kernel client modules:

/usr/src/linux/tools/usb/usbip$modprobe -v vhci-hcd

insmod /lib/modules/4.19.62/kernel/drivers/usb/usbip/usbip-core.ko

insmod /lib/modules/4.19.62/kernel/drivers/usb/usbip/vhci-hcd.ko

Let's say the server's IP address is 12.34.56.78. We can see what USB device on the server is available over IP:

/usr/src/linux/tools/usb/usbip$src/usbip list --remote=12.34.56.78

Exportable USB devices

======================

 - 12.34.56.78

      2-1.3: Chicony Electronics Co., Ltd : unknown product (04f2:b300)

           : /sys/devices/pci0000:00/0000:00:1a.0/usb2/2-1/2-1.3

           : Miscellaneous Device / ? / Interface Association (ef/02/01). 

 

Ah it is a webcam ;7) We attach to it:

/usr/src/linux/tools/usb/usbip# /usr/src/linux/tools/usb/usbip/src/usbip attach -debug --remote=12.34.56.78 --busid=2-1.3

And we play the remote (ie server) webcam by using mplayer:

/usr/src/linux/tools/usb/usbip$ mplayer -cache 128 -tv device=/dev/video2:driver=v4l2:width=640:height=480:outfmt=i420 -vo xv tv://

Note that here I access /dev/video2, since my client laptop also has a webcam at /dev/video1, and the USB/IP attach made a new one at /dev/video2.  Also the server webcam resolution is 1080 x 720 but that seemed to mave maxed out my old 100Mbps fiber link. Dropping the resolution down to 640 x 480 worked out well.

You can see your attached USB device using:

/usr/src/linux/tools/usb/usbip$src/usbip port --remote=12.34.56.78

Imported USB devices

====================

Port 00: <Port in Use> at High Speed(480Mbps)

       Chicony Electronics Co., Ltd : unknown product (04f2:b300)

       3-1 -> usbip://192.168.1.4:3240/3-1.3

           -> remote bus/dev 003/003

Notice the port number 00. We will be needing it. Lastly once you are through with the webcam you release it so another computer can use it:

/usr/src/linux/tools/usb/usbip$src/usbip detach --port=00

usbip: info: Port 0 is now detached!

There you have it. USB/IP comes standard with Linux, and lets you access USB devices remotely. I had great fun spending an afternoon doing this to the boogie blues. Here's a video of Daisy tapping her feet to it:

Click on picture for the video

Happy Trails.

Turning laptop into a keyboard: USB CDC Serial port to USB HID Keyboard Emulator

 Why on earth would I want to turn my laptop into a keyboard & monitor for another computer? It is easier to buy a usb keyboard and mobile monitor. Indeed I did exactly that. Years of dutifully humping keyboard, monitor, laptop and toolbox makes you dream of lightening the load. After all, the laptop already has a keyboard and screen; why carry more?

Most of the time I log into the onsite computers remotely via ssh (Teamviewer for Windows), but sometimes you need to debug the POST process or change the BIOS settings, before your OS is running.

Serial to USB HID Keyboard Emulator

My solution is to use a USB framegrabber for the monitor and a USB keyboard emulator. 

Watch video demo on youtube

You can of course buy a UART to USB Keyboard Converter like Inteletto's. 

Interletto RS232 to USB Keyboard cable

You can even build a superior one using the Bluefruit EZ-Key.  But the Inteletto costs USD99 and the EZ-Key (USD20) is discontinued. and I already have the parts to build my own.

Enter the Microchip PIC18F14K50. More importantly the free USB source code in MPLAB. I used my own miniaturized, code-compatible version of the Microchip Low Pin Count Development Kit. 

Microchip PIC18F14K50 Low Pin Count Development Board

Miniaturized PIC18F14K50 Low Pin Count Development Board 

The schematics is in the User Guide. Unlike the Bad Old Days there is even a Linux version of MPLAB now. My programmer is a PICkit 2.

Low Pin Count Development Board schematics

PICKit 2 Programmer (bottom)

The idea is to use two Low Pin Count Development boards connected back-to-back via Serial TTL or RS485; one programmed as USB CDC Serial Port and the other as USB HID Keyboard.

Two PIC18F14K50 boards back-to-back connected via 2-wire RS-485

You plug the serial port into your development system (ie main computer) and in a Linux system it comes up as /dev/ttyACM0. For Windows user, there is a device driver in the MPLAB library. You point minicom to /dev/ttyACM0, set the baudrate to 19200 1-stop no parity with no hardware flow control. You plug the USB 'Keyboard' into the target computer. Keys typed into minicom appears at the other end, the USB Keyboard.

Now this sounds expensive but a stripped-down and miniaturized PCB for is quite cheap to make, even in small quantities, and the PIC138F14K50 (SOIC-20 package) is only RM11 (USD2.76) at Digikey. You can also use any standard USB serial dongle, like the CH340. 

CH340 USB Serial TTL Dongle

I settled on the venerable ASCII as the serial protocol. It is easy to debug. This means minicom sends ASCII to the keyboard UART and this then needs to be translated to the corresponding USB Keyboard scancode. 

Full ASCII table

You can get the USB HID Keyboard whole nine yards from usb.org, but I used a handy summary from MightyPork. 

A small part of USB Keyboard scancodes

For example, ASCII code for 'A' is (hexadecimal) 0x41 and this is translated to 0x04. The software is a slightly modified version of Microchip's source code from the Low Pin Count Development Kit. A similar (but not identical) version is Microchip/USB/Device - HID - Keyboard/Firmware/Keyboard.c

You can download the file from my github repository. Notice my code uses a MAX323 RS485 to TTL converter IC, but this should be compatible with the schematic above. Minicom actually does not have enough keys to emulate a full-sized USB keyboard, so what you have here is a minimum subset of keystrokes needed to sucessfully change a BIOS. For example, the CapsLock key is emulated as 'Ctrl-N'. Other notable mappings:

CapsLock - Ctrl N
Ctrl-Alt-Del - Ctrl O
F1 - Ctrl Q
F2 - Ctrl R
F5 - Ctrl S
F6 - Ctrl T
F9 - Ctrl U
F10 - Ctrl V
Left Arrow - Ctrl W
Down Arrow - Ctrl X
Right Arrow - Ctrl Y
Up Arrow - Ctrl Z

If you do not have access to MPLAB there is a binary file keyboard.hex compatible with the PICKit 2.

To compile, I ran MPLAB from a qemu-kvm Virtual Machine running Windows XP. Using puTTY I sshfs into the MPLAB working directory to extract the binary file for programming.

 

$ sshfs -o reconnect -C user@12.34.56.78:c:/Program\ Files/ICW/home/user/keyboard/USB\ Device\ -\ HID\ -\ Keyboard/Firmware $HOME/pic18f14k50/keyboard/source

$ cp source/*.hex .

I use pk2cmd to program my PICkit 2:

PICkit 2 mounted on JTAG programming port

$./pk2cmd -PPIC18F14K50 -Fkeyboard.hex  -M 

PICkit 2 Program Report

7-3-2021, 14:57:12

Device Type: PIC18F14K50

Program Succeeded.

Operation Succeeded

There you have it- USB CDC Serial to USB HID Keyboard simulator. Happy Trails.

Hacking Trendnet TV-IP422WN IP Camera to use with Linux program

There are many good IP Cameras now, and I can replace them at very reasonable prices, but my 11 year old Trendnet TV-IP422WN cameras just keep running. Which surprised me considering I mounted them outdoors and the resident zebra doves use them as a convenient potty/perch.

 

Trendnet TV-IP422WN IP Camera

Resolution is only 640x480, and worse, the webpage uses ActiveX. But if I can use it from Linux then I can make my own webpage. And even consolidate several of them into one page, just like those CCTV screens.

Or perhaps a passive infrared sensor can be used to trigger a spotlight and cause the camera to capture a short video. All via MQTT. I found this really useful as it reduces the time required to view alarm footage.

 

Webpage viewed with Microsoft Internet Explorer: the live video display needs ActiveX

TV-IP422WN webpage using Google Chrome

Getting it to work in Linux bash turned out to be surprisingly easy. From zoneminder, type this into your browser (I used Chrome):

http://192.168.10.30/cgi/mjpg/mjpg.cgi

Note if you did a factory reset the default IP address is 192.168.10.30 user is admin and password is admin. You will need to change all three for security reasons.

From Linux bash I used:

$curl -m 5 -u admin:admin -o cctv_video.mpg -k http://192.168.10.30/cgi/mjpg/mjpg.cgi

Now this gives me an output file which I can view with mplayer. The '-m 5' option is used to record a 5-second video; otherwise the curl command will never exit. This is especially useful if you trigger recording with another sensor, maybe a passive infrared sensor.

If, instead of video record, you just want a live view,

$mplayer -fps 20 -demuxer lavf -user admin -passwd admin http://192.168.10.30/cgi/mjpg/mjpg.cgi

If you want a still snapshot, you aim your browser at:

http://192.168.10.30/cgi/jpg/image.cgi

Or from bash, you can use lynx:

$lynx -dump -auth=admin:admin  http://192.168.10.30/cgi/jpg/image.cgi > image.jpg

Usually, you will need different video settings for night and day. I could not find the settings I need on the Internet, but I got lucky: if I used the 'Debug' (ie F12) feature in Google Chrome, under the menu 'Network' with filter 'All', I was able to discern the url:

http://192.168.10.30/cgi/setup.cgi?page=camera

The data is in a Form posted as:

brightness=8&contrast=32&saturation=36&flicker=0&osd_enable=1

Google Chrome Debug mode for Video Settings page

My curl command for day then becomes:

$curl  -X POST -H "Content-Type: application/x-www-form-urlencoded" -u admin:admin -d "brightness=8&contrast=32&saturation=36&flicker=0&osd_enable=1" http://192.168.10.30/admin/camera.cgi

And for night:

$curl  -X POST -H "Content-Type: application/x-www-form-urlencoded" -u admin:admin -d "brightness=100&contrast=80&saturation=64&flicker=0&osd_enable=1" http://192.168.10.30/admin/camera.cgi

There is also a 'Night Mode' radio button which in bash is replicated as:

curl --connect-timeout 2 -u admin:admin -k http://192.168.10.30/admin/ptctl.cgi?night=1

Conversely 'Day' Mode is:

curl --connect-timeout 2 -u admin:admin -k http://192.168.10.30/admin/ptctl.cgi?night=0

Using the same method, the pan commands are:

$curl --connect-timeout 2 -u admin:admin -k http://192.168.10.30/admin/ptctl.cgi?move=left

$curl --connect-timeout 2 -u admin:admin -k http://192.168.10.30/admin/ptctl.cgi?move=right

Tilt commands are:

$curl --connect-timeout 2 -u admin:admin -k http://192.168.10.30/admin/ptctl.cgi?move=up

$curl --connect-timeout 2 -u admin:admin -k http://192.168.10.30/admin/ptctl.cgi?move=down

To center the camera:

$curl --connect-timeout 2 -u admin:admin -k http://192.168.10.30/admin/ptctl.cgi?move=h

If you want to set a particular camera angle at for example position 1 and name it 'Left':

http://192.168.10.30/admin/ptctl.cgi?position=0&positionname=Left

Note the position '1' is numbered as '0'. Again the data is uploaded as a Form:

$curl -u admin:admin  -H "Content-Type: application/x-www-form-urlencoded" -d "position=0&positionname=Left" -k http://192.168.10.30/admin/ptctl.cgi

Thereafter to return to your preset position at '1' you need to

$curl --connect-timeout 2 -u admin:admin -k http://192.168.10.30/admin/ptctl.cgi?move=p0

To get audio, go to Menu 'Video/Audio' then 'Video' and select '3GPP with Audio'. You can get the audio (and video) using:

mplayer rtsp://192.168.10.30/mpeg4

There you have it, Trendnet TV-IP422WN hacked. Happy Trails.

One rail to rule them all: Powering the NodeMCU ESP-12E with 12V

 

NodeMCU Motor Shield with ESP-12E Devkit and 6.8V 1W Zener diode

Sometimes it makes sense to use a single power rail, like when I was making an IoT dimmable LED lamp using the NodeMCU ESP-12E Devkit. The strip of LEDs required 12V and I was loath to use a 5V buck converter: it would just be another item to mount. So, can I run the NodeMCU Devkit on 12V?

NodeMCU ESP-12E Devkit. Note input power pin Vin at bottom left. Note this PCB version does not bring VUSB to the pins

The answer is yes. And not really. Let me explain. You can power it from the input power pins Vin and GND. Or you can use USB port. Or both even of them at once. At a pinch you can even power it from the 3.3V pin. Powering it with 12V at Vin will work, but the AMS1117-3.3 LDO regulator will heat up badly under normal operation. It might even do a thermal shutdown and cause the ESP8266 to reset.

Powering from the micro USB port is easiest. You attach a smartphone charger cable to it; there is no wiring to do and you can even use it to program the Devkit. The USB port supplies 5V, and the power rail is usually called VUSB.

The NodeMCU Devkit is usually sold as a 5V device. A quick look at the board shows an AMS1117-3.3 Low Drop-Out linear regulator is connected to Vin. It produces the 3.3V needed by the ESP-12E module. But the AMS1117 datasheet specifies a maximum input voltage of 18V:

AMS1117-3.3 maximum input voltage is 18V

Now it is possible that other components on the Devkit PCB might require Vin to be 5V. A look at the schematic is advisable. I got mine from here. The schematic names the Vin voltage rail VDD5V. The USB rail is VUSB and the 3.3V rail is VDD3V3. Happily there are only 3 components that use Vin:

NodeMCU Devkit Vin Power schematic 

Note the schematic even specifies a Vin maximum of 20V. This is because it uses an ON Semiconductor NCP1117ST33 LDO which has a maximum of 20V. Notice C7 is rated at 25V. 

VDDUSB and VDD5V are linked via the schottky diode 1N5819. The 1N5819 is not a problem; it is a 40V 1A device. The problem is it comes in a tiny SOD323 package just 1mm by 2mm in size. This is not going to dissipate much heat. Its thermal resistance is 380 degrees Celsius per Watt. Its forward voltage drop is 0.6V and if it were to carry just 500mA current the temperature would rise 114 degrees. 

So if you have both Vin and VDDUSB connected, and you did not happen to have Vin turned on, depending on your power supply, VDDUSB might supply too much current to it and burn up. It is probably safer not to use VDDUSB if Vin is much higher than 5V.

SOD323

The AMS1117-3.3 comes in a SOT-89 package. This looks a lot heftier than SOD323. From richtek, the thermal resistance is 135 degrees per Watt. Now we are likely to get better performance because the IC is soldered onto PCB copper traces which will help it dissipate heat, but from Torex this is no better than 76.9 degrees per Watt.

Now my NodeMCU Devkit running its IoT program at 80MHz will take 80mA at 3.3V. This is supplied by the AMS1117-3.3. Now if Vin is 12V, the AMS1117-3.3 is a linear (not buck) regulator, and it will have to dissipate a whopping 8.7V at 80mA or 696mW. At 76.9 degrees/W the temperature rise is 53 degrees. My ambient temperature is often 34 degrees in balmy Malaysia, so that makes 87 degrees Celsius at best.

At worst it is 128 degrees Celsius, perilously close to thermal shutdown at 150 degrees Celsius. So that was why Vin was specified as 5V by the manufacturer.

Since the issue is just heat and not voltage level, instead of using a 5V buck converter to lower my Vin, I could simply put a reverse-biased 6.8V Zener diode in series. Say a hefty 1N4736 weighing in at 1W. This will take 544mW off the AMS1117-3.3 which now should clock in at a comfy 46 degrees (ie 11.7 degrees rise).

Now many of the super-cheap NodeMCU have, shall we say, quality standards that are somewhat permissive. In my case AMS1117 was used in place of NCP1117, lowering the maximum Vin to 18V. C7 might also be derated to 6.3V. Your mileage may vary.

Also it is advisable not to debug with the USB port and 12V connected. If you have to do it connect it to your computer using a powered USB hub. That way if things goes South, you do not lose a motherboard. And if like me you develop your gizmos DevOps fashion, it is also advisable to install ArduinoOTA so you can update the ESP8266 program via WiFi. 

This removes a future temptation to stick a USB debug cable into a 12V system. If you are unlucky and the 1N4736 fails short-circuited (after all, it runs hot) there will be 12V at Vin. A thermally shut down ESP-12E often looks very much like it has faulty software! 

For my 12V LED lamp I used a NodeMCU L293 motor shield with my Devkit so I mounted the 1N4736 zener directly to the 12V and Vin terminals (see photo above). It worked well for me. After 3 hours at 33 degrees Celsius ambient, the zener diode got pretty hot; too hot to touch, but the ASM1117 was only slightly warm.

Happy Trails.

 

 

Fiber optics for the Home Network

 

“We cannot live only for ourselves. A thousand fibres connect us with our fellow men; and among those fibres, as sympathetic threads, our actions run as causes, and they come back to us as effects. ” – Henry Melville

Fiber optics networking, is normally expensive and fragile. Telecoms-grade equipment come to mind. Maybe we even have a broadband fiber to the house (helpfully called FTTH). This usually ends in a telecoms-supplied box, Passive Optical Network (PON) into which we plug out usual copper (ie RJ45 UTP) LAN cable.

Fiber broadband usually ends in copper LAN connection

But why would I even want fiber for home LAN? Regular readers will know my house is on a hill which regularly gets struck by lightning. We get used to being off-grid for the duration of the storm, which happily is not usually long. But it would be nice not to have damaged electronics. Even better if we can cut over to UPS and keep watching IP TV or youtube. If my copper LAN cable runs are too long (maybe 30m) lightning often damages the network switches, or even fuse the UTP connectors together.

A Huawei ONT 'fiber modem' commonly supplied with Unifi fiber broadband

Or you might want more reliable and secure links for your security cameras/CCTVs which are often outdoors and at the end of long cable runs. Maybe you want to share your neighbor's broadband connection. 

Or maybe you simply want to speed up/secure that wireless WiFi repeater for when you are at one end of the garden. High speed WiFi is well and good, but once your neighbors have theirs installed the airways can get pretty crowded. 

Fiber LAN often means using telecoms equipment which are not only expensive but often not available to the general public. First, the fiber optic cable. Your best chance would be to use the type that your local telecoms monopoly/behemoth uses. High manufacturing volumes usually mean lower prices. Here in Malaysia it is  G.657 Class A single mode fiber.  A 1000m roll of outdoor cable costs less than RM200 (USD50) and even RM100 (USD25) if you are willing to order from mainland China. That is comparable to a 300m roll of copper Cat 5 UTP LAN cable. 

1km roll of G.657 Class A single mode fiber

But fiber cable is more fragile? Yes, if you used the equivalent indoor drop cable. The outdoor cable is often extremely strong. Mine consisted of not one but 3 steel cables reinforcing the fiber cable. I have had tree branches pulling it almost to the ground and the fiber core remained unbroken. They are often stronger than the copper UTP cables.

Outdoor fiber cables are often extremely strong

The outdoor fiber cable is far more heavy but they are still smaller than the Cat 5 or 6 copper cable. It is surprisingly bendable, considering fiber optics is a glass. And since there is no ohmic contact, you can run the cable parallel to the mains cables, in the same cable trays or conduits. This greatly reduces cabling costs.

Most indoor fiber is limp and frail

OK, but what about the fiber optic interface to the computer? A single mode single core fiber cable specifications are something like this:

The last line reads 850/1300nm: it carries just 2 light frequencies. If you need more channels you need to run another cable. The traditional multi-mode cables carries lots of frequencies, but with a price tag to match.

Poor cousin: single-mode versus multi-mode fiber

We will be needing something to convert between UTP and fiber: two fiber modems, one at each end. As usual the Chinese have something cheap and cheerful (only RM25/USD5) called a media converter: the HTB-3100.

The HTB-3100A and HTB-3100B are sometimes sold as a matched  pair

Now this being a Chinese no-name box, be careful to look for a media converter that has only one fiber SC UPC port. The picture above shows two (marked TX and RX) but only the TX port can be uncovered. If you, like me, happen to buy the dual-port box by mistake you will need to lay 2 fiber cables for every copper UTP connection. You want to look for WDM (Wavelength Division Multiplexing). To carry 2 channels in one fiber, it transmits in one wavelength, 1550nm and receives at another, 1310nm. That is Type A. Type B is just the reverse, transmitting at 1310nm and receiving at 1550nm. You will be needing one unit of each type. They are often sold in matched pairs.

Both the single fiber and dual fiber media converters might be labelled as Half/Full Duplex. My guess is this refers to the UTP (ie RJ45 or copper) end. These days most CAT5 or CAT6 copper LAN cables come with both TX and RX pairs, and Half Duplex might be when Ethernet is negotiated down to CSMA/CD. 

Last but not least there is the complex matter of cutting, splicing and terminating your fiber optic cable. Years ago, it took expensive equipment, highly-trained operators and extremely clean conditions, which are sometimes difficult to do on-site. But for now there is and end-run, a workaround. Again from the Chinese. You can buy the cables already terminated for very low prices. Like RM36 for 50m. That is just USD9.

Pre-terminated outdoor single mode fiber cable.

Some suppliers will do it to a custom length. Just make sure the connector is SC UPC (it is easy to specify the incompatible SC APC or LC connectors). Now I had coax 10Base2 concealed LAN wiring installed in my house (yes, yes I am a dinosaur), so it was an easy matter to rip it out of the conduits and install the smaller fiber cable in it place.

The aim is to replace your long copper cable runs with fiber. One benefit is if the cable run is over 100m you do not need to install the repeaters (ie LAN switches) that UTP Ethernet needs. If you add in the cost of the power wiring, enclosures, the savings quickly pile up.

The aim is to replace long copper LAN cable runs with fiber (in red)

It worked so well I ran another 100m fiber cable outdoors to my home office so I can share the broadband. The system has been in place through several violent thunderstorms and one fallen tree and did not miss a beat.

If you live in Malaysia, you are in luck, for Talikom Malaysia, the telecoms monopoly mostly uses sub-contractors to install your FTTH fiber cable. For a very reasonable fee, not exceeding the cost of a roll of fiber cable, they can be persuaded to do your internal house fiber cabling. In my case it was money well spent for much of the work involved climbing onto the roof. A huge advantage is they will splice and terminate your fiber cable using proper equipment, resulting in a very good connection.

The only thing left would be an inexpensive way to cut, join and terminate a fiber cable myself. But that is for another post.

Happy Trails. 

 
 

RESTful IoT with privacy & security: How to set up a Debian HTTPS Server

 

"It is quiet here and restful, and the air is delicious. There are gardens everywhere and police spies lie in the bushes ... " - Maxim Gorky

It is very tempting to use the ubiquitous HTTP web protocol for IoT. It is easy to test, and lets you operate your IoT from smartphones, tablets, desktops and even a computer program. Such a setup is called RESTful. It simply means your IoT device speaks the language of the web browser and web server. 

So we rush ahead with our RESTful API, and the IoT device is soon working and indispensable. Pretty soon we realize we need security and privacy: it won't do to have a hacker open the voice-controlled garage door ...

Our first line of defense is our WiFi password. It is reasonable to assume those living in the house should have access to WiFi and IoT. But what if we had guests or lodgers? Changing WiFi passwords can be a real bear, especially if you have 20-odd IoT devices. In fact it makes sense to localize the changes to a dedicated IoT server. RESTful, naturally.  

We will be needing some form of authentication: account names and passwords should do for now. Next we will need a reasonable amount of privacy, i.e., encryption so that someone else should not be able to lift the IoT password off the WiFi. That means HTTPS, or HTTP with SSL.

We start by implementing HTTPS server on a Linux system. The ESP8266 is known to be a little wobbly running HTTPS. ESP32 is better, but we can do without the complication for now. The traditional way is to use Apache. There are other, easier ways (like nginx, nodejs and even python) but Apache lets you run multiple servers right off the bat. This means you can keep your bad old HTTP server, add another HTTPS server on top of that and lets you support both your HTTP and HTTPS IoT devices.

From a bog-standard Debian (mine is a Beaglebone on eMMC), do the usual:

# apt-get update

# apt-get upgrade

Next, get Apache:

# apt install apache2

And while you are at it, you might as well make sure you have ssh. I got my DNS from duckdns.

Apache should come up complete with the stock webpage at http://localhost. Put your webserver files at /var/www/html/

To access the webserver from outside your WiFi access point, you will need a DNS server, but once you get it organized, a bog standard browser will display a warning before it will display your home page:

That means you need SSL, which usually costs money. You can opt for a self-signed certificate but this will produce a warning with most browsers. You then elect to disregard the warning and proceed, but this is a real problem if you are trying to sell the IoT device.

One way out is to get a 90-day certificate free from sslforfree. You just have to register, input your domain name and prove that you have access to the webserver, usually by uploading an sslforfree file to it. After it checks out the certificates can be downloaded. sslforfree links to a youtube video describing the process.

Do check out the video. I will simply list the differences relevant to a Debian installation. In Debian it is a simple:

# a2enmod ssl

Considering dependency setenvif for ssl:

Module setenvif already enabled

Considering dependency mime for ssl:

Module mime already enabled

Considering dependency socache_shmcb for ssl:

Enabling module socache_shmcb.

Enabling module ssl.

See /usr/share/doc/apache2/README.Debian.gz on how to configure SSL and create s

elf-signed certificates.

To activate the new configuration, you need to run:

  systemctl restart apache2

I now need a configuration file for my HTTPS (or SSL) webserver. There is a template in Debian:

# cp /etc/apache2/sites-available/default-ssl.conf /etc/apache2/sites-available/secure.cmheong.duckdns.org.conf

secure.cmheong.duckdns.org being the domain name of my new HTTPS server. The parameters for the new server are put in:

# cat /etc/apache2/sites-available/secure.cmheong.duckdns.org.conf | head -n 16

<IfModule mod_ssl.c>

        <VirtualHost _default_:443>

                ServerName secure.cmheong.duckdns.org

                ServerAlias www.secure.cmheong.duckdns.org

                ServerAdmin webmaster@secure.cmheong.duckdns.org

                DocumentRoot /var/www/html

                # Available loglevels: trace8, ..., trace1, debug, info, notice, warn,

                # error, crit, alert, emerg.

                # It is also possible to configure the loglevel for particular

                # modules, e.g.

                #LogLevel info ssl:warn

                ErrorLog ${APACHE_LOG_DIR}/secure.cmheong.error.log

                CustomLog ${APACHE_LOG_DIR}/secure.cmheong.access.log combined

You then check your configuration and do not proceed further until this passes:

# apachectl configtest

Syntax OK

Make sure your webserver is now accessible from the Internet. Usually this means setting up Port Forwarding in your gateway to forward all port 443 traffic to your server IP address.

# systemctl restart apache2

You will then need to prepare for the sslforfree test of webserver and domain name ownership:

# mkdir /var/www/html/.well-known

# mkdir /var/www/html/.well-known/pki-validation

Register with sslforfree, download the challenge file they provided and put it in the new directory. This is where the Debian ssh installation comes in handy. 

If the sslforfree challenge succeeds, then the certificates and private key will be generated as a zip file.

# unzip secure.cmheong.duckdns.org.zip

Archive:  secure.cmheong.duckdns.org.zip

 extracting: certificate.crt

 extracting: ca_bundle.crt

 extracting: private.key

You then move them to their final secure directories:

# cp -v ./sslforfree/*.crt  /etc/ssl/certs

'./sslforfree/ca_bundle.crt' -> 'certs/ca_bundle.crt'

'./sslforfree/certificate.crt' -> 'certs/certificate.crt'

# cp  ./sslforfree/private.key  /etc/ssl/private/private.key

Remember to delete the ./sslforfree directory. If you want to put the certificates in a different place you will need to update the site config file accordingly:

# cat /etc/apache2/sites-available/secure.cmheong.duckdns.org.conf | grep -i SSLCerti

                #   SSLCertificateFile directive is needed.

                #SSLCertificateFile     /etc/ssl/certs/ssl-cert-snakeoil.pem

                #SSLCertificateKeyFile /etc/ssl/private/ssl-cert-snakeoil.key

                SSLCertificateFile      /etc/ssl/certs/certificate.crt

                SSLCertificateKeyFile /etc/ssl/private/private.key

                SSLCertificateChainFile /etc/ssl/certs/ca_bundle.crt

As usual test the Apache configuration:

# apachectl configtest

And then restart Apache:

# systemctl restart apache2

The just aim your Chrome browser at https://www.yoursecureserver.com. If it worked you get something like this:

 

Now the traffic to and from the IoT RESTful server is encrypted. Note the sslforfree certificates expire in 90 days, but you are free to generate a new set. They will even email you a reminder. 

There you have it: a secure Internet-facing RESTful IoT server.

Happy Trails.

AS3935 Lightning Detector with I2C and ESP8266 NodeMCU ESP-12E

 

AS3935 Lightning Detector with I2C on ESP8266 NodeMCU ESP-12E

There are several good posts on the AS3935 with Arduino Atmel CPUs, and some with ESP8266, including code, but these tend to publish only wiring diagrams for Arduino. So, this post no big deal, really to document a working example of CJMCU AS3935 with I2C on ESP8266 NodeMCU ESP-12E.

CJMCU AS3935 Lightning Detector

The wiring is:

AS3935                                  ESP-12E

    SCL                                    D1/GPIO5

    MISO                                  D2/GPIO4

    IRQ                                     D5/GPIO14

    VCC                                   3.3V

    GND                                   GND

In addition tie the AS3935 pins A0, A1 and SI to 3.3V

AS3935 I2C wiring for NodeMCU ESP-12E

The choice of D5 for interrupt line was guided by the excellent randomnerd reference. D3 and D4 seemed a lot more intuitive, but the ESP8266 failed to boot.

Unlike my previous posts on the AS3935, I thought it might make a change to use the one of the AS3935 libraries in the Arduino IDE database, specifically stevemarple's AS3935.h. Just navigate to 'Sketch' drop-down menu, then select 'Include Library' and then 'Manage Libraries'. 

Under 'File' and 'Preferences' you need to have the URL: http://arduino.esp8266.com/stable/package_esp8266com_index.json

The code is slightly modified from stevemarple's example to make it compile. I have uploaded a copy to github.

#include <AsyncDelay.h>
#include <SoftWire.h>
#include <AS3935.h>
#ifdef JTD
#include <DisableJTAG.h>
#endif
AS3935 as3935;
bool ledState = true;
AsyncDelay d;
ICACHE_RAM_ATTR void int2Handler(void) // 2020-11-22
{
  as3935.interruptHandler();
}

void readRegs(uint8_t start, uint8_t end)
{
  for (uint8_t reg = start; reg < end; ++reg) {
    delay(50);
    uint8_t val;
    as3935.readRegister(reg, val);
    Serial.print("Reg: 0x");
    Serial.print(reg, HEX);
    Serial.print(": 0x");
    Serial.println(val, HEX);
    Serial.flush();
  }
  Serial.print("State: ");
  Serial.println(as3935.getState(), DEC);
  Serial.println("-------------");
}
bool NoiseHigh = false;
bool Disturbed = false;
void printInterruptReason(Stream &s, uint8_t value, const char *prefix = nullptr)
{
  if (value & AS3935::intNoiseLevelTooHigh) {
    if (NoiseHigh == false) {
      if (prefix)
        s.print(prefix);
      s.println(F("Noise level too high"));
      // NoiseHigh = true;
    }  
  }
  if (value & AS3935::intDisturberDetected) {
    if (Disturbed == false) {
      if (prefix)
        s.print(prefix);
      s.println(F("Disturber detected"));
      // Disturbed = true;
    }
  }  
  if (value & AS3935::intLightningDetected) {
    if (prefix)
      s.print(prefix);
    s.println(F("Lightning detected"));
  }
}

void setup(void)
{
#ifdef JTD
  disableJTAG();
#endif
  Serial.begin(115200);
  // as3935.initialise(14, 17, 0x03, 3, true, NULL);
  as3935.initialise(4, 5, 0x03, 3, true, NULL); // 2020-11-22
  // as3935.calibrate();
  as3935.start();
  d.start(1000, AsyncDelay::MILLIS);
  while (!d.isExpired())
    as3935.process();
  Serial.println("Before setup:");
  readRegs(0, 0x09);
  // attachInterrupt(2, int2Handler, RISING); // 2 is believed to be D4, GPIO2
  attachInterrupt(14, int2Handler, RISING); // 2020-11-22 Chose D5 GPI14
  // attachInterrupt(digitalPinToInterrupt(5), int2Handler, RISING); // 2020-11-22 Chose D3 GPIO0
  d.start(1000, AsyncDelay::MILLIS);
  Serial.println("setup() done");
  readRegs(0, 0x09);
  as3935.setIndoor(true);
  as3935.setNoiseFloor(4);
  as3935.setThreshold(4);
  as3935.setSpikeRejection(0); // From AS3935_timestamp_demo.ino
  
  //as3935.calibrate();
  //Serial.println("Calibration done");
  //readRegs(0, 0x09);
  Serial.println("Masking Disturber interrupts ...");
  as3935.setRegisterBit(as3935.regInt, 5, true);
  readRegs(0, 0x09);
  pinMode(LED_BUILTIN, OUTPUT);
  digitalWrite(LED_BUILTIN, ledState);
}
uint8_t count = 0;
void loop(void)
{
  if (as3935.process()) {
    uint8_t flags = as3935.getInterruptFlags();
    uint8_t dist = as3935.getDistance();
    /*
    Serial.println("-------------------");
    Serial.println("Interrupt!");
    Serial.println("Reason(s):");
    */
    printInterruptReason(Serial, flags, "    ");
    
    if (AS3935::intLightningDetected & flags) {
      Serial.print("Distance: ");
      Serial.println(dist, DEC);
    }
  }
  if (as3935.getBusError()) {
    Serial.println("Bus error!");
    as3935.clearBusError();
  }
  // Flash the LED to show activity
  if (d.isExpired()) {
    ledState = !ledState;
    digitalWrite(LED_BUILTIN, ledState);
    // // Periodically output the AS3935 registers
    // if (++count > 5) {
    //  count = 0;
    //  readRegs(0, 0x09);
    // }
    d.start(1000, AsyncDelay::MILLIS);
  }
}

NodeMCU Base board Ver 1

I also used a baseboard V1 for my NodeMCU V3 Lua Lolin ESP-12E. Do take note of the different versions as the baseboard does not fit many of the cheap ESP-12E out there. The baseboard is just for convenience, really and you can wire the AS3935 directly to the ESP-12E. 

I was getting a little low on USB cables and had many 12V and 9V DC power supplies left over from dead TP-Link and Dlink ADSL modems. The voltage range is acceptable to the baseboard and the power jack fits. 

Have fun. I know I did. Happy Trails.