Did I leave the Air Conditioner On? Indicator LED detection using Raspberry Pi and OpenCV

 

Air Conditioner Indoor Unit with Yellow Indicator LED

Air conditioners are essential in hot and humid Malaysia, especially if you want to work from home. Most of us have occasionally wondered if we have left it on after we left the house: the resulting electricity bill can be a nasty surprise. Most times you cannot do much about it, save for going back home to check.

But then, I managed to hack its infrared remote using an ESP8266, which made it an Internet of Things (IoT) device, which lets me turn it on and off from my smartphone. Now I have a need to know if I left the aircond turned on at home.  

When the indoor unit comes on, there is a beep and an orange LED lights up. The standard way is to mount an optocoupler diode in series with the orange LED, wire the optocoupler output to an ESP8266 and the resulting IoT will reliably report the aircond on/off status every time.

But variety, they say, is the spice of life, and I happened to have an obsolete Raspberry Pi Model B with OpenCV installed. And lots of ancient 640x480 webcams. Granted the lighting conditions would change through the day, but surely it can recognize that round orange light with some consistency?

USB webcam looking at the indicator LED from 3 feet away

It would be a bonus if software can be added later to detect that beep. That would have other applications like detection of smoke alarms' beeps, thunder, doorbell chimes and other interesting sounds. But that is another blog post.

OpenCV Raspberry Pi Model B with LAN connection (ie 'headless' mode)

 Isaac Vidas looks like a good starting point, first using an HSV transform to isolate the color of interest, then using cv2.HoughCircles() to precisely locate the LED itself. 

Image after HSV transform

I did need some additional help in setting the color thresholds required in his code to:

# Get lower orange hue

lower_orange_hue = create_hue_mask(hsv_image, [0, 0, 255], [0, 255, 255]) 

# Get higher orange hue 

higher_orange_hue = create_hue_mask(hsv_image, [0, 0, 255], [38, 255, 255])

There is a handy python script by nathancy here, and together with a HSV color wheel, my threshold values could be determined using several images of the aircond LED under different lighting conditions.

HSV Color wheel

  

Image after color filtering

Firstof, you will be needing a programs to view the USB webcam video and still frames. I use mplayer and feh:

# apt-get install mplayer

# apt-get install feh

You set the Pi via raspi-config not to run the X Server (ie the GUI desktop), but it helps to have the X libraries installed. From your laptop/desktop you just ssh in:

fred@pi:~ $ ssh -t -Y 12.34.56.78

And from there, mplayer should display the video on your desktop. This lets you position the camera properly.

fred@pi:~ $ mplayer tv://

To get 10 still frames after 10s (some cameras auto-adjust brightness):

fred@pi:~ $ mplayer -vo jpeg -frames 10 -ss 10 -brightness 25 tv://

You can use 'mplayer -loop 0' to display the still images, but they flash on and off rather annoyingly. I much prefer something like feh:

fred@pi:~ $ feh .images/image_on.png

And best of all, the openCV code will execute as if you were using the Pi's console (ie HDMI).

Having selected your camera position, you should probably make a set of images under different lighting conditions. I used a fragment of Isaac Vidas's code to do this, in particular to see the effect of lighting on the separate operations like blurring and HSV transformation. This is named webcamTest.py and is available on my github repository. You typically do:

 

fred@pi:~/checkLed $ source ~/opencv/OpenCV-4.0-py3/bin/activate

(OpenCV-4.0-py3) fred@pi:~/checkLed $

(OpenCV-4.0-py3) fred@pi:~/checkLed $ python ./webcamTest.py image_on.png 

Next, use the nathancy code, which I named hsvThresholder.py. 

(OpenCV-4.0-py3) fred@pi:~/checkLed $ python hsvThresholder.py

hsvThresholder.py: adjust the sliders at the bottom. Runs very slowly on a Pi B, so be patient and watch the console output in the window below

You want to adjust the various sliders in order to mask out all other regions of different color to your LED. A Raspberry Pi 1 Model B will be extremely slow here so patience is required. One way is to watch the bash console messages as they are much quicker to update than the picture. Copy the final settings from the console, which will be something like:

(hMin = 0 , sMin = 0, vMin = 85), (hMax = 28 , sMax = 255, vMax = 255)

My version of Isaac Vidas's code is named checkAC_led.py. and pretty much works as advertised, except it required a much larger (something like 6x diameter) image of the LED. I would have needed to mount my camera much closer, just 17cm from the LED. The other problem is the camera needs to be square over the LED as cv2.HoughCircles() do not detect ellipses very well. And line (ie hollow) circles worked better than a solid one.

Image with test circle added: this is the minimum size circle cv2.HoughCircles() will detect

Mounting my camera closer and square-on the LED is the correct solution. This also minimizes false alarms and improves reliability of detection. This probably means some sort of mounting bracket on the wall, and might get in the way when the air conditioner is being serviced. A software solution would be great, and the future beep detector would help filter out those false alarms ...

This led me to cv2.SimpleBlobDetection() code, which does much better with smaller and deformed circles. Take care to set minArea as large as possible: I actually counted the number of LED pixels in my HSV transform.

The gotcha here is that the HSV image has to be inverted for blob detection to work:

    h, s, image_gray = cv2.split(full_image)

    image_gray_neg = cv2.bitwise_not(image_gray) 

    detector = cv2.SimpleBlobDetector_create(params)

After conversion to grayscale and inversion

After successful blob detection

The final version, checkACvideo_led.py reads from the webcam instead of still image files, filters out false alarms based on the blob x and y coordinates and prints the air conditioner status. In its IoT form the print statement just needs to be modified to publish to an MQTT server like mosquitto.

So did I leave the air conditioner on? Hey Mycroft, is my air conditioner on or off?

Happy Trails

USB over IP: How to remote-access your USB devices over the network

 

First appearing in 2005 in Takahiro Hirofuchi's kickass paper, there are much better guides on USB/IP, like Linux Magazine's, Ridgerun's, etc. From 2009 USB/IP was accepted into mainline Linux kernel and up-to-date documentation on it tends to get lost in the Linux haystack. The wrinkle here is USB/IP running on Slackware 14.2-current as of July 2019. Unlike Debian/Ubuntu you cannot usually seamlessly install USB/IP (or most other things) on Slackware. A little DIY is in order.

I know that is cold comfort compared to Debian's 'apt install'. Indeed, Slackware too has its (a little rickety) SlackBuild scripts. But in USB/IP case there is no SlackBuild as it was already included in the kernel. Well, after a fashion.

But take heart: in 30 years of Slackware I seldom fail to install the things I want. This is a chance to poke a little into the innards of Linux. You have the source code in all its glory. And it is built beautifully - I say this as one who has seen Microsoft Windows NT source code [shudder]. 

And the experience will prove useful in the rare instances Debian's installs fail, like the time I installed the Java development kit and could no longer log into my desktop. This means when things fail the same methods work: paste the error into google and look for the patches and workarounds. But I digress; back to USB/IP.  

I should test my Acer aspireM3's USB webcam before the install breaks things:

# mplayer tv:// -tv driver=v4l2:width=640:height=480 -vo xv -tv device=/dev/video1

Let's verify that my Slackware kernel actuall has USB/IP:

root@aspireM3:~# ls -lR /usr/src/linux/ | grep -i usbip
-rw-r--r-- 1 root root  1172 Jan 26  2019 sysfs-platform-usbip-vudc
-rw-r--r-- 1 root root 23365 Jan 26  2019 usbip_protocol.txt

drwxr-xr-x 2 root root  4096 Jan 26  2019 usbip/
/usr/src/linux/drivers/usb/usbip:
-rw-r--r-- 1 root root 18694 Jan 26  2019 usbip_common.c
-rw-r--r-- 1 root root 10117 Jan 26  2019 usbip_common.h
-rw-r--r-- 1 root root  3969 Jan 26  2019 usbip_event.c
drwxr-xr-x  3 root root   4096 Jan 27  2019 usbip/

Looks good. A look at the kernel's config file shows that the binary has not been left out at compile time:

root@aspireM3:~# grep -i usbip /usr/src/linux/.config

CONFIG_USBIP_CORE=m

CONFIG_USBIP_VHCI_HCD=m

CONFIG_USBIP_VHCI_HC_PORTS=8

CONFIG_USBIP_VHCI_NR_HCS=1

CONFIG_USBIP_HOST=m

# CONFIG_USBIP_DEBUG is not set

And finally, locate the kernel modules (ie device drivers) themselves:

# ls -l /lib/modules/4.19.18/kernel/drivers/usb/usbip

total 128

-rw-r--r-- 1 root root 22576 Jan 27  2019 usbip-core.ko

-rw-r--r-- 1 root root 42480 Jan 27  2019 usbip-host.ko

-rw-r--r-- 1 root root 58016 Jan 27  2019 vhci-hcd.ko

And it runs OK:

# modprobe -v usbip-core

insmod /lib/modules/4.19.18/kernel/drivers/usb/usbip/usbip-core.ko

Let's launch the USB/IP server daemon:

# usbipd -D

-su: usbipd: command not found

Oops. The userspace binaries are not installed. Let's see if my Slackware installation has the userspace source code:

# ls -l /usr/src/linux/tools/usb/usbip/src

total 96

-rw-r--r-- 1 root root   440 Jan 26  2019 Makefile.am

-rw-r--r-- 1 root root  4406 Jan 26  2019 usbip.c

-rw-r--r-- 1 root root  1292 Jan 26  2019 usbip.h

-rw-r--r-- 1 root root  5404 Jan 26  2019 usbip_attach.c

-rw-r--r-- 1 root root  5169 Jan 26  2019 usbip_bind.c

-rw-r--r-- 1 root root  2922 Jan 26  2019 usbip_detach.c

-rw-r--r-- 1 root root  9713 Jan 26  2019 usbip_list.c

-rw-r--r-- 1 root root  6179 Jan 26  2019 usbip_network.c

-rw-r--r-- 1 root root  5295 Jan 26  2019 usbip_network.h

-rw-r--r-- 1 root root  1536 Jan 26  2019 usbip_port.c

-rw-r--r-- 1 root root  3494 Jan 26  2019 usbip_unbind.c

-rw-r--r-- 1 root root 15083 Jan 26  2019 usbipd.c

-rw-r--r-- 1 root root  1630 Jan 26  2019 utils.c

-rw-r--r-- 1 root root   863 Jan 26  2019 utils.h

Yes, I do. I just need to compile it but it fails:

/usr/src/linux/tools/usb/usbip# ./autogen.sh

/usr/src/linux/tools/usb/usbip# ./configure

/usr/src/linux/tools/usb/usbip# make

make  all-recursive

make[1]: Entering directory '/usr/src/linux-4.19.18/tools/usb/usbip'

Making all in libsrc

make[2]: Entering directory '/usr/src/linux-4.19.18/tools/usb/usbip/libsrc'

  CC       libusbip_la-usbip_device_driver.lo

usbip_device_driver.c: In function �..read_usb_vudc_device�..:

usbip_device_driver.c:106:2: error: �..strncpy�.. specified bound 256 equals destination size [Werror=stringop-truncation]

  strncpy(dev->path, path, SYSFS_PATH_MAX);

  ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

usbip_device_driver.c:125:2: error: �..strncpy�.. specified bound 32 equals dest

ination size [-Werror=stringop-truncation]

  strncpy(dev->busid, name, SYSFS_BUS_ID_SIZE);

  ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

cc1: all warnings being treated as errors

make[2]: *** [Makefile:471: libusbip_la-usbip_device_driver.lo] Error 1

make[2]: Leaving directory '/usr/src/linux-4.19.18/tools/usb/usbip/libsrc'

make[1]: *** [Makefile:497: all-recursive] Error 1

make[1]: Leaving directory '/usr/src/linux-4.19.18/tools/usb/usbip'

make: *** [Makefile:365: all] Error 2

It does not look serious. What used to be a compiler warning has now been classified as an error so the compiler obligingly stops. A quick google comes up with a patch:

https://patchwork.kernel.org/project/linux-usb/patch/20180721021232.GR14131@deca

dent.org.uk/

> +++ b/tools/usb/usbip/libsrc/usbip_common.c

> @@ -226,8 +226,8 @@ int read_usb_device(struct udev_device *

>       path = udev_device_get_syspath(sdev);

>       name = udev_device_get_sysname(sdev);

>

> -     strncpy(udev->path,  path,  SYSFS_PATH_MAX);

> -     strncpy(udev->busid, name, SYSFS_BUS_ID_SIZE);

> +     snprintf(udev->path, SYSFS_PATH_MAX, "%s", path);

> +     snprintf(udev->busid, SYSFS_BUS_ID_SIZE, "%s", name);

So I just edit usbip_device_driver.c:

# vi libsrc/usbip_device_driver.c

 And at line 106

        strncpy(dev->path, path, SYSFS_PATH_MAX);

 Is changed to

        // strncpy(dev->path, path, SYSFS_PATH_MAX); cmheong 2021-03-27

        snprintf(dev->path, SYSFS_PATH_MAX, "%s", path);

At line 125:

strncpy(dev->busid, name, SYSFS_BUS_ID_SIZE);

           Became

snprintf(dev->busid, SYSFS_BUS_ID_SIZE, "%s", name);

Anow for the file usbip_common.c line 230:

        // strncpy(udev->path,  path,  SYSFS_PATH_MAX); // cmheong 2021-03-27

        // strncpy(udev->busid, name, SYSFS_BUS_ID_SIZE);

Becomes

        snprintf(udev->path, SYSFS_PATH_MAX, "%s", path);

        snprintf(udev->busid, SYSFS_BUS_ID_SIZE, "%s", name);

Now some Slackware versions, depending on the date of your install, you might get an additional error:

  CC       usbip_network.o

usbip_network.c: In function âusbip_net_pack_usb_deviceâ:

usbip_network.c:91:32: error: taking address of packed member of âstruct usbip_

usb_deviceâ may result in an unaligned pointer value [-Werror=address-of-packed

-member]

   91 |  usbip_net_pack_uint32_t(pack, &udev->busnum);

      |                                ^~~~~~~~~~~~~

We just need to tell the compiler not to freak out and treat the warning as an error. This can be done by changing line 12174 of the 'configure' file:

/usr/src/linux/tools/usb/usbip$vi configure

EXTRA_CFLAGS="-Wall -Wno-error=address-of-packed-member -Werror -Wextra -std=gnu

99"

After which the compile completes successfully:

# make

make  all-recursive

make[1]: Entering directory '/usr/src/linux-4.19.18/tools/usb/usbip'

Making all in libsrc

make[2]: Entering directory '/usr/src/linux-4.19.18/tools/usb/usbip/libsrc'

  CC       libusbip_la-usbip_common.lo

  CC       libusbip_la-usbip_host_common.lo

  CC       libusbip_la-vhci_driver.lo

  CC       libusbip_la-sysfs_utils.lo

  CCLD     libusbip.la

make[2]: Leaving directory '/usr/src/linux-4.19.18/tools/usb/usbip/libsrc'

Making all in src

make[2]: Entering directory '/usr/src/linux-4.19.18/tools/usb/usbip/src'

  CC       usbip.o

  CC       utils.o

  CC       usbip_network.o

  CC       usbip_attach.o

  CC       usbip_detach.o

  CC       usbip_list.o

  CC       usbip_bind.o

  CC       usbip_unbind.o

  CC       usbip_port.o

  CCLD     usbip

  CC       usbipd.o

  CCLD     usbipd

make[2]: Leaving directory '/usr/src/linux-4.19.18/tools/usb/usbip/src'

make[2]: Entering directory '/usr/src/linux-4.19.18/tools/usb/usbip'

make[2]: Leaving directory '/usr/src/linux-4.19.18/tools/usb/usbip'

make[1]: Leaving directory '/usr/src/linux-4.19.18/tools/usb/usbip'

Now the server daemon runs:

# /usr/src/linux/tools/usb/usbip/src/usbipd -D

A quick check:

/usr/src/linux/tools/usb/usbip# ps -ef | grep -e usbip

root     23597     2  0 14:57 ?        00:00:00 [usbip_event]

root     23608     1  0 19:53 ?        00:00:00 /usr/src/linux/tools/usb/usbip/src/.libs/lt-usbipd -D

root     23835  1533  0 19:54 pts/0    00:00:00 grep -e usbip

Next we list the USB devices available to the server:

/usr/src/linux/tools/usb/usbip# /usr/src/linux/tools/usb/usbip/src/usbip list --local

 - busid 1-3.1 (0835:8501)

   Action Star Enterprise Co., Ltd : unknown product (0835:8501)

 - busid 1-3.2 (046d:c077)

   Logitech, Inc. : M105 Optical Mouse (046d:c077)

 - busid 1-3.3 (13ba:0017)

   PCPlay : PS/2 Keyboard+Mouse Adapter (13ba:0017)

 - busid 1-3.4 (10c4:ea60)

   Cygnal Integrated Products, Inc. : CP2102/CP2109 UART Bridge Controller [CP21

0x family] (10c4:ea60)

 - busid 1-3.5.1 (0835:8502)

   Action Star Enterprise Co., Ltd : unknown product (0835:8502)

 - busid 1-3.5.2 (046d:0829)

   Logitech, Inc. : unknown product (046d:0829)

 - busid 1-3.5.4 (0c45:62f1)

   Microdia : unknown product (0c45:62f1)

 - busid 2-1.3 (04f2:b300)

   Chicony Electronics Co., Ltd : unknown product (04f2:b300)

 - busid 2-1.4 (04ca:3006)

   Lite-On Technology Corp. : unknown product (04ca:3006)

Using the displayed busis, we use 'lsusb -v' to zero in on the webcam:

Bus 002 Device 003: ID 04f2:b300 Chicony Electronics Co., Ltd

  iManufacturer           1 Chicony Electronics Co., Ltd.

  iProduct                2 HD WebCam

  iSerial                 3 SN0001

 - busid 2-1.3 (04f2:b300)

Now I need to launch another kernel module:

# modprobe -v usbip-host

insmod /lib/modules/4.19.18/kernel/drivers/usb/usbip/usbip-host.ko

We now have the USB/IP server grab the webcam:

/usr/src/linux/tools/usb/usbip# /usr/src/linux/tools/usb/usbip/src/usbip bind --busid=2-1.3

usbip: info: bind device on busid 2-1.3: complete

Now we move over to the other (ie client) computer, and repeat the process of installing USB/IP. We then launch the USB/IP kernel client modules:

/usr/src/linux/tools/usb/usbip$modprobe -v vhci-hcd

insmod /lib/modules/4.19.62/kernel/drivers/usb/usbip/usbip-core.ko

insmod /lib/modules/4.19.62/kernel/drivers/usb/usbip/vhci-hcd.ko

Let's say the server's IP address is 12.34.56.78. We can see what USB device on the server is available over IP:

/usr/src/linux/tools/usb/usbip$src/usbip list --remote=12.34.56.78

Exportable USB devices

======================

 - 12.34.56.78

      2-1.3: Chicony Electronics Co., Ltd : unknown product (04f2:b300)

           : /sys/devices/pci0000:00/0000:00:1a.0/usb2/2-1/2-1.3

           : Miscellaneous Device / ? / Interface Association (ef/02/01). 

 

Ah it is a webcam ;7) We attach to it:

/usr/src/linux/tools/usb/usbip# /usr/src/linux/tools/usb/usbip/src/usbip attach -debug --remote=12.34.56.78 --busid=2-1.3

And we play the remote (ie server) webcam by using mplayer:

/usr/src/linux/tools/usb/usbip$ mplayer -cache 128 -tv device=/dev/video2:driver=v4l2:width=640:height=480:outfmt=i420 -vo xv tv://

Note that here I access /dev/video2, since my client laptop also has a webcam at /dev/video1, and the USB/IP attach made a new one at /dev/video2.  Also the server webcam resolution is 1080 x 720 but that seemed to mave maxed out my old 100Mbps fiber link. Dropping the resolution down to 640 x 480 worked out well.

You can see your attached USB device using:

/usr/src/linux/tools/usb/usbip$src/usbip port --remote=12.34.56.78

Imported USB devices

====================

Port 00: <Port in Use> at High Speed(480Mbps)

       Chicony Electronics Co., Ltd : unknown product (04f2:b300)

       3-1 -> usbip://192.168.1.4:3240/3-1.3

           -> remote bus/dev 003/003

Notice the port number 00. We will be needing it. Lastly once you are through with the webcam you release it so another computer can use it:

/usr/src/linux/tools/usb/usbip$src/usbip detach --port=00

usbip: info: Port 0 is now detached!

There you have it. USB/IP comes standard with Linux, and lets you access USB devices remotely. I had great fun spending an afternoon doing this to the boogie blues. Here's a video of Daisy tapping her feet to it:

Click on picture for the video

Happy Trails.