Did I leave the Air Conditioner On? Indicator LED detection using Raspberry Pi and OpenCV

 

Air Conditioner Indoor Unit with Yellow Indicator LED

Air conditioners are essential in hot and humid Malaysia, especially if you want to work from home. Most of us have occasionally wondered if we have left it on after we left the house: the resulting electricity bill can be a nasty surprise. Most times you cannot do much about it, save for going back home to check.

But then, I managed to hack its infrared remote using an ESP8266, which made it an Internet of Things (IoT) device, which lets me turn it on and off from my smartphone. Now I have a need to know if I left the aircond turned on at home.  

When the indoor unit comes on, there is a beep and an orange LED lights up. The standard way is to mount an optocoupler diode in series with the orange LED, wire the optocoupler output to an ESP8266 and the resulting IoT will reliably report the aircond on/off status every time.

But variety, they say, is the spice of life, and I happened to have an obsolete Raspberry Pi Model B with OpenCV installed. And lots of ancient 640x480 webcams. Granted the lighting conditions would change through the day, but surely it can recognize that round orange light with some consistency?

USB webcam looking at the indicator LED from 3 feet away

It would be a bonus if software can be added later to detect that beep. That would have other applications like detection of smoke alarms' beeps, thunder, doorbell chimes and other interesting sounds. But that is another blog post.

OpenCV Raspberry Pi Model B with LAN connection (ie 'headless' mode)

 Isaac Vidas looks like a good starting point, first using an HSV transform to isolate the color of interest, then using cv2.HoughCircles() to precisely locate the LED itself. 

Image after HSV transform

I did need some additional help in setting the color thresholds required in his code to:

# Get lower orange hue

lower_orange_hue = create_hue_mask(hsv_image, [0, 0, 255], [0, 255, 255]) 

# Get higher orange hue 

higher_orange_hue = create_hue_mask(hsv_image, [0, 0, 255], [38, 255, 255])

There is a handy python script by nathancy here, and together with a HSV color wheel, my threshold values could be determined using several images of the aircond LED under different lighting conditions.

HSV Color wheel

  

Image after color filtering

Firstof, you will be needing a programs to view the USB webcam video and still frames. I use mplayer and feh:

# apt-get install mplayer

# apt-get install feh

You set the Pi via raspi-config not to run the X Server (ie the GUI desktop), but it helps to have the X libraries installed. From your laptop/desktop you just ssh in:

fred@pi:~ $ ssh -t -Y 12.34.56.78

And from there, mplayer should display the video on your desktop. This lets you position the camera properly.

fred@pi:~ $ mplayer tv://

To get 10 still frames after 10s (some cameras auto-adjust brightness):

fred@pi:~ $ mplayer -vo jpeg -frames 10 -ss 10 -brightness 25 tv://

You can use 'mplayer -loop 0' to display the still images, but they flash on and off rather annoyingly. I much prefer something like feh:

fred@pi:~ $ feh .images/image_on.png

And best of all, the openCV code will execute as if you were using the Pi's console (ie HDMI).

Having selected your camera position, you should probably make a set of images under different lighting conditions. I used a fragment of Isaac Vidas's code to do this, in particular to see the effect of lighting on the separate operations like blurring and HSV transformation. This is named webcamTest.py and is available on my github repository. You typically do:

 

fred@pi:~/checkLed $ source ~/opencv/OpenCV-4.0-py3/bin/activate

(OpenCV-4.0-py3) fred@pi:~/checkLed $

(OpenCV-4.0-py3) fred@pi:~/checkLed $ python ./webcamTest.py image_on.png 

Next, use the nathancy code, which I named hsvThresholder.py. 

(OpenCV-4.0-py3) fred@pi:~/checkLed $ python hsvThresholder.py

hsvThresholder.py: adjust the sliders at the bottom. Runs very slowly on a Pi B, so be patient and watch the console output in the window below

You want to adjust the various sliders in order to mask out all other regions of different color to your LED. A Raspberry Pi 1 Model B will be extremely slow here so patience is required. One way is to watch the bash console messages as they are much quicker to update than the picture. Copy the final settings from the console, which will be something like:

(hMin = 0 , sMin = 0, vMin = 85), (hMax = 28 , sMax = 255, vMax = 255)

My version of Isaac Vidas's code is named checkAC_led.py. and pretty much works as advertised, except it required a much larger (something like 6x diameter) image of the LED. I would have needed to mount my camera much closer, just 17cm from the LED. The other problem is the camera needs to be square over the LED as cv2.HoughCircles() do not detect ellipses very well. And line (ie hollow) circles worked better than a solid one.

Image with test circle added: this is the minimum size circle cv2.HoughCircles() will detect

Mounting my camera closer and square-on the LED is the correct solution. This also minimizes false alarms and improves reliability of detection. This probably means some sort of mounting bracket on the wall, and might get in the way when the air conditioner is being serviced. A software solution would be great, and the future beep detector would help filter out those false alarms ...

This led me to cv2.SimpleBlobDetection() code, which does much better with smaller and deformed circles. Take care to set minArea as large as possible: I actually counted the number of LED pixels in my HSV transform.

The gotcha here is that the HSV image has to be inverted for blob detection to work:

    h, s, image_gray = cv2.split(full_image)

    image_gray_neg = cv2.bitwise_not(image_gray) 

    detector = cv2.SimpleBlobDetector_create(params)

After conversion to grayscale and inversion

After successful blob detection

The final version, checkACvideo_led.py reads from the webcam instead of still image files, filters out false alarms based on the blob x and y coordinates and prints the air conditioner status. In its IoT form the print statement just needs to be modified to publish to an MQTT server like mosquitto.

So did I leave the air conditioner on? Hey Mycroft, is my air conditioner on or off?

Happy Trails