Thursday, 28 December 2017

IoT Bluetooth Autogate using Modbus

The IoT Bluetooth Relay Board tests well enough with a simple python script, but every time the script finishes bluetoothctl drops its connection to the HC-06. When you run the test script again, it reconnects to the HC-06 well enough, but the connection takes a variable number of seconds, and sometimes the script fails to connect. For an application like the autogate this is not acceptable.

However if the script does not exit the connection is reliable over many days, so the script should not exit. That means a server (Linux daemon) process, and its accompanying client script. There is one such python solution and that is pymodbus, a python implementation of Modbus client and server.

Modbus for an autogate? Modbus is a popular industrial communications protocol. Isn't that overkill?. It probably is.

Or is it? As we have seen the remote sensor/actuator portion is done using bluetooth device and a Microchip PIC18F14K50, which consumes 10mA at 5V. Modbus lives on the IoT gateway, ie the bluetooth master which happens to be an old laptop running Slackware Linux. Pymodbus makes my homebrew autogate remote opener compatible with IoT for Industry. The client python script will then be run by a PHP script from the laptop's apache webserver, in the same way as the Raspberry Pi Robot platform.

Security is via the WPA WiFi password on my home WiFi router. Notice that this system will still work without a broadband connection, as long as your smartphone is within range of the WiFi. The HC-06 will not accept a second bluetooth connection once it is paired to the laptop, so access is via the WiFi alone. With a ADSL connection this is a true IoT, able to accept commands from the Internet.

Remote autogate operation may make sense if you say, want to let the gardener into the yard, or the electricity/water utility person to read the meter. In my case my dogs will probably terrorize the meter reader   before galloping off to poop in the neighbour's yard. So its main advantage is an extended-range autogate remote.

The installation for Slackware is probably subject for another post, but say you had it installed. There is a sample program pymodbus-master/examples/common/ which we can use as a template.

The first few lines of code are important:
# import the modbus libraries we need
from pymodbus.server.async import StartTcpServer
from pymodbus.device import ModbusDeviceIdentification
from pymodbus.datastore import ModbusSequentialDataBlock
from pymodbus.datastore import ModbusSlaveContext, ModbusServerContext
from pymodbus.transaction import ModbusRtuFramer, ModbusAsciiFramer

# import the twisted libraries we need
from twisted.internet.task import LoopingCall

Next we import our bluetooth module, which shall look suspiciously like our previous python test script.

# communicate with bluetooth rs-485 pic18f14k50 2017-12-19
import autogate_bluetooth

The main server loop code is simply; we will add our bluetooth code later:
# define your callback process
def updating_writer(a):
    ''' A worker process that runs every so often and
    updates live values of the context. It should be noted
    that there is a race condition for the update.

    :param arguments: The input arguments to the call
    log.debug("updating the context")
    context  = a[0]
    register = 3
    slave_id = 0x00
    address  = 0x10
    values   = context[slave_id].getValues(register, address, count=5)
    values   = [v + 1 for v in values]
    log.debug("new values: " + str(values))
    context[slave_id].setValues(register, address, values)

Next there is our Modbus database, which we reduce from 100 to 16 to reduce memory usage, a initialize to 0 (except for the 2 bits reversed logic for the Arduino relays):
# initialize your data store
store = ModbusSlaveContext(
    di = ModbusSequentialDataBlock(0, [0]*16),
    co = ModbusSequentialDataBlock(0, [True, True, False, False]*4), # True because relay logic reversed
    hr = ModbusSequentialDataBlock(0, [i for i in range(16)]),
    ir = ModbusSequentialDataBlock(0, [45]*16))
context = ModbusServerContext(slaves=store, single=True)

Jumping to the end of the file for now, we have:
if __name__ == "__main__":
  # run the server you want
  looptime = 0.2 # 
  loop = LoopingCall(f=updating_writer, a=(context,))
  loop.start(looptime, now=True) # initially delay by time

  StartTcpServer(context, identity=identity, address=("localhost", 5020)) #2017-12-19

Note the loop time has been speeded up from 5s to 0.2s. StartTcpServer() is set to use the non-root network port 5020, which our client script will need to listen to.

That is all there is to it. Most of the pymodbus server code is done. Now for that bluetooth code in ./, which I have put in the same directory so that it imports correctly. Here it is in full:

import serial
from time import localtime, strftime, sleep

Sends bluetooth rs-485 PIC18F14K50 commands and receives the replies.    

  ./             to test
  Uses pyserial module
  The ems command is single-char, reply is a double sequence of 3 binary bytes
  followed by check strings
  Replies to 'z' are 2 3-byte analog input readings. 

port1 = serial.Serial('/dev/rfcomm0',timeout= 1) #2017-12-19 HC-06 bluetooth   
print 'Waiting for bluetooth connect ...',
sleep(8) # Wait for bluetooth to connect
print 'Done'

cmd = ['z', '1', '2', '3', '0' ]
ChkStr1 = 'HCM'
ChkStr2 = 'LJP'

import struct
import binascii

def check_frame(reply):
  if reply[2:5] != 'HCM':
    print 'Checkbyte HCM Fail', reply
    return 0
  if reply[7:10] != 'LJP':
    print 'Checkbyte LJP Fail', reply
    return 0

  return 1
def ParseReply(reply):
  Battery_Voltage=int(format(ord(reply[0]), '02x'), 16)*256 + \
                  int(format(ord(reply[1]), '02x'), 16)
  Calibrated_Battery_Voltage = Battery_Voltage * 13.75 / 689
  PIC_Voltage=int(format(ord(reply[5]), '02x'), 16)*256 + \
              int(format(ord(reply[6]), '02x'), 16)
  Calibrated_PIC_Voltage = PIC_Voltage * 13.75 / 689

  return [int(format(ord(reply[0]))), int(format(ord(reply[1]))), \
          int(format(ord(reply[5]))), int(format(ord(reply[6]))) ]
  #return [Calibrated_Battery_Voltage, Calibrated_PIC_Voltage] 

def Send_Cmd(cmd):
  # sleep(1)
  reply =
  if reply != "":
    print 'Reply:', reply, 'len', len(reply)
  # for j in range(0, len(reply)):
  #   print format(ord(reply[j]), '02x'),
  # print '...'

    if (check_frame(reply) != 0):
      data = ParseReply(reply)
      data = 0
    print '... Noreply to cmd', cmd
    data = 0
  return data

import sys

# main program loop
if __name__ == "__main__":

  if (len(sys.argv) == 2):
    Send_Cmd(cmd= sys.argv[1])
    for i in cmd:
      answer = Send_Cmd(i)
      print 'result', answer


The main function here is Send_Cmd(). The PIC18F14K50 has been programmed to accept the commands 'z' (read voltages), '0' (both relays on), '1' (first relay on), '2' (second relay on) and '3' (both relays off).

Now let us go back to so we can modify the main server loop to communicate with the HC-06. We need to add a new module update_output() to read the Modbus database. If there is a change (caused by the client script requesting the Arduino relay to turn on) it will then send the correct command to the HC-06. The code is:

import struct

def get_muxoutputBuf():
    register = 1      # digital output (modbus 'coils')
    slave_id = 0x00
    address  = 0
    values   = context[slave_id].getValues(register, address, count=8)#from 48
    # print 'values=', values
    return values

def get_muxscratchBuf():
    register = 1      # use holding registers for convenience
    slave_id = 0x00
    address  = 0x0008
    values   = context[slave_id].getValues(register, address, count=8)
    return values

def set_muxscratchBuf(values):
    register = 1      # use holding registers for convenience
    slave_id = 0x00
    address  = 0x0008 # used as scratch buffer
    context[slave_id].setValues(register, address, values)
    return values

def update_output(a): # new function 2016-08-10 09:15
    #context = a
    str = 'z'
    log.debug("Checking the output buffer")
    scratchBuf = get_muxscratchBuf()
    #print 'scratchBuf', scratchBuf
    outputBuf = get_muxoutputBuf()
    #print 'outputbuffer', outputBuf

    data = [0]*1
    #print 'data length', len(data),

    # pack the bits into the integer/byte array 2015-08-23
    for i in range(0, len(data)*8):
        j = i / 8
        k = i % 8
        # print 'j', j
        if outputBuf[i] == True:
            data[j] |= 1 << k
            data[j] &= ~(1 << k)

    datastr= [format(data[i], '02X') for i in range(0, len(data))]
    str = datastr[0][1] # 2017-12-20

    #print 'data is',data,'datastr',datastr,'output Cmd data string', str
    if (scratchBuf != outputBuf):
        autogate_bluetooth.Send_Cmd(str) # 2017-12-21 note answer is not stored
        print 'Hey, it is here!!!!'
        print '!!!!!!!'
        print '.......'
        print 'setting scratchBuf to', outputBuf
        print 'scratchBuf', scratchBuf
        print 'scratchBuf', scratchBuf,'outputBuf', outputBuf

    return str

Next there is the code to update the Modbus database with the raw values read from the HC-06. These are the 2-byte values from the analog inputs. There are two inputs that makes 4 buyes in total.

def update_database(context, answer=[]):
    ''' Updated 2017-12-21
    raw = []
    for byte in answer:
      raw += byte
    print 'raw =', raw

    register = 3 # input register. *Not* 4
    address = 0; # 2017-12-20
    values   = context[0x01].getValues(register, address, count=4)# 2017-12-20
    print 'stored values=', str(values)

    log.debug(str(address) + " input register values: " + str(answer))
    print 'address=', address, ' answer: ', answer

    #context[0x01].setValues(register, address, raw) # 0x01 replaced slave_id
    context[0x01].setValues(register, address, answer) # 0x01 replaced slave_id


battery_scan_interval = 3600 # set at 1 hour

The new server loop now becomes:

def updating_writer(a):
    global battery_scan_interval

    # read battery voltage every hour
    if battery_scan_interval == 3600:
        answer = [0*2] # 2017-12-19
        answer = autogate_bluetooth.Send_Cmd('z') # 2017-12-20
        if answer != 0:
            print '\n\n Cmd z, voltages are', answer
            update_database(context, answer)
            CPU_Voltage = answer[0] * 256 + answer[1]
            Calibrated_CPU_Voltage = (CPU_Voltage * 5.0) / 1024
            Battery_Voltage = answer[2] * 256 + answer[3]
            Calibrated_Battery_Voltage = (Battery_Voltage * 13.53) / 652
            print 'CPU', answer[0] * 256 + answer[1], 'Battery', \
                  answer[2] * 256 + answer[3]
            print 'CPU', "{:4.2f}".format(Calibrated_CPU_Voltage), \
                  'Battery', "{:4.2f}".format(Calibrated_Battery_Voltage),\
                  'Volts',strftime("%Y-%m-%d %H:%M:%S", localtime())

            print 'no answer'
        battery_scan_interval = 0
        battery_scan_interval += 1

    str = update_output(a) # 2017-12-20

And that is it. You run the server from a command line (remembering to run bluetoothctl and 'rfcomm bind /dev/rfcomm0' first) :

$python ./
Waiting for bluetooth connect ... Done
Reply: ûHCM¥LJP len 10

 Cmd z, voltages are [3, 251, 2, 165]
stored values= [1, 2, 3, 4]
address= 0  answer:  [3, 251, 2, 165]
CPU 1019 Battery 677
CPU 4.98 Battery 14.05 Volts 2017-12-28 17:09:54
... Noreply to cmd 9
Hey, it is here!!!!
setting scratchBuf to [True, False, False, True, True, False, False, True]
scratchBuf [True, False, False, True, True, False, False]
Reply: ÿHCM§LJP len 10

The client script is much simpler. Let's call it You only use the TCP/IP version:
# import the various server implementations
from pymodbus.client.sync import ModbusTcpClient as ModbusClient

You connect to the server thus:
client = ModbusClient('localhost', port=5020)

import sys
from time import localtime, strftime, sleep
if __name__ == "__main__":
    if len(sys.argv) == 2 :
        if sys.argv[1] == 'open' or sys.argv[1] == 'close' \
           or sys.argv[1] == 'Open' or sys.argv[1] == 'Close':
            coils = [True, False, False, False, False, False, False, False]
            rq = client.write_coils(0, coils, unit=0) # 2017-12-20
            print sys.argv[1], 'autogate'
            coils = [True, True, False, False, False, False, False, False]
            rq = client.write_coils(0, coils, unit=0) # 2017-12-20
        elif sys.argv[1] == 'ajar' or sys.argv[1] == 'Ajar':
            coils = [True, False, False, False, False, False, False, False]
            rq = client.write_coils(0, coils, unit=0) # 2017-12-20
            print sys.argv[1], 'autogate'
            coils = [True, True, False, False, False, False, False, False]
            rq = client.write_coils(0, coils, unit=0) # 2017-12-20
            coils = [True, False, False, False, False, False, False, False]
            rq = client.write_coils(0, coils, unit=0) # 2017-12-20
            print sys.argv[1], 'autogate'
            coils = [True, True, False, False, False, False, False, False]
            rq = client.write_coils(0, coils, unit=0) # 2017-12-20
            # python rocks! evaluate argument as python expression
            coils = eval(sys.argv[1])
            rq = client.write_coils(0, coils, unit=0) # 2017-12-20
            print 'writing', coils

        rr = client.read_holding_registers(0,4,unit=0x01)# 2016-12-20
        #print "read_holding_registers 1", rr.registers
        #print "read_holding_registers type", type(rr.registers)
        #print "read_holding_registers list data type", type(rr.registers[0])
        CPU_Voltage = rr.registers[0] * 256 + rr.registers[1]
        Calibrated_CPU_Voltage = (CPU_Voltage * 5.0) / 1024
        Battery_Voltage = rr.registers[2] * 256 + rr.registers[3]
        Calibrated_Battery_Voltage = (Battery_Voltage * 13.53) / 652
        print "read_holding_registers 1", rr.registers, 'CPU', \
              (rr.registers[0] * 256 + rr.registers[1]), 'Battery', \
              rr.registers[2] * 256 + rr.registers[3]
        print 'CPU', "{:4.2f}".format(Calibrated_CPU_Voltage), \
              'Battery', "{:4.2f}".format(Calibrated_Battery_Voltage),\
              'Volts',strftime("%Y-%m-%d %H:%M:%S", localtime())
        print 'Wrong/no arguments', len(sys.argv)

The main code is:
            coils = [True, False, False, False, False, False, False, False]
            rq = client.write_coils(0, coils, unit=0) # 2017-12-20
You set the bits you want (only the first 2 are implemented) and use client.write_coils() to write to the Modbus database. The server process does the rest. 

Typical output is:
$python ./ open
open autogate
read_holding_registers 1 [3, 251, 2, 62] CPU 1019 Battery 574
CPU 4.98 Battery 11.91 Volts 2017-12-28 16:14:06

To close the autogate you run the client again:
$python ./ open
open autogate
read_holding_registers 1 [3, 251, 2, 62] CPU 1019 Battery 574
CPU 4.98 Battery 11.91 Volts 2017-12-28 16:31:13

Now when I am on foot I find it useful not to open the gate wide, as a dog might then be tempted to bolt. This opens a man-size opening by triggering the relay twice:
$python ./ ajar
ajar autogate
ajar autogate
read_holding_registers 1 [3, 251, 2, 62] CPU 1019 Battery 574
CPU 4.98 Battery 11.91 Volts 2017-12-28 16:31:07

And that is all there is to it. Now for the PHP script.

$cat  ~/autogate/bluetooth/html/autogate.html 
   <!-- \/ starthtml -->
   <META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=iso-8859-1">
   <META NAME="keywords" CONTENT="Heong Chee Meng electronics engineer system sof
tware hardware digital design analog design FPGA VHDL parametric tester Windows N
T realtime device driver Linux kernel hacking semiconductor manufacturing SCADA e
mbedded Seremban Malaysia KM48T02 DAC71">
   <META NAME="description" CONTENT="Heong Chee Meng's Autogate Remote Control website.">
   <META NAME="author" CONTENT="Heong Chee Meng">
   <TITLE>Heong's Autogate Remote Control Website</TITLE>
Heong's Autogate Remote Control Website

<p style="float: left; width: 33.3%; text-align: "center>
<form action="openbutton.php" method="post">
  <button type="submit" name="open" value="Connect"><img width="360" height="360" alt="Connect" src="./open.svg" align="left"></button> 
<p style="float: left; width: 33.3%; text-align: center">
<form action="openbutton.php" method="post">
  <button type="submit" name="ajar" value="Connect"><img width="360" height="360" alt="Connect" src="./ajar.svg" align="right"></button>

<!-- /\ end html  -->

And the PHP script is:

$cat  ~/autogate/bluetooth/html/openbutton.php 
  if (isset($_POST['open'])) {
    $result = shell_exec('python open');
    echo "Done!<pre>$result</pre>";
  if (isset($_POST['ajar'])) {
    $result = shell_exec('python ajar');
    echo "Done!<pre>$result</pre>";
  header("Location: ./autogate.html");

Aim your browser (I used Google Chrome) at the webserver: xx.xx.xx.xx/autogate/autogate.html and there you have it- an IoT bluetooth Autogate remote. 

Happy Trails

Saturday, 23 December 2017

IoT Bluetooth Relay Board for Autogate

Clockwise from top: Arduino dual relay, LM2956 boost-buck DC-DC converter, bluetooth USB PIC18F14K50 and HC-06 bluetooth module
The IoT Autogate is simply a combination of two systems, the remote-controlled Autogate and the Bluetooth Solar Battery Voltmeter. with the addition of a cheap (only RM10) Arduino dual-channel relay board.
Arduino 5V opto-isolated dual relay PCB
The PIC18F14K50 (it is really an incarnation of the Microchip Low Pincount Development Kit)  can drive the optoisolator input directly using two of its spare pins configured as digital output. Best of all the relay coil consumes little power: 80mA per active relay. When it is not active the power used is 10mA at 5V or 0.05W. This is very useful because in the event of a power cut and the autogate is running on battery, you want the gate to remain powered for as long as possible.

The relay board logic is reversed, that is a 'low' or 0V output by the PIC18F14K50 turns the relay on. I happened to have the pins RC4 and RC5 free so I used them. This is the same port as the voltmeter's analog input pins (RC6 & RC7).

I first tested it from the USB port of my laptop. This lets me start testing from a clean USB Microchip Low Pincount Development Kit software and helps prevent errors in my bluetooth code from affecting the relay board code. It is safer to use a USB hub, preferably one which has its own 5V power in case a wiring mistake damages your laptop.

First of, simply set TRISC correctly for both analog inout and digital output:
    #define mInitAllLEDs()      LATC &= 0xC0; TRISC &= 0xC0;
If you need to delve further into this code you first need to read the (rather thick) PIC18F14K50 datasheet.

We put a little veneer of C over the new analog output bits:
    #define digital_output_1    LATCbits.LATC4 // 2017-12-03
    #define digital_output_2    LATCbits.LATC5 // 2017-12-03

So, to turn both relays off (like on power up) we simply do:
    digital_output_1 = 1; //Make sure relays are off
    digital_output_2 = 1; //Make sure relays are off

I simply added a counter to the USB code and wrote the value of the counter to the PIC18F14K50 output port. The first time a voltmeter command is issued, only one relay comes on (01). The second command causes the first relay to go off and the second relay to come on (10). The third command causes both relays to go off (11). The forth command causes both relays to come on (00).

The actual test is simplicity itself (python rocks!) and took just a few minutes. The relays duly clicked in their assigned order:

Python 2.7.11 (default, Mar  3 2016, 13:35:30)
[GCC 5.3.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import serial
>>> port=serial.Serial('/dev/ttyACM0', 19200, timeout=1)
>>> port.write('z');;
>>> port.write('z');;
>>> port.write('z');;
>>> port.write('z');;

Now with the new digital output function working it is time to test it with the bluetooth code. To do so I needed to power on from a non-USB 5V. Otherwise the PIC18F14K50 would detect the USB port and would not run the bluetooth code. This is easily done with a power bank.


As before, use hciconfig and bluetoothctl.

[bluetooth]# devices
[bluetooth]# paired-devices
[bluetooth]# power on
Changing power on succeeded
[bluetooth]# scan onm
Invalid argument onm
[bluetooth]# scan on
Discovery started
[CHG] Controller C8:FF:28:27:7D:2C Discovering: yes
[NEW] Device 98:D3:32:20:BB:7B HC-06
[bluetooth]# agent on
Agent registered
[bluetooth]# default-agent
Default agent request successful
[bluetooth]# pair 98:D3:32:20:BB:7B
Attempting to pair with 98:D3:32:20:BB:7B
[CHG] Device 98:D3:32:20:BB:7B Connected: yes
Request PIN code
[agent] Enter PIN code: 1234
[CHG] Device 98:D3:32:20:BB:7B UUIDs: 00001101-0000-1000-8000-00805f9b34fb
[CHG] Device 98:D3:32:20:BB:7B Paired: yes
Pairing successful
[CHG] Device 98:D3:32:20:BB:7B Connected: no


$rfcomm unbind /dev/rfcomm0 98:D3:32:20:BB:7B 1
$rfcomm bind /dev/rfcomm0 98:D3:32:20:BB:7B 1

Back to python (did I say python rocks?)

Python 2.7.11 (default, Mar  3 2016, 13:35:30)
[GCC 5.3.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import serial
>>> port=serial.Serial('/dev/rfcomm0', timeout=3)
>>> port.write('0');
>>> port.write('1');
>>> port.write('2');
>>> port.write('3');
>>> port.write('0');

Finally, I needed a 12V to 5V DC to DC power supply. The autogate runs off a 12V backup battery, and while I could wire to the autogate controller's 5V (it is also a Microchip) that would mean soldering two wires to every time I changed controllers. Again Arduino provided the answer: the LM2596 module that cost RM3.50 on sale.

Do power it up and set the output before you connect it to your PIC18F14K50, for the LM2596 is a step-up (boost) as well as step-down (buck) converter. The trimpot is multiturn, so it might take a bit of turning to get it down to 5V. Hook it up to the PIC18F14K50 and we are ready to mount it into the autogate

You can go faster and jump right to the bluetooth section, but I find while these methodical baby-steps may take a little longer, it sure beats two weeks of confusion while you sorted out the interacting bugs from hardware build, digital output code and bluetooth code. 

Perhaps you can spot the bug in my code, which I will address in the next post. 

Happy trails.

Monday, 18 December 2017

Autogate: remote-control Gate

Autogate: control panel on far left. Note the swing-arm actuator anchored to the bottom of the gatepost

A couple of years ago, we had a remote-control gate installed. Known here as an autogate, it is quite popular, can be retro-fitted and is and not too expensive. The parts are available here online.

It gave no trouble for a year or so, then all of a sudden it would open on its own. Things got a little fraught because at about the same time, the wife had rescued three street dogs, and they would get out and fight with about six other dogs up the road.

Naturally we rushed up the road to get them back in, but they figured we had come to fight the good fight, and not only routed the other dogs but were having a grand time chasing them down and were in no mood to come home.

I had the repairman in and he fixed it, but within a month it opened again. This time I had a different repairman in and he swapped out the controller board and the power transformer. Within six months a lightning bolt hit the gatepost and you guessed it: it opened the gate. Luckily a rainstorm followed and the dogs did not fancy getting wet so we were spared the dogfight.

So I thought it is about time I got up close and personal with that autogate.  It has two swing-arm actuators which was mounted to the bottom of the gates and anchored at the gateposts. When 12V DC was fed to it, it contracted to open the gate or extended to close it.

The controller board has a Microchip controller which operated the relays that powered the actuators. It has two inputs, which when shorted to 0V, opened both gates or just one gate.

Top right: autogate controller. Top left:12V transformer
Power was via a 12V transformer, which the controller board also rectified and generated the 12V for the actuator, 5V for the Microchip CPU and 13.8V DC to charge a 12V sealed lead acid backup battery.

Remote operation is via a separate radio receiver, which takes in 12-15V and when the transmitter buttons are activated will put out a low impedance on its two output lines. Match these lines to the inputs of the controller and you have an autogate system.

12Vdc Actuators

The controller board is RM250, the transformer RM25, the remote control module is RM100. The actuators are about RM250 apiece, depending on the stroke length. For less than RM1000 including labor, you can get an autogate fitted.

Wired up controller board

But what about the problems? The first cause were ants. There are loads of them all over the garden. I find them very useful - get an ant colony living in the roots of your tree and they will keep the termites from eating the tree alive from the ground up. A year after the autogate was up, we had a very wet rainy season. The ants got used to walking along the gates (they are normally shut anyway) to avoid the runoff rainwater.

The control panel was nice dry and warm: ideal way station for wet ants. Trouble is, sometimes they shorted the remote control outputs. A shorted remote is a 'gate open' signal. The first repairman did a warranty repair - he replaced the damaged controller board and sealed up the panel door with sealant.

When the problem recurred, the next repairman found the ants had returned via the air vents in the panel. This time he put in some mothballs and sealed the panel again. Things were fine until the lightning strike, which blew out the fuze and the battery charger circuit.

I got tired of calling autogate repairmen - they are nice people and have reasonable charges, but they are never on time. Their idea of an appointment is sometime within a week. Besides, it is about time the gate got permanently fixed. Or so I thought.

Autogate controllers: Super i (left) and S-1 (right)

Since the controller boards were cheap at RM190 or so I bought a couple. They are different makes but all work the same way. I did not really choose them - they are simply the same ones the repairmen used. I just swapped the boards out when there is a problem. After a while I found it was quite easy to repair them, so I ended up with four boards, two spare, one in service and one a parts carcass.

The wife found the solution the the lightning problem: she simply found the circuit breaker to the autogate panel and turned it off when a rainstorm looked likely. Over the years we developed a routine- the broadband, satellite TV, telephone and autogate got disconnected before a storm and reconnected after.

The last rainy season was so wet water went into the sealed-up panel anyway and soaked the transformer into a leaky sorry mess. This I solved by bagging up the replacement in a freezer bag before remounting it. For good measure I duct-taped a bin liner over the whole panel. Not pretty, but a green bin liner kind of disappeared into the hedge next to the panel.

Problems solved, right? Well, kind of. We finally trained the dogs not to rush out whenever the gates opened (actually each dog was convinced only after she got caught in a closing autogate). We soon learned it was easier not to stand by the gate when it was being opened- the dogs tended not to bolt. Unfortunately this meant operating the remotes right at the limit of their range. Even so it was tolerable until we had a record wet season this year.

Heavy rains soaked the ground year round, and the remotes range decreased a lot. Plus I had been messing about with bluetooth Microchip PIC18F14K50 controllers. The range was so good the webserver in the study could reach the bluetooth PIC18F14K50 at the gatepost!

It is time to upgrade a bluetooth-equipped PIC18F14K50 to output to the autogate controller. With a PHP script in the webserver (my WiFi extended a few meters beyond my gates) we can then use our smartphones as an autogate remote. This will also make it an IoT device. Since my bluetooth PIC already read analog inputs I might as well include a battery voltmeter function.

Stay tuned for the next post the IoT Autogate. Happy trails.

Sunday, 12 November 2017

Tropical Pallet Wood Carpentry

When the MRT project finished last July I had been almost three years continuously on-site and needed a change. A couple of new houses were being built near us and I noticed a lot of lumber were being burnt. Some of it had been used as concrete molds but most of it were used pallets.

I dragged some of the lumber home bought some cheap hand tools and dismantled the pallets.
It is quite a satisfying experience ripping apart stuff with crowbars and sledgehammers but I quickly found that extracting nails reliably required some finesse. The claw hammer and the crowbar broke the nail heads off too often, and all too often in the best pieces.

I bought an old-fashioned pair of pincers and by the third pallet could get every nail out. Even when the claw hammer broke the heads off the pincers could still save the day. This meant I could work the pallets quickly with the crowbar and hammer and revert to the pincers when I had a stuck nail.

When the wood was planed, much to my surprise some were hardwood: keruing, meranti and nyatoh. Some were almost scrap wood (and rightly so given most are burned), part-bark or knotty but a lot were usable. Meranti is endangered; it seems almost criminal to burn it.

In secondary school (high school) I did three years of Industrial Arts. It is a little like trade school- we did electrical wiring, engine maintenance, metalwork and woodwork. I loved the wood work and became really good at it. It helps if your family has a hardware store and you can get your lumber replaced when you mess your project up.

I approached my new hobby like I do my design: I searched the Internet. It was a real eye-opener. Back in the 1970s we used hand tools: jack planes, mitre gauges, set squares, handsaw, chisels. Now there are power tools for everything and you can work really fast. There are power jointers, table saws, power sanders, jigsaws, circular saws. For measurement you now have lasers which actually trace out a true line on the workpiece itself without marking it.

And youtube now has a video for nearly any aspect of woodwork, some by master carpenters. Instructables has some really good designs with complete instructions. It is good to be alive.

I bought a power jigsaw- I thought I would start small in case I lost interest in it. That enabled me to repair a long-broken shed door and seal up the shed so that rats could no longer set up there.
Power Jigsaw

Next I got a power handplane. That was less satisfying- youtube had not reckoned on tropical hardwoods. In no time the blade was blunt and fairly skipped over the hardwood like a water skier. The jigsaw suffered less - it was capable of cutting through metal.
Power plane
That dented my confidence in the new power tools. I had planned to get a circular saw next, but the circular saw looked like a very dangerous bit of kit (danger is relative - I have a chainsaw for over 20 years now) and I had visions of it jamming on tropical hardwood, kicking back and amputating some critical piece of anatomy.
Circular saw can kick back
Remembering my success with the pincers, I went old-school. I bought a tenon saw instead.
Old school: tenon saw
I practiced the youtube master carpenters' techniques on pallet wood with some success, starting on the foreign temperate soft woods. The hardwood was extremely tough going, especially an antique piece of chengal (chengal is nearly extinct in Peninsular Malaysia. Our family store used to sell it but I have not seen it for 30 years). The chengal was so hard the tenon saw was heating up like it was sawing metal. I was not going to give up on it. 

Razor sharp: Bahco Superior 2600

I got out the sharpest saw I have, a Bahco Superior 2600 which for 20 years I have been using to trim hardwood trees in the garden. That did the trick: between the youtube sawing techniques and the Bahco I could make true cuts without getting too tired.Because the Bahco saw groove was quite big, I needed to be very careful marking the wood- it is easy to end up with a piece that was too small.

I needed something to replace that power plane, and was looking for the only thing I was familiar with, a Jack Plane. But the power planes are all the rage these days and the jack plane was more expensive (over RM300) than its powered replacement! And it took two weeks to arrive.
Jack Plane

Happily I found an extremely cheap and suspect pair of hand planes from my favorite neighborhood hardware store. They were so cheap and nasty (RM40 for two) that the proprietor was reluctant to sell it, fearing I would have a bad experience with it.

Cheap and cheerful
True enough the handle was too small and skinned my right hand after a day's use. And the depth setting would not hold when in heavy use, i.e., when used on hardwood. Maybe it was because I did not expect much from the planes, but they did the trick and finished up where the power plane failed. I had to sharpen it a few times a day, but I was already good at sharpening (another youtube skill). The constant dismantling also made the wandering depth setting a little more tolerable.

I was all set. First I needed a workbench so the first batch of reclaimed wood became just that.

Palletwood workbench

Next I made a little stool to go with the bench. This time instead of building it with butt joints and holding it together with steel screws I got a little more ambitious and went for lap joints and wood glue. The glue was a revelation. It was incredibly strong, stronger than the wood and hardwood at that.

I had to relearn my jointing and sawing; the stool ended up a little squiffy but was very strong.

That was August. It is now mid-November and the rainy season means I am indoors a lot and pallet woodwork is starting to look inviting again. I thought I would try for another table, but this time better finished, even stained and polished. I put in an online order for that Jack Plane.

The tabletop is mostly 4"x0.5" pallet planks planed true by hand and glued together. It is a little rough in spots because of the odd knot and because I had failed to match the grain when I lined up the boards to be glued. But it is mostly there and I figure it can be smoothed with a sander.

Palletwood table top. Note the magic glue. You can never have too many clamps
 But I would get to try out my new sash clamps. Another revelation: the old ones were very heavy and expensive. These new clamps are a dream.

Because of the size of the top, the plane was to hold it in place with glue, then reinforce it with a few steel screws. Fastening hardwood with screws with precision was unexpectedly difficult. The holes need to be drilled in advance or the wood might split especially at the ends.
More power! The battery powered drills do not play well with hardwood.

More often the wood was so hard the screw heads simply stripped or the screw got stuck. There are very few thing more disheartening than stuck screws. Another unpleasant surprise was a stuck power drill (used as a power screwdriver) can blow the house mains fuse, but mostly you just released the magic smoke.

Going back to basics was not an option this time. Using a hand drill on hardwood is far too exhausting.
One for the museum: a hand drill

After the glue was applied, aligning the holes became difficult, after which the screw might again get stuck. Sometimes a screw does not penetrate the second piece properly and when forced in can act like a little crowbar and pry the pieces apart.

Then there is the drilling. Power drill bits for woodwork was unexpectedly difficult to buy, especially the smaller sizes. And the smaller bits are the ones that tend to break. After breaking all the bits in a single session I ended up using bits meant for metalwork. These went in readily enough, but the flutes do not excavate the waste wood well enough, resulting in a stuck bit. Even the carpenter's pincers were no good on a stuck bit- the high-speed steel is too brittle and snaps off.

Eventually, I found the waste wood can be excavated by repeatedly withdrawing the bit. The advantage of the metalwork bit is that if the wood is too hard to drill properly, I can simply set the speed to high and the resulting friction simply burns the bit through the hardwood. This should be done sparingly to minimize the burnt area. Adding small amounts of WD-40 also seems to help. This is very hard on the power drill so you need a good one.
WD-40: your magic cure for all things stuck

The glue dries in 24 hours, so wish me luck. In the meantime, in the spirit of Tim Allen's Home Improvement, "More power!" to you.

Wednesday, 8 November 2017

Controlling the Raspberry Pi Robot Platform from an Android Smartphone

It is sometimes more convenient to control the raspberry pi robot platform from the smartphone.
Raspberry Pi robot platform with smartphone HTML RC control

Perhaps it is because I am now lugging around a 17" Asus X751L laptop.

There are several options. The quickest way is to ssh into the Raspberry Pi using an app like Juicessh. I can then reuse the C programs forward, left and right.

This turned out to be quite limiting, especially at my age. The smartphone keyboard and screen are really too small for the Linux command line interface to be used for a useful length of time.

But that's what rapid prototyping is all about. We build another one as fast as possible, preferably with available resources. Time for the next iteration.

Rather typing a separate command (in this case a bash script) perhaps it is better to make a program to issue direction commands on a single key press. This takes us to the C program arrowkeys.c

The process is really quite quick and simple: I just googled for the code, starting with general terms lile 'linux', 'keypress', 'C language'. C just happened to be the language I am most familiar with. 'python' or your favoorite language should work just as well.

This search leads me to ncurses, the library that I need. We refine the search further by adding 'sample' and 'code', and this time google leads us to to the function I need, wgetch. This in turn leads us to the article 'A Simple Key Usage Example'. I copied the code wholesale, and just had to add the C library stlib.h so that the keypress code can use system() to invoke my direction programs.

            There is the program. You compile it thus:
gcc -lncurses -o arrowkeys  arrowkeys.c

 *                                                                            *
 * Simple program to issue motor commands in response to keyboard arrow keys  *
 * Copied from:                                                               *
 *                  *
 *                                                                            *
 * 2017-10-24 CM Heong                                                        *
 *                                                                            *
 *                                                                            *
 *                                                                            *
 *                                                                            *
 *                                                                            *
 *                                                                            *
 *                                                                            *
 *                                                                            *
#include <stdio.h>
#include <ncurses.h>
#include <stdlib.h>

#define WIDTH 30
#define HEIGHT 10

int startx = 0;
int starty = 0;

char *choices[] =
  "Back    ",
  "Left    ",
  "Right   ",

int n_choices = sizeof(choices) / sizeof(char *);
void print_menu(WINDOW *menu_win, int highlight);

int main()
  WINDOW *menu_win;
  int highlight = 1;
  int choice = 0;
  int c;

  cbreak();     /* Line buffering disabled. pass on everything */
  startx = (80 - WIDTH) / 2;
  starty = (24 - HEIGHT) / 2;

  menu_win = newwin(HEIGHT, WIDTH, starty, startx);
  keypad(menu_win, TRUE);
  mvprintw(0, 0, "Use arrow keys to go up and down, Any other key to exit");
  print_menu(menu_win, highlight);
    c = wgetch(menu_win);
      case KEY_UP:
         system("/home/heong/piface/piface-master/c/forward 100");/*2017-10-31*/
      case KEY_DOWN:
      case KEY_LEFT:
         system("/home/heong/piface/piface-master/c/left 100");/*2017-10-31*/
      case KEY_RIGHT:
         system("/home/heong/piface/piface-master/c/right 100");/*2017-10-31*/
        highlight=n_choices; /* exit */
    print_menu(menu_win, highlight);
    if(highlight==n_choices)  /* Exit chosen - out of the loop */
  mvprintw(23, 0, "You chose choice %d with choice string %s\n", choice, choices[choice - 1]);
  return 0;

void print_menu(WINDOW *menu_win, int highlight)
  int x, y, i; 

  x = 2;
  y = 2;
  box(menu_win, 0, 0);
  for(i = 0; i < n_choices; ++i)
  {  if(highlight == i + 1) /* High light the present choice */
    {  wattron(menu_win, A_REVERSE);
      mvwprintw(menu_win, y, x, "%s", choices[i]);
      wattroff(menu_win, A_REVERSE);
      mvwprintw(menu_win, y, x, "%s", choices[i]);

Now arrowkeys worked great on the laptop. But on the smartphone, I had to take my eyes off the robot and look at the screen, and this gets wearing very quickly as there a a lot of motion commands.

What I need is just 3 very big buttons, for forwards, left and right. The obvious way is to use an App, perhaps even write one using the MIT App Inventor.

At this point my smartphone,a Nexus 5 failed. I quickly reverted to my trusty Nexus 1, but in my haste to install the SIM card, I broke the phone's SIM connector. The second backup is a Leonovo A390, which did not have much disk space left after installing Whatsapp.

But we can work around this: if I ran a website from the Raspberry Pi, I can invoke the C motion programs from php, via the function exec(), very similar to the trusty system() function.

This means however I have to come up with the button program in html. The added advantage is it will work both in the laptop and the smartphone.

As you suspected, it is back to google. 'html', 'button' and 'system' eventually led me to the 'form', 'exec' keywords. From there the search narrows down to w3schools.

Not knowing web programming should not stop you: it didn't stop me. If you have a background in at least one computer language you should get by. The result would not be pretty code, but you will have a workable (well, sort of) prototype you can show as plausible progress.

From there on, a full day's frobbing to get the buttons screen just right, in the process discovering CSS 'style' in the process. The resulting program, robot.html and upbutton.php:

$cat robot/piface/piface-master/robot2/robot.html
   <!-- \/ starthtml -->
   <META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=iso-8859-1">
   <META NAME="keywords" CONTENT="Heong Chee Meng electronics engineer Android Raspberry Pi Robot Platform Control">
   <META NAME="description" CONTENT="Heong Chee Meng's robot control website.">
   <META NAME="author" CONTENT="Heong Chee Meng">
   <TITLE>Heong's Robotic Control Website</TITLE>
Heong's Robot Platform Control Website
<p style="float: left; width: 33.3%; text-align: center">
<form action="upbutton.php" method="post">
  <button type="submit" name="buttonu" value="Connect"><img width="180" height="180" alt="Connect" src="./upbutton.svg" align="center"></button>
<! </form>
<p style="float: left; width: 33.3%; text-align: left">
<form action="leftbutton.php" method="post">
  <button type="submit" name="buttonl" value="Connect"><img width="180" height="180" alt="Connect" src="./leftbutton.svg" align="left"></button>
<! </form>

<p style="float: right; width: 33.3%; text-align: right">
<form action="rightbutton.php" method="post">
  <button type="submit" name="buttonr" value="Connect"><img width="180" height="180" alt="Connect" src="./rightbutton.svg" align="right"></button>
<form action="bleh.php" method="post">

<!-- /\ end html  -->

$cat robot/piface/piface-master/robot2/upbutton.php
  if (isset($_POST['buttonu'])) {
    $result = shell_exec('ls -l upbutton.svg');
    echo "Done!<pre>$result</pre>";
  if (isset($_POST['buttonl'])) {
    $result = shell_exec('ls -l leftbutton.svg');
    echo "Done!<pre>$result</pre>";
  if (isset($_POST['buttonr'])) {
    $result = shell_exec('ls -l rightbutton.svg');
    echo "Done!<pre>$result</pre>";

You can get the button images from here. Put the files into a subdirectory 'robot' in your apache directory, which in Slackware should be:


You start the webserver with

chmod +x /etc/rc.d/rc.httpd

/etc/rc.d/rc.httpd restart

The next step would be to have the phone and the Raspberry Pi log into the same wifi hotspot. This is straightforward if you are at home, just have both your Pi and smartphone log into your wifi hotspot.

If you are on the move (say you need to demo your new prototype at a client's place), just set your smartphone to be a wifi hotspot and have Raspberry Pi connect to it.

If you use static IP (say your Pi is or if you use a DHCP server, the command

dhclient -v wlan0

will display the Pi's IP.

From there it is just a matter of typing in the link into your favourite browser - I used Firefox:

By now I have a spanking new Samsung Note 5, which unhappily was slimmer than the stack of RM50 bank notes needed to buy it.

After playing RC car with the robot (I've always wanted one!) for a while it would be nice to be able to do this remotely, like, over the Internet. This would have telepresence features. Happily, you just need to configure your home wifi modem router to host the Raspberry Pi website, and the same setup should work.

Now that I have the Note 6 I just needed to complete my search for the Android App to do ssh commands with big buttons. The first hit for 'ssh', 'button', 'app' turned up HotButton SSH. Now this should be a lot less painful to get working than html and the only damage would be to my pride.

The HTML version's advantage is that it is the same whether you used a desktop, laptop or a smartphone. Best of all, it is easy to link up as an Internet of Things (IoT) device. If you were deploying an interface for, say an MRT SCADA system for both the Station Control Room as well as mobile workers this would be an advantage, especially with training costs.

The moral here is rapid prototyping simply uses the resources on hand. Time is the essence- often it is much easier to find out the faults if you (or the customer) have some form of the prototype to work on. The individual motion programs were not suitable for repeated use, and arrowkeys.c although easy to do (with my skill-set) proved OK for laptop use, but using a laptop proved inconvenient.

It sounds trendy. Indeed DevOps uses a similar approach for its 'Continuous Delivery'  portion, but this approach would sound familiar in the US during the Great Depression.

Happy trails.

Sunday, 29 October 2017

The Wild Wild East: Perils of Online Shopping

Fresh from the heady successes of the low-cost Arduino parts, all purchased online, I bought a power bank which not only puts out the standard 5V but 12V, 19V and can even jump-start the car.

Good deal? Note the 30000mAh capacity listed
MY55MART the seller at lelong promised a 30000mAh lithium power bank. The link for the product will probably not be up for that long, but at the moment it is still here.

It arrived very quickly and the first indication something is not right was that it was lighter than my 20000mAh power bank. The original China manufacturer's printing on the box does not mention 30000mAh.

The manufacturer does not mention 30000mAh
When unboxed, the product looked the same as the picture on the original link, down to every last accessory. I topped up the charge overnight.

The next morning, I tested it on my laptop, an Acer AspireF15 with a battery low on charge. Set the power bank output to 19V and it started up well, and proceeded to charge the laptop as well as a computationally-intensive kernel compile. So far so good. After about an hour the power bank was still going strong but the charge indicator on it went from 4 bars to 3. That is 25% for an hour. That seems a little fishy.

Lithium batteries capacity are usually rated at the lithium cell voltage, 3.7V. It should really be rated at nominal output voltage, 5V like a lead acid battery, but oh well.

Now capacity changes depending on the rate of output. I intend to compare it against a 20000mAh as well as a 8000mAh power bank. Useful load would be that drawn by a Raspberry Pi with a Piface cape, about 360mA.

The plan would be to power up a Raspberry Pi with its Piface cape, and print a message every 20 minutes or so until the battery runs out. I topped up the charge again, overnight.

Power bank capacity test: Raspberry Pi and Piface as load. Note the network cable connection to LAN

The voltage and current is checked (but not recorded) using a USB Charge doctor, as the load is largely the same at 360mA.

Note the Charge Doctor reading the load current at 360mA
 The program is simply a bash script. Just ssh into the Raspberry Pi via its copper LAN: 

ssh -t xx.xx.xx.xx

Set the date and time - the Pi does not have a realtime clock:
root@modem:/home/heong# date 1101094317
Wed Nov  1 09:43:00 MYT 2017

Notice here I have set the date wrong (it is actually Oct 29) but the time is correct.

The bash script is simply:

root@modem:/home/heong# while [ 1 ]; do date; sleep 1200; done
Wed Nov  1 09:45:50 MYT 2017
Wed Nov  1 10:05:50 MYT 2017
Wed Nov  1 10:25:50 MYT 2017
Wed Nov  1 10:45:50 MYT 2017
Wed Nov  1 11:05:50 MYT 2017
Wed Nov  1 11:25:50 MYT 2017
Wed Nov  1 11:45:50 MYT 2017
Wed Nov  1 12:05:50 MYT 2017
Wed Nov  1 12:25:50 MYT 2017
Wed Nov  1 12:45:50 MYT 2017
Wed Nov  1 13:05:50 MYT 2017
Wed Nov  1 13:25:50 MYT 2017
Wed Nov  1 13:45:50 MYT 2017
Wed Nov  1 14:05:50 MYT 2017
Wed Nov  1 14:25:50 MYT 2017
Wed Nov  1 14:45:50 MYT 2017
Wed Nov  1 15:05:50 MYT 2017
Wed Nov  1 15:25:50 MYT 2017
Wed Nov  1 15:45:50 MYT 2017
Wed Nov  1 16:05:50 MYT 2017
Wed Nov  1 16:25:50 MYT 2017
Wed Nov  1 16:45:50 MYT 2017
Wed Nov  1 17:05:50 MYT 2017
Wed Nov  1 17:25:50 MYT 2017
Wed Nov  1 17:45:50 MYT 2017
Wed Nov  1 18:05:50 MYT 2017
Wed Nov  1 18:25:50 MYT 2017
Wed Nov  1 18:45:50 MYT 2017
Wed Nov  1 19:05:50 MYT 2017
Wed Nov  1 19:25:50 MYT 2017
Wed Nov  1 19:45:50 MYT 2017
Wed Nov  1 20:05:50 MYT 2017
Wed Nov  1 20:25:50 MYT 2017

The difference between the last output time message and 09:39 would be taken as the duration of the load. Despite the printed start time of 09:45 the actual start time was recorded separately using my watch, as the Pi takes a little while to start up. The measurement error here is 20min.

If the power bank was indeed rated at 30Ah, 3.7V derated to 5V (neglecting the inefficiency of the power supply) means 30 x 3.7 / 5 or 22.2Ah. At consumption of 360mA this means 61.7 hours.

For a short while, maybe a couple of minutes on startup, the Pi actually drew 410mA, but this quickly settled down to 360mA. At shutdown it drew 170mA.

The Raspberry Pi stopped at 20:25 that makes 10.7 hours or 5200mAh (10.7 x 0.36 x 5 / 3.7), woefully short of the promised 30000mAh.

It does not mean the product did not work - it did power my laptop as well as charge an empty battery. It did jump start a car. The China manufacturer did not claim 30000mAh capacity, rather it seems to be the Malaysian seller doing it.

Here is another notice for roughly the same product by the same seller but without the contentious claim:

Such are the perils of online shopping in the Wild East.