Raspberry Pi Zero W as IoT Bluetooth Gateway Part3: Replacing Slackware with Raspbian

Something's missing ... yes the USB bluetooth dongle

Nowadays I use the iterative development method. Following Part 1 and Part 2, we used the Raspberry Pi Zero W as an IoT Bluetooth Gateway daily for about six months. It was more secure than using the smartphone to directly pair to the autogate bluetooth module HC-06 - the PIN is only 4 digits and easily brute-forced.. Using a WiFi repeater, its range was much better. We could now reliably operate the autogate without getting out of the car.

In addition the Pi Zero W gateway hogged the HC-06 all the time, and prevented any intruder from pairing to it. To gain access the intruder would now have to brute-force the WPA2 passphrase in the WiFi repeater, a much tougher opponent.

It also got turned on and off a lot to avoid being damaged by lightning, and sometimes it would fail to start up properly. This is usually due to HC-06 connectivity issues- perhaps it was not unpaired properly or was later that usual being discovered. This allowed the startup to be adjusted using my bash script in /etc/rc.d/rc.local.

Now that it was stable enough it was time to improve it further. From Part 2, Slackware 14.2 did not recognize the Raspberry Pi Zero W's builtin bluetooth module. Perhaps the Raspbian kernel I used was not current enough or perhaps my Slackware installation lacked something, but having to use an RM32 bluetooth dongle with an RM42.20 Raspberry Pi Zero W rankled.

I downloaded the Raspberry Pi NOOBS, and following the installation guide, installed the latest Raspbian OS into by Pi Zero W. Make sure to set up the WiFi connection to your broadband. The builtin bluetooth controller worked first time and had no trouble working with the HC-06 in the autogate. I repeated the procedure in Part 1, and upgraded the firmware/BIOS in the Slackware 14.1 image, but the builtin bluetooth controller did not come up.

Now I could have gone further and upgraded the Linux kernel as well, but perhaps it is time to work with Raspbian/Debian for a while. At least I would not have to keep up with the updates.

To use Raspbian for the IoT Bluetooth Gateway, I needed pymodbus, apache, and php. In Slackware, pymodbus needed upgrades to python 2.7, pip and pysetuptools. Checked in Raspbian:

root@raspi-0-w-2:/root# pip --version
pip 9.0.1 from /usr/lib/python2.7/dist-packages (python 2.7)
root@raspi-0-w-2:/root# python --version
Python 2.7.13

Now that was a pleasant surprise: no upgrades necessary. Now for pysetuptools: Make sure your WiFi connection is working before you try this:

root@raspi-0-w-2:/root# apt-get upgrade python-setuptools

Reading package lists... Done

Building dependency tree

Reading state information... Done

python-setuptools is already the newest version (33.1.1-1).

python-setuptools set to manually installed.

Calculating upgrade... Done

0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

Again, no upgrades necessary. Might as well go ahead with pymodbus:

root@raspi-0-w-2:/root# pip install -U pymodbus

Installed successfully, no fuss. Now my Modbus server program needed twisted, so:

root@raspi-0-w-2:/root# pip install twisted

No problems there. Also required is cryptography:

root@raspi-0-w-2:/root# pip install cryptography

Now for apache. Before that just to be sure I checked for upgrades to Raspbian:

root@raspi-0-w-2:/root# apt-get update

No upgrades necessary- I am beginning to like Debian. Now for apache:

root@raspi-0-w-2:/root# apt-get install apache2 -y

No fuss, apache ran right out of the box (check using http://localhost/index.html):

root@raspi-0-w-2:/root# find /var -name index.html

/var/www/html/index.html

root@raspi-0-w-2:/root# ps -ef

root     20567     2  0 14:14 ?        00:00:00 [kworker/0:3]

root     21396     1  0 14:16 ?        00:00:00 /usr/sbin/apache2 -k start

www-data 21398 21396  0 14:16 ?        00:00:00 /usr/sbin/apache2 -k start

www-data 21399 21396  0 14:16 ?        00:00:00 /usr/sbin/apache2 -k start

root     21723  1705  0 14:18 pts/1    00:00:00 ps -ef

Similarly, php just worked, striaght from the box:

root@raspi-0-w-2:/root# apt-get install libapache2-mod-php php

The web server was linked to pymodbus using the same method as before. Now Raspbian defaults to using dynamic IP, and ssh disabled, which is not very useful if you are using the Pi in "headless" (ie without monitor, mouse or keyboard). ssh is easily enabled via the Raspbian desktop or using raspi-config. To get it to use static IP simply add the following lines to /etc/dhcpcd.conf:

interface wlan0

static ip_address=192.168.1.2/24

static routers=192.168.1.1

static domain_name_servers=8.8.8.8

Lastly Raspbian uses the much improved bluetoothctl, which is an interactive program and not easily included in a bash script like bluez. Luckily Linux has just the thing we need: 'expect' can be used to fool bluetoothctl into thinking it is interacting with a human via the keyboard.

root@raspi-0-w-2:/root# apt-get install expect

Put it in a little bash script and you have a much more robust startup script than simple-agent:

root@raspi-0-w-2:/root# cat ./hc-06.sh

#!/usr/bin/expect -f

set prompt "#"

set address [lindex $argv 0]

spawn sudo bluetoothctl -a

expect -re $prompt

send "remove $address\r"

sleep 1

expect -re $prompt

send "scan on\r"

send_user "\nSleeping\r"

sleep 5

send_user "\nDone waiting for controller\r"

expect "Controller"

send_user "\nSleeping ... waiting for autogate\r"

sleep 5

send_user "\nDone waiting for autogate\r"

expect "HC-06"

send "scan off\r"

send "trust $address\r"

sleep 2

send "pair $address\r"

sleep 2

send "0000\r"

sleep 3

send_user "\nShould be paired now.\r"

send "quit\r"

expect eof

Note: replace '0000' with your password and launch the script thus:

root@raspi-0-w-2:/root# ./hc-06.sh 12:34:56:78:9A:BC

Lastly bind to your bluetooth device to get the serial port /dev/rfcomm0:

rfcomm bind 0 12:34:56:78:9A:BC 1

And you are ready for the pymodbus program, which I launched in the background and captured its output to a log file:

root@raspi-0-w-2:/root# python ./autogate_server.py >> ./autogate.log 2>> ./autogate.log &

Raspbian turned out to be quite pleasant to use, but then it is Linux, so we expect nothing less. 

Happy Trails.

The SPI who came in from the cold

The film was based on John Le Carre's novel of the same name

“Don't give it to them all at once, make them work for it. Confuse them with detail, leave things out, go back on your tracks" -  John le Carré, The Spy Who Came In from the Cold

I had always wanted a wall clock which self-corrects from the Internet using NTP. A few weeks ago I bought a cheap MAX7219 8-digit 7-segment display. It seems a perfect match for the ESP8266, which will access NTP from my home WiFi. The MAX7219 uses SPI which should link to the ESP-12E and can then be scaled up to wall clock size.

Straightforward project, perfect for IoT. It started well- I linked two ESP-12E using SPI and the sample programs in Arduino IDE. That worked a treat. I may not even need to learn the Lua programming language. Then real life intervened.

It did not work. The Lua sample code had worked when I used ESP-12Es as master and slave. Did I get a bad MAX7219? It certainly was a cheap clone from a no-name China supplier. Was it the level translator, the Lua program or the ESP-12E? 

It was weird- at times it seems to start working (the display comes on) only when I power it off.It seemed to work even when the MAX7219 did not have power - the MAX7219 seemed to be able to leech enough power from its signal lines. Randomly disconnecting a signal cable sometimes produced a fleeting glimpse of a working display.

But after three days, I had to admit- I was stuck. But what is really needed is an admission that the easy way failed. 
I needed to know whether it is a problem with the MAX7129 or the software.One way was to get the MAX7219 working with different software. The other would be to buy another MAX7219 module, preferably from a different supplier. To rule out the ESP-12E's flakey start I need a different controller. 

I had SPI source code for the PIC18F14K50 CPU from Microchip- it came with their development kit. I even have a finished PIC18F14K50 prototype PCB I designed myself some years ago but had not gotten round to testing it.

Testing the SPI PIC18F14K50. Being USB devices they came up in Linux as /dev/ttyACM0 and /dev/ttyACM1

The SPI PIC18F14K50 and their software (sample code from the Microchip MCC18 compiler) did not work either. Time to back off SPI. The PIC18F14K50's SPI pins can be reprogrammed to use the I2C protocol. Maybe I will have better luck - at least it will prove I have working PIC18F14K50.

Fell back to I2C. Did not work either! It is starting to look like I am going backwards. Time to bring up the Death Star, my oscilloscope. Nothing gets by the oscilloscope. I love my scope, but just setting it up takes one day. It forces me to take things real slow, step by steady step, soldering test points, checking voltages pin by pin. Oh, and you now have to read the datasheets. Ugh. Buzzkill. Time to RTFM. 

And true enough, the scope showed problem is the MCC18 code. It does not work. That is quickly fixed, and now I2C between two PIC18F14K50 work. The SPI worked soon after, but the datasheets said there is no real standard for SPI, there can be 4 variants. The timing cycles for the PIC18F14K50 and the MAX7219 needs to be carefully examined for a match.

The 4 SPI modes supported by the PIC18F14K50

Luckily, there is a match: the second mode in the PIC18F14K50 matches:

MAX7219 Timing Diagram. 

Set up the PIC18F14K50 with the MAX7219. It did not work! In fact the symptoms are much the same as the ESP-12E, the display flashes working on power down, the MAX7219 starting up without power. But this time I have the scope. Turned out the PCB labelling was wrong- SPI MISO and MOSI were swapped round. And more MCC18 code needed to be replaced, but eventually it worked:

MAX7219 working with the PIC18F14K50

It's time to get back to the source of my grief- the software in the ESP-12E sample code. Now this is not a reflection on the software quality or the author- SPI simply had too many variants for the code to work right off the bat. Plus the author was interfacing to his SPI device using 32-bit SPI cycles. The MAX7219 needed 16-bit cycles.

Bottom: working SPI  on ESP-12E and MAX7219. Top: the mighty Death Star

You can get lucky most times and have your project work just using Internet resources. Indeed, you should and must use the optimistic approach first. But sometimes you just have to get out the Death Star and do the slow but steady steamroller. 



Later, I found the LEDControl already has working software for the ESP8266. It had not come up with my Google searches becasue it did not mention SPI, just the MAX7219.

The ESP-12E SPI interface to MAX7219 has had twists and turns worthy of a John Le Carre movie. Such is life. Happy Trails. 

Heong, when is the prototype going to be finished?

I learned long ago, never to wrestle with a pig. You get dirty, and besides, the pig likes it.
- George Bernard Shaw

Blast from the past: the Macsym 150, running Macbasic was an early embedded computer

Back when I was at my first job designing SCADA systems for Oil & Gas installations, a new system perhaps took two to three years to complete. You got from the customer what he wanted in detail, the 'requirements', produced a massive document, the 'specifications' and then proceeded with the 'design', and then the hellish part, 'testing' and hopefully the delivery and installation. These days it is called the waterfall model.

This was so with my first project 1985-1987, a SCADA system to monitor 4 Daniel 2234 flow computers. It was based on the ancient MACSYM 150 by Analog Devices, both now extinct. While wallowing in the now-familiar post-project blues, I asked to tag along for the big one- a building automation system for a whole building complex comprising tower block, museum, theater and library. It was critical to the company- in fact the Technical Director handled it himself. Being the FNG, I had to promise not to touch anything.

Heavy metal- PDP-11/23+

It was based on a Digital Equipment PDP11/23, heavy metal in those days. It ran Ultrix. A BAS is not that different from a SCADA. It required microprocessor-based RTUs.

By 1987 the project was in trouble. The company by then could not afford to import the rest of the RTUs nor pay for the software, but was still bound by contract to deliver the BAS. The performance penalties alone would spell the end. Desperate times required desperate people- going three months without pay does things to you.

We had the nucleus of the BAS- the PDP11/23+, one MTU and 6 RTU, so we knew how it should work. There was no choice- we produced the remaining 4 MTUs and 25 RTUs, copied it lock stock and barrel. We installed it and got a shock reject from the customer - the system was too slow. Despite the agreed-upon 'Requirements', and 'Specifications'.

There was little time to think. All I had on hand were IBM PCs. I got the newest one, an IBM PC/AT. The only choice of OS was SCO Xenix/286 and I pretty much wrote the software on the fly, alternating between customer site and office. The result was a 4X improvement in speed, and we got it accepted by the skin of our teeth.

Small beer- IBM PC/AT

 We had taken a properly engineered system, replaced major components willy-nilly and got away with it! It was a one-off, we had dodged a bullet and it was time to go back to the straight and narrow, the waterfall method. I could not have been more wrong.

Over the years working at Westinghouse, HP, Motorola, then ON Semiconductor this pattern was to be repeated again and again. The waterfall method does not work.

So, what worked?

Being close to the customers did. You visited them often. Better yet get a desk there too. This is because often the customers does not know what they want. They know what they don't want- if they can see it and use it they will soon tell you.

You then need to whip up a prototype to show the customer. The faster the better. It does not have to meet all the requirements - the customers are certain to reject even the perfect ones. But nearly every time they will tell you why they hate it.

You then repeat the process - rush back to the office, and repeat the entire process and rush back to the customer with a new prototype. Eventually they will relent and you will succeed. Most customers are flattered you are willing to develop a bespoke system for them. In fact most times the customers do not get to meet the designers. They just keep hearing the mantra from the sales or operations guys: "You have agreed to the specifications and this is what we will deliver".

But what of the cost, the failed prototypes, the components discarded? There is no choice but to reduce cost throughout the process. The man-hours are about the same, the delivery times are often fixed in advance. You start with the components on hand. Failing that, off-the-shelf parts. You use the minimum of custom parts. Software is a godsend. There is little material cost and it can be changed very fast, often right there in front of the customer.

And that is why prototypes are never going to be finished. For there is always the next customer. Sometimes over many customers the design may settle to a happy equilibrium, and you get to declare victory and move on, but don't bet on it.

You will see this pattern repeated in this blog- the prototypes sort of work but there is always something lacking. If I am lucky this results in fresh insight on the design. It helps if you like design and prototyping. For me, this is pachinko - something done for fun. Like wrestling in the mud.

Over the years, especially in software, this method has been given respectability and names, 'iterative development', 'extreme programming'. You automate it up the wazoo and apply it to large teams(indeed the whole company), and it becomes 'DevOps'.

That used to be just my fangs out hair on fire approach, a beast to be let out only when necessary.

Happy Trails, and see you at the pachinko.

Klang Valley MRT Opens, a Project Ends

Page 3 of the Sin Chew

17 July 2017 was opening day, and it was supposed to be a doddle. We had finished a month ahead of schedule; but then they ordered 4 more escalators at the last minute.

The fastest way to get around was to use the MRT itself. On opening day I finished up at Station 5 and was expecting to take the train to Station 26 but there was a snag: the trains did not run all the way from Stations 5 to 34. The new segment only went from 16 (Museum Negara) to 34 (Kajang).

Station 5

So at opening time 16:00 hours I was at  Museum Negara and went through as fast as I could. A reporter took a picture as I went through and the next day, it made page 3 of the local Chinese paper, the Sin Chew Jit Poh. 

Post Project Blues

Project C1313K started in 2013 and required remote control of over 200 escalators over 34 stations spanning an area of over 50km. The feature did not exist so it would have to be designed, built and tested on the fly. 

Senior R&D Engineer Edward Boo at ease

For the last 2 years we ate, slept and dreamed C1313K. There were setbacks, attrition, additional requirements, variation orders, but these are teething pains. You focus: eyes on the prize...

The team in 2015

You get used to being occupied by a design for years. Then it is over. OK, it succeeded, it works and is delivered on time and we get to fight another day. There is the high, the celebration, the adrenaline rush.  

Northern Zone passed Field Tests Oct 2016

Southern Zone passed field tests May 2017

So where do you go from there? It is strange not to think about C1313K when you get up or just before you go to sleep. Welcome to the Post Project Blues.

 

Reprieve

 And then you get days like these, 18 July 2017 when you make page 3 of a national paper. Did I say life was good?

Moving On

Time for MRT2 (KVMRT SSP Line) then. Goodbye, C1313K.

But with the word the time will bring on summer,
When briers shall have leaves as well as thorns,
And be as sweet as sharp. We must away;
Our wagon is prepared, and time revives us:
All's well that ends well; still the fine's the crown;
Whate'er the course, the end is the renown.

- William Shakespeare