In my last post I tested the ESP8266-based ESP-12E using its partially-available AT Command set, and showed that we can make a simple IoT device with it.
Alternatively we can directly program the 80MHz 32-bit Tensilica CPU inside the ESP8266. After all, the NodeMCU folks wrote ESP8266 firmware so that we can run Lua programs directly on the ESP8266.
If you are new to all this, don't be fazed- all this new stuff is really easy to learn and use. They are designed like that. This is also my first experience of NodeMCU, Lua and ESP8266.
We program the ESP-12E using the Arduino IDE. It supports much more than the ESP8266, which is just a tiny part of the Arduino ecosystem.
Screenshot of Arduino IDE on Slackware 14.2 (bottom left). and the Arduino IDE Serial Port Monitor (top left). Henry's Bench website (right)
I would have preferred a smaller program, the command line-based nodemcu-uploader would have been perfect, but it was unable to work with my ESP-12E. I suspect I would first have to change the firmware.
Since I am running Linux Slackware 14.2, we first need to install Arduino IDE. I got the source files arduino-1.8.5-linux64.tar.xz from here. Using my root account, I simply ran the install script:
$./install.sh
Still as root (otherwise you need to give yourself the permissions to access the USB device /dev/ttyUSB0), and still using the same directory you unpacked your source files:
$arduino-1.8.5/arduino
Henry's Bench does a great job explaining how to set up Arduino IDE for the ESP8266. I would just add that you need a working Internet connection to get it done. His sample program, which I copied and pasted into my Arduino IDE worked right out of the box.
To program and run the stock sample Blink sketch (Lua programs are called sketches), again please refer to the excellent example in Henry's Bench.
While you are setting up the LED circuit, it is sometimes helpful to have an idea how the Blink program is running. I have modified the Blink sketch slightly so that it outputs messages to the Arduino Serial Monitor as well:
void setup() {
pinMode(LED_BUILTIN, OUTPUT); // Initialize the LED_BUILTIN pin as an output
Serial.begin(115200);
Serial.print("Setup done on ");
Serial.println(LED_BUILTIN);
}
// the loop function runs over and over again forever
void loop() {
digitalWrite(LED_BUILTIN, LOW); // Turn the LED on (Note that LOW is the voltage level
// but actually the LED is on; this is because
// it is active low on the ESP-01)
Serial.println("Low");
delay(1000); // Wait for a second
digitalWrite(LED_BUILTIN, HIGH); // Turn the LED off by making the voltage HIGH
Serial.println("High");
delay(2000); // Wait for two seconds (to demonstrate the active low LED)
}
Rather than going to the trouble of building another 3.3V power module for this test, I have simply directly connected the LED to pin D0 of the ESP-12E. The circuit in Henry's Corner is a much better one; it results in a bright light and will work reliably without over-stressing the ESP8266's GPIO output pin.
Often modern CPUs have short-circuit protection for their IO pins, and often the output pins are strong enough to light up an LED dimly. It helps to chose a red LED (they are the oldest and most efficient). Luckily, in this case I got away with it; the LED was just bright enough to be seen, although I had to turn off my study lights for it to come out in the photo.
One possible pitfall with this method is the LED will take up precious mA of power from your ESP-12E PCB. The ESP8266 is effifcient enough but it seems to have trouble powering up from many laptops' USB ports and most USB hubs.
I am powering my ESP-12E from a D-Link USB 3.0 hub with 4A of external power. If your system gets all flakey, you are probably better off using the method in Henry's Bench. The current draw of the ESP-12E is normally about 60mA. When the LED turns on this increased to 80mA. This means the ESP8266 IC is putting out 20mA to the LED which is very close to its maximum running current.
The Blink sketch LED circuit from Henry's Corner
You now have an ESP8266 running a native (ie stored onboard) Lua program on NodeMCU firmware. If you now exit from Arduino IDE, the ESP-12E program should continue to run on its own. In fact the program will now start up every time you turn the device on.
ESP8266 as IoT Digital Output controlling an LED
Next to make it an IoT application, navigate thus:
/*
* This sketch demonstrates how to set up a simple HTTP-like server.
* The server will set a GPIO pin depending on the request
* http://server_ip/gpio/0 will set the GPIO2 low,
* http://server_ip/gpio/1 will set the GPIO2 high
* server_ip is the IP address of the ESP8266 module, will be
* printed to Serial when the module is connected.
*/
// Start the server
server.begin();
Serial.println("Server started");
// Print the IP address
Serial.println(WiFi.localIP());
}
void loop() {
// Check if a client has connected
WiFiClient client = server.available();
if (!client) {
return;
}
// Wait until the client sends some data
Serial.println("new client");
while(!client.available()){
delay(1);
}
// Read the first line of the request
String req = client.readStringUntil('\r');
Serial.println(req);
client.flush();
// Match the request
int val;
if (req.indexOf("/gpio/0") != -1)
val = 0;
else if (req.indexOf("/gpio/1") != -1)
val = 1;
else {
Serial.println("invalid request");
client.stop();
return;
}
// Set GPIO2 according to the request
digitalWrite(16, val);
client.flush();
// Prepare the response
String s = "HTTP/1.1 200 OK\r\nContent-Type: text/html\r\n\r\n<!DOCTYPE HTML>\r\n<html>\r\nGPIO is now ";
s += (val)?"high":"low";
s += "</html>\n";
// Send the response to the client
client.print(s);
delay(1);
Serial.println("Client disonnected");
// The client will actually be disconnected
// when the function returns and 'client' object is detroyed
}
I have simple changed the GPIO pin to the D0 pin I connected my LED to:
// Set GPIO2 according to the request
digitalWrite(16, val);
Compile and upload the sketch to the ESP-12E. Over at a convenient browser in your phone, laptop or desktop type out:
http://IPaddress:8080/gpio/1
And the LED turns on, as before. You might notice I use the webserver port 8080, whereas the normal port to use is 80. This is because my website www.cmheong.com is already using port 80, and I reconfigured my modem router to map port 8080 in the browser to port 80 in the ESP-12E.
There you have it: a simple IoT control of an LED. Add a cheap Arduino relay PCB, and you should be able to switch useful things like light bulbs.
When I finished the last post on the ESP8266, I bought an ESP-12E from robotedu.my for just RM21.90, and it arrived the day after! How could I resist the new toy?
Communication should be simple. There is a CH340G USB to serial converter IC on the ESP-12E and a micro USB connector to boot, so I plugged it into my laptop. True enough a new device file came up: /dev/ttyUSB0. No surprises there.
I ran minicom, and this time it failed with 'Device /dev/ttyUSB0 is locked'. Started up minicom without resetting the device with
minicom -o
The ESP-12E was not responsive. Now there is a reset button on it and every time it was pressed the blue led flashed, and minicom detected something. Eventually, by trying the baud rates one by one, at 115200 baud I got part of a message when it was reset: 'ready'. Well and good, but it refused to go any further.
So it is not going to be simple. A few weeks later, I tried again. This time by first searching the web for startup problems with this board. This link mentioned the board was a little prone to power problems.
Luckily, by this time, yet another new toy had arrived, the D-Link DUB-1370 USB 3.0 Hub, which came with a stonking 4A 5V external power supply. Time to upgrade the ESP-12E's power supply.
This time minicom had no trouble with the device file (/dev/ttyUSB0). The PCB has 'use 9600bps' helpfully printed on it, but it did not work. Luckily I noticed the Reset button, marked 'RST'. On pressing it there are a couple of satisfying flashes from a blue blinkenlight.
A search of WiFi access points came up with a new one:
So there are some signs of life. An Internet search for ESP_42C155 turned up nothing. But luckily the reset button produced some rubbish characters. I simply kept changing the serial port (/dev/USB0)'s baud rate until the output made sense on reset. Eventually, it produced 'ready' at 115200 baud, 8-bit, 1-stop and no parity.
followed by Ctrl-M and Ctrl-J. And got a very satisfying output:
AT
OK
It was time to look in the manual. This produced the version numbers:
AT+GMR
AT version:1.1.0.0(May 11 2016 18:09:56)
SDK version:1.5.4(baaeaebb)
compile time:May 20 2016 15:08:19
OK
Now since the Access Point ESP_42C155 was not encrypted, I logged my laptop into it. dhclient ran successfully and assigned the IP address of 192.168.4.2 to my laptop:
$dhclient -v -1 wlan0
Internet Systems Consortium DHCP Client 4.3.4
Copyright 2004-2016 Internet Systems Consortium.
All rights reserved.
For info, please visit https://www.isc.org/software/dhcp/
Listening on LPF/wlan0/54:27:1e:50:38:c7
Sending on LPF/wlan0/54:27:1e:50:38:c7
Sending on Socket/fallback
DHCPDISCOVER on wlan0 to 255.255.255.255 port 67 interval 8
DHCPREQUEST on wlan0 to 255.255.255.255 port 67
DHCPOFFER from 192.168.4.1
DHCPACK from 192.168.4.1
bound to 192.168.4.2 -- renewal in 2863 seconds
So the little PCB not only has an Access Point, but has a DHCP server. Many of the AT commands in the manual did not work, but a good many did. There are probably a lot of variations in the NodeMCU firmware. This AT command confirms the DHCP server:
AT+CWDHCP?
+CWDHCP:3
OK
And I can even ping my laptop:
AT+PING="192.168.4.2"
+1
OK
However trying the link http://192.168.4.1 from a browser failed. Some posts had suggested some ESP-12E boards also had a webserver. Just not this one.
No matter. Now an IoT device needs to be able to connect to the Internet. I set the ESP-12E to be a WiFi client as well as an Access Point:
AT+CWMODE=3
OK
And set it to log into my house WiFi Access Point:
AT+CWJAP="HouseAP","password"
WIFI CONNECTED
WIFI GOT IP
OK
AT+CIFSR
+CIFSR:APIP,"192.168.4.1"
+CIFSR:APMAC,"2e:3a:e8:42:c1:55"
+CIFSR:STAIP,"172.16.1.102"
+CIFSR:STAMAC,"2c:3a:e8:42:c1:55"
OK
Cool. It looks like the house AP has assigned it an IP. To confirm Internet access I pinged the Google nameserver 8.8.8.8:
AT+PING="8.8.8.8"
+38
OK
This looks good. The manual mentions the ESP-12E can be made to communicate with a WeChat App (the China version of WhatsApp). It can also be upgraded from the Internet. But to keep this blog post short, we go with the AT command manual for now.
An IoT device also needs to be able to receive commands from my phone or laptop. The ESP-12E not having a webserver or ssh server running, perhaps we can try a raw network socket. From the manual, we first dip a toe in the water:
AT+CIPSTART=?
+CIPSTART:(id)("type"),("ip address"),(port)
+CIPSTART:((id)"type"),("domain name"),(port)
OK
That looked good, so let's try to connect to an arbitrary network socket 3344 on the laptop. We can use TCP/IP but let's try the socket raw, using UDP:
AT+CIPSTART="UDP","192.168.4.2",3344
CONNECT
OK
A quick check shows the connection is good:
AT+CIPSTATUS
STATUS:4
+CIPSTATUS:0,"UDP","192.168.4.2",3344,43972,0
OK
Over at the laptop I ran a bash command nc to read the port:
$nc -ul -p 3344
Back at the minicom terminal, I issue:
AT+CIPSEND=11
OK
>
This looks good. The string 'Hello world' is exactly 11 characters, and on typing it into minicom, over at the laptop I get:
$nc -ul -p 3344
Hello world
So we have the basic elements of an IoT device. It can send messages to my laptop, albeit with much typing (in particular the use of uppercase AT commands terminated with Ctrl-M Ctrl-J made things quite ponderous. This can be automated from the laptop using a python script:
$python
Python 2.7.11 (default, Mar 3 2016, 13:35:30)
[GCC 5.3.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
Much work remains. The GPIO commands like AT+SYSGPIOREAD and AT+SYSGPIOWRITE did not work, otherwise a simple (if a little ponderous) IoT application can be written in python using the AT commands in the manual. Without ssh there are probably serious security issues to be addressed first. The ESP-12E firmware should probably be updated, and it would be nice to be able to communicate with it using a browser.
USB Keyboard Emulator. Top, RS485 USB Dongle and bottom: RS485 USB Dongle reprogrammed as USB HID Keyboard
We have a few hundred embedded Linux systems installed by now, and we have always maintained them remotely where possible, by connecting them to a GSM/LTE/4G modem, and then logging in from the office. Where this is not possibly we plugged in via CAT5 copper LAN.
Since we do not use the monitor or keyboard in these cases we often do not install them. The Linux CPU itself might not be easily accessible; for example, it might be mounted on top of a lift.
There are some types of failure where Linux would not start sufficiently to run its network stack or even boot up. This means we could not log in at all. In these cases we would need to attach a monitor and keyboard.
I got tired of having to lug around a laptop, monitor, full-sized keyboard in addition to a rucksack filled with tools and spares. The laptop has an LCD screen as well as a keyboard. Why should I have to carry two of those?
The laptop screen can be connected to the embedded system using a USB HDMI Capture dongle. I got a no-name Chinese make to start with.
USB Video Capture Device HDMI Capture Card USB 2.0 1080P HD Game Video HDMI Capture Card
It cost only RM58 but did not come with an HDMI cable. And it pretty much worked right out of the box and came up as /dev/video1. I simply treated it like a TV dongle:
mplayer -cache 128 -tv driver=v4l2:width=1080:height=720:outfmt=i420 -vo xv tv://
The picture isn't great, it wobbles somewhat and I had to crank up the gamma setting on the laptop LCD, but hey, it sure beats lugging a monitor around.
Pretty soon the keyboard started getting on my nerves. After all it is a much simpler device, why shouldn't I be able to get a USB dongle, plug one end into my laptop and the other into the embedded Linux CPU? It was surprisingly hard to find, in fact I could not find one at all.
We make a lot of USB to RS485 dongles, used mostly for Modbus interfacing. I could just take two of them, connect them back-to-back via their RS-485 ports and plug one into my laptop. It would come up as /dev/ttyACM0, a serial port. At a pinch I could use minicom to act as a crude keyboard.
The other USB to RS485 dongle would be a little more tricky. I would need to reprogram it as a USB HID Keyboard so that the embedded Linux system (indeed the BIOS before that) thinks it is a keyboard.
Luckily I had based design of the USB to RS485 dongle very closely on Microchip's Low Pin Count Development System using the PIC18F14K50 CPU. And Microchip had provided sample code for it including code for USB HID Keyboard.
I immediately reprogrammed my dongle as a keyboard and plugged it in. And sure enough it came up as:
[Wed Jan 31 19:48:48 2018] usb 1-4.5.4: New USB device found, idVendor=04d8, idPr
oduct=0055
[Wed Jan 31 19:48:48 2018] usb 1-4.5.4: New USB device strings: Mfr=1, Product=2,
SerialNumber=0
[Wed Jan 31 19:48:48 2018] usb 1-4.5.4: Product: Keyboard Demo
[Wed Jan 31 19:48:48 2018] usb 1-4.5.4: Manufacturer: Microchip Technology Inc.
[Wed Jan 31 19:48:48 2018] input: Microchip Technology Inc. Keyboard Demo as /dev
ices/pci0000:00/0000:00:14.0/usb1/1-4/1-4.5/1-4.5.4/1-4.5.4:1.0/0003:04D8:0055.00
08/input/input22
[Wed Jan 31 19:48:48 2018] hid-generic 0003:04D8:0055.0008: input,hidraw5: USB HI
D v1.11 Keyboard [Microchip Technology Inc. Keyboard Demo] on usb-0000:00:14.0-4.
5.4/input0
Next I copied over the RS232 code from the same Microchip sample set, 'USB Device - CDC - Serial Emulator', and that worked too.
Lastly I needed to match the minicom keystrokes (where were RS485 ASCII) to the keyboard scan code. The former can be found here and the latter here. It took just two days to prototype, a USB keyboard emulator.
[Updated 2021-03-09]
Despite initial misgivings about using minicom, it is possible to work around this by assigning the less-commonly used ASCII codes, like SI (Shift In), decimal value 15, which minicom can generate as Ctrl-O. Similarly for commonly-used BIOS keys like F2, F5, F6, F9 and F10.
Raspberry Pi Zero W as Bluetooth IoT Gateway installed in the front porch
The original Bluetooth Autogate Remote used a recycled Acer Aspire F15 laptop as an IoT gateway. The laptop ran an Apache webserver for the html/php program. This went very well until a fortnight ago when we had a cool rainy spell and the garden got waterlogged, which always seem to affect wireless communications at the Autogate control box located in the gatepost.
The Bluetooth Autogate Remote started to fail, first on individual button presses, then the laptop bluetooth controller lost the link to the HC-06 in the gatepost. A typical bluetoothctl command sequence and output now goes something like this:
[bluetooth]# scan on
Discovery started
[CHG] Controller C8:FF:28:27:7D:2C Discovering: yes
[CHG] Device 98:D3:32:20:BB:7B RSSI: -90
[bluetooth]# devices
Device 98:D3:32:20:BB:7B HC-06
[bluetooth]# pair 98:D3:32:20:BB:7B
Attempting to pair with 98:D3:32:20:BB:7B
Failed to pair: org.bluez.Error.ConnectionAttemptFailed
When RSSI value got to about -86 the pairing would work but dropped every few days or so. This handy link explained quite well that RSSI meant 'Received Signal Strength Indicator' and has an RSSI table:
Acceptable Signal Strengths
Signal Strength
TL;DR
Required for
-30 dBm
Amazing
Max achievable signal strength. The client can only be a few feet from the AP to achieve this. Not typical or desirable in the real world.
N/A
-67 dBm
Very Good
Minimum signal strength for applications that require very reliable, timely delivery of data packets.
VoIP/VoWiFi, streaming video
-70 dBm
Okay
Minimum signal strength for reliable packet delivery.
Email, web
-80 dBm
Not Good
Minimum signal strength for basic connectivity. Packet delivery may be unreliable.
N/A
-90 dBm
Unusable
Approaching or drowning in the noise floor. Any functionality is highly unlikely.
N/A
So my RSSI went from 'Not Good' to 'Unusable'. Now the laptop's bluetooth controller had 2 walls and a plastic enclosure between it and the Autogate Bluetooth HC-06, so I though it might be a good idea to have the IoT Gateway on the front porch, which not only is only 30m away but had line-of-sight to the HC-06.
That was the purpose of the Raspberry Pi Zero W. My last post showed Slackware 14.1 can run on it. This is handy because my laptop ran Slackware 14.2.
First I need to get the WiFi running on the Raspberry Pi Zero W. That was easy enough. It turned out I was missing some bios files:
-rwxr-xr-x 1 root root 6551 Oct 30 18:36 fixup.dat
-rwxr-xr-x 1 root root 2578 Oct 30 18:36 fixup_cd.dat
-rwxr-xr-x 1 root root 9694 Oct 30 18:36 fixup_db.dat
-rwxr-xr-x 1 root root 9694 Oct 30 18:36 fixup_x.dat
-rwxr-xr-x 1 root root 2820196 Oct 30 18:36 start.elf
-rwxr-xr-x 1 root root 667460 Oct 30 18:36 start_cd.elf
-rwxr-xr-x 1 root root 4956676 Oct 30 18:36 start_db.elf
-rwxr-xr-x 1 root root 3904228 Oct 30 18:36 start_x.elf
These files were really a set (startup.elf needed fixup.dat and so on) and in my case I had start.elf but not fixup.dat. In addition I was missing the file:
-rwxr-xr-x 1 root root 15830 Oct 30 18:36 bcm2708-rpi-0-w.dtb
Once corrected the WiFi interface came up and was connected to my home WiFi hotspot with no fuss. I could now ssh into the Zero W and use it without the monitor or USB keyboard.
The bluetooth controller was a whole different ballgame. After spending a whole Sunday on it, I gave up and decided to use a Vztech bluetooth dongle instead. This I connected to the micro USB OTG connector and worked straight out of the box.
The bluetooth software was the same Bluez used in Slackware 14.2, but being the 14.1 version did not have bluetoothctl. The function I missed most was the bluetooth agent required to pair the HC-06 to the Pi Zero W. Happily a little digging showed that Bluez tarball for Slackware 14.1 has a workable agent, simple-agent which was built but not included when the Slackware upgradepkg program installed Bluez.
I just needed to unpack the tarball and install it:
The next steps involve setting up python, pymodbus and apache in Slackware 14.1, and are best treated separately. You can also refer to other online links for this. The end result is much the same as the previous post, and worked about the same. Not bad for the RM42 Raspberry Pi Zero W, which replaced an RM3,200 Acer AspireF15 as the IoT Bluetooth Gateway.
Now all this seemed a little fly-by-the-seat-of-your-pants, but this is the nature of DevOps/Rapid Development. It is iterative and relies on incremental redesigns with the emphasis on putting out the prototype as quickly as possible. Notice it took a few weeks operating the prototype for the bluetooth wireless problem to be obvious.
Some problems are immediately obvious: the Pi Zero W does not have battery backup while the laptop did. Installing a battery would require other things: a proper enclosure, perhaps a shelf. But in the meantime up goes the latest prototype to fish for other problems.
Perhaps one of the most important things about Rapid Development is customer involvement. One indication the project is going well is the wife got quite used to opening the autogate from her car, and complained immediately when the Bluetooth Autogate stopped working in wet weather.
Happy Trails.
Update: as feared, every few days or so the Raspberry Pi Zero W rebooted by itself. This was tracked by having the pymodbus server program print out regular heatbeat messages:
Cmd z, voltages are [3, 255, 2, 157]
stored values= [3, 252, 2, 154]
address= 0 answer: [3, 255, 2, 157]
CPU 1023 Battery 669
CPU 5.00 Battery 13.88 Volts 2018-01-26 04:34:13
Reply: ÿHCM¡LJP len 10
Cmd z, voltages are [3, 255, 2, 161]
stored values= [3, 255, 2, 157]
address= 0 answer: [3, 255, 2, 161]
CPU 1023 Battery 673
CPU 5.00 Battery 13.97 Volts 2018-01-26 04:46:14
Reply: ïHCMLJP len 10
Cmd z, voltages are [3, 239, 2, 156]
stored values= [3, 255, 2, 161]
address= 0 answer: [3, 239, 2, 156]
CPU 1007 Battery 668
CPU 4.92 Battery 13.86 Volts 2018-01-26 04:58:15
packet_write_wait: Connection to 172.16.1.25 port 22: Broken pipe
We had gotten really used to that WiFi autogate remote, so to keep it going (while I am trying to figure out what went wrong) I simply added this to /etc/rc.d/rc.local so that the IoT gateway program restarted itself. Not using bluetoothctl had a silver lining after all:
I ordered in quick succession a NodeMCU ESP-12E and a Raspberry Pi Zero W. Both came very quickly but the Zero W got off the ground first.
I powered it using its micro USB socket (far right in above picture) with my favourite Slackware distribution in micro SD card, and nothing happened. It drew only 20-30mA current. I used a feisty 2A Samsung USB charger that put out 5.2V.
I had expected a little power LED at the very least. A little concerned I might have got a dud, I shorted the bare 'reset' PCB vias using a pair of steel tweezers.
'Reset' PCB vias (bottom left) marked 'RUN'
Again, nothing. After 30 years of electronics you get used to having your blinkenlights, chief amongst them is your 'Power' LED.
The usual rats' nest
As a last resort, and just because I had a Sandisk Class 10 32GB microSD card, I loaded it with Ubuntu Mate, a recent Linux distribution for the Raspberry Pi B. And, bingo, after a few seconds I have my little green light.
So instead of connecting a 'Power' LED directly to the power supply (usually in series with a current-limiting resistor) the designers must have used the CPU's GPIO pin to turn on the power LED. This kind of works, but makes it a little harder to troubleshoot power supply problems, and the Raspberry Pi has no shortage of those.
Nothing much happened after the power LED came on. No WiFi access point, no bluetooth device came up. Hmmm. I bought a mini HDMI adapter and hooked it up to my monitor and, aha, I have a 'kernel panic- Attempted to kill init!' message. Looks like Ubuntu Mate does not run on the Pi Zero W. This is confirmed by this handy article. Rather, the Pi Zero W (and its close relative the Zero) is compatible with the older Raspberry Pi A and A+.
Time to try my trusty Slackware 14.1 Raspberry Pi image. A few years ago I got Slackware running on the Raspberry Pi A by using an Ubuntu kernel with a Slackware root filesystem. I loaded it into my new microSD card, powered up, and nothing happened. No blinkenlight. Not even a kernel panic. Time to dig a little deeper.
A look at my Slackware image showed it consisted of 2 partitions, a Windows-type FAT32 boot partition and a much larger Linux ext4 root partition.
The Windows FAT32 boot partition looks like this:
$ls -l ../slackware/boot
total 31076
-rwxr-xr-x 1 root root 1447 Jan 19 15:29 LICENCE.broadcom
-rwxr-xr-x 1 root root 2013 Jan 19 15:29 README
-rwxr-xr-x 1 root root 1479 Jan 19 15:29 README.initrd
-rwxr-xr-x 1 root root 777094 Jan 19 15:29 System.map
-rwxr-xr-x 1 root root 16536 Jan 19 15:29 bootcode.bin
-rwxr-xr-x 1 root root 82 Jan 19 15:29 cmdline.txt
-rwxr-xr-x 1 root root 91 Jan 19 15:29 config.txt
-rwxr-xr-x 1 root root 5282 Jan 19 15:29 fixup.dat
-rwxr-xr-x 1 root root 2020 Jan 19 15:29 fixup_cd.dat
-rwxr-xr-x 1 root root 137 Jan 19 15:29 issue.txt
-rwxr-xr-x 1 root root 3112676 Jan 19 15:29 kernel.img
-rwxr-xr-x 1 root root 2104952 Jan 19 15:29 kernel_cutdown.img
-rwxr-xr-x 1 root root 16264692 Jan 19 15:29 kernel_emergency.img
-rwxr-xr-x 1 root root 275235 Jan 19 15:29 loader.bin
-rwxr-xr-x 1 root root 2011408 Jan 19 15:29 start.elf
-rwxr-xr-x 1 root root 523144 Jan 19 15:29 start_cd.elf
-rwxr-xr-x 1 root root 101 Jan 19 15:29 version-kernel_raspi.txt
-rwxr-xr-x 1 root root 92 Jan 19 15:29 version-raspi-boot.txt
And an ext4 Slackware 14.1 root partition:
$ls -l ../slackware/root
total 164
drwxr-xr-x 2 root root 4096 Jan 19 15:30 bin
drwxr-xr-x 2 root root 4096 Jan 19 15:30 boot
drwxr-xr-x 17 root root 69632 Jan 19 15:30 dev
drwxr-xr-x 3 root root 4096 Jan 19 15:30 doc
drwxr-xr-x 89 root root 12288 Jan 19 15:30 etc
drwxr-xr-x 6 root root 4096 Jan 19 15:30 home
drwxr-xr-x 7 root root 4096 Jan 19 15:30 lib
drwx------ 2 root root 4096 Jan 19 15:30 lost+found
drwxr-xr-x 16 root root 4096 Jan 19 15:30 media
drwxr-xr-x 11 root root 4096 Jan 19 15:30 mnt
drwxr-xr-x 2 root root 4096 Jan 19 15:30 opt
drwxr-xr-x 2 root root 4096 Jan 19 15:30 proc
drwx--x--- 9 root root 4096 Jan 19 15:30 root
drwxr-xr-x 2 root root 4096 Jan 19 15:30 run
drwxr-xr-x 2 root root 12288 Jan 19 15:30 sbin
drwxr-xr-x 2 root root 4096 Jan 19 15:30 srv
drwxr-xr-x 2 root root 4096 Jan 19 15:30 sys
drwxr-xr-t 5 root root 4096 Jan 19 15:30 tmp
drwxr-xr-x 20 root root 4096 Jan 19 15:40 usr
drwxr-xr-x 18 root root 4096 Jan 19 15:41 var
A quick look at the Ubuntu Mate image showed a similar organization, a FAT32 boot partition followed by a Linux ext4 root partition. The boot partition is:
$ls -l ../ubuntu_mate/flash
total 20432
-rwxr-xr-x 1 root root 18693 Jan 19 15:11 COPYING.linux
-rwxr-xr-x 1 root root 1494 Jan 19 15:11 LICENCE.broadcom
-rwxr-xr-x 1 root root 14273 Jan 19 15:11 bcm2708-rpi-b-plus.dtb
-rwxr-xr-x 1 root root 14010 Jan 19 15:11 bcm2708-rpi-b.dtb
-rwxr-xr-x 1 root root 13964 Jan 19 15:11 bcm2708-rpi-cm.dtb
-rwxr-xr-x 1 root root 15356 Jan 19 15:11 bcm2709-rpi-2-b.dtb
-rwxr-xr-x 1 root root 15992 Jan 19 15:11 bcm2710-rpi-3-b.dtb
-rwxr-xr-x 1 root root 15350 Jan 19 15:11 bcm2710-rpi-cm3.dtb
-rwxr-xr-x 1 root root 17932 Jan 19 15:11 bootcode.bin
-rwxr-xr-x 1 root root 223 Jan 19 15:11 cmdline.txt
-rwxr-xr-x 1 root root 36783 Jan 19 15:11 config.txt
-rwxr-xr-x 1 root root 6622 Jan 19 15:11 fixup.dat
-rwxr-xr-x 1 root root 2535 Jan 19 15:11 fixup_cd.dat
-rwxr-xr-x 1 root root 9753 Jan 19 15:11 fixup_db.dat
-rwxr-xr-x 1 root root 9753 Jan 19 15:11 fixup_x.dat
-rwxr-xr-x 1 root root 4130008 Jan 19 15:11 kernel.img
-rwxr-xr-x 1 root root 4231256 Jan 19 15:11 kernel7.img
drwxr-xr-x 2 root root 4096 Jan 19 15:11 overlays
-rwxr-xr-x 1 root root 2823396 Jan 19 15:11 start.elf
-rwxr-xr-x 1 root root 634532 Jan 19 15:11 start_cd.elf
-rwxr-xr-x 1 root root 4956996 Jan 19 15:11 start_db.elf
-rwxr-xr-x 1 root root 3906116 Jan 19 15:11 start_x.elf
In an old-fashioned desktop or laptop, the startup process is controlled by a non-volatile flash, eeprom or eprom program called the BIOS. The ARM CPU in the Pi has a much smaller BIOS and has the rest of it stored on the microSD card. Now since I got a kernel panic from Ubuntu Mate, this meant that the ARM CPU managed to load the kernel and run it, only to have the kernel fail later. Since it is the job of the BIOS to load the kernel, this mean my Ubuntu Mate image had the correct BIOS files.
The boot sequence of the Raspberry Pi is basically this:
Stage 1 boot is in the on-chip ROM. Loads Stage 2 in the L2 cache
Stage 2 is bootcode.bin. Enables SDRAM and loads Stage 3
Stage 3 is loader.bin. It knows about the .elf format and loads start.elf
start.elf loads kernel.img. It then also reads config.txt, cmdline.txt and bcm2835.dtb If the dtb file exists, it is loaded at 0×100 & kernel @ 0×8000 If disable_commandline_tags is set it loads kernel @ 0×0 Otherwise it loads kernel @ 0×8000 and put ATAGS at 0×100
kernel.img is then run on the ARM.
The Ubuntu Mate image did not have loader.bin or bcm2835.dtb but it did have a kernel7.img in addition to the kernel.img file. So I copied the files bootloader.bin, start.elf, kernel.img and kernel7.img over to my Slackware image, taking care to keep my Slackware config.txt and cmdline.txt. Since the Slackware image did not have any dtb files at all, I copied them as well.
Now the image worked, and Slackware was able to start. A little problem is it did not recognize the WiFi hardware and its WiFi interface did not come up. But that is a problem for another post.
If you would like to run Slackware on your Raspberry Pi Zero or Zero W, leave me a comment in this blog post and I will put up the 8GB image for download.
In my previous posts I started making IoT devices from full Linux systems, like the Raspberry Pi solar battery voltmeter. Then we progressed to a bluetooth sensor/actuator with a Linux IoT gateway like the IoT Autogate remote.
The NodeMCUdevelopment kit promises to provide both: a remote sensor/actuator that provides it own WiFi IoT gateway. The NodeMCU is open source firmware based on Expressif System's ESP8266, a System-on-a-Chip (SoC) which has an 80MHz 32-bit Tensilica CPU paired with a full-stack WiFi networking at an extraordinarily low cost. The Lua ESP-12E NodeMCU costs a mere RM21.80
ESP-12E prices online in Malaysia
Watch out for the ESP-12E V1, and V2 versions; their physical sizes are different! There is a write-up here.
Be careful of the ESP-12 V1 and V2 differences
While my previous Microchip PIC18F14K50 plus Arduino HC-06 IoT can be commercialized, no one else has done it like that. The reason may be something like the ESP8266. ITEAD's Sonoff mains power switches and sockets actually uses the ESP8266.
I am starting with the ESP-01 and a CH340 USB to RS232 TTL dongle. Just because I happen to have them lying around. Okay maybe it was because the ESP-01 cost me a mere RM14. The CH340 is even better at RM7.90.
Earlier ESP modules
ESP-01 at RM14
CH340 at RM7.90
The ESP-01 and CH340 setup provides a bridge from my Linux comfort zone to explore the ESP8266, after which I fully expect the ESP8266 to be a standalone IoT system. For this I will need NodeMCU, which is the software framework to base future IoT work.
First we plug the CH340 dongle into my laptop. Slackware 14.2 Linux recognized the device immediately:
[Tue Jan 9 06:38:49 2018] usb 1-4.5.4: new full-speed USB device number 113 using xhci_hcd
[Tue Jan 9 06:38:49 2018] usb 1-4.5.4: New USB device found, idVendor=1a86, idProduct=7523
[Tue Jan 9 06:38:49 2018] usb 1-4.5.4: New USB device strings: Mfr=0, Product=2, SerialNumber=0
[Tue Jan 9 06:38:49 2018] usb 1-4.5.4: Product: USB2.0-Serial
[Tue Jan 9 06:38:50 2018] usbcore: registered new interface driver usbserial
[Tue Jan 9 06:38:50 2018] usbcore: registered new interface driver usbserial_generic
[Tue Jan 9 06:38:50 2018] usbserial: USB Serial support registered for generic
[Tue Jan 9 06:38:50 2018] usbcore: registered new interface driver ch341
[Tue Jan 9 06:38:50 2018] usbserial: USB Serial support registered for ch341-uart
[Tue Jan 9 06:38:50 2018] ch341 1-4.5.4:1.0: ch341-uart converter detected
[Tue Jan 9 06:38:50 2018] usb 1-4.5.4: ch341-uart converter now attached to ttyUSB0
The pinouts are:
The wiring will be (in format ESP-01 to CH340):
RX to TX, VCC to VCC, GPIO0 to VCC (Note this is different from the following picture), RST to GND, GPIO2 No connection (N/C), CH_PD to VCC, GND to GND, and TX to RX.
Next I will need to make a cable to connect the ESP-01. The connection should be similar to that for the other USB to serial dongle, the FTDI:
The ESP-01 and the FTDI USB to Serial TTL dongle. Note GPIO0 should be high and not as shown.
The ESP-01 will draw power from the CH340 which in turn gets its power from the laptop's USB socket. The important thing to remember is the ESP-01 is a 3.3V device. Luckily the CH340 (like the PIC18F14K50) can be set to run at 3.3V. The ESP-01 can draw a non-trivial 170mA(a typical external USB drive will draw 300mA), so I plan to use the laptop's USB2 connector and connect the setup via an externally-powered DLINK USB hub as a backup and for additional safety.
I powered on, and ... nothing happened. Oh the power LED in the CH340 came on. So did the red power LED in the ESP-01. Now disconnect RST from GND and the ESP-01's blue LED flashed like it was supposed to, but the CH340 would not respond. Its Linux device file, /dev/ttyUSB0 would come up when the CH340 is powered on.
But when I hooked up the ESP-01 the file disappeared.
[Wed Jan 10 12:23:34 2018] usb 1-4.5.4: USB disconnect, device number 38
[Wed Jan 10 12:23:34 2018] ch341-uart ttyUSB0: ch341-uart converter now disconnected from ttyUSB0
[Wed Jan 10 12:23:34 2018] ch341 1-4.5.4:1.0: device disconnected
The ESP-01 appeared OK though and I noticed a new WiFi Access Point had appeared:
I could connected the laptop to it, as there is no password required, and once in a little probing showed an active IP address 192.168.4.1:
$nmap -O 192.168.4.1-254
Starting Nmap 7.12 ( https://nmap.org ) at 2018-01-10 20:14 MYT
Nmap scan report for 192.168.4.1
Host is up (0.0010s latency).
All 1000 scanned ports on 192.168.4.1 are closed
MAC Address: 5E:CF:7F:FD:1D:EB (Unknown)
Warning: OSScan results may be unreliable because we could not find at least 1 o
pen and 1 closed port
Aggressive OS guesses: Philips Hue Bridge (lwIP stack v1.4.0) (95%), 2N Helios I
P VoIP doorbell (91%), Advanced Illumination DCS-100E lighting controller (91%),
British Gas GS-Z3 data logger (91%), Espressif WiFi system-on-a-chip (91%), Gra
ndstream GXP1105 VoIP phone (91%), LaSAT satellite receiver (91%), lwIP 1.4.0 li
ghtweight TCP/IP stack (91%), m3 muvid IR 715-2 Internet radio receiver (91%), M
ilight WiFi Receiver bridge (91%)
No exact OS matches for host (test conditions non-ideal).
Network Distance: 1 hop
Voltage seems normal at 3.3V and only 70mA current was being drawn. A little digging in the datasheets of the ESP8266 and CH340 showed the minimum operating voltage of the ESP8266 was 3V and the CH340 3.3V. It is possible that on power up the power surge of the ESP8266 dragged the voltage down enough for the CH340 to panic and go to bed, but I could not be sure without hooking up the oscilloscope.
Instead I popped out one of those deliciously cheap Arduino LM2596 buck converters and set it to take in 5V and put out 3.3V just for the ESP8266. I now needed to connect the GND of the CH340 and the ESP8266 to provide a common signal reference, and did not change any other connections.
Rats' nest: clockwise from top LM2956, CH340 and ESP-01
I connected the CH340 to my laptop, and ran minicom, setting it to /dev/ttyUSB0 and baud rate 115200. And lo and behold, on turning on the ESP-01:
ESP-01 screen on power on
And just to make sure it is no fluke I type 'AT' followed by Ctrl-M and Ctrl-J, and it replied with 'OK'.
And to connect it to the Internet like any good IoT device (the commands are in bold and the ESP-01's replies in normal font; at the end of every command line type Ctrl-M and Ctrl-J):
AT
OK AT+CWMODE=3
OK AT+CWLAP
+CWLAP:(1,"AP1",-83,"xx::xx::xx:xx:xx:xx",1,-17,0)
+CWLAP:(3,"......",-80,"yy:yy:yy:yy:yy:yy",8,-21,0)
+CWLAP:(3,"AP2",-65,"zz:zz:zz:zz:zz:zz",9,-14,0)
AT+CWJAP="AT2","verysecretpassword"
WIFI CONNECTED
WIFI GOT IP
It connected to my home WiFi network, assigned it an IP address different from its own () access point; 172.16.1.105. And to make doubly sure we launch a network probe at the new IP address:
$nmap -O 172.16.1.105
Starting Nmap 7.12 ( https://nmap.org ) at 2018-01-11 19:39 MYT
Nmap scan report for 172.16.1.105
Host is up (0.012s latency).
All 1000 scanned ports on 172.16.1.105 are closed
MAC Address: 70:62:B8:A9:C0:CA (D-Link International)
Warning: OSScan results may be unreliable because we could not find at least 1 o
pen and 1 closed port
Aggressive OS guesses: Philips Hue Bridge (lwIP stack v1.4.0) (95%), 2N Helios I
P VoIP doorbell (91%), Advanced Illumination DCS-100E lighting controller (91%),
British Gas GS-Z3 data logger (91%), Espressif WiFi system-on-a-chip (91%), Gra
ndstream GXP1105 VoIP phone (91%), LaSAT satellite receiver (91%), lwIP 1.4.0 li
ghtweight TCP/IP stack (91%), m3 muvid IR 715-2 Internet radio receiver (91%), M
ilight WiFi Receiver bridge (91%)
No exact OS matches for host (test conditions non-ideal).
Network Distance: 1 hop
OS detection performed. Please report any incorrect results at https://nmap.org/
submit/ .
Nmap done: 1 IP address (1 host up) scanned in 4.43 second
And there you have it. The ESP-01 with its ESP8266 SoC certainly lived up to its reputation. Next I shall try to program it to control a relay from WiFi, much like the IoT bluetooth Autogate Remote project.
The IoT Bluetooth Relay Board tests well enough with a simple python script, but every time the script finishes bluetoothctl drops its connection to the HC-06. When you run the test script again, it reconnects to the HC-06 well enough, but the connection takes a variable number of seconds, and sometimes the script fails to connect. For an application like the autogate this is not acceptable.
However if the script does not exit the connection is reliable over many days, so the script should not exit. That means a server (Linux daemon) process, and its accompanying client script. There is one such python solution and that is pymodbus, a python implementation of Modbus client and server.
Modbus for an autogate? Modbus is a popular industrial communications protocol. Isn't that overkill?. It probably is.
Or is it? As we have seen the remote sensor/actuator portion is done using bluetooth device and a Microchip PIC18F14K50, which consumes 10mA at 5V. Modbus lives on the IoT gateway, ie the bluetooth master which happens to be an old laptop running Slackware Linux. Pymodbus makes my homebrew autogate remote opener compatible with IoT for Industry. The client python script will then be run by a PHP script from the laptop's apache webserver, in the same way as the Raspberry Pi Robot platform.
Security is via the WPA WiFi password on my home WiFi router. Notice that this system will still work without a broadband connection, as long as your smartphone is within range of the WiFi. The HC-06 will not accept a second bluetooth connection once it is paired to the laptop, so access is via the WiFi alone. With a ADSL connection this is a true IoT, able to accept commands from the Internet.
Remote autogate operation may make sense if you say, want to let the gardener into the yard, or the electricity/water utility person to read the meter. In my case my dogs will probably terrorize the meter reader before galloping off to poop in the neighbour's yard. So its main advantage is an extended-range autogate remote.
The installation for Slackware is probably subject for another post, but say you had it installed. There is a sample program pymodbus-master/examples/common/updating-server.py which we can use as a template.
The first few lines of code are important:
#---------------------------------------------------------------------------#
# import the modbus libraries we need
#---------------------------------------------------------------------------#
from pymodbus.server.async import StartTcpServer
from pymodbus.device import ModbusDeviceIdentification
from pymodbus.datastore import ModbusSequentialDataBlock
from pymodbus.datastore import ModbusSlaveContext, ModbusServerContext
from pymodbus.transaction import ModbusRtuFramer, ModbusAsciiFramer
#---------------------------------------------------------------------------#
# import the twisted libraries we need
#---------------------------------------------------------------------------#
from twisted.internet.task import LoopingCall
Next we import our bluetooth module, which shall look suspiciously like our previous python test script.
Next there is our Modbus database, which we reduce from 100 to 16 to reduce memory usage, a initialize to 0 (except for the 2 bits reversed logic for the Arduino relays):
Note the loop time has been speeded up from 5s to 0.2s. StartTcpServer() is set to use the non-root network port 5020, which our client script will need to listen to.
That is all there is to it. Most of the pymodbus server code is done. Now for that bluetooth code in ./autogate_bluetooth.py, which I have put in the same directory so that it imports correctly. Here it is in full:
#!/usr/bin/python
import serial
from time import localtime, strftime, sleep
'''
Sends bluetooth rs-485 PIC18F14K50 commands and receives the replies.
Usage:
import autogate_bluetooth.py
./autogate_bluetooth.py to test
Uses pyserial module
The ems command is single-char, reply is a double sequence of 3 binary bytes
followed by check strings
Replies to 'z' are 2 3-byte analog input readings.
'''
port1 = serial.Serial('/dev/rfcomm0',timeout= 1) #2017-12-19 HC-06 bluetooth
The main function here is Send_Cmd(). The PIC18F14K50 has been programmed to accept the commands 'z' (read voltages), '0' (both relays on), '1' (first relay on), '2' (second relay on) and '3' (both relays off).
Now let us go back to updating_server.py so we can modify the main server loop to communicate with the HC-06. We need to add a new module update_output() to read the Modbus database. If there is a change (caused by the client script requesting the Arduino relay to turn on) it will then send the correct command to the HC-06. The code is:
Next there is the code to update the Modbus database with the raw values read from the HC-06. These are the 2-byte values from the analog inputs. There are two inputs that makes 4 buyes in total.
You set the bits you want (only the first 2 are implemented) and use client.write_coils() to write to the Modbus database. The server process does the rest.
Typical output is:
$python ./autogate_client.py open
open autogate
read_holding_registers 1 [3, 251, 2, 62] CPU 1019 Battery 574
CPU 4.98 Battery 11.91 Volts 2017-12-28 16:14:06
To close the autogate you run the client again:
$python ./autogate_client.py open
open autogate
read_holding_registers 1 [3, 251, 2, 62] CPU 1019 Battery 574
CPU 4.98 Battery 11.91 Volts 2017-12-28 16:31:13
Now when I am on foot I find it useful not to open the gate wide, as a dog might then be tempted to bolt. This opens a man-size opening by triggering the relay twice:
$python ./autogate_client.py ajar
ajar autogate
ajar autogate
read_holding_registers 1 [3, 251, 2, 62] CPU 1019 Battery 574
CPU 4.98 Battery 11.91 Volts 2017-12-28 16:31:07
And that is all there is to it. Now for the PHP script.
