Lightning detector AS3935: Part 3 of 3

Clockwise from top left: ADSL splitter, AS3935 (in vise), ESP-12E, relay module, powered USB hub, modem and POTS analog phone. All blinkenlights are go.

In the last part of this series, the AS3935 detector is hooked up to the ADSL line, and the ESP-12E is programmed to disconnect if there is a thunderstorm overhead. When the storm moves more than 8km away it will reconnect after 30 minutes.

As first mentioned in Part 2, it publishes to a local MQTT server, and also subscribes to it. This is to allow it to disconnect and reconnect the ADSL modem remotely on demand. For the last six months or so my TM Streamyx (now 'Unifi') ADSL2+ line has been unstable; dropping and reconnecting every few hours or worse. This constant failover to 3G tended to overload the TP-Link MR3420. Disconnecting the phone line during the bad periods seemed to give it some relief.

As before, the AS3935 can be quite sensitive to inteference. I ended up using separate shielded USB charger cables for 5V for both the ESP-12E and the relay module. At worst, excessive interference resulted in permanent false alarms for 'storm overhead'.

Also you can reduce false alarms by additional shielding. I partially encased the AS3935 module in a steel vise to stop it triggering when I turned on my fluorescent lights. Do not shield it completely or there will be nothing detected.

Jury-rigged steel vise shield for the AS3935

Connecting to the telephone line

My incoming POTS line has two wires, but a standard US RJ11 phone jack has at least 4. Our benighted Talikon Malaysia uses British standard (BS 6312), but most devices (not phone line) comes in the US RJ11 4-wire format, which is what I used.

Lanshack has a good page on the pinout:

The 4 wires in my RJ11 jacks cover pins 2 to 5 corresponding to Pair 1 and Pair 2. Pair 1, ie R1 & T1 at pins 3 & 4  (ie red & green wires) seem like a good starting point. I took out the modem ADSL cable, ie one with RJ11 jacks on both ends and cut it in two. The red and green cables are easily accessible.

Start with your modem phone cable, and cut in in half 

Modem cable with just Pair 1 reconnected

I verified my guess by reconnecting just Pair 1 and used my modified cable to reconnect the modem to the phone line. My ISP TM (TMnuts for short) uses ADSL2+ and that did not seem to affect my modem operation but your mileage may vary. Since broadband runs at 1Mbps up to 20Mbps changing the phone cables characteristic impedance with discontinuities like connectors and joints might affect it big-time.

Also, it would be wise to hook up the ADSL splitter and the POTS analog phone to see if they still work. If the joints did not affect your phone or modem setup, go ahead and hook up the relay module. A note of caution: disconnect the phone cable from your ISP line before you rewire. Phone line voltages are around 40Vdc and are usually safe to handle. This can change to as much as 100Vac if someone tried to call you on that phone. Be careful- phone lines can be hazardous! so you really do not want to touch the relay module when the phone is ringing.

Relay module cabling, clockwise from top left: 5V power, R1, T1 and TTL serial

I wired up the ISP side to relay Normally Open and the modem side to Relay Common. This means the relays need to be on most of the time and more power (about 100mA at 5V) will be used. The advantage is if the first strike took out the power mains, the modem will not be reconnected to the ISP.

If you only have the single relay version just switch T1 to break the circuit. This should help a lot and reduce the power of a strike, but my guess is the lightning strike can still reach the modem via R1 and if it is near enough will then complete the circuit to modem Earth or failing that to modem DC GND.

Of course if a strike is almost on top of the modem it will probably jump the relay air gaps, but the modem will be the least of your problems then. It is advisable not to go near the setup in a thunderstorm, even after the relay module has disconnected the modem. I have noticed the electronics twitching on following strikes even without power, ie after the first strike has taken out the power mains.

Software  

For esp8266 MQTT client I used pubsubclient. It was quite painless to use and its sample code worked first time with mosquitto so I will simply skip ahead to the complete version. For details on the mosquitto MQTT server, see Part 2. The nice thing about a local MQTT is that the AS3935 can then be used to disconnect more than one IoT device, for example the autogate, POTS phone extension and the satellite TV.

You can get a copy of the software from github.

To look at the MQTT output:

$mosquitto_sub -t 'AS3935/messages' -v

To turn on the relays

$mosquitto_pub -t 'AS3935/commands' -m '1'

Or you can do it from a smartphone App. I used MyMQTT. For now, you need to be in the same WiFi access point as the AS3936.

I used MyMQTT on my Android phone

As usual we keep monitoring for bugs and tuning the calibration settings. I tend to desensitize the alarms as Malaysia gets many violent but localized (ie usually not widespread) thunderstorms, and I do not want to trigger a disconnect until it is right on top of the AS3935. I also have fluorescent lights and a hot shower which tends to set it off unless I use a threshold setting of 9 and above.

This is my typical output for a real storm:

AS3935/messages 14:08:59 2019.11.19 Lightning detected 8 km away
AS3935/messages 14:09:04 2019.11.19 Lightning detected 6 km away
AS3935/messages 14:09:13 2019.11.19 Disturber detected
AS3935/messages 14:09:21 2019.11.19 Disturber detected
AS3935/messages 14:09:22 2019.11.19 Disturber detected
AS3935/messages 14:09:30 2019.11.19 Lightning detected 6 km away
AS3935/messages 14:09:45 2019.11.19 Disturber detected
AS3935/messages 14:10:07 2019.11.19 Storm overhead, watch out! Disconnecting the phone lines ..

AS3935/messages 14:20:13 2019.11.19 Disturber detected
AS3935/messages 14:31:10 2019.11.19 Lightning detected 12 km away
AS3935/messages 14:32:40 2019.11.19 Lightning detected 12 km away
AS3935/messages 14:33:18 2019.11.19 Disturber detected
AS3935/messages 14:36:02 2019.11.19 Disturber detected
AS3935/messages 14:40:08 2019.11.19 Reconnecting the phone lines ...

Once it disconnects, the software waits for the storm to move 9km away, waits a further 30min and then automatically reconnects.

At this point we just sit back and watch the light show. Happy Trails.

Lightning Detector AS3935: adding Serial Relay Module and MQTT Server Part 2 of 3

In Part 1, we interfaced the AS3935 lightning detector to an ESP-12E using SPI. In Part 2 here, we add an ESP-01 2-channel WiFi relay board and also set up an MQTT server.

But first, some test results. After running the system in Part 1 for a few weeks we know that:
1. There are many false (ie 'disturber') alarms
2. Some lightning strikes are incorrectly classed as 'disturber'
3. The little antenna is directional, so be careful positioning it
4. The AS3935 and relay module are best on separate 5V wires, otherwise there may be too many false alarms
5. The settings that worked for me are: Noise Floor: 3, Spike Rejection: 0, and Watchdog Threshold:2
6. There is good correlation between actual lightning strikes and the AS3935 readings
7. I have good results disconnecting the relays when the lightning is less than 5km (ie 'storm overhead') and reconnecting it 30 minutes after  lightning strikes are further than 8km away.

2-channel wifi relay module

ESP-01 2 Channel WiFi Relay Module

I got mine from lazada.com for RM27 (about USD4) each. Unlike the other ESP8266 relay boards this one has a separate CPU to activate the relays and instead of using precious GPIO pins you can do it with the serial port UART0.

makerrelay has a good writeup on a very similar module, complete with schematics and identical relay switching commands. Even better, libretto (Sergiy Zaschipas) shows you how to reprogram the relay CPU.

I can not only replace the ESP-01 with a NodeMCU ESP-12E, but I can now add relay 'channels' by simply adding more relay boards. I do not even have to use extra GPIO pins - the same TTL serial port UART0 can drive all the relay boards.

Another convenience is the relay outputs need not be reset especially since the ESP8266 is sometimes best reset if it is unable to reconnect to WiFi. If the relay CPU is reprogrammed, the ESP8266 may even read back the relay states. Since the AS3935 drops the phone line when a storm is overhead it can actually cause WiFi disconnects.

Based on some of the board markings, it is possible the original design is by LC Tech. LC Tech also provides documentation.

NodeMCU ESP-12E Pinout

You connect GP101 (also TXD0 or TX) to the pin (ESP-12E bottom right) marked 'TX' on the relay board. Connect up the adjacent GND pin to the correcponding relay module pin and you are good to go.

Relay module, solder side

Note that if you want to supply power to the relay board you need 5V which is on the other column of pins (VIN) of the ESP-12E, and not the very tempting 3.3V pin.

The relay commands are:

//Hex command to send to serial for close relay
byte relON[]  = {0xA0, 0x01, 0x01, 0xA2};
//Hex command to send to serial for open relay
byte relOFF[] = {0xA0, 0x01, 0x00, 0xA1};
//Hex command to send to serial for close relay
byte rel2ON[]  = {0xA0, 0x02, 0x01, 0xA3};
//Hex command to send to serial for open relay
byte rel2OFF[] = {0xA0, 0x02, 0x00, 0xA2};

To turn on relay 1:

Serial.write (relON, sizeof(relON));

It is that simple. My setup now looks like this:

AS3935 SPI and Relay serial port cables consolidated to one 2-headed monster. Note the 5V VIN cable on the far right. System power is via the USB cable (silver)

Top left to right: ESP-12E (partially hidden), relay module and AS3935

MQTT Server

Now that I have a way of disconnecting the ADSL modem, a convenient way to monitor the AS3935 results would be nice. Initially I simply printed the results using Serial.print() and got the output via the Arduino IDE Serial Monitor.

This worked when I am at my work station but pretty soon I was printing out to my Adafruit MQTT feed. This worked much better until a lightning storm came close enough for the relay module to disconnect the ADSL modem at which point the Adafruit connection got interrupted.

Now my WiFi does have a 3G failover, but it takes precious seconds for the Adafruit feed to re-establish itself and vital AS3935 messages are often lost. So one of the things to try would seem to be a local MQTT server, which could capture and retain my MQTT messages and if necessary upload them to my Adafruit feed for when I am offsite.

It sounds like a lot of work, until along came Mosquitto. Since my main station is still Slackware 14.2, I got my SlackBuild here:

As usual douwnload the SlackBuild tarball and unzip it:

$tar -xvpzf mosquitto.tar.gz
mosquitto/
mosquitto/slack-desc
mosquitto/README
mosquitto/mosquitto.info
mosquitto/doinst.sh
mosquitto/mosquitto.SlackBuild
usr/man/man8/mosquitto.8.gz
usr/sbin/
usr/sbin/mosquitto
usr/share/

Then download the mosquitto source code (from the SlackBuild link not the Eclipse one) and copy it into the mosquitto slackbuild directory. Then all you need is

$./mosquitto.SlackBuild

And then a Slackware install:

$upgradepkg --install-new /tmp/mosquitto-1.6.7-x86_64-1_SBo.tgz

Next you need to set up a mosquitto user:
$vi /etc/mosquitto/mosquitto.conf

And add the line:
user mosquittouser

If necessary create the new user:
$adduser mosquittouser

You start the server thus:
$mosquitto -c /etc/mosquitto/mosquitto.conf
1572678454: mosquitto version 1.6.7 starting
1572678454: Config loaded from /etc/mosquitto/mosquitto.conf.
1572678454: Opening ipv4 listen socket on port 1883.
1572678454: Opening ipv6 listen socket on port 1883.

And to check use the included client subscriber:

$mosquitto_sub -t 'test/topic' -v

The publish:

$mosquitto_pub -t 'test/topic' -m 'hello world of mosquitto'

And that was all you need. I suspect it is even easier in Debian/Ubuntu. The next thing I need would be an MQTT client for the ESP-12E, but that is another story.

So till then, Happy Trails.

Thunderbolts and Lightning Detector AS3935 Part 1 of 3

Bohemian Rhapsody, Queen

"Thunderbolts and lightning, very very frightening me" - Queen, Bohemian Rhapsody

Being near the equator, Malaysia is very frequently struck by lightning and has a flash rate density close to 100. ADSL broadband modems, cordless telephones, network switches, autogate controllers and monitors regularly get damaged.

A few times a year the storms get really bad: you can hear the lightning hiss followed by the thunderclap. The phone cables are fried, electric power is out and the dogs try to get on your lap, never mind the cat.

I often repair the electronics, but there is something about lightning damage: they are never quite the same. TVs seem more immune, but if you use them with a computer (as a monitor), they get damaged by lightning. Go figure.

But the most annoying damage is to the ADSL modem. This is because I run an IoT server 24/7. I have backup power via UPS but a fried modem often results in service interruption. Eventually I installed a backup 3G modem using a TP-Link MR-3420.

For some reason, things get a lot better if we simply disconnected the telephone line from the ADSL modem and phone. This causes the MR-3420 to fail over to 3G, and I put in a little fix to ensure a failback once the ADSL modem is reconnected. Things do not get fried as much maybe once a year a LAN switch or a monitor cops it.

And this works very well for us unless we are away on holiday. What we need is a system to disconnect the modem from the phone line when the storm is close and reconnect when it passes. And sure enough, cometh the hour cometh the chip: AS3935 from Austria Microsystems.

I got my AS3935 for RM92 (about USD20)

But why not stay on 3G? The problem is in Malaysia we still have a data cap on 3G, maybe 10GB per month, and if your IoT device is a security camera it quickly adds up.

Since I had good results with the SPI interface on the NodeMCU ESP-12E before,
I made the following SPI cable to the CJMCU AS3935:

PIN#   AS3935                               diagram  ESP-12E    Arduino IDE no
 1          VCC                                      5V
 2          GND                                     GND
 3          SCL                    HSCLK     D5      GPIO14                     14
 5          MOSI                  HMISO     D6      GPIO12                     12
 4          MISO                  HMOSI     D7      GPIO13                     13
 6          CS                       HCS         D8      GPIO15                     15
 7          SI                                         GND

 8          IRQ                                      D2      GPIO2                         4

 9          EN_VREG                          5V

The AS3935 allows for 3V3 to be used but the CJMCU board did not bring out VREG to the header, so I needed to supply 5V and enable the onboard regulator by tying EN_VREG to 5V. The last column, 'Arduino IDE number' is the pin number to use for the Arduino sketch.  I used a 12-way header with a 10-way ribbon cable because I wanted to use GPIO1 and GPIO3 to control a relay board.

For software, I used Eva Schindling's Thunder and Lightning. Even though it was written five years ago for the Arduino, it pretty much worked first time for the ESP-12E. I used her AS3935_example.ino sketch with just a few tweaks to accommodate my SPI cable and the ESP-12E.

Download her code in a zip file, unzip it and the make a symbolic link to your Arduino IDE library:

ln -s /home/guest/as3935/ThunderAndLightning-master/library/AS3935 /home/guest/Arduino/libraries/AS3935

And that was it. I knew Arduino SPI code did not travel very well to the ESP8266 and was well, thunderstruck it worked first time. And unlike Eva who back in 2014 seemed to have lacked good thunderstorms to test with, a big storm arrived just in time for testing.

Life indeed is good. Happy Trails.

/*

  LightningDetector.pde - AS3935 Franklin Lightning Sensor™ IC by AMS library demo code

  Copyright (c) 2012 Raivis Rengelis (raivis [at] rrkb.lv). All rights reserved.

  This library is free software; you can redistribute it and/or

  modify it under the terms of the GNU Lesser General Public

  License as published by the Free Software Foundation; either

  version 3 of the License, or (at your option) any later version.

  This library is distributed in the hope that it will be useful,

  but WITHOUT ANY WARRANTY; without even the implied warranty of

  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU

  Lesser General Public License for more details.

  You should have received a copy of the GNU Lesser General Public

  License along with this library; if not, write to the Free Software

  Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA

*/

#include <SPI.h>

#include <AS3935.h>

void printAS3935Registers();

// Function prototype that provides SPI transfer and is passed to

// AS3935 to be used from within library, it is defined later in main sketch.

// That is up to user to deal with specific implementation of SPI

// Note that AS3935 library requires this function to have exactly this signature

// and it can not be member function of any C++ class, which happens

// to be almost any Arduino library

// Please make sure your implementation of choice does not deal with CS pin,

// library takes care about it on it's own

byte SPItransfer(byte sendByte);

// Iterrupt handler for AS3935 irqs

// and flag variable that indicates interrupt has been triggered

// Variables that get changed in interrupt routines need to be declared volatile

// otherwise compiler can optimize them away, assuming they never get changed

void AS3935Irq();

volatile int AS3935IrqTriggered;

// First parameter - SPI transfer function, second - Arduino pin used for CS

// and finally third argument - Arduino pin used for IRQ

// It is good idea to chose pin that has interrupts attached, that way one can use

// attachInterrupt in sketch to detect interrupt

// Library internally polls this pin when doing calibration, so being an interrupt pin

// is not a requirement

#define IRQpin 4 // ESP-12E

#define CSpin 15

AS3935 AS3935(SPItransfer,CSpin,IRQpin);

void setup()

{

  // Serial.begin(9600);

  Serial.begin(115200); // 2019-10-19 ESP-12E default

  // first begin, then set parameters

  SPI.begin();

  // NB! chip uses SPI MODE1

  SPI.setDataMode(SPI_MODE1);

  // NB! max SPI clock speed that chip supports is 2MHz,

  // but never use 500kHz, because that will cause interference

  // to lightning detection circuit

  SPI.setClockDivider(SPI_CLOCK_DIV16);

  // and chip is MSB first

  SPI.setBitOrder(MSBFIRST);

  // reset all internal register values to defaults

  AS3935.reset();

  // and run calibration

  // if lightning detector can not tune tank circuit to required tolerance,

  // calibration function will return false

  

  

  //if(!AS3935.calibrate())

  //  Serial.println("Tuning out of range, check your wiring, your sensor and make sure physics laws have not changed!");

  outputCalibrationValues();

  recalibrate();

  AS3935.setNoiseFloor(1);

  AS3935.setSpikeRejection(2);

  AS3935.setWatchdogThreshold(1);

  

  outputCalibrationValues();

  recalibrate();

  // since this is demo code, we just go on minding our own business and ignore the fact that someone divided by zero

  // first let's turn on disturber indication and print some register values from AS3935

  // tell AS3935 we are indoors, for outdoors use setOutdoors() function

  AS3935.setIndoors();

  // AS3935.setOutdoors();

  // turn on indication of distrubers, once you have AS3935 all tuned, you can turn those off with disableDisturbers()

  AS3935.enableDisturbers();

  // AS3935.disableDisturbers();

  printAS3935Registers();

  AS3935IrqTriggered = 0; 

  // Using interrupts means you do not have to check for pin being set continiously, chip does that for you and

  // notifies your code

  // demo is written and tested on ChipKit MAX32, irq pin is connected to max32 pin 2, that corresponds to interrupt 1

  // look up what pins can be used as interrupts on your specific board and how pins map to int numbers

  // ChipKit Max32 - irq connected to pin 2

  // attachInterrupt(1,AS3935Irq,RISING);

  // uncomment line below and comment out line above for Arduino Mega 2560, irq still connected to pin 2

  attachInterrupt(digitalPinToInterrupt(IRQpin),AS3935Irq,RISING); // ESP-12E

}

void loop()

{

  // here we go into loop checking if interrupt has been triggered, which kind of defeats

  // the whole purpose of interrupts, but in real life you could put your chip to sleep

  // and lower power consumption or do other nifty things

  if(AS3935IrqTriggered)  

  {

    // reset the flag

    AS3935IrqTriggered = 0;

    // wait 2 ms before reading register (according to datasheet?)

    delay(2);

    // first step is to find out what caused interrupt

    // as soon as we read interrupt cause register, irq pin goes low

    int irqSource = AS3935.interruptSource();

    // returned value is bitmap field, bit 0 - noise level too high, bit 2 - disturber detected, and finally bit 3 - lightning!

    if (irqSource & 0b0001)

      Serial.println("Noise level too high, try adjusting noise floor");

    if (irqSource & 0b0100)

      Serial.println("Disturber detected");

    if (irqSource & 0b1000)

    {

      // need to find how far that lightning stroke, function returns approximate distance in kilometers,

      // where value 1 represents storm in detector's near victinity, and 63 - very distant, out of range stroke

      // everything in between is just distance in kilometers

      int strokeDistance = AS3935.lightningDistanceKm();

      if (strokeDistance == 1)

        Serial.println("Storm overhead, watch out!");

      if (strokeDistance == 63)

        Serial.println("Out of range lightning detected.");

      if (strokeDistance < 63 && strokeDistance > 1)

      {

        Serial.print("Lightning detected ");

        Serial.print(strokeDistance,DEC);

        Serial.println(" kilometers away.");

      }

    }

  }

}

void printAS3935Registers()

{

  int noiseFloor = AS3935.getNoiseFloor();

  int spikeRejection = AS3935.getSpikeRejection();

  int watchdogThreshold = AS3935.getWatchdogThreshold();

  int minLightning = AS3935.getMinimumLightnings();

  Serial.print("Noise floor is: ");

  Serial.println(noiseFloor,DEC);

  Serial.print("Spike rejection is: ");

  Serial.println(spikeRejection,DEC);

  Serial.print("Watchdog threshold is: ");

  Serial.println(watchdogThreshold,DEC); 

  Serial.print("Minimum Lightning is: ");

  Serial.println(minLightning,DEC);   

}

// this is implementation of SPI transfer that gets passed to AS3935

// you can (hopefully) wrap any SPI implementation in this

byte SPItransfer(byte sendByte)

{

  return SPI.transfer(sendByte);

}

// this is irq handler for AS3935 interrupts, has to return void and take no arguments

// always make code in interrupt handlers fast and short

void ICACHE_RAM_ATTR AS3935Irq()

{

  AS3935IrqTriggered = 1;

}

void recalibrate() {

  delay(50);

  Serial.println();

  int calCap = AS3935.getBestTune();

  Serial.print("antenna calibration picks value:\t ");

  Serial.println(calCap);

  delay(50);

}

void outputCalibrationValues() {

   // output the frequencies that the different capacitor values set:

  delay(50);

  Serial.println();

  for (byte i = 0; i <= 0x0F; i++) {

    int frequency = AS3935.tuneAntenna(i);

    Serial.print("tune antenna to capacitor ");

    Serial.print(i);

    Serial.print("\t gives frequency: ");

    Serial.print(frequency);

    Serial.print(" = ");

    long fullFreq = (long) frequency*160;  // multiply with clock-divider, and 10 (because measurement is for 100ms)

    Serial.print(fullFreq,DEC);

    Serial.println(" Hz");

    delay(10);

  }

}

mDNS: Bonjour Slackware

Bonjour Sourire by Henri Salvador

"Adieu tristesse, bonjour sourire" - Henri Salvador

Goodbye sadness, hello smile. Somehow it seems apt.

For 20 years now I have used static IP rather than a Name Server for my home network. When I started my home network, computers were expensive and few. But then came the Raspberry Pi and the Espressif ESP8266.

In practice I maintained a 'name server' by hand, updating my /etc/resolv.conf every time I added a new computer to the network. There are now over 100 entries in 3 sub-nets but you do get used to it. What put me over the edge was the IoT device.

The need for periodic security updates meant reprogramming: the device had to be taken down and the ESP8266 plugged into its programmer. The most useful ones always seem to be the most inaccessible, like upside down on the ceiling or out in the elements like the autogate controller.

Then along came Over The Air programming, just like you would update your App in your smartphone. But now every IoT device came with its unique static IP address, while I might have 10 IoT lights installed, there is only 1 version source code and there was a high chance I would brick a device by using the wrong address.

But adieu tristesse, there is mDNS or Multicast DNS to the rescue. mDNS was first implemented as Apple Inc's Bonjour. Actually I unknowingly used it in ArduinoOTA and Raspbian. It is just my main workstations ran Slackware, which did not come with mDNS pre-installed. It would be nice to update IoT devices directly from my development machine.

To install mDNS onto my Slackware 14.2-current (which does not automatically resolve dependencies) you will need to install in this order: libdaemon, avahi and  nss-mdns.

But first you need to create the user 'avahi':

$groupadd -g 214 avahi

$useradd -u 214 -g 214 -c "Avahi User" -d /dev/null -s /bin/false avahi

libdaemon:

$tar -xvpzf libdaemon.tar.gz

$cd libdaemon

Download the source code libdaemon-0.14.tar.gz into libdaemon directory and:

$./libdaemon.SlackBuild

which produced /tmp/libdaemon-0.14-x86_64-1_SBo.tgz and can be installed into Slackware thus:

$upgradepkg --install-new /tmp/libdaemon-0.14-x86_64-1_SBo.tgz

avahi:

$tar -xvpzf avahi.tar.gz

$cd avahi

Download your source code

$ ./avahi.SlackBuild

$upgradepkg --install-new /tmp/avahi-0.7-x86_64-1_SBo.tgz

In my case I ring-fenced my IoT devices in their own WiFi Access Point. Many consumer WiFi routers (like my TP-Link TL-MR3420) randomly drops connections to the WiFi clients when there are too many of them, like around 30 devices.

This means my Slackware server (ie Google smart home nodejs, MQTT, and webhook servers) can only access the IoT devices from behind their router. In this case I had to set just one of the mDNS computers in the IoT subnet to reflector. That is in /etc/avahi/avahi-daemon.conf add:

[reflector]
enable-reflector=yes
  

$/etc/rc.d/rc.avahidaemon start

Starting Avahi mDNS/DNS-SD Daemon:  /usr/sbin/avahi-daemon -D

$/etc/rc.d/rc.avahidnsconfd start

Add the following lines to your /etc/rc.d/rc.local:

# Start avahidaemon

if [ -x /etc/rc.d/rc.avahidaemon ]; then

  /etc/rc.d/rc.avahidaemon start

fi

# Start avahidnsconfd

if [ -x /etc/rc.d/rc.avahidnsconfd ]; then

  /etc/rc.d/rc.avahidnsconfd start

fi

nss-mdns:

$tar -xvzf nss-mdns.tar.gz

$cd nss-mdns

Download your source code

$./nss-mdns.SlackBuild

$upgradepkg --install-new /tmp/nss-mdns-0.10-x86_64-2_SBo.tgz

Now your  /etc/nsswitch.conf will have a line line this:

hosts:          files dns

Which you need to change to 

hosts:          files mdns4_minimal [NOTFOUND=return] dns

After which a command like getent will work:

$getent hosts MyComputer.local

123.45.67.89   MyComputer.local

And you should be able to remotely access it by name without a DNS server:

$ssh -t MyComputer.local

Even better, it works with the ArduinoOTA code for the ESP8266. This sets me up nicely for Over The Air software upgrades for my home-brew IoT devices. 

Pour éclairer un ciel trop gris,
Pour cueillir un bout de printemps
Il suffit dans la vie
Il suffit bien souvent
De dire adieu tristesse, bonjour sourire.

Happy Trails.

To illuminate a sky too gray,
To pick a piece of spring
It's enough in life
It is often enough
To say goodbye sadness, hello smile.

ArduinoOTA: Programming the ESP-01S via WiFi Part 1

Radio Ga Ga, Queen Youtube video 

"You had your time, you had the power
You've yet to have your finest hour ...

Radio, what's new?

Radio, someone still loves you." - Queen, Radio Ga Ga.

WiFi is a radio communications standard. The ArduinoOTA Library enables the ESP8266 to be reprogrammed over WiFi. This is especially handy if you used the ESP8266 for IoT devices and as a result mounted them in very useful but inaccessible places.

Regular IoT program updates is probably a necessity, as a cheap, portable and concealable Deauth Jammer can disable WiFi IoT devices within range. The recent Matheus Garbelini ESP32 and ESP8266 exploits mean ESP8266-based IoT devices will have to be re-programmed.

ArduinoOTA readily reprograms the NodeMCU ESP-12, but ArduinoOTA will not update the ESP-01, which has too little memory and will not update the ESP-01S with default settings. Now I could change to the ESP-12E - it is only slightly more expensive, but this also involves changing the IoT enclosure. Basically a rebuild.

It turned out that the Arduino IDE default settings for the 'Generic ESP8266 Module' assumes an ESP-01 which only has 512K memory and is not enough for ArduinoOTA. But the later and much more common modules are ESP-01S which has 1M memory.

ESP-01 and ESP-01S. Click for video. If you do not speak Spanish, just turn on the captions and set to auto-translate.

I checked mine using esptool.py:

$esptool.py --baud=115200 --port=/dev/ttyUSB0 flash_id

esptool.py v2.7

Serial port /dev/ttyUSB0

Connecting....

Detecting chip type... ESP8266

Chip is ESP8266EX

Features: WiFi

Crystal is 26MHz

MAC: dc:4f:22:56:d9:bc

Uploading stub...

Running stub...

Stub running...

Manufacturer: 85

Device: 6014

Detected flash size: 1MB

Hard resetting via RTS pin...

To program it, first flash the ArduinoOTA Library BasicOTA using the usual programmer. On power-cycling the CH340, a new programming port should come up on your Arduino IDE. something like 'esp8266-xxxxxx at your_esp_ip_address'. Change your programming port to it.

To program the ESP-01S, first set Tools->Flash Size to 1M no SPIFFS. You can use the ArduinoOTA Library OTALeds sample program. Since OTALeds is for the ESP-12, make sure to change the line thus, so that the ESP-01S led gets assigned properly:

int led_pin = 1

The generic ESP-01S boards seem to work more reliably if I added a couple of Serial.print() statements in loop(). The modified program is:

#include <ESP8266WiFi.h>

#include <ESP8266mDNS.h>

#include <WiFiUdp.h>

#include <ArduinoOTA.h>

const char* ssid = "your_wifi_hotspot";

const char* password = "password";

//variabls for blinking an LED with Millis

const short LED_BUILTIN_ESP01 = 1; //GPIO1

const int led =LED_BUILTIN_ESP01; // ESP8266 Pin to which onboard LED is connected

unsigned long previousMillis = 0;  // will store last time LED was updated

const long interval = 1000;  // interval at which to blink (milliseconds)

int ledState = LOW;  // ledState used to set the LED

void setup() {

pinMode(led, OUTPUT);

    

  Serial.begin(115200);

  Serial.println("Booting");

  WiFi.mode(WIFI_STA);

  WiFi.begin(ssid, password);

  while (WiFi.waitForConnectResult() != WL_CONNECTED) {

    Serial.println("Connection Failed! Rebooting...");

    delay(5000);

    ESP.restart();

  }

  // Port defaults to 8266

  // ArduinoOTA.setPort(8266);

  // Hostname defaults to esp8266-[ChipID]

  // ArduinoOTA.setHostname("myesp8266");

  // No authentication by default

  // ArduinoOTA.setPassword("admin");

  // Password can be set with it's md5 value as well

  // MD5(admin) = 21232f297a57a5a743894a0e4a801fc3

  // ArduinoOTA.setPasswordHash("21232f297a57a5a743894a0e4a801fc3");

  ArduinoOTA.onStart([]() {

    String type;

    if (ArduinoOTA.getCommand() == U_FLASH)

      type = "sketch";

    else // U_SPIFFS

      type = "filesystem";

    // NOTE: if updating SPIFFS this would be the place to unmount SPIFFS using SPIFFS.end()

    Serial.println("Start updating " + type);

  });

  ArduinoOTA.onEnd([]() {

    Serial.println("\nEnd");

  });

  ArduinoOTA.onProgress([](unsigned int progress, unsigned int total) {

    Serial.printf("Progress: %u%%\r", (progress / (total / 100)));

  });

  ArduinoOTA.onError([](ota_error_t error) {

    Serial.printf("Error[%u]: ", error);

    if (error == OTA_AUTH_ERROR) Serial.println("Auth Failed");

    else if (error == OTA_BEGIN_ERROR) Serial.println("Begin Failed");

    else if (error == OTA_CONNECT_ERROR) Serial.println("Connect Failed");

    else if (error == OTA_RECEIVE_ERROR) Serial.println("Receive Failed");

    else if (error == OTA_END_ERROR) Serial.println("End Failed");

  });

  ArduinoOTA.begin();

  Serial.println("Ready");

  Serial.print("IP address: ");

  Serial.println(WiFi.localIP());

}

  void loop() {

    ArduinoOTA.handle();

  

  //loop to blink without delay

    unsigned long currentMillis = millis();

    if (currentMillis - previousMillis >= interval) {

    // save the last time you blinked the LED

    previousMillis = currentMillis;

    // if the LED is off turn it on and vice-versa:

    ledState = not(ledState);

    // set the LED with the ledState of the variable:

    Serial.print("Writing to LED: "); // Seems to work better if I added debug statements

    Serial.println(ledState);  // Seems to work better if I added debug statements

    digitalWrite(led,  ledState);

    }

  }

 Good luck with your ESP-01S. Happy Trails.

The second life of a laptop / Node.js on Slackware

Youtube video

You only live twice, or so it seems
One life for yourself, and one for your dreams

This dream is for you, so pay the price
Make one dream come true, you only live twice

- Nancy Sinatra, "You Only Live Twice"

My three-year old laptop, an Acer Aspire F15 was meant as a cheap, expendable laptop for rough, almost sacrificial use on-site. It just about survived the KV-MRT Phase 1 project, but only just.  Its screen failed - the entire display was compressed into a line about 5mm high.

Acer AspireF15 (with repaired screen and 2 extra monitors) running a Slackware-based Actions on Google IoT server 

It had been defenestrated and ran Slackware 14.1, and by attaching external monitors to its HDMI and VGA auxiliary displays, I re-purposed it as an Actions on Google IoT server. Google had recently included Firebase in its Smart Home code, and the Aspire F15's version of Chrome no longer displayed Firebase webpages correctly.

Even worse, the Spectre and Heartbleed exploits meant Slackware 14.1 desperately needed an upgrade if it were to stand a chance as an IoT production server on the Internet. But of course it did not get upgraded, and it chugged along until last month when Google's accumulated upgrades broke my server program.

Well, no problem. I have installed Slackware for over 20 years, and went to the Slackware mirrors page for a suitable mirror and made a site rip of the latest and greatest version:

wget -r -c --tries=100 --read-timeout=300 \
-np https://mirror.math.princeton.edu/pub/slackware/slackware64-current/

Not all mirrors like to be ripped, so do use a robot.txt friendly program like wget and try another mirror until you get a tolerant one like princeton.edu.

Next make an ISO filesystem:
$xorriso -as mkisofs   -iso-level 3   -full-iso9660-filenames   -R -J -A "Slackware Install"
 -hide-rr-moved   -v -d -N   -eltorito-boot isolinux/isolinux.bin    -no-emul-boot -boot-load-size 4
-boot-info-table   -isohybrid-mbr /usr/share/syslinux/isohdpfx.bin   -eltorito-alt-boot
 -e isolinux/efiboot.img   -no-emul-boot -isohybrid-gpt-basdat -m 'source' 
-volid "SlackDVD"   -output ../slackware64-current-20190801.iso   .

And burn it into a DVD:
$growisofs -speed=1 -dvd-compat -Z /dev/sr0=slackware64-current-20190801.iso

Now a USB thumbdrive would probably be appropriate, and is known to work, but many of the slackware64-current mirrors do not have programs for a working boot USB drive. So the DVD-ROM would probably get you further, and anyway the Aspire F15 has a DVD drive.

So into the DVD drive it goes and the laptop rebooted, and the minute it is not running Linux (it was running the BIOS), reality struck: the installation process needs the laptop's native screen. Ouch.

Luckily FixItSam's Youtube video showed that the Aspire F15's screen is easily replaced.

Acer Aspire F15's screen is easily replaced

It only took about 15 minutes to remove, and is a Taiwan Innolux N156HGE-EAL Rev.C1. I bought a replacement screen from lunarkim for RM198 (about USD47) and since product links often disappear fast, here is a screenshot as well:

Click on image for the web page

It turned out to be a China Innolux N156BGA-EB2 Rev C.1 but it fitted the screen mount, and worked quite well.

Slackware installed quite painlessly after that; and Google Smart Home code could start, but when Google Assistant attempted to link to the server, the server program failed with the message:

node: symbol lookup error: /home/xxx/smart-home-nodejs-master/node_m
odules/grpc/src/node/extension_binary/node-v57-linux-x64-glibc/grpc_node.node: undefined symbol: SSL_library_init

The server code ran on node.js, which  seems to have replaced old faithful, Apache. I had installed node.js from a SlackBuild, a decades-old way of installing Slackware code. I had run version 8.12.0 with no trouble, but the new version 8.16.0 stubbornly would not run.

Now I could regress back to 8.10.0 but after a well-known node.js exploit, an older version might not inspire confidence. After a fair bit of trying and searching, this thread seems relevant. It is not really a node.js problem but can be fixed by certain node.js versions. In particular:

Your best bet in using gRPC at the moment under Ubuntu 18.04 is to:

• install nodejs using nvm
  or
• install gRPC by compiling from source: npm install --build-from-source=grpc.

'npm install --build-from-source=grpc' did not work for me, but reinstalling nodejs using nvm did.

You install nvm thus:

$wget -qO- https://raw.githubusercontent.com/nvm-sh/nvm/v0.34.0/install.sh | bash
=> Downloading nvm from git to '/home/xxx/.nvm'
=> Cloning into '/home/xxx/.nvm'...
remote: Enumerating objects: 278, done.
remote: Counting objects: 100% (278/278), done.
remote: Compressing objects: 100% (249/249), done.
remote: Total 278 (delta 33), reused 88 (delta 16), pack-reused 0
Receiving objects: 100% (278/278), 142.36 KiB | 121.00 KiB/s, done.
Resolving deltas: 100% (33/33), done.
=> Compressing and cleaning up git repository

=> Appending nvm source string to /home/xxx/.bashrc
=> Appending bash_completion source string to /home/xxx/.bashrc
=> Close and reopen your terminal to start using nvm or run the following to use
 it now:

export NVM_DIR="$HOME/.nvm"
[ -s "$NVM_DIR/nvm.sh" ] && \. "$NVM_DIR/nvm.sh"  # This loads nvm
[ -s "$NVM_DIR/bash_completion" ] && \. "$NVM_DIR/bash_completion"  # This loads
 nvm bash_completion

To install node.js:

$nvm install node

Downloading and installing node v12.8.0...

Downloading https://nodejs.org/dist/v12.8.0/node-v12.8.0-linux-x64.tar.xz...

######################################################################### 100.0%

Computing checksum with sha256sum

Checksums matched!

Now using node v12.8.0 (npm v6.10.2)

Creating default alias: default -> node (-> v12.8.0)

The SSL_library_init problem went away; the Actions on Google server program ran and the Aspire F15, and Slackware got a new lease of life.

Now it seems incongruous to run node.js from Slackware. Debian would have been much easier. But after over twenty years, the least I can do is a long goodbye for Slackware. 

After all, you only live twice. Happy Trails.

Internet over Easy: TP-Link MR3420 as 3G Failover

The TP-Link MR-3420 takes a 3G USB Modem

(2020-01-22 Update): See here for an updated version.
The last six months I have been building and installing Internet of Things devices around the house and my nearby office. I run them from a single 8Mbps ADSL copper line from the Telekom Malaysia (TM), which connects to a TP-Link Archer D50 and which in turn acts as WAN for a TP-Link MR-3420. This provides ADSL with a failover to 3G.

TP-Link Archer D50 is a budget ADSL WiFi modem router

Now I did not really put much thought in selecting TP-Link or other models. They happened to be what was available in the Seremban stores over the years and I just happened to buy one. Or rather, a few, as I live on top of a little hill and the house tends to get struck by lightning a few times a year, usually killing the ADSL modem. I usually preset each modem with identical settings so that they can be swapped out in a few minutes.

If we are at home during a lightning storm we often disconnect the modem from the ADSL line as this seems to save it from getting destroyed in a strike. There is little to lose as the ADSL line often cuts out during an electrical storm anyway. When this happens, the MR-3420 fails over to a 3G connection using an MA260 3G USB dongle.

TP-Link MA260 3G USB dongle

The MR-3420 has an ADSL to 3G failover function:

MR-3420 ADSL to 3G failover

Now both the D50 and MR-3420 can put out WiFi hotspots but only the MR-3420 hotspot is backed up over 3G. This became really useful with the IoT devices because losing your Internet connection now meant losing access to your lights, fans and other IoT gizmos.

What's more being reliant on Google Assistant for reminders and note-taking makes you really feel the loss of an Internet connection. The failover to 3G is quite smooth and you really only notice a change in speed and responsiveness of the Internet. It costs me an extra RM29 a month for a basic 3GB data-only 3G connection.

When we went away for a week-long holiday we noticed that the MR-3420 reliably fails over, but will not fail back. Which means once it cuts over to the backup 3G line it will not cut back to the restored ADSL connection. This especially happens if the ADSL (ie WAN) connection went down but the network connection with the D50 is still up. Upgrading the firmware did not seem to help. However, disconnecting and reconnecting  the LAN cable to the D50 will result in a failback.

We used to get by 'failing back' by reconnecting the LAN cable by hand, but of late the ADSL has taken to intermittent disconnects several times a day. This pretty much leaves the MR-3420 in failover mode permanently and messing up our IoT chi.

Now I can probably buy a proper modem with 3G failover (and working failback!), but I am loath to junk electronics that still have life in them, especially having bought a few identical spare units. Ideally we get it to work by adding a little more junk ...

I have an orphaned Raspberry Pi Model B+ after having upgraded to the superlative Raspberry Pi 3 B+.

A picture of the Raspberry Pi B+ might seem redundant, but the various models can be hard to distinguish

 I also happened to have a Piface Digital 2, made redundant by those cheap ESP8266 relay boards.

PiFace Digital 2

And if I threw in a D-Link DES-1005A LAN switch, I can use traceroute to detect a restored ADSL connection to the D50 and use the Piface Digital 2 to power-cycle the LAN connection  (DES-1005A) between the D50 and the MR-3420. This should cause the MR-3420 to fail back.

D-Link DES-1005A uses a 5Vdc power supply

Now you can use any LAN switch you like, but the DES-1005A only requires 5Vdc at 60mA or so, which means the whole setup will run off the Raspberry Pi power supply (via a USB port). You can just as easily switch the mains power to the DES-1005A, but not having lethal voltages exposed (around the Piface connector) makes things a lot safer.

The D50 is necessary because the MR-3420 does not have an ADSL modem.

To prepare any Raspberry Pi, you just need to set it up with an Internet connection (in my case a D-Link DWA-123 USB WiFi dongle) and do:

# apt-get update
# apt-get upgrade

Next, using raspi-config I set it up to log into my D50 hotspot whose IP address for the purpose of this post is 'xx.xx.xx.1'. Next set up the Pi's Ethernet connection to the MR-3420, whose IP is 'yy.yy.yy.1'.  In your /etc/dhcpcd.conf add:

interface eth0
static ip_address=yy.yy.yy.2/24

interface wlan0
static ip_address=yy.yy.yy.2/24
static routers=yy.yy.yy.1
static domain_name_servers=8.8.8.8

Now set it so that access to 8.8.8.8 is routed via the TL-MR3420:

route add -host 8.8.8.8 gw yy.yy.yy.1 dev eth0

On reboot, your primary link to the Internet is now via the D50 WiFi hotspot. You are also linked to the MR-3420 via Ethernet. 

Now set up the Piface Digital 2. There is a slight hitch, but we can get it to work. Connect the DES 1005A LAN switch 5V line to Relay0:

Piface Digital 2 Pinout. Only Relay0 NC and CO are used

The 0V return line does not need to be switched.

The Piface Digital 2's program is simplicity itself;

# cat relay1.py
from time import sleep
import pifacedigitalio

DELAY = 2.0  # seconds

if __name__ == "__main__":
    pifacedigital = pifacedigitalio.PiFaceDigital()
    #  power-cycle LAN switch
    pifacedigital.leds[0].turn_on()
    sleep(DELAY)
    pifacedigital.leds[0].turn_off()

You run it using the command:

# python3 ./relay1.py

Now we will need to detect a failover condition. We try to connect to an always available server, say Google's DNS server 8.8.8.8. Now the Pi will default to using the D50 WiFi, so to force it to use Ethernet we do:
# route add -host 8.8.8.8 gw yy.yy.yy.1 dev eth0

The traceroute should now produce:
# traceroute 8.8.8.8
traceroute to 8.8.8.8 (8.8.8.8), 30 hops max, 60 byte packets
 1  yy.yy.yy.1 (yy.yy.yy.1)  0.768 ms  0.939 ms  0.612 ms
 2  xx.xx.xx.1 (xx.xx.xx.1)  0.893 ms  0.706 ms  0.630 ms
 3  115.134.254.254 (115.134.254.254)  40.218 ms  42.331 ms  43.880 ms
 4  10.55.106.61 (10.55.106.61)  53.481 ms 10.55.106.63 (10.55.106.63)  53.336 m
s  55.914 ms
 5  10.55.39.196 (10.55.39.196)  55.378 ms 10.55.39.146 (10.55.39.146)  55.988 m
s 10.55.39.196 (10.55.39.196)  57.889 ms
 6  10.55.48.64 (10.55.48.64)  61.153 ms 10.55.48.66 (10.55.48.66)  46.202 ms 10
.55.48.64 (10.55.48.64)  48.096 ms
 7  72.14.197.66 (72.14.197.66)  50.004 ms  51.596 ms  53.118 ms
 8  108.170.249.225 (108.170.249.225)  55.585 ms 108.170.249.241 (108.170.249.24
1)  46.389 ms 108.170.250.1 (108.170.250.1)  49.268 ms
 9  dns.google (8.8.8.8)  50.805 ms  52.660 ms  54.818 ms

Now we only need to look for the D50's IP address so this is sufficient:
# traceroute 8.8.8.8 | grep -w 'xx\.xx\.xx\.1'
 2  xx.xx.xx.1 (xx.xx.xx.1)  0.893 ms  0.706 ms  0.630 ms

Now if the above command produces nothing we now know we are in failover mode and we now need to check if the ADSL is restored. Here is the output of a restored D50:

# ping -c 4 -I wlan0 8.8.8.8 

PING 8.8.8.8 (8.8.8.8) from xx.xx.xx.2 wlan0: 56(84) bytes of data. 

64 bytes from 8.8.8.8: icmp_seq=1 ttl=57 time=57.1 ms 

64 bytes from 8.8.8.8: icmp_seq=2 ttl=57 time=52.6 ms 

64 bytes from 8.8.8.8: icmp_seq=3 ttl=57 time=51.4 ms 

64 bytes from 8.8.8.8: icmp_seq=4 ttl=57 time=49.1 ms 

--- 8.8.8.8 ping statistics --- 4 packets transmitted, 4 received, 0% packet loss, time 3005ms

rttmin/avg/max/mdev = 49.103/52.587/57.140/2.931 ms

Again, when there is no ADSL service there is no output. After confirming ADSL is back we just need to power cycle the DES-1005A LAN switch:

# python3 ./relay1.py

A bash script can do this:

# cat failover

#!/bin/bash

ADSL=$(traceroute 8.8.8.8 | grep -w 'xx\.xx\.xx\.1')

echo "result is "  $ADSL

if  [[ "$ADSL" == "" ]]

then

  date

  echo "Streamyx failed ..."

  ping -c 4 -I wlan0 8.8.8.8

  if [ $? == 0 ]

  then

    echo "Need to cut back!"

    python3 ./relay1.py

  fi

else

  echo "all's well"

fi

And for continuous operation:

# cat failover_loop

#!/bin/bash

echo on

while [ 1 ]

do 

  console_output=$(/home/heong/failover/failover)

  echo $console_output

  echo "Sleeping 60s"

  sleep 60

done

3G failback: from left, TL-MR3420 with MA260 3G dongle, Raspberry Pi with Piface 2 shield, DES-1005A LAN switch and Archer D50

And there you have it. MR-3420 failback. Internet over easy.

Note if you are a moderate Internet user you can get by with just the 3G line and you would not have to bother with this ADSL failover palaver. Or if you are blessed with a fiber optic connection, or any other service provider besides TM ... 

Happy Trails.