Wednesday 18 December 2019

AS3935 Lightning Detector: The Sequel

“Look aloft!” cried Starbuck. “The corpusants! the corpusants!"
 "The lightning flashes through my skull; mine eyeballs ache ... " - Captain Ahab

The rainy season came late and in its very first acid test, the AS3935 detected the oncoming storm and disconnected the ADSL modem in good time. However a really close strike after that took out the AS3935's relay module; the lightning must have arced over the open relay contacts into the 5V line. I had unwisely left my laptop connected to it to collect the debug messages, so the lightning took out my laptop as well as its power module, and plunging the neighborhood into darkness.

And to prove it was no fluke, repeated strikes on the now-unpowered system produced white arcs on a twitching relay board, a real sight to behold in the darkness. "The corpusants", like Starbuck said in Melville's 'Moby Dick'. This is probably very risky, so do proceed with caution if you ever have the misfortune to encounter this.


Scorch marks on the relay module

The over-current on the 5V line is such the heat lifted the copper trace right off the PCB.


The surge must have arced over to the nearby 5V pin on its way to the USB power line

And welded the USB connector to the hub.




The USB hub socket was welded to the cable. Note the melted contact on the left.
The ESP-12E had no obvious signs of damage but was unresponsive. Happily the AS3935 was undamaged.


Burst mains 'Live' wire in the power extension cable

The 'Live' wire in the mains power extension (to laptop power adapter) open-circuited, although the 13A fuze was intact. The 3A 230V fuze in the laptop adapter blew. This calls for a rethink:
  1. the relay contacts need to be a lot better isolated from each other
  2. more isolation is needed between the relay module and the ESP-12E
  3. the power modules need to be better isolated

(Update 2020-01-22): The lightning strike propagated along the 5V rail and damaged the LM1117-3.3 regulator in the ESP-12E. It also seemed to have slightly damaged the AS3935 (which also takes 5V directly to the IC) and although its registers read normal and it can be calibrated, it no longer classified 'disturber' events as lightning, no matter how near. New parts have been ordered - this is not over!

(Update 2020-02-15): Dismantled my laptop, an Asus X751L. 


Asus X751L disassembled - one of the worst things about laptop repair is the disassembly. There are myriad flimsy plastic tabs to break and also various irreplaceable flexible cables to rip off by accident
As suspected, the fuses had blown:


Blown fuses, middle, just beside the 6-pin 19V power adapter connector. The power MOS QM3024M3 are the square 8-pin IC just to the left
The fuses lead to the drain of a QM3024M power MOSFET rated at 30V and a stonking 46A. The source is connected to a second QM3024M3, and this second MOSFET has a short-circuit from drain to source. Ouch.

The first MOSFET is likely used to turn on 19V from power adapter to the mainboard power supply unit, and the other MOSFET to connect the battery when there is no mains supply. The short-circuit means that the lightning strike short-circuited the 19V adapter output and battery output simultaneously to 0V. Indeed the battery was completely drained. Looks like I have a dead power MOSFET as well. 

Dead bug: QM3024M3 upside down. On removal the mainboard is no longer shorted


I confirmed this by de-soldering it from the mainboard.

Watch this space. Happy Trails.

"... I will do my endeavor. I try all things; I achieve what I can" - Ishmael,  'Moby Dick' by Herman Melville



Monday 16 December 2019

IEEE 802.11w: Securing your WiFi from Deauth Attack Part 2: Electric Kool-Aid Acid Test


"You are hereby empowered!!!" - Tom Wolfe, The Electric Kool-Aid Acid Test

In Part 1, we tested for cheap devices compatible with IEEE 802.11w-2009. Here, we will just detail the configuration files needed to build a Ieee 802.11w-209 compatible Raspberry Pi 3 B+ WiFi Bridge. The actual steps are detailed in the official Raspberry Pi site.  We will need:
1. Raspberry Pi 3 B+
2. Very good 5V 2.5A USB power module (e.g. the Raspberry Pi 2.5A module). This requirement is important! I used a stonking 5V 6A TDK RDM05-6R0
3. Realtek RTL8812BU generic WiFi dongle. Or a dongle with any of the Atheros chips tested in Part 1.
4. ADSL modem router with wired (ie copper) LAN interface.


For simplicity, the above diagram omits the power supplies and the powered hub. The RTL8812BU can draw a lot of power, especially if connected to an outdoor antenna.

Note it is also possible to set up the above devices as WiFi repeater by using the Raspberry Pi's built-in WiFi chip (wlan0) as WAN.

Top middle: TDK 5V 6A power module. Center: Raspberry Pi 3 B+ with RTL8812BU. Bottom left: thick USB power cable with ammeter showing 920mA current draw

First, set up your ADSL modem and make sure you have Internet WAN access via the copper LAN (ie wired Ethernet). I used a TP-Link Archer D20.

Next, set up your Raspberry Pi 3 B+ with the latest and greatest version of Raspbian. The Linux version of the installation guide worked for me, but there is also a Windows version. It is really worth using a fast sdcard (16GB is sufficient and 32GB is plenty), Class 10 or better if you can manage it.  You will need to set it up to log into your Internet connection. My Pi connected to the Internet when I plugged in the LAN cable. After it has finished installation you need to update it immediately:

# apt-get update
# apt-get upgrade

This can take hours depending on your Internet connection and you will have to reboot your Pi. Next, set up your root password:
# sudo vi /etc/passwd

Remove the 'x' from the line
root:x:0:0::/root:/bin/bash

Next set the root password using:
#sudo passwd root

I usually use the wifi bridge in 'headless' (ie no monitor or keyboard) so I usually turn off the GUI using
#sudo raspi-config

To control it, I usually enable the ssh server (again using raspi-config). Now to run it headless I make sure my laptop is connected to the same network and if it is also running Debian (Rasbian is a version of Debian) I simply do:
$ssh -t pi@raspi.local

It is also handy to have your first setup connected to copper LAN as well as keyboard and monitor, as we will be messing about with networking tools and a mistake is likely to freeze up your remote login.

If wlan1 the rtl8812bu does not come up, refer to Part 1.

Next you will need to stop systemd from messing with your network interfaces.
# systemctl mask wpa_supplicant.service
Created symlink /etc/systemd/system/wpa_supplicant.service � /dev/null.

In /etc/dhcpcd.conf add the lines:
interface eth0
static ip_address=192.168.1.1/24

interface wlan0
  denyinterfaces wlan0
  nohook wpa_supplicant

interface wlan1
  denyinterfaces wlan1
  nohook wpa_supplicant

Reboot, and you should be ready for the next step.

We need to use hostapd for our bridge, so we stop systemd from messing with it:
# systemctl mask hostapd

We make a bridge:
# brctl addbr br0
# brctl addif br0 wlan1

If all went well, you should get:
# brctl show br0
bridge name     bridge id               STP enabled     interfaces
br0             8000.1cbfce5d51a0       no              wlan1

Add the copper LAN to your new bridge in case you want to connect client devices by wire
# brctl addif br0 eth0
# ifconfig eth0 0.0.0.0 up
# ifconfig wlan1 0.0.0.0 up

Where 192.168.1.1 is the IP address of your Pi at eth0 and 192.168.0.1 is the address of your WAN router (ie the D-Link Archer D50 in the diagram):
# ifconfig br0 192.168.1.1 up

Next set up your dnsmasq config file with:
interface=br0
except-interface=lo
listen-address=192.168.1.1
bind-interfaces

As usual you need to tell systemd to keeps its grubby hands to itself:
# systemctl mask dnsmasq
# killall dnsmasq 
# dnsmasq -C /etc/dnsmasq.conf 

Next we get the Pi to start forwarding. Add the following lines to /etc/sysctl.conf:
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv4.ip_forward = 1
net.ipv4.conf.all.rp_filter = 1
net.ipv4.conf.default.rp_filter = 1
net.ipv4.ip_dynaddr = 1
net.ipv4.conf.all.accept_source_route = 0
net.ipv4.conf.default.accept_source_route = 0
net.ipv4.conf.all.accept_redirects = 0
net.ipv4.conf.default.accept_redirects = 0

# sysctl -p /etc/sysctl.conf

Next is hostapd. Set up the config file /etc/hostapd/hostapd.conf thus:
interface=wlan1
driver=nl80211
ssid=ElectricKoolAid
hw_mode=g
channel=1
macaddr_acl=0
wpa=2
wpa_passphrase=VerySecretPassword
rsn_pairwise=CCMP
ieee80211w=2
wmm_enabled=1
auth_algs=3
ignore_broadcast_ssid=1
wpa_key_mgmt=WPA-PSK-SHA256 WPA-EAP-SHA256
wpa_pairwise=CCMP

# killall hostapd
# hostapd -dd -P /var/run/hostapd.pid /etc/hostapd/hostapd.conf -B

Next set up /etc/firewall.conf thus:

*filter
:INPUT DROP [39:4576]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [42055:10283301]
-A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A INPUT -i br0 -j ACCEPT
-A INPUT -i eth0 -j ACCEPT
-A FORWARD -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -i br0 -j ACCEPT
-A FORWARD -i eth0 -j ACCEPT
COMMIT
# Generated by iptables-save v1.6.0 
*nat
:PREROUTING ACCEPT [78732:17589805]
:INPUT ACCEPT [29742:7228146]
:OUTPUT ACCEPT [2937:514776]
:POSTROUTING ACCEPT [985:97284]
-A POSTROUTING -s 192.168.1.0/24 -j MASQUERADE
COMMIT
# Generated by iptables-save v1.6.0 
*mangle
:PREROUTING ACCEPT [1318452:373291697]
:INPUT ACCEPT [623742:150528221]
:FORWARD ACCEPT [120385:79521689]
:OUTPUT ACCEPT [42068:10285133]
:POSTROUTING ACCEPT [168801:90872517]
-A POSTROUTING -p tcp -m tcp --tcp-flags SYN,RST SYN -m comment --comment "fix packet size for stuff that\'s being routed through this box (SEE NOTE *)" -j TCPMSS --clamp-mss-to-pmtu
COMMIT

# iptables-restore < /etc/firewall.conf

Your IEEE 802.11w-2009 bridge AP "ElectricKoolAid" with "VerySecretPassword" should be up by now.

Over at the client (in my case the Acer Aspire E1) make a wpa_supplicant config file wpa_supplicant.conf. Notice we make IEEE 802.11w compulsory:

ctrl_interface_group=0
eapol_version=1
ap_scan=1
fast_reauth=1

# WPA protected network, supply your own ESSID and WPAPSK here:
network={
  scan_ssid=1
  ssid="ElectricKoolAid"
  proto=RSN  
  key_mgmt=WPA-PSK-SHA256
  pairwise=CCMP TKIP 
  group=CCMP TKIP
  psk="VerySecretPassword"
  ieee80211w=2
  priority=10
}

And, assuming you have already wrestled with systemd (and won!) there too:
# wpa_supplicant -d -Dnl80211 -iwlan0 -c/etc/wpa_SIKAMAT7.conf -B

Note: for some reason, nl80211 worked a lot better than wext

And you should have a working link to the Pi bridge. 

# wpa_cli -iwlan0 status   
bssid=11:22:33:44:55:66
freq=2412
ssid=ElectricKoolAid
id=0
mode=station
pairwise_cipher=CCMP
group_cipher=CCMP
key_mgmt=WPA2-PSK-SHA256
pmf=2
mgmt_group_cipher=BIP
wpa_state=COMPLETED
ip_address=192.168.0.114

And there should be Internet access; provided you have set up your WAN router's dhcp server, the client at the end of the bridge "ElectricKoolAid" should get its dhclient requests routed straight through.

ping -c 4 -I wlan0 8.8.8.8
PING 8.8.8.8 (8.8.8.8) from 192.168.0.114 wlan0: 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=56 time=524 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=56 time=518 ms
64 bytes from 8.8.8.8: icmp_seq=3 ttl=56 time=380 ms
64 bytes from 8.8.8.8: icmp_seq=4 ttl=56 time=346 ms

--- 8.8.8.8 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3001ms
rtt min/avg/max/mdev = 346.240/442.040/524.243/79.921 ms

To get it to start on power up, just put the commands in /etc/rc.local

There you have it, IEEE 802.11w-2009 AP and client, immune from the dreaded Deauth Attack. In part 3 we will launch an aircrack-ng deauth attack at the the bridge AP.

Or, as Tom Wolfe said: "You are hereby empowered!!!". Happy Trails.

Thursday 12 December 2019

IEEE 802.11w: Securing your WiFi from Deauth Attack Part 1 - The Right Stuff (Updated 2019-12-31)

The Right Stuff by Tom Wolfe

"No bucks, no Bucks Rogers" - Gus Grissom, The Right Stuff


One of the guilty pleasures of Linux is aircrack-ng,  software tools to -cough- test WiFi security. In the good old days with WEP WiFi passwords you can get a WiFi password within 2 minutes. It was so convenient it was much quicker than getting the password the legal way from your company sysadmin. Back in the day, the best hotspots are for Top Management, people who don't browse or torrent, dictate their emails and think that typing is for secretaries.


aircrack-ng on Kali Linux


IEEE 802.11 is the standard for WiFi networks. Amongst other things it specifies Management Frames, ie data packets sent by both the client and the Access Point to 'manage' or maintain communications with each other, for example, logging in, authentication and when one party wishes to terminate the connection, 'deauthenticate'.

With WPA (in particular WPA2), cracking WiFi hotspots became much much harder, but if the password is weak (most passwords are weak) you can speed things up by getting aircrack-ng to issue 'deauthenticate' Management Frames in your victim's name (ie MAC address). This causes the target to re-issue authentication frames. The more frames you sniff out, the quicker you will crack it.



Now if you do nothing but issue deauth frames, say with a bash script, you effectively jam WiFi for the luckless target, and if you target the AP MAC address you jam the whole hotspot if you are close enough. Normally you did it for a purpose, like persuading the legal users to quit using their WiFi so you can, -cough-  check-in your flight online. People who deauth used to have a higher purpose ... besides it is easy to spot the pasty-faced user on the Linux laptop with an out-sized WiFi antenna.

ESP8266 deauth jammer

But with the advent of Espressif's ESP8266  you can now buy or build a cheap portable deauth jammer that will fit inside your pocket, and now all you have to do is walk up to a hotspot to jam it. And since many security cameras these days are WiFi, an intruder can disable them by just driving up to your gate.

Now having made quite a few ESP8266-based IoT devices, it is time to secure my WiFi from deauth attacks: IEEE 802.11w-2009. With 802.11w, the Management Frame is encrypted and becomes a Protected Management Frame (PMF). This means a WiFi outsider now cannot deauthenticate you with impunity.

IEEE 802.11w stops deauth attack by encrypting the 'deauthenticate' command

Now you would think you can buy a new reasonably-priced IEEE 802.11w-2009 router for home use. After all the standard is 10 years old. And you would be wrong. It is common enough in Big Iron like Cisco or Aruba (one of dear old Hewlett Packard's random name changes) but it is not common in say, a D-Link or TP-Link. OpenWrt is a shining exception, as easy as checking an option in its base distribution. But OpenWrt often only runs on the more expensive models, and then there is the problem of compatibility with WiFi client devices.

The usual culprit would be production cost. As Gus Grissom said, "No bucks, no Buck Rogers". It turned out it is probably a compatibility issue, specifically of encryption, a missing cipher. IEEE 802.11w uses CCMP-128, SHA-256 and AES128-CMAC encryption for Management Frame protection.

To tell if your hardware has The Right Stuff, you need to execute the Linux command 'iw phy', shown here for a no-name WiFi USB dongle based on the Realtek RTL8812BU:

# iw phy
Wiphy phy8
        max # scan SSIDs: 9
        max scan IEs length: 2304 bytes
        max # sched scan SSIDs: 0
        max # match sets: 0
        max # scan plans: 1
        max scan plan interval: -1
        max scan plan iterations: 0
        Retry short limit: 7
        Retry long limit: 4
        Coverage class: 0 (up to 0m)
        Supported Ciphers:
                * WEP40 (00-0f-ac:1)
                * WEP104 (00-0f-ac:5)
                * TKIP (00-0f-ac:2)
                * CCMP-128 (00-0f-ac:4)
                * CMAC (00-0f-ac:6)

And listed under the 'Supported Ciphers' the bare minimum you need is CCMP-128 (note OUI 00-0f-ac:4) and CMAC (OUI 00-0f-ac:6). There is a helpful write-up here. Compare this to the output for the delicious Qualcomm AR9565 from my Acer Aspire E1 laptop:

$iw phy                                                
Wiphy phy0
        max # scan SSIDs: 4
        max scan IEs length: 2257 bytes
        max # sched scan SSIDs: 0
        max # match sets: 0
        max # scan plans: 1
        max scan plan interval: -1
        max scan plan iterations: 0
        Retry short limit: 7
        Retry long limit: 4
        Coverage class: 0 (up to 0m)
        Device supports RSN-IBSS.
        Device supports AP-side u-APSD.
        Device supports T-DLS.
        Supported Ciphers:
                * WEP40 (00-0f-ac:1)
                * WEP104 (00-0f-ac:5)
                * TKIP (00-0f-ac:2)
                * CCMP-128 (00-0f-ac:4)
                * CCMP-256 (00-0f-ac:10)
                * GCMP-128 (00-0f-ac:8)
                * GCMP-256 (00-0f-ac:9)
                * CMAC (00-0f-ac:6)
                * CMAC-256 (00-0f-ac:13)
                * GMAC-128 (00-0f-ac:11)
                * GMAC-256 (00-0f-ac:12)

CCMP-256 and in particular GCMP-256 looks really good. Now your mileage may vary with 'iw phy', as some device drivers are known not to be listed. I had best results with a late model Raspberry Pi (3 or 4) running the latest Rasbian (ie Debian).

You can do an online search for compatibility with this link.

From my very limited survey here in Seremban Malaysia only a minority of WiFi devices on the market supports this. And since some device drivers do not come up in 'iw phy' (in particular deliciously cheap Realteks), in the end I had to take my Linux laptop to my friendly local computer shop and test all 5 WiFi USB dongles models he had. None passed the test; in the end an ancient TP-Link TL-822N v1.1 proved up to the task.

TP-Link TL-W822N. Note only v1 worked, and possibly v2. v3 & v4 have Ralink chips

So did the Samsung Galaxy Note 5, with default setings:

Samsung Galaxy Note 5 will work as IEEE 802.11w client


A very pleasant surprise was the Raspberry Pi 3 Model B Plus' BCM4345:

        Supported Ciphers:
                * WEP40 (00-0f-ac:1)
                * WEP104 (00-0f-ac:5)
                * TKIP (00-0f-ac:2)
                * CCMP-128 (00-0f-ac:4)
                * CMAC (00-0f-ac:6)

Unfortunately the results were spotty. Upgrades to Rasbian caused it to stop working, but subsequent updates should fix this, so watch this space for updates.

And since I meant to link my deauth-proof routers to outdoor antennae I took a chance on the RTL8812BU generic USB dongles, from a hint here

Generic no-name RTL8812BU USB dongle also worked, but only as AP and not client/STA
This proved to be the toughest to use, as even my up-to-the-minute Rasbian (Buster) did not come with the device driver for it. The dongles came with driver source code, but cilynx (Randy C Wills) has a much better version here. cilynx's instructions (in README.md) worked very well for my Raspberry Pi 3 B+; I just needed to add in the Makefile

EXTRA_CFLAGS += -Wno-error=date-time

And change 'n' to 'y' for

CONFIG_80211W = y

Perhaps because this is a new driver, the default debug logs are very many and to prevent it from wearing out your micro sdcard, you might want to do:

# echo 3 > /proc/net/rtl88x2bu/log_level

And you are good to go.

In summary the WiFi hardware having the Right Stuff for deauth-proof IEEE 802.11w are:
1. Qualcomm Atheros AR9565 (Acer Aspire E1, Asus X751L)
2. Qualcomm Atheros QCA9377 (Acer Aspire F15)
4. Ralink RT5370 (Raspberry Pi 3 Model B Plus)
5. Realtek RTL8812BU (no-name China USB dongle)
6. TP-Link TL-WN822N v1.1
7. Samsung Galaxy Note 5 (and Fan Edition) as client device. The Galaxy Note 3 does not work.

Seven devices with the Right Stuff, just like the seven Project Mercury astronauts. The next step is to rebuild my WiFi Bridges to comply with IEEE 802.11w. But that is another story. 

Happy Trails.

Update: Removed the D-Link DWA-123 from the list, as it did not work. In the Raspberry Pi 3, weird kernel driver interactions caused the built-in  BCM4345 driver (brcmfmac) to work, but only with a Realtel RTL8188EU device plugged in. The last Raspbian update caused it to stop working so BCM4345 results are still pending

Monday 18 November 2019

Lightning detector AS3935: Part 3 of 3


Clockwise from top left: ADSL splitter, AS3935 (in vise), ESP-12E, relay module, powered USB hub, modem and POTS analog phone. All blinkenlights are go.


In the last part of this series, the AS3935 detector is hooked up to the ADSL line, and the ESP-12E is programmed to disconnect if there is a thunderstorm overhead. When the storm moves more than 8km away it will reconnect after 30 minutes.

As first mentioned in Part 2, it publishes to a local MQTT server, and also subscribes to it. This is to allow it to disconnect and reconnect the ADSL modem remotely on demand. For the last six months or so my TM Streamyx (now 'Unifi') ADSL2+ line has been unstable; dropping and reconnecting every few hours or worse. This constant failover to 3G tended to overload the TP-Link MR3420. Disconnecting the phone line during the bad periods seemed to give it some relief.

As before, the AS3935 can be quite sensitive to inteference. I ended up using separate shielded USB charger cables for 5V for both the ESP-12E and the relay module. At worst, excessive interference resulted in permanent false alarms for 'storm overhead'.

Also you can reduce false alarms by additional shielding. I partially encased the AS3935 module in a steel vise to stop it triggering when I turned on my fluorescent lights. Do not shield it completely or there will be nothing detected.

Jury-rigged steel vise shield for the AS3935


Connecting to the telephone line


My incoming POTS line has two wires, but a standard US RJ11 phone jack has at least 4. Our benighted Talikon Malaysia uses British standard (BS 6312), but most devices (not phone line) comes in the US RJ11 4-wire format, which is what I used.

Lanshack has a good page on the pinout:


The 4 wires in my RJ11 jacks cover pins 2 to 5 corresponding to Pair 1 and Pair 2. Pair 1, ie R1 & T1 at pins 3 & 4  (ie red & green wires) seem like a good starting point. I took out the modem ADSL cable, ie one with RJ11 jacks on both ends and cut it in two. The red and green cables are easily accessible.

Start with your modem phone cable, and cut in in half 


Modem cable with just Pair 1 reconnected

I verified my guess by reconnecting just Pair 1 and used my modified cable to reconnect the modem to the phone line. My ISP TM (TMnuts for short) uses ADSL2+ and that did not seem to affect my modem operation but your mileage may vary. Since broadband runs at 1Mbps up to 20Mbps changing the phone cables characteristic impedance with discontinuities like connectors and joints might affect it big-time.

Also, it would be wise to hook up the ADSL splitter and the POTS analog phone to see if they still work. If the joints did not affect your phone or modem setup, go ahead and hook up the relay module. A note of caution: disconnect the phone cable from your ISP line before you rewire. Phone line voltages are around 40Vdc and are usually safe to handle. This can change to as much as 100Vac if someone tried to call you on that phone. Be careful- phone lines can be hazardous! so you really do not want to touch the relay module when the phone is ringing.

Relay module cabling, clockwise from top left: 5V power, R1, T1 and TTL serial

I wired up the ISP side to relay Normally Open and the modem side to Relay Common. This means the relays need to be on most of the time and more power (about 100mA at 5V) will be used. The advantage is if the first strike took out the power mains, the modem will not be reconnected to the ISP.

If you only have the single relay version just switch T1 to break the circuit. This should help a lot and reduce the power of a strike, but my guess is the lightning strike can still reach the modem via R1 and if it is near enough will then complete the circuit to modem Earth or failing that to modem DC GND.

Of course if a strike is almost on top of the modem it will probably jump the relay air gaps, but the modem will be the least of your problems then. It is advisable not to go near the setup in a thunderstorm, even after the relay module has disconnected the modem. I have noticed the electronics twitching on following strikes even without power, ie after the first strike has taken out the power mains.

Software  

For esp8266 MQTT client I used pubsubclient. It was quite painless to use and its sample code worked first time with mosquitto so I will simply skip ahead to the complete version. For details on the mosquitto MQTT server, see Part 2. The nice thing about a local MQTT is that the AS3935 can then be used to disconnect more than one IoT device, for example the autogate, POTS phone extension and the satellite TV.

You can get a copy of the software from github.

To look at the MQTT output:

$mosquitto_sub -t 'AS3935/messages' -v

To turn on the relays

$mosquitto_pub -t 'AS3935/commands' -m '1'

Or you can do it from a smartphone App. I used MyMQTT. For now, you need to be in the same WiFi access point as the AS3936.

I used MyMQTT on my Android phone


As usual we keep monitoring for bugs and tuning the calibration settings. I tend to desensitize the alarms as Malaysia gets many violent but localized (ie usually not widespread) thunderstorms, and I do not want to trigger a disconnect until it is right on top of the AS3935. I also have fluorescent lights and a hot shower which tends to set it off unless I use a threshold setting of 9 and above.

This is my typical output for a real storm:

AS3935/messages 14:08:59 2019.11.19 Lightning detected 8 km away
AS3935/messages 14:09:04 2019.11.19 Lightning detected 6 km away
AS3935/messages 14:09:13 2019.11.19 Disturber detected
AS3935/messages 14:09:21 2019.11.19 Disturber detected
AS3935/messages 14:09:22 2019.11.19 Disturber detected
AS3935/messages 14:09:30 2019.11.19 Lightning detected 6 km away
AS3935/messages 14:09:45 2019.11.19 Disturber detected
AS3935/messages 14:10:07 2019.11.19 Storm overhead, watch out! Disconnecting the phone lines ..
AS3935/messages 14:20:13 2019.11.19 Disturber detected
AS3935/messages 14:31:10 2019.11.19 Lightning detected 12 km away
AS3935/messages 14:32:40 2019.11.19 Lightning detected 12 km away
AS3935/messages 14:33:18 2019.11.19 Disturber detected
AS3935/messages 14:36:02 2019.11.19 Disturber detected
AS3935/messages 14:40:08 2019.11.19 Reconnecting the phone lines ...

Once it disconnects, the software waits for the storm to move 9km away, waits a further 30min and then automatically reconnects.

At this point we just sit back and watch the light show. Happy Trails.

Wednesday 13 November 2019

Lightning Detector AS3935: adding Serial Relay Module and MQTT Server Part 2 of 3


In Part 1, we interfaced the AS3935 lightning detector to an ESP-12E using SPI. In Part 2 here, we add an ESP-01 2-channel WiFi relay board and also set up an MQTT server.

But first, some test results. After running the system in Part 1 for a few weeks we know that:
1. There are many false (ie 'disturber') alarms
2. Some lightning strikes are incorrectly classed as 'disturber'
3. The little antenna is directional, so be careful positioning it
4. The AS3935 and relay module are best on separate 5V wires, otherwise there may be too many false alarms
5. The settings that worked for me are: Noise Floor: 3, Spike Rejection: 0, and Watchdog Threshold:2
6. There is good correlation between actual lightning strikes and the AS3935 readings
7. I have good results disconnecting the relays when the lightning is less than 5km (ie 'storm overhead') and reconnecting it 30 minutes after  lightning strikes are further than 8km away.

2-channel wifi relay module


ESP-01 2 Channel WiFi Relay Module

I got mine from lazada.com for RM27 (about USD4) each. Unlike the other ESP8266 relay boards this one has a separate CPU to activate the relays and instead of using precious GPIO pins you can do it with the serial port UART0.

makerrelay has a good writeup on a very similar module, complete with schematics and identical relay switching commands. Even better, libretto (Sergiy Zaschipas) shows you how to reprogram the relay CPU.

I can not only replace the ESP-01 with a NodeMCU ESP-12E, but I can now add relay 'channels' by simply adding more relay boards. I do not even have to use extra GPIO pins - the same TTL serial port UART0 can drive all the relay boards.

Another convenience is the relay outputs need not be reset especially since the ESP8266 is sometimes best reset if it is unable to reconnect to WiFi. If the relay CPU is reprogrammed, the ESP8266 may even read back the relay states. Since the AS3935 drops the phone line when a storm is overhead it can actually cause WiFi disconnects.

Based on some of the board markings, it is possible the original design is by LC Tech. LC Tech also provides documentation.

NodeMCU ESP-12E Pinout
You connect GP101 (also TXD0 or TX) to the pin (ESP-12E bottom right) marked 'TX' on the relay board. Connect up the adjacent GND pin to the correcponding relay module pin and you are good to go.

Relay module, solder side
Note that if you want to supply power to the relay board you need 5V which is on the other column of pins (VIN) of the ESP-12E, and not the very tempting 3.3V pin.

The relay commands are:

//Hex command to send to serial for close relay
byte relON[]  = {0xA0, 0x01, 0x01, 0xA2};
//Hex command to send to serial for open relay
byte relOFF[] = {0xA0, 0x01, 0x00, 0xA1};
//Hex command to send to serial for close relay
byte rel2ON[]  = {0xA0, 0x02, 0x01, 0xA3};
//Hex command to send to serial for open relay
byte rel2OFF[] = {0xA0, 0x02, 0x00, 0xA2};

To turn on relay 1:

Serial.write (relON, sizeof(relON));

It is that simple. My setup now looks like this:

AS3935 SPI and Relay serial port cables consolidated to one 2-headed monster. Note the 5V VIN cable on the far right. System power is via the USB cable (silver)


Top left to right: ESP-12E (partially hidden), relay module and AS3935

MQTT Server


Now that I have a way of disconnecting the ADSL modem, a convenient way to monitor the AS3935 results would be nice. Initially I simply printed the results using Serial.print() and got the output via the Arduino IDE Serial Monitor.

This worked when I am at my work station but pretty soon I was printing out to my Adafruit MQTT feed. This worked much better until a lightning storm came close enough for the relay module to disconnect the ADSL modem at which point the Adafruit connection got interrupted.

Now my WiFi does have a 3G failover, but it takes precious seconds for the Adafruit feed to re-establish itself and vital AS3935 messages are often lost. So one of the things to try would seem to be a local MQTT server, which could capture and retain my MQTT messages and if necessary upload them to my Adafruit feed for when I am offsite.

It sounds like a lot of work, until along came Mosquitto. Since my main station is still Slackware 14.2, I got my SlackBuild here:

As usual douwnload the SlackBuild tarball and unzip it:

$tar -xvpzf mosquitto.tar.gz
mosquitto/
mosquitto/slack-desc
mosquitto/README
mosquitto/mosquitto.info
mosquitto/doinst.sh
mosquitto/mosquitto.SlackBuild
usr/man/man8/mosquitto.8.gz
usr/sbin/
usr/sbin/mosquitto
usr/share/

Then download the mosquitto source code (from the SlackBuild link not the Eclipse one) and copy it into the mosquitto slackbuild directory. Then all you need is

$./mosquitto.SlackBuild

And then a Slackware install:

$upgradepkg --install-new /tmp/mosquitto-1.6.7-x86_64-1_SBo.tgz

Next you need to set up a mosquitto user:
$vi /etc/mosquitto/mosquitto.conf

And add the line:
user mosquittouser

If necessary create the new user:
$adduser mosquittouser

You start the server thus:
$mosquitto -c /etc/mosquitto/mosquitto.conf
1572678454: mosquitto version 1.6.7 starting
1572678454: Config loaded from /etc/mosquitto/mosquitto.conf.
1572678454: Opening ipv4 listen socket on port 1883.
1572678454: Opening ipv6 listen socket on port 1883.

And to check use the included client subscriber:
$mosquitto_sub -t 'test/topic' -v

The publish:
$mosquitto_pub -t 'test/topic' -m 'hello world of mosquitto'

And that was all you need. I suspect it is even easier in Debian/Ubuntu. The next thing I need would be an MQTT client for the ESP-12E, but that is another story.

So till then, Happy Trails.

Sunday 20 October 2019

Thunderbolts and Lightning Detector AS3935 Part 1 of 3

Bohemian Rhapsody, Queen
"Thunderbolts and lightning, very very frightening me" - Queen, Bohemian Rhapsody

Being near the equator, Malaysia is very frequently struck by lightning and has a flash rate density close to 100. ADSL broadband modems, cordless telephones, network switches, autogate controllers and monitors regularly get damaged.

A few times a year the storms get really bad: you can hear the lightning hiss followed by the thunderclap. The phone cables are fried, electric power is out and the dogs try to get on your lap, never mind the cat.

I often repair the electronics, but there is something about lightning damage: they are never quite the same. TVs seem more immune, but if you use them with a computer (as a monitor), they get damaged by lightning. Go figure.

But the most annoying damage is to the ADSL modem. This is because I run an IoT server 24/7. I have backup power via UPS but a fried modem often results in service interruption. Eventually I installed a backup 3G modem using a TP-Link MR-3420.

For some reason, things get a lot better if we simply disconnected the telephone line from the ADSL modem and phone. This causes the MR-3420 to fail over to 3G, and I put in a little fix to ensure a failback once the ADSL modem is reconnected. Things do not get fried as much maybe once a year a LAN switch or a monitor cops it.

And this works very well for us unless we are away on holiday. What we need is a system to disconnect the modem from the phone line when the storm is close and reconnect when it passes. And sure enough, cometh the hour cometh the chip: AS3935 from Austria Microsystems.

I got my AS3935 for RM92 (about USD20)

But why not stay on 3G? The problem is in Malaysia we still have a data cap on 3G, maybe 10GB per month, and if your IoT device is a security camera it quickly adds up.

Since I had good results with the SPI interface on the NodeMCU ESP-12E before,
I made the following SPI cable to the CJMCU AS3935:

PIN#   AS3935                               diagram  ESP-12E    Arduino IDE no
 1          VCC                                      5V
 2          GND                                     GND
 3          SCL                    HSCLK     D5      GPIO14                     14
 5          MOSI                  HMISO     D6      GPIO12                     12
 4          MISO                  HMOSI     D7      GPIO13                     13
 6          CS                       HCS         D8      GPIO15                     15
 7          SI                                         GND
 8          IRQ                                      D2      GPIO2                         4
 9          EN_VREG                          5V

The AS3935 allows for 3V3 to be used but the CJMCU board did not bring out VREG to the header, so I needed to supply 5V and enable the onboard regulator by tying EN_VREG to 5V. The last column, 'Arduino IDE number' is the pin number to use for the Arduino sketch.  I used a 12-way header with a 10-way ribbon cable because I wanted to use GPIO1 and GPIO3 to control a relay board.

For software, I used Eva Schindling's Thunder and Lightning. Even though it was written five years ago for the Arduino, it pretty much worked first time for the ESP-12E. I used her AS3935_example.ino sketch with just a few tweaks to accommodate my SPI cable and the ESP-12E.

Download her code in a zip file, unzip it and the make a symbolic link to your Arduino IDE library:

ln -s /home/guest/as3935/ThunderAndLightning-master/library/AS3935 /home/guest/Arduino/libraries/AS3935

And that was it. I knew Arduino SPI code did not travel very well to the ESP8266 and was well, thunderstruck it worked first time. And unlike Eva who back in 2014 seemed to have lacked good thunderstorms to test with, a big storm arrived just in time for testing.

Life indeed is good. Happy Trails.
/*
  LightningDetector.pde - AS3935 Franklin Lightning Sensor™ IC by AMS library demo code
  Copyright (c) 2012 Raivis Rengelis (raivis [at] rrkb.lv). All rights reserved.

  This library is free software; you can redistribute it and/or
  modify it under the terms of the GNU Lesser General Public
  License as published by the Free Software Foundation; either
  version 3 of the License, or (at your option) any later version.

  This library is distributed in the hope that it will be useful,
  but WITHOUT ANY WARRANTY; without even the implied warranty of
  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
  Lesser General Public License for more details.

  You should have received a copy of the GNU Lesser General Public
  License along with this library; if not, write to the Free Software
  Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
*/

#include <SPI.h>
#include <AS3935.h>

void printAS3935Registers();

// Function prototype that provides SPI transfer and is passed to
// AS3935 to be used from within library, it is defined later in main sketch.
// That is up to user to deal with specific implementation of SPI
// Note that AS3935 library requires this function to have exactly this signature
// and it can not be member function of any C++ class, which happens
// to be almost any Arduino library
// Please make sure your implementation of choice does not deal with CS pin,
// library takes care about it on it's own
byte SPItransfer(byte sendByte);

// Iterrupt handler for AS3935 irqs
// and flag variable that indicates interrupt has been triggered
// Variables that get changed in interrupt routines need to be declared volatile
// otherwise compiler can optimize them away, assuming they never get changed
void AS3935Irq();
volatile int AS3935IrqTriggered;

// First parameter - SPI transfer function, second - Arduino pin used for CS
// and finally third argument - Arduino pin used for IRQ
// It is good idea to chose pin that has interrupts attached, that way one can use
// attachInterrupt in sketch to detect interrupt
// Library internally polls this pin when doing calibration, so being an interrupt pin
// is not a requirement

#define IRQpin 4 // ESP-12E
#define CSpin 15

AS3935 AS3935(SPItransfer,CSpin,IRQpin);

void setup()
{
  // Serial.begin(9600);
  Serial.begin(115200); // 2019-10-19 ESP-12E default
  // first begin, then set parameters
  SPI.begin();
  // NB! chip uses SPI MODE1
  SPI.setDataMode(SPI_MODE1);
  // NB! max SPI clock speed that chip supports is 2MHz,
  // but never use 500kHz, because that will cause interference
  // to lightning detection circuit
  SPI.setClockDivider(SPI_CLOCK_DIV16);
  // and chip is MSB first
  SPI.setBitOrder(MSBFIRST);
  // reset all internal register values to defaults
  AS3935.reset();
  // and run calibration
  // if lightning detector can not tune tank circuit to required tolerance,
  // calibration function will return false
  
  
  //if(!AS3935.calibrate())
  //  Serial.println("Tuning out of range, check your wiring, your sensor and make sure physics laws have not changed!");



  outputCalibrationValues();
  recalibrate();

  AS3935.setNoiseFloor(1);
  AS3935.setSpikeRejection(2);
  AS3935.setWatchdogThreshold(1);
  
  outputCalibrationValues();
  recalibrate();

  // since this is demo code, we just go on minding our own business and ignore the fact that someone divided by zero

  // first let's turn on disturber indication and print some register values from AS3935
  // tell AS3935 we are indoors, for outdoors use setOutdoors() function
  AS3935.setIndoors();
  // AS3935.setOutdoors();
  // turn on indication of distrubers, once you have AS3935 all tuned, you can turn those off with disableDisturbers()
  AS3935.enableDisturbers();
  // AS3935.disableDisturbers();
  printAS3935Registers();
  AS3935IrqTriggered = 0; 
  // Using interrupts means you do not have to check for pin being set continiously, chip does that for you and
  // notifies your code
  // demo is written and tested on ChipKit MAX32, irq pin is connected to max32 pin 2, that corresponds to interrupt 1
  // look up what pins can be used as interrupts on your specific board and how pins map to int numbers

  // ChipKit Max32 - irq connected to pin 2
  // attachInterrupt(1,AS3935Irq,RISING);
  // uncomment line below and comment out line above for Arduino Mega 2560, irq still connected to pin 2
  attachInterrupt(digitalPinToInterrupt(IRQpin),AS3935Irq,RISING); // ESP-12E
}

void loop()
{
  // here we go into loop checking if interrupt has been triggered, which kind of defeats
  // the whole purpose of interrupts, but in real life you could put your chip to sleep
  // and lower power consumption or do other nifty things
  if(AS3935IrqTriggered)  
  {
    // reset the flag
    AS3935IrqTriggered = 0;
    // wait 2 ms before reading register (according to datasheet?)
    delay(2);
    // first step is to find out what caused interrupt
    // as soon as we read interrupt cause register, irq pin goes low
    int irqSource = AS3935.interruptSource();
    // returned value is bitmap field, bit 0 - noise level too high, bit 2 - disturber detected, and finally bit 3 - lightning!
    if (irqSource & 0b0001)
      Serial.println("Noise level too high, try adjusting noise floor");
    if (irqSource & 0b0100)
      Serial.println("Disturber detected");
    if (irqSource & 0b1000)
    {
      // need to find how far that lightning stroke, function returns approximate distance in kilometers,
      // where value 1 represents storm in detector's near victinity, and 63 - very distant, out of range stroke
      // everything in between is just distance in kilometers
      int strokeDistance = AS3935.lightningDistanceKm();
      if (strokeDistance == 1)
        Serial.println("Storm overhead, watch out!");
      if (strokeDistance == 63)
        Serial.println("Out of range lightning detected.");
      if (strokeDistance < 63 && strokeDistance > 1)
      {
        Serial.print("Lightning detected ");
        Serial.print(strokeDistance,DEC);
        Serial.println(" kilometers away.");
      }
    }
  }
}

void printAS3935Registers()
{
  int noiseFloor = AS3935.getNoiseFloor();
  int spikeRejection = AS3935.getSpikeRejection();
  int watchdogThreshold = AS3935.getWatchdogThreshold();
  int minLightning = AS3935.getMinimumLightnings();
  Serial.print("Noise floor is: ");
  Serial.println(noiseFloor,DEC);
  Serial.print("Spike rejection is: ");
  Serial.println(spikeRejection,DEC);
  Serial.print("Watchdog threshold is: ");
  Serial.println(watchdogThreshold,DEC); 
  Serial.print("Minimum Lightning is: ");
  Serial.println(minLightning,DEC);   
}

// this is implementation of SPI transfer that gets passed to AS3935
// you can (hopefully) wrap any SPI implementation in this
byte SPItransfer(byte sendByte)
{
  return SPI.transfer(sendByte);
}

// this is irq handler for AS3935 interrupts, has to return void and take no arguments
// always make code in interrupt handlers fast and short
void ICACHE_RAM_ATTR AS3935Irq()
{
  AS3935IrqTriggered = 1;
}


void recalibrate() {
  delay(50);
  Serial.println();
  int calCap = AS3935.getBestTune();
  Serial.print("antenna calibration picks value:\t ");
  Serial.println(calCap);
  delay(50);
}

void outputCalibrationValues() {
   // output the frequencies that the different capacitor values set:
  delay(50);
  Serial.println();
  for (byte i = 0; i <= 0x0F; i++) {
    int frequency = AS3935.tuneAntenna(i);
    Serial.print("tune antenna to capacitor ");
    Serial.print(i);
    Serial.print("\t gives frequency: ");
    Serial.print(frequency);
    Serial.print(" = ");
    long fullFreq = (long) frequency*160;  // multiply with clock-divider, and 10 (because measurement is for 100ms)
    Serial.print(fullFreq,DEC);
    Serial.println(" Hz");
    delay(10);
  }
}

Wednesday 2 October 2019

mDNS: Bonjour Slackware

Bonjour Sourire by Henri Salvador
"Adieu tristesse, bonjour sourire" - Henri Salvador

Goodbye sadness, hello smile. Somehow it seems apt.

For 20 years now I have used static IP rather than a Name Server for my home network. When I started my home network, computers were expensive and few. But then came the Raspberry Pi and the Espressif ESP8266.

In practice I maintained a 'name server' by hand, updating my /etc/resolv.conf every time I added a new computer to the network. There are now over 100 entries in 3 sub-nets but you do get used to it. What put me over the edge was the IoT device.

The need for periodic security updates meant reprogramming: the device had to be taken down and the ESP8266 plugged into its programmer. The most useful ones always seem to be the most inaccessible, like upside down on the ceiling or out in the elements like the autogate controller.

Then along came Over The Air programming, just like you would update your App in your smartphone. But now every IoT device came with its unique static IP address, while I might have 10 IoT lights installed, there is only 1 version source code and there was a high chance I would brick a device by using the wrong address.

But adieu tristesse, there is mDNS or Multicast DNS to the rescue. mDNS was first implemented as Apple Inc's Bonjour. Actually I unknowingly used it in ArduinoOTA and Raspbian. It is just my main workstations ran Slackware, which did not come with mDNS pre-installed. It would be nice to update IoT devices directly from my development machine.

To install mDNS onto my Slackware 14.2-current (which does not automatically resolve dependencies) you will need to install in this order: libdaemon, avahi and  nss-mdns.

But first you need to create the user 'avahi':

$groupadd -g 214 avahi
$useradd -u 214 -g 214 -c "Avahi User" -d /dev/null -s /bin/false avahi

libdaemon:


$tar -xvpzf libdaemon.tar.gz
$cd libdaemon

Download the source code libdaemon-0.14.tar.gz into libdaemon directory and:

$./libdaemon.SlackBuild

which produced /tmp/libdaemon-0.14-x86_64-1_SBo.tgz and can be installed into Slackware thus:

$upgradepkg --install-new /tmp/libdaemon-0.14-x86_64-1_SBo.tgz

avahi:


$tar -xvpzf avahi.tar.gz
$cd avahi

Download your source code

$ ./avahi.SlackBuild
$upgradepkg --install-new /tmp/avahi-0.7-x86_64-1_SBo.tgz

In my case I ring-fenced my IoT devices in their own WiFi Access Point. Many consumer WiFi routers (like my TP-Link TL-MR3420) randomly drops connections to the WiFi clients when there are too many of them, like around 30 devices.

This means my Slackware server (ie Google smart home nodejs, MQTT, and webhook servers) can only access the IoT devices from behind their router. In this case I had to set just one of the mDNS computers in the IoT subnet to reflector. That is in /etc/avahi/avahi-daemon.conf add:

[reflector]
enable-reflector=yes
  
$/etc/rc.d/rc.avahidaemon start
Starting Avahi mDNS/DNS-SD Daemon:  /usr/sbin/avahi-daemon -D
$/etc/rc.d/rc.avahidnsconfd start

Add the following lines to your /etc/rc.d/rc.local:

# Start avahidaemon
if [ -x /etc/rc.d/rc.avahidaemon ]; then
  /etc/rc.d/rc.avahidaemon start
fi
# Start avahidnsconfd
if [ -x /etc/rc.d/rc.avahidnsconfd ]; then
  /etc/rc.d/rc.avahidnsconfd start
fi

nss-mdns:


$tar -xvzf nss-mdns.tar.gz
$cd nss-mdns

Download your source code

$./nss-mdns.SlackBuild
$upgradepkg --install-new /tmp/nss-mdns-0.10-x86_64-2_SBo.tgz

Now your  /etc/nsswitch.conf will have a line line this:

hosts:          files dns

Which you need to change to 

hosts:          files mdns4_minimal [NOTFOUND=return] dns

After which a command like getent will work:

$getent hosts MyComputer.local
123.45.67.89   MyComputer.local

And you should be able to remotely access it by name without a DNS server:

$ssh -t MyComputer.local

Even better, it works with the ArduinoOTA code for the ESP8266. This sets me up nicely for Over The Air software upgrades for my home-brew IoT devices. 

Pour éclairer un ciel trop gris,
Pour cueillir un bout de printemps
Il suffit dans la vie
Il suffit bien souvent
De dire adieu tristesse, bonjour sourire.

Happy Trails.

To illuminate a sky too gray,
To pick a piece of spring
It's enough in life
It is often enough
To say goodbye sadness, hello smile.