Showing posts with label Raspbian. Show all posts
Showing posts with label Raspbian. Show all posts

Thursday, 2 July 2020

Raspberry Pi 4 Voice Assistant Part 2 of 4: Google Text to Speech




gTTS: The Empire Strikes Back


In Part 1, one of my goals was to have my laptop issue voice commands to to my Google Home smart speaker. After trying out Mycroft, Jasper seems like the logical next step. The other text to speech systems voice quality were something like the Texas Instruments Speak & Spell products: especially ESP32Talkie. Even Mycroft sounded a bit sad next to my Home Mini speaker.

And then I stumbled upon gTTS, Google Text to Speech.  This python interface to Google's text to speech can be installed using:

pip install gTTS

And requires a program, of only ten lines:

from gtts import gTTS

def text_to_speech(input_name, output_name, language):
    file = open(input_name, 'r')
    content = file.read()
    file.close()
    sound = gTTS(text=content, lang=language)
    sound.save(output_name + '.mp3')
    #https://pypi.org/project/gTTS/
text_to_speech('input_en.txt', 'sound_en', 'en')
#text_to_speech('input_tr.txt', 'sound_tr', 'tr')
print('Done!')

You put your text in a file, input_en.txt:
$cat ./input_en.txt
Hey, Google

$python tts.py

And you get back an mp3 file, sound_en.mp3

And all you need to do now to trigger the Google Home smart speaker is:

$mplayer sound_en.mp3

A typical complete command would be something like:
$mplayer HeyGoogle.mp3; sleep 1; mplayer OfficeLampOn.mp3

For some reason, the other trigger phrase 'OK Google' did not work, but for very little effort I can now integrate disparate IoT devices, be they home brewed, Alexa, or Google Home into one Voice Assistant that rules them all.

Here's what it sounds like:



Happy Trails.

Luke: Vader... Is the dark side stronger?

Yoda: No, no, no. Quicker, easier, more seductive.

Thursday, 14 May 2020

Raspberry Pi 4 Voice Assistant: Mycroft Part 1 of 3


(Shown a photo of a baby)
Mycroft: "Yes, looks very ... fully functioning."

Sherlock: "Is that the best you can do?"
Mycroft: "Sorry, I've never been very good with them."
Sherlock: "Babies?"
Mycroft: "Humans."

My Seeed Studio Respeaker 4-Mic Array arrived during the 2020 Covid-19 Lockdown, which pretty much guaranteed it some immediate attention.




The Seeed Studio link has some good instructions, and it was smooth sailing until the section "Alexa/Baidu/Snowboy SDK". I have Google Home Mini smart speakers and so was loath to register for Alexa. Baidu seemed like a good alternative, for in 2016 Baidu published a stunning paper on Deep Speech on using deep learning on speech to text.

Getting Baidu authorization keys, however proved way too slow so I took a quick look at Mozilla's implementation of Deep Speech, using Google's Tensorflow.

The purpose of all this (besides having some fun) is to see if I can voice-control my IoT devices without an Internet link. Also the added security and privacy seems worthwhile. And it is not like I'm going anywhere for a few days.

At this point Mycroft looks tempting, and since the instructions are straightforward, I downloaded the image file. There is a typo in the image write to sdcard; just replace /dev/sdb1 with /dev/sdb:

sudo dd if=path-to-your-image.img of=/dev/sdb bs=20M

Per the instructions, you will have to register with their website so keep a note of your registration
code on the screen. Keep following until the section "Selecting audio output and audio input".

Respeaker is not listed in the microphones' list, But Dimitry Maslov comes to the rescue:

sudo apt-get update
sudo apt-get upgrade
git clone https://github.com/respeaker/seeed-voicecard.git
cd /home/pi/seeed-voicecard
./install.sh 4mic

Next, go back to Mycroft: a quick and clean way is to reboot. Reconfigure it again with

mycroft-setup-wizard

And select 'Other'. Mycroft should now work. Here's a video of mine


Mycroft/Picroft on Raspberry Pi 4 and Respeaker 4-mic Array


Mycroft seems to run a lot slower than Google Assistant. This is because it also uploads the audio to cloud servers and Mycroft servers probably have a lot less oomph.

Next we want Mycroft to turn on an IoT lamp. We can used a few services for this, for example, Adafruit but for simplicity we can use an esp8266 1-channel relay and a webhook. We use 'mycroft-msk create' and fill in the questionnaire:

(.venv) pi@picroft:~ $ mycroft-msk create
Enter a short unique skill name (ie. "siren alarm" or "pizza orderer"): soldering station lamp

Class name: SolderingStationLampSkill
Repo name: soldering-station-lamp-skill

Looks good? (Y/n) n
Enter a short unique skill name (ie. "siren alarm" or "pizza orderer"): soldering station lamp on

Class name: SolderingStationLampOnSkill
Repo name: soldering-station-lamp-on-skill

Looks good? (Y/n) y
Enter some example phrases to trigger your skill:
- Soldering station lamp on
- Soldering station light on
- Turn on the soldering station lamp
- Turn on the soldering station light
-
Enter what your skill should say to respond:
- The soldering station light is now on
- Turning on the soldering station light
-
Enter a one line description for your skill (ie. Orders fresh pizzas from the store):
- Turns on the light on the soldering station
Enter a long description:
> Turns on the light on the soldering station
>
Enter author: cmheong
Go to Font Awesome (fontawesome.com/cheatsheet) and choose an icon.
Enter the name of the icon: lightbulb
Pick a color for your icon. Find a color that matches the color scheme at mycroft.ai/colors, or pick a color at: color-hex.com.
Enter the color hex code (including the #): #fff68f

Categories define where the skill will display in the Marketplace. It must be one of the following:
Daily, Configuration, Entertainment, Information, IoT, Music & Audio, Media, Productivity, Transport.
Enter the primary category for your skill:
- IoT
Enter additional categories (optional):
-
Enter tags to make it easier to search for your skill (optional):
- IoT
- Smart Home
- Home Assistant
-
For uploading a skill a license is required.
Choose one of the licenses listed below or add one later.

1: Apache v2.0
2: GPL v3.0
3: MIT
Choose license above or press Enter to skip? 3

Some of these require that you insert the project name and/or author's name. Please check the license file and add the appropriate information.

Does this Skill depend on Python Packages (PyPI), System Packages (apt-get/others), or other skills?
This will create a manifest.yml file for you to define the dependencies for your
 Skill.
Check the Mycroft documentation at mycroft.ai/to/skill-dependencies to learn more about including dependencies, and the manifest.yml file, in Skills. (y/N) y
Would you like to create a GitHub repo for it? (Y/n) y

Enumerating objects: 12, done.
Counting objects: 100% (12/12), done.
Delta compression using up to 4 threads
Compressing objects: 100% (10/10), done.
Writing objects: 100% (12/12), 2.52 KiB | 322.00 KiB/s, done.
Total 12 (delta 0), reused 0 (delta 0)
To https://github.com/cmheong/soldering-station-lamp-on-skill
 * [new branch]      master -> master
Branch 'master' set up to track remote branch 'master' from 'origin'.
Created GitHub repo: https://github.com/cmheong/soldering-station-lamp-on-skill
Created skill at: /opt/mycroft/skills/soldering-station-lamp-on-skill

And that is all there is to it. Don't worry about uploading to github - it is optional. You will now get a whole bunch of smallish files:

(.venv) pi@picroft:~ $ ls -l /opt/mycroft/skills/soldering-station-lamp-on-skill
/
total 32
-rw-r--r-- 1 pi pi  393 May 14 15:15 __init__.py
-rw-r--r-- 1 pi pi 1058 May 14 15:20 LICENSE.md
drwxr-xr-x 3 pi pi 4096 May 14 15:13 locale
-rw-r--r-- 1 pi pi 1009 May 14 15:20 manifest.yml
drwxr-xr-x 2 pi pi 4096 May 14 15:15 __pycache__
-rw-r--r-- 1 pi pi  531 May 14 15:17 README.md
-rw-r--r-- 1 pi pi   35 May 14 15:17 settings.json
-rw-r--r-- 1 pi pi  631 May 14 15:20 settingsmeta.yaml

The file we are interested in is __init__.py. Since this is a toy example to get you going, we are going to use the crudest possible and most insecure method, using a bash shell to launch our webhook. The modified file is in my github repository, but it is so small I'll also list it here:.

$cat __init__.py
from mycroft import MycroftSkill, intent_file_handler
import subprocess

class SolderingStationLampOn(MycroftSkill):
    def __init__(self):
        MycroftSkill.__init__(self)

    @intent_file_handler('off.lamp.station.soldering.intent')
    def handle_off_lamp_station_soldering(self, message):
        cmd = "curl -k http://ww.xx.yy.zz:8080/1/on"
        answer = ""
        try:
            answer = subprocess.check_output(cmd, shell=True)
        except:
            print(str(answer))
        print(str(answer))

        self.speak_dialog('off.lamp.station.soldering')


def create_skill():

    return SolderingStationLampOn()

Here's a video of the result. You will notice there is another skill to turn off the lamp.

Mycroft with IoT Skil

But what I really wanted was Mycroft to function offline. There is some talk of a "Personal Server" version, but as this forum shows, there is a lot of code by the redoubtable JarbasAI, but is not quite ready yet. 

So, it is back to my Respeaker and DeepSpeech image: we will look at Jasper in Part 2.

Happy Trails

Friday, 8 May 2020

Hacking the Raspberry Pi Model B to use with Geekworm UPS Hat


Raspberry Pi Model B (not Plus) with Composite Video socket dismounted
I recently bought a Geekworm UPS Hat for the Raspberry Pi Models B Plus and later. It has, shall we say a few rough edges. Especially running loads of 500mA and above. It often functions as a UPS at loads of say 480mA. If I paired it with a say, Raspberry Pi Model B, it works quite well.

Now I have always wanted a solar-powered wifi repeater. In the day, a solar panel provides DC power and also charges up an NS60 (nominal 60Ah at 12v) car battery. At night it runs off the battery. It seems reasonable enough: the Raspberry Pi 3 Model B Plus wifi repeater took up more than 500mA at 5V 24 hours a day, whilst the solar panel might supply 2A at 19V for maybe 6 hours a day.

But before I could buy the NS60, the Covid-19 pandemic of 2019 intervened. Rather than wait for lockdown to pass, why not reconfigure it to run from solar power in the day, and seamlessly switch over to mains power at night. I would need a couple of relays: one each for mains and solar power 5V DC-DC buck converters. And to ensure a trouble-free switchover, a UPS Hat for the Raspberry Pi would be nice ...

The Geekworm UPS Hat worked well at 500mA, and misbehaved over 600mA: the WiFi Repeater would reset on switchover. Or it would not charge the lithium battery on switching back to mains power. Now this is actually self-recovering: on the battery running down it would reset the load and the battery would charge again. There might be a minute of WiFi repeater service interruption. TM Net my service provider certainly does that a few times a day. But this is humiliating; not tolerable for anyone other than TM Net.

My Raspberry Pi Model B Plus drew 510mA clean and 600mA once the WiFi dongle started firing up. On the other hand a Raspberry Pi Model B drew only  440mA and might just work. The trouble is, the Geekworm UPS Hat has a 40-pin socket and the Model B only has a 26-pin header.

Raspberry Pi Model B's 26-pin Header

Many hardware designers seek backward-compatibility when upgrading their designs. Often old hats will work on new models, but new hats will not. But this means the headers will have a lot of similarity. Sometimes enough to work. A quick comparison shows that the first 26 pins of the 50-pin header is pretty much identical, except for GPIO 19-21. The I2C pins are the same, and crucially, so are the power and ground pins.

Raspberry Pi Model B Plus' 40-pin Header
And as long as the 3 contentious GPIO pins are not used, they will default to GPIO input, and inputs when mis-wired and unused are harmless.

But there is another problem: the Model B's composite video output, an RCA socket is too tall and gets in the way of the Geekworm 40-pin socket. This is easily de-soldered.

Remember, an electronic engineer's favorite programming language is solder!
Once the RCA socket is removed, the Geekworm UPS Hat mounts nicely onto a Model B.

Geekworm UPS Hat on Raspberry Pi Model B
The Model B powers on nicely from battery. But the proof of the pudding is in the eating. Besides the Geekworm UPS I2C device at address 0x48 I also had an ADS1115 4-channel analog input card at address 0x36.

# i2cdetect -y 1
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f
00:          -- -- -- -- -- -- -- -- -- -- -- -- -- 
10: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
20: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
30: -- -- -- -- -- -- 36 -- -- -- -- -- -- -- -- -- 
40: -- -- -- -- -- -- -- -- 48 -- -- -- -- -- -- -- 
50: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
60: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
70: -- -- -- -- -- -- -- --             

And from my last post

# ./ups_read
97

# ./ups_read -vc
4.161250V 96.550781%


Blinkenlights galore: Raspberry Pi Model B booting from battery power

To probe it a little further, I printed out a few more registers of the MAX17048:
# ./ups_read2 -a
MODE 10 00
CFG 97 1c
CRATE ff f9
VER 00 12
STAT 00 ff
VRST 00 0c
VCELL cf f0
CREG 60 65
4.158750V 96.394531%

The code is in my github repository.

Note that the 5V input into the Geekworm UPS Hat greatly affects its operation. In the picture of the WiFi extender below, a smaller 5V DC-DC buck converter (red PCB) was used, and this resulted in the Geekworm Hat not charging. Swapping it out for a 5V 3A unit (blue PCB) did the trick. Note that when the battery is at full charge, it takes a little while, maybe a few minutes, for the Hat to start charging.

Raspberry Pi Model B with Geekworm UPS Hat, installed as daytime solar powered WiFi extender . From top: ADS1115 I2C analog converter, WiFi dongle, Pi Model B with Geekworm UPS Hat, 2A buck converter(disconnected), and 3A CC CV buck converter

And yes, the Raspberry Pi daytime solar WiFi Repeater works for now. It's early days yet and there are many switchovers and switchbacks yet to come.

And there you have it: how to hack a 26-pin Raspberry Pi Model B for the Geekworm UPS Hat.

Happy Trails



Thursday, 30 April 2020

Geekworm Raspberry Pi UPS Hat

I did not really have much faith in the Geekworm Raspberry Pi UPS Hat. It cost just RM48 and change and it came with a 3.7V 2700mAh lithium battery. The software links did not work (the correct link is here) and the driver looked out of date.

Plus there has been some rather harsh comments about it. Some like this one, seems reasonable. Now I am not saying they are wrong, but that Geekworm Raspberry Pi UPS Hat worked for me, much to my surprise.

It is quite well described here and I will not repeat the information. I originally had the idea of using a store-bought power bank for the very same purpose. I thought if I ran the power bank with the charger always connected and the raspberry Pi always drawing power, it should work like a cheap UPS. Not bad for an RM30 no-name 3000mAh power bank.
Power bank as UPS. Note the Qi receiver coil
I found one while walking my dogs. It's cover was cracked open and it looked like it had been thrown out of a car, but when it was dried out it worked. The battery looked intact, did not overheat and was not bulging.

But sadly it did not work consistently. When the mains charger was powered off it ran from battery well enough. The problem was when the charger was switched on, it sometimes rebooted. Looks like the output power was not quite stable enough during the switchover. A common enough problem with regular UPS.

Now I can still use it as a UPS for systems that can tolerate a reboot. In some cases I simply put a little restart code in /etc/rc.local. And I can't really complain about the price. Do be careful of discarded lithium batteries though. They have been known to explode, or burn white-hot.

But a real UPS would switch over and back without a glitch. And it would be nice to have an indication of the battery state. The Geekworm UPS actually looked like (I am not sure; I did not check) it was adapted from a power bank circuit  and it would be good so see where I fell short.

Having heard some comments about bad batteries, miswired batteries and faulty UPS boards, I checked both the battery output and the polarity before I hooked it up. It all checked out and powered on a Raspberry Pi 3 Model B+ quite nicely while still on charge.

Geekworm UPS Hat installed
It ran nicely on battery, and did not reset when the charger is reconnected. Even when running X Windows, Chrome and a youtube video at 1900x1080 resolution.

The extremely brief 'manual' (http://raspberrypiwiki.com/File:UserManual.pdf) called for installation of a driver:

User Guide:
1. Upgrade software:
sudo apt-get update
sudo apt-get upgrade
2. Enable the I2C function via raspi-config tool.
3. Install wiringPi .
git clone git://git.drogon.net/wiringPi
cd wiringPi
git pull origin
cd wiringPi
./build
4. Download the zip package;?rpi-ups-hat.zip?
unzip rpi-ups-hat.zip
cd rpi-ups-hat
5. Run the tested program
sudo python example.py

But there is also a C program, which I reproduce in full here:
$cat main.c

#include <unistd.h>                     // close read write
#include <stdio.h>                      // printf
#include <fcntl.h>                      // open
#include <linux/i2c-dev.h>
#include <sys/ioctl.h>
#include <getopt.h>


#define VREG 2
#define CREG 4
#define BUFSIZE 16
#define DEV "/dev/i2c-1"
#define ADRS 0x36


static int readReg(int busfd, __uint16_t reg, unsigned char *buf, int bufsize)
{
    unsigned char reg_buf[2];

    reg_buf[0] = (reg >> 0) & 0xFF;
    reg_buf[1] = (reg >> 8) & 0xFF;

    int ret = write(busfd, reg_buf, 2);

    if (ret < 0) {
        printf("Write failed trying to read reg: %04x (0x%02x 0x%02x)\n", reg, r
eg_buf[0], reg_buf[1], reg);
        return ret;
    }

    return read(busfd, buf, bufsize);
}

int main(int argc, char **argv)
{
    int vOpt = 0, cOpt = 0, o;

    while ((o = getopt (argc, argv, "vc")) != -1) {
        switch (o)
        {
        case 'v':
            vOpt = 1;
            break;
        case 'c':
            cOpt = 1;
            break;


        }
    }

    int bus = 1;
    unsigned char buf[BUFSIZE] = {0};

    int busfd;
    if ((busfd = open(DEV, O_RDWR)) < 0) {
        printf("can't open %s (running as root?)\n",DEV);
        return(-1);
    }

    int ret = ioctl(busfd, I2C_SLAVE, ADRS);
    if (ret < 0)
        printf("i2c device initialisation failed\n");

    if (ret < 0) return(-1);

    readReg(busfd, VREG, buf, 2);

    int hi,lo;
    hi = buf[0];
    lo = buf[1];
    int v = (hi << 8)+lo;
    if (vOpt) {
                printf("%fV ",(((float)v)* 78.125 / 1000000.0));
        }

    readReg(busfd, CREG, buf, 2);
    hi = buf[0];
    lo = buf[1];
    v = (hi << 8)+lo;
    if (!cOpt && !vOpt) {
                printf("%i",(int)(((float)v) / 256.0));
        }
        if (cOpt) {
                printf("%f%%",(((float)v) / 256.0));
        }

        printf("\n");

    close(busfd);
    return 0;

}

I started with the usual:

sudo apt-get update
sudo apt-get upgrade

And used raspi-config to turn on i2c. As soon as I did that an i2c device came up:

# ls -l /dev/i2*
crw-rw---- 1 root i2c 89, 1 Apr 30 20:28 /dev/i2c-1

A quick scan of the C program main.c showed that that was all it needed to run. So, I skipped all the other steps and went straight to:

# gcc main.c -o ups_read

And it runs:
# ./ups_read
97

When I disconnected the charger and ran on battery it actually reads a little higher:
# ./ups_read
98

But it quickly starts to show a correct, discharging trend:
# ./ups_read
97

I ran it for some 6 minutes at pretty much full power and it went to 83%:
# date;./ups_read
Thu Apr 30 20:46:49 +08 2020
83

When I plugged the charger in I get the wobble:
# date;./ups_read
Thu Apr 30 20:47:27 +08 2020
82
            And yes, it charges:
# date;./ups_read
Thu Apr 30 20:47:49 +08 2020
83

# ./ups_read -vc
4.161250V 96.550781%

And that was all it took. Raspberry Pi UPS Hat on the cheap.

Happy Trails.


[Update 2020-05-06]
Managed to reproduce some of the problems mentioned above by increasing the current drawn from the UPS. Decreasing the UPS input power did not affect it much, except it increased the charging time. But, if I increased the UPS load too much (by loading up the Pi USB ports with for example a WiFi dongle at full power), a switchover from mains to battery supply now caused the Pi 1 to reset. And on reboot, the Geekworm UPS often failed to charge unless the load is power-cycled.

600mA peak current draw from the Geekworm UPS Hat was enough to trigger the problems mentioned. A 510mA peak draw was more or less OK for trouble-free operation. Treat the numbers as a guide: I only used a USB Charge Doctor to measure the peak current consumption. This almost guarantees an incorrect reading. An oscilloscope would be a better choice.  

The 4 LED do not accurately reflect the battery state of charge. The I2C value read by ups_read is much more accurate.  

Thursday, 2 April 2020

Microchip ENC28J60 SPI Ethernet controller

From bottom: ENC28J60 SPI Ethernet Controller, Raspberry Pi 3 B and 5V 6A power module
The Microchip ENC28J60 Ethernet Controller has an SPI interface which makes it possible to retrofit LAN functionality to microcontroller systems, especially legacy ones. Now the ESP8266 is a much more obvious choice, but sometimes it is handy to use a wired, or copper LAN.

As is often the case it turned out it is easier to test the ENC28J60 using a Raspberry Pi. Usually because the software is easily available, but in this case because TheSpotShed has a great writeup on it. My board is a little different from his, but even so it worked first time, so for more details hie you hence to TheSpotShed.

Now a late-model Pi is far from a tiddly microcontroller, and we often forget that the Network Stack takes up more than one third of the 15-million plus lines of Linux kernel source code. It is a measure of how far we have come to even consider implementing networking in an embedded microcontroller system. Besides being a slam-dunk, implementing it for a Pi lets you gauge where the bottlenecks are: the SPI interface, the LAN controller or the microcontroller.

My ENC28J50 board had different pinout from the one in TheSpotShed, and it runs on 5V instead of 3.3V. There are a few spelling errors, e.g. LNT instead of INT, SL instead SI

I bought my ENC28J60 from lelong.com.my's enewground before it was removed from sale
Plus the pinouts were different. My cable is thus:

Pi                      ENC28J60     Colour
------------------------------------------------
+3V3                  VCC          Brown            <--- Note my PCB is *5V*
GPIO10/MOSI    SI              Grey
GPIO9/MISO     SO             Green
GPIO11/SCLK   SCK          Purple
GND                  GND          Red

GPIO25              INT           Orange
CE0#/GPIO8      CS            Black

Working from 5V also meant that my 10-way Molex header (SL Modular Connector, 70066 Series IDC/IDT 2.54mm) no longer sufficed. I had to run a couple of easyhooks to the top of the Raspberry Pi header for my 5V. Other than that, everything worked on the first try, so kudos to TheSpotShed.

First I checked for the enc28j60.dtbo overlay:

# mount /dev/mmcblk0p1 /mnt/flash
# ls -l /mnt/flash/overlays/enc*
-rwxr-xr-x 1 root root 1403 Nov 19  2018 /mnt/flash/overlays/enc28j60.dtbo
-rwxr-xr-x 1 root root 1279 Nov 19  2018 /mnt/flash/overlays/enc28j60-spi2.dtbo

It is not only there but very promisingly there looks to be provision for a second spi interface, spi2.

A quick edit of /boot/config.txt to add:
dtparam=spi=on
dtoverlay=enc28j60

And on reboot, it came up immediately as eth1:

# ifconfig -a
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet xx.xx.xx.xx  netmask 255.255.255.0  broadcast xx.xx.xx.255
        ether b8:27:eb:a4:ab:0b  txqueuelen 1000  (Ethernet)
        RX packets 85  bytes 9631 (9.4 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 68  bytes 8877 (8.6 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

eth1: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        ether 4e:cd:f6:c2:4e:3a  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        device interrupt 167

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
wlan0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet zz.zz.zz.zz  netmask 255.255.0.0  broadcast zz.zz.255.255
        ether b8:27:eb:f1:fe:5e  txqueuelen 1000  (Ethernet)
        RX packets 22  bytes 2202 (2.1 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 26  bytes 3454 (3.3 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

On connecting it to my modem router the link came up immediately. I tested it with Youtube with Firefox, and it pretty much ran a full HD (1080p) music video, with the odd hiccup or two. But overall very impressive throughput, compared to my previous SPI link to my SIM7000C 3G modem.

Sistar's 'Give it to Me' in fullscreen mode is a brutal workout for anything less than a Linux workstation 
The next thing to do would be to hook the ENC28J60 to an ESP8266, but that is another blog post, so watch this space.

Keep safe, and Happy Trails.

Tuesday, 21 January 2020

3G Failover using the Huawei B310

Huawei B310 3G router
I first implemented a 3G failover using the TP-Link MR3420, but that had a few problems. It got a little erratic with less than 20 IoT devices connected. When in backup 3G mode the uplink often failed when run for too long and the 3G USB modem then needed to be power cycled. In a thunderstorm 3G reception often got a lot worse. But worst of all I could not alter the failover criteria.

This is because my benighted ADSL service provider Talikom Malaysia plumbed news depths in quality of service. The ADSL connection would get erratic, going on and off within a few minutes. Sometimes there would be an actual ADSL connection but it would not be able to reach the TM gateway and from there the Internet. Sometimes the link simply got slow.

This would sometimes cause the MR3420 to remain in ADSL mode and not fail over to 3G. Worse this sometimes caused it to randomly drop connections to some client WiFi devices. Power-cycling it usually helps, but the random dropouts are nevertheless annoying.

Huawei B310 with external antennae


The Huawei B310 has a simple failover criteria: it needed its WAN input cable disconnected before it goes into 3G mode. I could do that with an electromechanical relay; this allowed me to make a custom failover program to handle TM's myriad failure modes. Also the B310 has two connectors for 3G external antennae which are separately bought. And it does not disconnect clients in either 3G or ADSL mode.

The setup is much the same as before:


The Huawei B310's WAN uplink is taken from the Archer D50 via the DES-100S LAN switch. This allows me to disconnect the WAN link by turning off the DC power to the LAN switch. I do this using a Raspberry Pi Model B and a Piface hat.

Because of frequent electrical storms, the ADSL line can also be disconnected by an AS3935 lightning detector relay module.

The Raspberry Pi runs Raspbian. The Piface is controlled by the pifacedigitalio library and the failover program for now is a simple bash script. Here is a first-pass version of the failover program, it did not take more than an afternoon to code and test.

Because of the AS3935 lightning detector, the failover program subscribes to and publishes to a local MQTT server. The piface digitalio library is not one of you best supported ones, but there are a few hints if you run into trouble.

As always the choice of components is not necessarily optimal, if not downright obsolescent. Most of the time they are used simply because they are available on hand. Indeed you can probably get an all-singing all-dancing 3G failover modem router for a very reasonable price somewhere. But it goes against grain not to use working parts simply because they are a little too old.

Much better to go out with your boots on. Happy Trails.

Monday, 16 December 2019

IEEE 802.11w: Securing your WiFi from Deauth Attack Part 2: Electric Kool-Aid Acid Test


"You are hereby empowered!!!" - Tom Wolfe, The Electric Kool-Aid Acid Test

In Part 1, we tested for cheap devices compatible with IEEE 802.11w-2009. Here, we will just detail the configuration files needed to build a Ieee 802.11w-209 compatible Raspberry Pi 3 B+ WiFi Bridge. The actual steps are detailed in the official Raspberry Pi site.  We will need:
1. Raspberry Pi 3 B+
2. Very good 5V 2.5A USB power module (e.g. the Raspberry Pi 2.5A module). This requirement is important! I used a stonking 5V 6A TDK RDM05-6R0
3. Realtek RTL8812BU generic WiFi dongle. Or a dongle with any of the Atheros chips tested in Part 1.
4. ADSL modem router with wired (ie copper) LAN interface.


For simplicity, the above diagram omits the power supplies and the powered hub. The RTL8812BU can draw a lot of power, especially if connected to an outdoor antenna.

Note it is also possible to set up the above devices as WiFi repeater by using the Raspberry Pi's built-in WiFi chip (wlan0) as WAN.

Top middle: TDK 5V 6A power module. Center: Raspberry Pi 3 B+ with RTL8812BU. Bottom left: thick USB power cable with ammeter showing 920mA current draw

First, set up your ADSL modem and make sure you have Internet WAN access via the copper LAN (ie wired Ethernet). I used a TP-Link Archer D20.

Next, set up your Raspberry Pi 3 B+ with the latest and greatest version of Raspbian. The Linux version of the installation guide worked for me, but there is also a Windows version. It is really worth using a fast sdcard (16GB is sufficient and 32GB is plenty), Class 10 or better if you can manage it.  You will need to set it up to log into your Internet connection. My Pi connected to the Internet when I plugged in the LAN cable. After it has finished installation you need to update it immediately:

# apt-get update
# apt-get upgrade

This can take hours depending on your Internet connection and you will have to reboot your Pi. Next, set up your root password:
# sudo vi /etc/passwd

Remove the 'x' from the line
root:x:0:0::/root:/bin/bash

Next set the root password using:
#sudo passwd root

I usually use the wifi bridge in 'headless' (ie no monitor or keyboard) so I usually turn off the GUI using
#sudo raspi-config

To control it, I usually enable the ssh server (again using raspi-config). Now to run it headless I make sure my laptop is connected to the same network and if it is also running Debian (Rasbian is a version of Debian) I simply do:
$ssh -t pi@raspi.local

It is also handy to have your first setup connected to copper LAN as well as keyboard and monitor, as we will be messing about with networking tools and a mistake is likely to freeze up your remote login.

If wlan1 the rtl8812bu does not come up, refer to Part 1.

Next you will need to stop systemd from messing with your network interfaces.
# systemctl mask wpa_supplicant.service
Created symlink /etc/systemd/system/wpa_supplicant.service � /dev/null.

In /etc/dhcpcd.conf add the lines:
interface eth0
static ip_address=192.168.1.1/24

interface wlan0
  denyinterfaces wlan0
  nohook wpa_supplicant

interface wlan1
  denyinterfaces wlan1
  nohook wpa_supplicant

Reboot, and you should be ready for the next step.

We need to use hostapd for our bridge, so we stop systemd from messing with it:
# systemctl mask hostapd

We make a bridge:
# brctl addbr br0
# brctl addif br0 wlan1

If all went well, you should get:
# brctl show br0
bridge name     bridge id               STP enabled     interfaces
br0             8000.1cbfce5d51a0       no              wlan1

Add the copper LAN to your new bridge in case you want to connect client devices by wire
# brctl addif br0 eth0
# ifconfig eth0 0.0.0.0 up
# ifconfig wlan1 0.0.0.0 up

Where 192.168.1.1 is the IP address of your Pi at eth0 and 192.168.0.1 is the address of your WAN router (ie the D-Link Archer D50 in the diagram):
# ifconfig br0 192.168.1.1 up

Next set up your dnsmasq config file with:
interface=br0
except-interface=lo
listen-address=192.168.1.1
bind-interfaces

As usual you need to tell systemd to keeps its grubby hands to itself:
# systemctl mask dnsmasq
# killall dnsmasq 
# dnsmasq -C /etc/dnsmasq.conf 

Next we get the Pi to start forwarding. Add the following lines to /etc/sysctl.conf:
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv4.ip_forward = 1
net.ipv4.conf.all.rp_filter = 1
net.ipv4.conf.default.rp_filter = 1
net.ipv4.ip_dynaddr = 1
net.ipv4.conf.all.accept_source_route = 0
net.ipv4.conf.default.accept_source_route = 0
net.ipv4.conf.all.accept_redirects = 0
net.ipv4.conf.default.accept_redirects = 0

# sysctl -p /etc/sysctl.conf

Next is hostapd. Set up the config file /etc/hostapd/hostapd.conf thus:
interface=wlan1
driver=nl80211
ssid=ElectricKoolAid
hw_mode=g
channel=1
macaddr_acl=0
wpa=2
wpa_passphrase=VerySecretPassword
rsn_pairwise=CCMP
ieee80211w=2
wmm_enabled=1
auth_algs=3
ignore_broadcast_ssid=1
wpa_key_mgmt=WPA-PSK-SHA256 WPA-EAP-SHA256
wpa_pairwise=CCMP

# killall hostapd
# hostapd -dd -P /var/run/hostapd.pid /etc/hostapd/hostapd.conf -B

Next set up /etc/firewall.conf thus:

*filter
:INPUT DROP [39:4576]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [42055:10283301]
-A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A INPUT -i br0 -j ACCEPT
-A INPUT -i eth0 -j ACCEPT
-A FORWARD -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -i br0 -j ACCEPT
-A FORWARD -i eth0 -j ACCEPT
COMMIT
# Generated by iptables-save v1.6.0 
*nat
:PREROUTING ACCEPT [78732:17589805]
:INPUT ACCEPT [29742:7228146]
:OUTPUT ACCEPT [2937:514776]
:POSTROUTING ACCEPT [985:97284]
-A POSTROUTING -s 192.168.1.0/24 -j MASQUERADE
COMMIT
# Generated by iptables-save v1.6.0 
*mangle
:PREROUTING ACCEPT [1318452:373291697]
:INPUT ACCEPT [623742:150528221]
:FORWARD ACCEPT [120385:79521689]
:OUTPUT ACCEPT [42068:10285133]
:POSTROUTING ACCEPT [168801:90872517]
-A POSTROUTING -p tcp -m tcp --tcp-flags SYN,RST SYN -m comment --comment "fix packet size for stuff that\'s being routed through this box (SEE NOTE *)" -j TCPMSS --clamp-mss-to-pmtu
COMMIT

# iptables-restore < /etc/firewall.conf

Your IEEE 802.11w-2009 bridge AP "ElectricKoolAid" with "VerySecretPassword" should be up by now.

Over at the client (in my case the Acer Aspire E1) make a wpa_supplicant config file wpa_supplicant.conf. Notice we make IEEE 802.11w compulsory:

ctrl_interface_group=0
eapol_version=1
ap_scan=1
fast_reauth=1

# WPA protected network, supply your own ESSID and WPAPSK here:
network={
  scan_ssid=1
  ssid="ElectricKoolAid"
  proto=RSN  
  key_mgmt=WPA-PSK-SHA256
  pairwise=CCMP TKIP 
  group=CCMP TKIP
  psk="VerySecretPassword"
  ieee80211w=2
  priority=10
}

And, assuming you have already wrestled with systemd (and won!) there too:
# wpa_supplicant -d -Dnl80211 -iwlan0 -c/etc/wpa_SIKAMAT7.conf -B

Note: for some reason, nl80211 worked a lot better than wext

And you should have a working link to the Pi bridge. 

# wpa_cli -iwlan0 status   
bssid=11:22:33:44:55:66
freq=2412
ssid=ElectricKoolAid
id=0
mode=station
pairwise_cipher=CCMP
group_cipher=CCMP
key_mgmt=WPA2-PSK-SHA256
pmf=2
mgmt_group_cipher=BIP
wpa_state=COMPLETED
ip_address=192.168.0.114

And there should be Internet access; provided you have set up your WAN router's dhcp server, the client at the end of the bridge "ElectricKoolAid" should get its dhclient requests routed straight through.

ping -c 4 -I wlan0 8.8.8.8
PING 8.8.8.8 (8.8.8.8) from 192.168.0.114 wlan0: 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=56 time=524 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=56 time=518 ms
64 bytes from 8.8.8.8: icmp_seq=3 ttl=56 time=380 ms
64 bytes from 8.8.8.8: icmp_seq=4 ttl=56 time=346 ms

--- 8.8.8.8 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3001ms
rtt min/avg/max/mdev = 346.240/442.040/524.243/79.921 ms

To get it to start on power up, just put the commands in /etc/rc.local

There you have it, IEEE 802.11w-2009 AP and client, immune from the dreaded Deauth Attack. In part 3 we will launch an aircrack-ng deauth attack at the the bridge AP.

Or, as Tom Wolfe said: "You are hereby empowered!!!". Happy Trails.

Thursday, 12 December 2019

IEEE 802.11w: Securing your WiFi from Deauth Attack Part 1 - The Right Stuff (Updated 2019-12-31)

The Right Stuff by Tom Wolfe

"No bucks, no Bucks Rogers" - Gus Grissom, The Right Stuff


One of the guilty pleasures of Linux is aircrack-ng,  software tools to -cough- test WiFi security. In the good old days with WEP WiFi passwords you can get a WiFi password within 2 minutes. It was so convenient it was much quicker than getting the password the legal way from your company sysadmin. Back in the day, the best hotspots are for Top Management, people who don't browse or torrent, dictate their emails and think that typing is for secretaries.


aircrack-ng on Kali Linux


IEEE 802.11 is the standard for WiFi networks. Amongst other things it specifies Management Frames, ie data packets sent by both the client and the Access Point to 'manage' or maintain communications with each other, for example, logging in, authentication and when one party wishes to terminate the connection, 'deauthenticate'.

With WPA (in particular WPA2), cracking WiFi hotspots became much much harder, but if the password is weak (most passwords are weak) you can speed things up by getting aircrack-ng to issue 'deauthenticate' Management Frames in your victim's name (ie MAC address). This causes the target to re-issue authentication frames. The more frames you sniff out, the quicker you will crack it.



Now if you do nothing but issue deauth frames, say with a bash script, you effectively jam WiFi for the luckless target, and if you target the AP MAC address you jam the whole hotspot if you are close enough. Normally you did it for a purpose, like persuading the legal users to quit using their WiFi so you can, -cough-  check-in your flight online. People who deauth used to have a higher purpose ... besides it is easy to spot the pasty-faced user on the Linux laptop with an out-sized WiFi antenna.

ESP8266 deauth jammer

But with the advent of Espressif's ESP8266  you can now buy or build a cheap portable deauth jammer that will fit inside your pocket, and now all you have to do is walk up to a hotspot to jam it. And since many security cameras these days are WiFi, an intruder can disable them by just driving up to your gate.

Now having made quite a few ESP8266-based IoT devices, it is time to secure my WiFi from deauth attacks: IEEE 802.11w-2009. With 802.11w, the Management Frame is encrypted and becomes a Protected Management Frame (PMF). This means a WiFi outsider now cannot deauthenticate you with impunity.

IEEE 802.11w stops deauth attack by encrypting the 'deauthenticate' command

Now you would think you can buy a new reasonably-priced IEEE 802.11w-2009 router for home use. After all the standard is 10 years old. And you would be wrong. It is common enough in Big Iron like Cisco or Aruba (one of dear old Hewlett Packard's random name changes) but it is not common in say, a D-Link or TP-Link. OpenWrt is a shining exception, as easy as checking an option in its base distribution. But OpenWrt often only runs on the more expensive models, and then there is the problem of compatibility with WiFi client devices.

The usual culprit would be production cost. As Gus Grissom said, "No bucks, no Buck Rogers". It turned out it is probably a compatibility issue, specifically of encryption, a missing cipher. IEEE 802.11w uses CCMP-128, SHA-256 and AES128-CMAC encryption for Management Frame protection.

To tell if your hardware has The Right Stuff, you need to execute the Linux command 'iw phy', shown here for a no-name WiFi USB dongle based on the Realtek RTL8812BU:

# iw phy
Wiphy phy8
        max # scan SSIDs: 9
        max scan IEs length: 2304 bytes
        max # sched scan SSIDs: 0
        max # match sets: 0
        max # scan plans: 1
        max scan plan interval: -1
        max scan plan iterations: 0
        Retry short limit: 7
        Retry long limit: 4
        Coverage class: 0 (up to 0m)
        Supported Ciphers:
                * WEP40 (00-0f-ac:1)
                * WEP104 (00-0f-ac:5)
                * TKIP (00-0f-ac:2)
                * CCMP-128 (00-0f-ac:4)
                * CMAC (00-0f-ac:6)

And listed under the 'Supported Ciphers' the bare minimum you need is CCMP-128 (note OUI 00-0f-ac:4) and CMAC (OUI 00-0f-ac:6). There is a helpful write-up here. Compare this to the output for the delicious Qualcomm AR9565 from my Acer Aspire E1 laptop:

$iw phy                                                
Wiphy phy0
        max # scan SSIDs: 4
        max scan IEs length: 2257 bytes
        max # sched scan SSIDs: 0
        max # match sets: 0
        max # scan plans: 1
        max scan plan interval: -1
        max scan plan iterations: 0
        Retry short limit: 7
        Retry long limit: 4
        Coverage class: 0 (up to 0m)
        Device supports RSN-IBSS.
        Device supports AP-side u-APSD.
        Device supports T-DLS.
        Supported Ciphers:
                * WEP40 (00-0f-ac:1)
                * WEP104 (00-0f-ac:5)
                * TKIP (00-0f-ac:2)
                * CCMP-128 (00-0f-ac:4)
                * CCMP-256 (00-0f-ac:10)
                * GCMP-128 (00-0f-ac:8)
                * GCMP-256 (00-0f-ac:9)
                * CMAC (00-0f-ac:6)
                * CMAC-256 (00-0f-ac:13)
                * GMAC-128 (00-0f-ac:11)
                * GMAC-256 (00-0f-ac:12)

CCMP-256 and in particular GCMP-256 looks really good. Now your mileage may vary with 'iw phy', as some device drivers are known not to be listed. I had best results with a late model Raspberry Pi (3 or 4) running the latest Rasbian (ie Debian).

You can do an online search for compatibility with this link.

From my very limited survey here in Seremban Malaysia only a minority of WiFi devices on the market supports this. And since some device drivers do not come up in 'iw phy' (in particular deliciously cheap Realteks), in the end I had to take my Linux laptop to my friendly local computer shop and test all 5 WiFi USB dongles models he had. None passed the test; in the end an ancient TP-Link TL-822N v1.1 proved up to the task.

TP-Link TL-W822N. Note only v1 worked, and possibly v2. v3 & v4 have Ralink chips

So did the Samsung Galaxy Note 5, with default setings:

Samsung Galaxy Note 5 will work as IEEE 802.11w client


A very pleasant surprise was the Raspberry Pi 3 Model B Plus' BCM4345:

        Supported Ciphers:
                * WEP40 (00-0f-ac:1)
                * WEP104 (00-0f-ac:5)
                * TKIP (00-0f-ac:2)
                * CCMP-128 (00-0f-ac:4)
                * CMAC (00-0f-ac:6)

Unfortunately the results were spotty. Upgrades to Rasbian caused it to stop working, but subsequent updates should fix this, so watch this space for updates.

And since I meant to link my deauth-proof routers to outdoor antennae I took a chance on the RTL8812BU generic USB dongles, from a hint here

Generic no-name RTL8812BU USB dongle also worked, but only as AP and not client/STA
This proved to be the toughest to use, as even my up-to-the-minute Rasbian (Buster) did not come with the device driver for it. The dongles came with driver source code, but cilynx (Randy C Wills) has a much better version here. cilynx's instructions (in README.md) worked very well for my Raspberry Pi 3 B+; I just needed to add in the Makefile

EXTRA_CFLAGS += -Wno-error=date-time

And change 'n' to 'y' for

CONFIG_80211W = y

Perhaps because this is a new driver, the default debug logs are very many and to prevent it from wearing out your micro sdcard, you might want to do:

# echo 3 > /proc/net/rtl88x2bu/log_level

And you are good to go.

In summary the WiFi hardware having the Right Stuff for deauth-proof IEEE 802.11w are:
1. Qualcomm Atheros AR9565 (Acer Aspire E1, Asus X751L)
2. Qualcomm Atheros QCA9377 (Acer Aspire F15)
4. Ralink RT5370 (Raspberry Pi 3 Model B Plus)
5. Realtek RTL8812BU (no-name China USB dongle)
6. TP-Link TL-WN822N v1.1
7. Samsung Galaxy Note 5 (and Fan Edition) as client device. The Galaxy Note 3 does not work.

Seven devices with the Right Stuff, just like the seven Project Mercury astronauts. The next step is to rebuild my WiFi Bridges to comply with IEEE 802.11w. But that is another story. 

Happy Trails.

Update: Removed the D-Link DWA-123 from the list, as it did not work. In the Raspberry Pi 3, weird kernel driver interactions caused the built-in  BCM4345 driver (brcmfmac) to work, but only with a Realtel RTL8188EU device plugged in. The last Raspbian update caused it to stop working so BCM4345 results are still pending