Monday, 28 January 2019

Webhook! controlling the generic ESP8266 IoT device with Google Home


Nick Felke's youtube introductoin

The usual method to link your ESP8266 to Google Home is via Adafruit and IFTTT. It is free to start with and the sample code is easier to implement.

My autogate works beautifully on voice control, especially when driving. I preferred the homebrew version to the Sonoff version, mainly because I could add a function to crack open the gate ajar, wide enough for a person putting out the rubbish, but not so wide as to tempt the dogs and/or cat to rush out.

To accommodate my wife, I implemented a 2-feed version, with 2 separate Adafruit and IFTTT accounts. It ran a little slower as it now has 2 feeds to monitor, but was stable over WiFi disconnects and worked quite well. That is, until my sister came to visit.

To accommodate my sister, I had to dismantle my autogate setup, remove the ESP-01S relay adapter and reprogram it. And add a third feed. This is obviously not going to scale.



One solution would be to hook the ESP-01S relay PCB directly to Google Assistant, just like Sonoff. To add users I simply add them as 'Home Member'. Users are easily revoked via my Google Home settings.

Adding home members is much easier than reprogramming the ESP8266

Luckily Google has provided all you need to do this, a tutorial and sample code. Unlike other code, I found it slower to pick up as there was simply so much to learn (for me anyway). Things like Conversational User Interface, JSON and node.js were quite new to me. But it can be done on the fly mainly because the tutorial and sample code is complete and it works! It helps to start by watching Nick Felke's video.

You will need to sign up for Actions on Google and ngrok. Both have free versions. You do not really need to sign up for ngrok - you can simply start using it, but if you do, that temporary ngrok URL they give you will not expire every few hours, which can be quite annoying especially if you have the whole house's devices set up.
You MUST CLICK TEST whether or not you intend to use the Test Menu
You will need to copy and paste 3 URLs from your smart-home-nodejs-master program output into the Actions on Google site. And you must click on Test, whether or not you intend to use the Test page. This is because unless you do your smartphone's Google Assistant will not use the new ngrok links.



When you try to link your Google Assistant on the phone to smart-home-nodejs-master you should get a login screen like above. It is likely it will login slowly (probably because of ngrok) and it helps to wait for it to finish before pressing 'Login' again. You can monitor smart-home-nodejs-master's output to be sure it has finished.

At this stage you should have a working template, and all the IoT 'devices' are being simulated by smart-home-nodejs-master.

To add real ESP8266 devices you will need to add a webhook to the file smart-home-nodejs-master/smart-home-provider/smart-home-app.js. Don't worry!. webhooks are new to me as well.

Recall that the IoT ESP-01S Relay is turned on from the browser with a command like:

http://12.34.56.78:8080/gpio/1

Or off:

http://12.34.56.78:8080/gpio/0

If you need it run from the command line you can use the Linux program lynx thus:

lynx -dump -cmd_script=/home/fred/lynx_script http://12.34.56.78:8080/gpio/1

You will first need to generate the file lynx_script thus:

echo "exit" > /home/fred/lynx_script

Now you will need to get the program smart-home-nodejs-master/smart-home-provider/smart-home-app.js to call you lynx command. This you can do by using the 'child_process' library. You need to insert the code into the execDevice() function (typically at line 520):

    if ( execDevice[device.id].properties.name.nicknames == "My Test Lamp" && execDevice[device.id].states.on) {
      var exec = require('child_process').exec; 
      var cmd = "lynx -dump -cmd_script=/home/fred/lynx_script http://12.34.56.78:8080/gpio/1"
      lampon = exec(cmd, function(err, stdout, stderr) {
        console.log(stdout);
      });
    }
    if ( execDevice[device.id].properties.name.nicknames == "My Test Lamp" && !execDevice[device.id].states.on) {
      var exec = require('child_process').exec; 
      var cmd = "lynx -dump -cmd_script=/home/heong/IoT/lynx_script http://12.34.56.78:8080/gpio/0"
      lampon = exec(cmd, function(err, stdout, stderr) {
        console.log(stdout);
      });
    }

Next log into your ngrok url (https://12345678.ngrok.io) and add a new device, the Monochrome Light. Change the nickname to 'My Test Lamp'

Add a new device, 'Monochrome Light' and change the nickname to 'My Test Lamp'
You should now be able to control the ESP-01S Relay PCB using the voice command "OK Google, turn on My Test Lamp"

There you have it, a voice-activated IoT device easily expanded to include other Home Members and even the occasional dinner guest. There is no reason why this should be limited to little dinky IoT devices. You can webhook desktop or server functions the same way: "OK Google backup my main server".

Happy Trails.

Saturday, 22 December 2018

Autogate Adventures: Voice Control via Google Home. Part 3 of 3: Debugging Adafruit MQTT disconnects

One reason why I love doing software is the instant gratification: the speed at which you can try out something. And when a prototype works, the impulse is to immediately move to something else.

But a prototype is seldom suitable for customer use, and there are long weeks of grinding work where you monitor for failures and fix them as they appear.  A robust system is usually achieved sometimes years of constant fixing. If you are lucky enough to have a customer for your prototype the key is responsiveness- you need to fix it as fast as possible.

In Part 2, the MQTT-based ESP8266 IoT autogate was usable in less than a day. But it would not respond after a few days, usually after I disconnected the ADSL modem during thunderstorms (we get really spectacular lightning strikes and the ADSL modem is usually the first to go).

I can get the IoT Autogate to work again by power-cycling it, but it really looked bad next to the Sonoff switches, which seemed to recover after no more than 30 minutes as long as the broadband connection is restored. Since the Sonoff is also based on the ESP8266 and they both used the same ADSL modem and routers the problem must be the software.

Debug setup: ESP-01S on its CH340 USB Adapter


To debug the software, it would be nice to get to the Arduino IDE Serial Monitor, but since the  MQTT-based ESP8266 IoT autogate is out of reach inside the autogate enclosure. One way is to program another ESP-01S and run it on its CH340 USB adapter. When Google Assistant triggers the autogate (via IFTTT and Adafruit MQTT) both devices will now respond.

You will get messages like these every 5 seconds (Wife being the other House Member):

Wife Connecting to MQTT... Wife MQTT Connected!
Connecting to MQTT... MQTT Connected!
Wife Connecting to MQTT... Wife MQTT Connected!
Connecting to MQTT... MQTT Connected!

It would be nice to monitor it while at work, and since Adafruit's MQTT broker allows an IoT device to send messages to a feed, and since I have to make an IoT report it state back to Google Home at some point, I might as well use it to report debug messages.

In Adafruit-speak, when we want Google Assistant to control a device the ESP8266 IoT program (ie its Arduino sketch) needs to 'subscribe' to a feed. When it is sending data back to Google Assistant, it needs to 'publish'. So publish we shall. I added another feed, 'log' to my Adafruit account. 

Adafruit_MQTT_Publish mqtt_log = Adafruit_MQTT_Publish(&mqtt, AIO_USERNAME"/feeds/log");

No setup() code is necessary and you simple insert the following code snippet in loop():


{
    char logmsg[50]; 

          sprintf(logmsg, "YOUR_LOG_MESSAGE %d", Your_Log_Number); 
          if (! mqtt_log.publish(logmsg)) {
            Serial.println(F("publish Failed"));
          } else {
            Serial.println(F("publish OK!"));
          } 
}

Now log into your Adafruit account, navigate to

https://io.adafruit.com/youraccountname/feeds/log

And you have a remote Serial Monitor. It is that simple.

Logging debug messages in the Adafruit feed


But what of the ESP8266 MQTT IoT device going unresponsive problem? It turned out that just having the ESP8266 publish regularly (preferably not less than once every five minutes) made the problem go away. 

For now I can only guess when a device goes silently unresponsive, the MQTT broker may sometimes be unaware that the connection is broken, ie the mqtt.connect() function returned 0 (meaning connected) even when it is not. On the other hand we know when a periodic MQTT report does not arrive on schedule.

Notice I have disabled the "watchdog reset" to make the IoT keep trying to connect without restarting the WiFi code.    

void loop() {
  MQTT_connect();

    .
    .
    .
 publish++; 

  if (publish >= 20) //  ping every 4min
  {
    // Now we publish stuff! ping limit is 300s so 60s seems plenty
    Serial.print(F("\nPinging adafruit mqtt ... State "));
    Serial.print(State);
    Serial.print("...");
    sprintf(logmsg, "State is %d %d", State, Wife_Autogate_State);
    if (! mqtt_log.publish(logmsg)) {
      Serial.println(F("publish Failed"));
    } else {
      Serial.println(F("publish OK!"));
    }
    publish = 0;
  }
}

void MQTT_connect() {
  int8_t ret;

  // Stop if already connected.
  if (mqtt.connected()) {
    return;
  }
  
  Serial.print("Connecting to MQTT... ");
  
  uint8_t retries = 3;
  
  while ((ret = mqtt.connect()) != 0) { // connect will return 0 for connected
    Serial.println(mqtt.connectErrorString(ret));
    Serial.println("Retrying MQTT connection in 5 seconds...");
    mqtt.disconnect();
    delay(5000);  // wait 5 seconds
    //  retries--;
    if (retries == 0) {
      // basically die and wait for WDT to reset me
      while (1);
    }
  }
  Serial.println("MQTT Connected!");
}

So, for now the ESP8266 MQTT IoT Autogate goes back for more testing. We still do not know exactly what caused the problem- it might still be there for all we know. At least it is reconnecting when the Internet comes back, just like the Sonoff devices. Dave has some good work on ESP8266 reconnection problems. There is some talk here that it may be due to problems in the Expressif SDK.

By the way if you suspect the ESP8266 Adafruit MQTT library itself, you can turn on its debug messages:
 #define MQTT_DEBUG

Happily my customer (in this case Home Member Wife) does not seem to mind. We quickly got used to the Google Home/Assistant voice control of the autogate. It is especially handy when you have your hands full driving or carrying the groceries. Now if only I can get Google Assistant to recognize the dogs' voices ...

Happy Christmas, and Happy Trails.   

Sunday, 9 December 2018

Driving Inductive loads with the Sonoff Basic

The Sonoff Basic Smart Home power switch is very handy and has a claimed output rating of 10A at 230Vac. The basis for this is the output relay rating, a SONGLE SRD-05VDC-SL-C.

SONGLE SRD-05VDC-SL-C Miniature Relay


Now in tropical Malaysia, one of the most handy home appliances is a fan. There is usually quietly ruthless jostling for fans so there is usually at least one in use.

Poppy the dog usually wins pole position


One of my favourites is a Pensonic PFF-20B, rated at a modest 200W or  0.87A.

Pensonic PFF-20B 230V 200W

Trouble is the Sonoff Basic does not last long switching the PFF-20B, even at a modest 1A load. The contacts often get stuck, ie the relay might turn on but fail to turn off. A sharp knock on the Sonoff Basic will often jar the contacts back into position, but it will invariably fail again. Inductive loads cause arcing of the contacts resulting in damaging micro-welds.

On the other end of the spectrum is a magnetic contactor, like the LSIS MC-9b. Modestly rated at 9A for inductive loads (AC3), it is a veritable King Kong next to the SONGLE SRD-05VDC-SL-C.

LSIS MC-9b 230V 9A (AC3)

The obvious answer is to use the Sonoff Basic to drive the MC-9b. The MC-9b has a 230Vac input relay coil. I picked one up at RM36 from my local store, but you can find them online for as low as RM28. The wiring is straightforward, so I'll dispense with the schematics.


The contactor jumps about quite a bit, so it is necessary to mount it securely and to ensure the jolting does not transfer too much to the Sonoff Basic.

Here's a youtube video of a voice controlled Pensonic PFF-20B. The MC-9b did not faze Poppy at all. In Malaysia, it takes a lot to give up one's spot in front of the fan.

Happy Trails.

Friday, 30 November 2018

Autogate Adventures: Voice Control via Google Home. Part 2 of 3: ESP8266, MQTT and IFTTT

In Part 1, the autogate became a smarthome device, controlled by voice via my Google Home speaker or Google Assistant on my phone. However every time we went out, it needed four voice commands to open and close the gate.

I can also use an ESP8266 relay board, MQTT and IFTTT like I did for my ancient lantern, but there were a couple of problems. First the MQTT code would stop working after a few days. Secondly I had not figured out how to share the device on Google Home with my wife.

Top: 230Vac to 5V@1A power supply. Bottom: ESP-01S with 1-channel relay adapter

It did not take long to find out why the program stopped running. The routine MQTT_connect() is run in a loop and normally returns non-zero for a valid connection to MQTT server. However if the connection should drop (maybe WiFi interference, power outtage affecting the router, it tries to reconnect 3 times and then goes into an infinite loop waiting for a watchdog timer.

The watchdog timer is enabled by default, and is serviced by various routines in the library, so there is really no good way to use it except as a delayed reset. There is a good article on it here.

  while ((ret = mqtt.connect()) != 0) { // connect will return 0 for connected
    Serial.println(mqtt.connectErrorString(ret));
    Serial.println("Retrying MQTT connection in 5 seconds...");
    mqtt.disconnect();
    delay(5000);  // wait 5 seconds
    retries--;
    if (retries == 0) {
      // basically die and wait for WDT to reset me
      while (1);
    }
  }

The reason for the failure seems to be mqtt.connect() or perhaps the WiFi router sometimes cannot detect there is no longer a connection. For now the best workaround seems to be to have an active MQTT input, that is to regularly 'publish' to the MQTT broker Adafruit. After a few minutes (normally less than 30) mqtt.connect() will eventually work and the device will reconnect.

Getting Google Home to share the device took a little more time. On the face of it, IFTTT allows me to share: I can publish my Applet for unlimited sharing. But that is too much sharing: letting a random user open my gate from the Internet is a not good idea.

I get my MQTT service from Adafruit, (which IFTTT needs to link the ESP8266 to Google Home) and I can share just my Adafruit feed for the autogate with my wife. Indeed she can activate the autogate from the Adafruit App, but Google Home will not link to a shared feed. Furthermore the Adafruit share turned out to be a global share as well, again an unacceptable security risk.

It turned out that if I got the wife to register with IFTTT and Adafruit, and if I duplicated the ESP8266 code using my wife's Adafruit account and secret key (AIO) the device will support both Adafruit feeds.

Adafruit_MQTT_Client mqtt(&client, AIO_SERVER, AIO_SERVERPORT, AIO_USERNAME, AIO_KEY);
Adafruit_MQTT_Client wife_mqtt(&wife_client, AIO_SERVER, AIO_SERVERPORT, AIO_WIFENAME, AIO_WIFEKEY);

// Setup feeds Adafruit_MQTT_Subscribe lantern = Adafruit_MQTT_Subscribe(&mqtt, AIO_USERNAME"/feeds/frontgate");
Adafruit_MQTT_Subscribe wife_autogate = Adafruit_MQTT_Subscribe(&wife_mqtt, AIO_WIFENAME"/feeds/frontgate");

Converting the Open/Close Google Home commands to autogate pulses is now easy:

while ((subscription = mqtt.readSubscription(2000))) {
    if (subscription == &gate) {
      Serial.print(F("Got: "));
      Serial.println((char *)gate.lastread);
      int gate_State = atoi((char *)gate.lastread);
      Serial.print("Autogate openclose ");
      Serial.println(gate_State);
      digitalWrite(0, 1); // Issue a command to toggle gate
      delay(500);
      digitalWrite(0, 0); // Autogate CPU input is monostable. Issue command to reset  
    }
  }

Tight fit: clockwise from top left: autogate control board, RF remote control module, 12V battery and ESP8266 relay module

Here's a little youtube video of the result.

I am still a little unhappy with it: I would it like to feedback (ie 'publish' in MQTT-speak) actual gate open/close state, and I would have to manually dismantle and reprogram it should my parents come to visit. Also it would be nice to keep a log of the autogate usage, but perhaps that is a future project, an MQTT server. But in the spirit of DevOps, let's just put it up and look for those all-important gotchas that are so critical to a product's eventual success.

There you have it- a voice-controlled autogate. I have not looked very hard, but with a little luck, this should be one of the first homebrew Google Home autogate smart remotes.

Happy Trails.

Thursday, 29 November 2018

Autogate Adventures: Voice Control via Google Home. Part 1 of 3: Sonoff Basic

About a year ago I decided to maintain the autogate myself, and naturally could not resist adding a Bluetooth remote function, in addition to its keychain RF remote control. Later I added a Raspberry Pi Zero W IoT gateway, which let me control it from my smartphone. The next step would be to make it a smarthome device - voice control via my Google Home speaker or Google Assistant on my phone.

The obvious way would be to use an ESP8266 relay board, MQTT and IFTTT like I did for my ancient lantern, but there were a couple of problems. First the MQTT code would stop working after a few days. Secondly I had not figured out how to share the device on Google Home with my wife. So I though I would use a commercially-available IoT device, a Sonoff Basic.

Sonoff Basic was RM23 from Lazada
There was no trouble sharing the Sonoff Basic with home members and it was a lot more reliable. I had been using it to control some of the lights in the house. The Basic takes 230Vac input and puts out 230Vac when it turns on. But the autogate needed a dry (ie voltage-free) contact, so I simply added a 230Vac relay , an Omron MY2N.

Omron MY2N 230Vac coil relay, RM10 
I found difficult to mount in the available space in the autogate enclosure, so eventually replaced it with a solid-state relay, the Omron G3TA-IAZR02S.

The OMRON G3TA-IAZR025 was a whopping RM45
Sonoff Basic converted to output dry contact
It worked ... here's a video ... sort of. Trouble was the autogate controller needed a pulse to toggle the gate open and shut. The Sonoff Basic puts out a steady on or off command. Opening the gate was easy:

"OK Google Open autogate"

Now I need to deactivate the signal to the autogate controller, so

"OK Google Close autogate"

But the gate is still open, so now it is time to close it:

"OK Google Open autogate"

And then

"OK Google Close autogate"

It took all of 15 minutes before I got tired of it. I guess it is time for Part 2.


Saturday, 27 October 2018

Voice-controlled ESP8266 IoT AC Mains Power Extension Part 3 of 3

230Vac Fluorescent battery-backed lantern  
I have had this lantern for over 20 years. It is obsolete- dual 230Vac fluorescent lamps running from a 6V 7Ah lead-acid battery. Yet it has never failed - I replaced the battery many times but the lamps never blew. It is always plugged, in float-charging the battery and when the mains power goes the lamps automatically comes on.

It would be nice to have a bit of extra light sometimes during normal conditions, so why not mount it on the ESP8266 IoT Extension? And it would even be nicer if I could control it by voice command.

I do not have Google Home or Amazon Echo yet, but my Android smartphone does have Google Assistant which will capture my voice commands. I just need to forward the command words to the ESP8266 IoT device.

Robosapien has a great writeup on using the ESP8266 with Google Assistant, IFTTT and Adafruit IO. I really wish I could improve on it, but I believe Robosapien has had the last word on the subject. I simply followed all the steps and replaced his hardware with my ESP8266 IoT AC Mains Power Extension.

The process just took half a day, and I now have a voice controlled 20-year old lantern!

Update (2018-11-27): after running the IoT lantern a few days, I noticed a new WiFi access point with an extremely strong signal:

iwlist wlan0 scan
          Cell 04 - Address: xx:xx:xx:xx:xx:xx
                    Channel:10
                    Frequency:2.457 GHz (Channel 10)
                    Quality=70/70  Signal level=-34 dBm
                    Encryption key:off
                    ESSID:"ESP_1CC60D"
                    Bit Rates:5.5 Mb/s; 11 Mb/s; 1 Mb/s; 2 Mb/s; 6 Mb/s
                              12 Mb/s; 24 Mb/s; 48 Mb/s
                    Bit Rates:54 Mb/s; 9 Mb/s; 18 Mb/s; 36 Mb/s
                    Mode:Master
                    Extra:tsf=0000000008b54204
                    Extra: Last beacon: 4442ms ago
                    IE: Unknown: 000A4553505F314343363044
                    IE: Unknown: 01088B9682840C183060
                    IE: Unknown: 03010A
                    IE: Unknown: 0706534720010D14
                    IE: Unknown: 32046C122448
                    IE: Unknown: DD0918FE34030100000000

A signal level of -34dBm was stronger than my own access point! In fact I can detect and log into it two houses down, some 150m away through 2 concrete walls.

It turned out to come from the lantern code, and while it does not interfere with its operation, it is also a security risk as it is an open Access Point. As usual the Internet provided an answer; there is a bug in the ESP8266 Arduino library. There is a workaround: you just need to insert WiFi.mode(WIFI_STA) after calling WiFi.begin().

My setup code now looks like:

void setup() {
  Serial.begin(115200);

  pinMode(0, OUTPUT); // 2018-10-26 set gpio to o/p

  // Connect to WiFi access point.
  Serial.println(); Serial.println();
  Serial.print("Connecting to ");
  Serial.println(WLAN_SSID);

  WiFi.begin(WLAN_SSID, WLAN_PASS);
  WiFi.mode(WIFI_STA); // 2018-11-27 remove rogue AP
  while (WiFi.status() != WL_CONNECTED) {
    delay(500);
    Serial.print(".");
  }
  Serial.println();

  Serial.println("WiFi connected");
  Serial.println("IP address: ");
  Serial.println(WiFi.localIP());


  // Setup MQTT subscription for onoff feed.
  mqtt.subscribe(&lantern);
}



Lantern with IoT Power Extension
Now the logic is reversed for the lantern as it comes on when there is no mains power. This I did by entering a data '0' when the IFTTT trigger is on.

Note reversed (data zero) logic
Google Assistant, IFTTT and Adafruit IO all worked first time. Here is a youtube video of it:

Youtube video
There you have it, a voice-controlled IoT lantern. It is not ideal and much work needs to be done on it, but it still works as before without the Internet, and in the spirit of DevOps, just put it out there as fast as we can ...  Happy Trails.

Sunday, 7 October 2018

ESP8266 IoT AC Mains Power Extension Part 2 of 3




ESP8288 ESP-01S IoT AC Mains Power Extension, shown with light bulb in E27 adapter
Rather than building separate Internet of Things (IoT) lamps, TVs, speakers, etc I found it much more flexible to make an IoT power extension. This lets me test the various IoT devices at leisure.

In Part 1, half the space is taken up by the IoT power supply. Here, in Part 2, is a proper 1-gang IoT AC mains power extension using the ESP8266 NodeMCU ESP-01S and a very handy 700mA 5V AC mains power supply.


5VDC 700mA adapter from 220VAC 
I bought mine from autobotic at lelong.com for about RM7 each. They were so handy I actually bought up all the available stock in Malaysia (sorry). The 5V 1A version is over twice in size and not as useful.

Assembly is straightforward. The power socket fits nicely over the power module and the ESP-01S relay modle without coming into contact. Note the cable ties to hold the modules in place
Assemble is simple. Note the cable ties used to hold the modules in place. This is advisable as there is Live voltages on the power board and you do not want them in contact with the other bits. The modules should not be exposed from the bottom either, as some of the holes there are quite large.

You may need to drill a hole in the side for the power cord, and maybe enlarge the various holes in the bottom enclosure to fit your cable ties.

Assembled IoT mains power extension
There you have it, a proper IoT AC mains power extension. In Part 3 of this series I hope to detail how to remotely control it from your smartphone or your Google Home smart speaker.

Happy Trails.