Using the lwIP stack with RF24Ethernet and RF24 radios - Another experiment in progress

 Using the lwIP stack with RF24Ethernet and RF24 radios

Another experiment in networking 

 So I finally got some time to play around with the lwIP stack over the holidays, and decided to try my hand at getting it working with the RF24Ethernet/RF24Gateway layers, so users can more conveniently handle TCP/IP traffic over the nRF24 radios.

 

 Overview:

 The RF24Gateway and RF24Ethernet libraries work together to handle TCP/IP and UDP traffic over nRF24 or nRF52 radio links. Users can communicate using standard protocols and techniques, without additional programming or advanced knowledge of the radio software or hardware.

The RF24Gateway library runs on more powerful systems with their own IP stack like a Raspberry Pi or other Linux devices with GPIO and SPI capabilities. The RF24Ethernet library is designed for Micro-controllers like those found using the Arduino platform. Now it supports the faster devices with a more advanced IP stack, lwIP.

 In studying and playing around with lwIP and the Arduino platform, most if not all of the client examples for Arduino utilize a single connection at a time, whereas lwIP supports multiple connections. In server mode, the default behaviour is to listen on a given port, with a backlog. This means that in the current configuration, lwIP will handle the first connection normally, then accept a second connection, but won't complete the second connection until the first connection has finished.

I've also introduced default timeouts, 30 seconds server side, so that if no data is sent or received for a given timeout period the server will disconnect the client. For the server side, call server.setTimeout(30000); . For the client, users need to implement their own timeout conditions. The server timeout can be set to 0 to disable it.

Usage: 

 Usage remains exactly the same as before, with users utilizing the Arduino Ethernet API and client/server functions to connect, disconnect and transfer data between systems. This means that RF24Ethernet users don't have to learn a new API to use the library, as long as they are familiar with the standard Arduino Ethernet API.

As of this writing UDP support is not included but will be added at a later date, TCP/IP support is functional and tested via MQTT and HTTP. The Arduino lwIP library can be installed using the Arduino Library Manager.

 Conclusion:

 I've been wanting to configure lwIP to go along with the RF24Ethernet library for a long time, but just didn't have the time or expertise to sit down and make it happen. With the recent holidays, I was able to put things together, and it came out working very nicely. It is still in prototyping stages, likely with some bugs to work out, but it is functional and available for testing at https://github.com/TMRh20/Sketches/tree/master/RF24Ethernet-lwIP

 NEW: 

The RF24Ethernet library no longer requires RF24Gateway running! Nodes can simply run RF24Ethernet and establish TCP/IP connections between themselves! See the new Headless examples found in the examples/Headless directory. 

Instructions:

 1. Download the RF24Ethernet library from the link above and place into your Documents/Arduino/libaries/ folder

2. Download and install the RF24Network library from ZIP from https://github.com/nRF24/RF24Network

 3. Edit the RF24Network_config.h file to set MAX_PAYLOAD_SIZE to 1514

4. The library is configured to detect CPU speeds over 50Mhz and use lwIP automatically, but users can edit RF24Ethernet.h and #define USE_LWIP as required

5. Enjoy!!! 

 

 

 

 

 

 

 

 

 

Automation etc with Node Red - How to make your Dashboard2 chart data persistent using a SQLite database

 How to make your chart data persistent using a SQLite database

Reliable charts and graphs with Node Red

 So I've been using Node Red for quite some time now as part of my general IoT network, involving wireless radio modules and numerous sensors etc. One main drawback of using Node Red, is that by default, it doesn't store data long term, and your charts etc will go blank after a restart or shutdown.

 Using a SQLite database combined with Node Red, I've been able to store information for the long term that persists even after restarts and shutdowns and this is how I did it.

 

Above is an image of a simple Node Red flow, logging and retrieving information from a database.

  So first things first, I will go through the process of the above flow:

1. Arduino and other sensors send in data over MQTT (Purple "houseTemp" and "houseHum") 

2. We need to combine the information into a single record and insert it into a database. (Light orange "Filter Dups & Insert")

 3. The data is then logged to a SQLite database

.4 At the same time, we select data from the database, so we can insert the newest data into some charts

5.  The results of that selection are fed directly into some charts:

 

 

 Detailed Process & Code: 

1. An Arduino sends in a message via MQTT with the msg.topic "houseTemp" and a float that represents the temperature (msg.payload), and another message with the msg.topic "houseHum" and a float that represents the humidity as the msg.payload.

2.  Then we combine the 2 results into 1, and insert the data using a function node:

 let temp = context.get('temp3') || null;

let hum = context.get('hum3') || null;

let ctr = context.get('ctr3') || 0;

if (msg.topic == "houseTemp") {

   context.set('temp3', msg.payload);

} else {

   context.set('hum3', msg.payload);

}

let sendMsg = false;

ctr = ctr + 1;

if (ctr >= 2) {

   ctr = 0;

   sendMsg = true;

}

context.set('ctr3', ctr);

msg.topic = "INSERT INTO HOUSETEMP (TIMESTAMP, TEMPERATURE, HUMIDITY) VALUES (" + Date.now() + ", " + temp + ", " + hum + ")";

if (sendMsg == true) {   

if (hum != null && temp != null) {

      return msg;

   }

} else {

   return;

} 

3. The data is logged to a SQL database, per the "INSERT" command in the previous function. 

4. Lets "select" or retrieve all the data including the data that was just logged:

msg.topic = "SELECT * FROM HOUSETEMP ORDER BY TIMESTAMP DESC LIMIT 1000";

return msg;

 

4. The database node responds immediately to the previous function, returning data

 

5. Display data in the charts:

 The main things to consider are having all of your variable names correct, and

 using the correct "properties" configuration in your Chart Node.

 

General DB Maintenance: 

In order to create a table, you need to add an "inject" node and set the msg.topic to something like:

CREATE TABLE HOUSETEMP ( TIMESTAMP INT PRIMARY KEY NOT NULL, TEMPERATURE float NOT NULL, HUMIDITY float NOT NULL)

 Since we are using a Linux timestamp, we can use the following formula to clean up old data via another "inject" node:

DELETE FROM HOUSETEMP WHERE TIMESTAMP < strftime('%s', 'now', '-14 days'); 

  

 Overview:

This generally works pretty well and keeps the graphs & charts up to date with good data, storing it for up to 2 weeks. Of course, this is a heavy-handed approach, replacing all the chart data each time. A better approach is identified below, using an external MySQL server to offload activity.

A bit better solution would be to make the SELECT function node only return 1 record at a time. Then, add a blank inject node that triggers only once on startup, that both resets the charts ( send JSON msg.payload of []  ), and loads all existing data into the charts. The arrays coming out of the database at the end would need to be reversed as well, so the newest data is at the end of the array.

This is a much less intensive way to go about it, but a little more complicated to set up. 

 

RF24: An advanced example for the Arduino Uno Q

 RF24: An advanced example for the Arduino Uno Q

 A new take on the RF24 GettingStarted example

In testing out the RF24 library and related features, I managed to get a few different things working together on the Arduino Uno Q to provide a more advanced example of RF24 interaction. 

 

 The main thing the sketch does is transfer data over RF24 radio link along with a standard Arduino device running the RF24 GettingStarted sketch included with the RF24 library.

Then, any data received is displayed in the Serial Monitor, Python Console and an associated webpage to demonstrate usage of incoming data etc.

 A button is included on the webpage that allows users to toggle the LED matrix on/off, which demonstrates how to pass data from Webpage to Python to Microcontroller. 

All in all its a simple concept, but it did take some figuring to work out how the functions and stuff all relate through JavaScript, HTML, Python, C++ etc.

My Arduino Q sketches can all be found here:

https://github.com/TMRh20/Sketches/tree/master/ArduinoQ 

For a limited time, a live real-time demo of a similar sketch will be available online here:

TMRh20s Example Webpage

The main functionality is tied into 4 files:

sketch/sketch.ino - The Arduino sketch handles radio input/output & passes its data to Python

python/main.py - The middleman between the webserver and the Arduino sketch

assets/app.js - JavaScript handling, communicates between index.html and Python via websockets

assets/index.html - The main webpage, provides variables that can be modified via JavaScript

Trying out the Arduino Uno Q - A quick and dirty overview

 Trying out the Arduino Uno Q - A quick and dirty overview

 Love it or hate it, it... exists 

 With the release of the new Arduino Uno Q board, I decided to get one and try it out. After a few days of playing around and experimentation I have to say I'm less than impressed.

 The concept seems to be a solid one, combining a standard Micro-Processor Unit (MPU) with a Microcontroller Unit (MCU), but the execution has been a little rough.

 The first and main thing to probably address is the Bridge concept, which is a way for users to make the MCU 'talk' to the MPU. It is driven by a serial connection running at 115200 baud, using RPC, which altogether is far too slow to pass copious amounts of data, or do it really quickly. It is also limited in what can be sent, so trying to pass arrays or data structures between the systems is out of the question. On the other hand, it can be used to pass information between programs, be it Python to Python or C++ (Arduino) to Python.

 One of the first things I tried, of course, is running RF24 radio based applications. It appears to work OK initially with the core RF24 driver and RF24Network, but once I got into the RF24Mesh layers, I started to notice problems. I thought it was basic speed issues, but it turned out to be a radio power supply related issue it seems. After changing out the radio and calling 'radio.setPALevel(RF24_PA_MIN,0);', it worked.

Next, I attempted to create a basic TUN network interface in Linux using Python, to see about creating a RF24Gateway like interaction, but realized quickly that this too was outside the general capabilities of the device. It runs a Python virtual environment within a Docker container, with no apparent way to configure things, so I can't seem to get it working. That means an RF24Gateway style library is out of the question for now.

It seems that so much of what I've tried do to so far just doesn't work. There was also a forced, breaking update that happened recently, to version 3.0, where users have to manually download a new version of App Lab in order to program their devices.

 One essential thing to note, is that when the App Lab stops responding, or the Serial/Monitor output gets corrupted, I need to close App Lab, and run `arduino-app-cli system cleanup` from the command line to get it working again.

 Some of the features still seem promising, the ability to easily incorporate "Bricks" and features into your programs, and the underlying software for the MPU is Debian/Linux along with Arduino, so you can still do a lot with it. 

I don't think I'll get another one, but I will definitely be playing around with the one I've got for a while. 

In the meantime, I've included a new example in the RF24 core library that users can try with the Arduino Uno Q that allows users to configure the radio and print debug information.

I've also added an example sketch to go along with the stock RF24 GettingStarted example here:

https://github.com/TMRh20/Sketches/tree/master/ArduinoQ 

 

RF24Gateway - What Does It Do And How Do I Use It?

 RF24Gateway - What Does It Do And How Do I Use It?

 A simplified explanation...

 So maybe you've taken a look at the RF24 communication stack and have seen RF24Gateway, along with the RF24Ethernet libraries, and are wondering what they do? I've noticed some confusion among users who aren't sure what the libraries do exactly.

 The simplest explanation, is that it allows users to use nRF24L01 radios as standard Network Interface Cards (NICs). Once RF24Gateway is installed, its like having a standard NIC linked into the wireless RF24 radio network.

This means users don't need to understand programming, RF24 APIs, or anything much about how the RF24 comm. stack operates in order to use RF24Gateway &/or RF24Ethernet. It means users can use standard networking tools like the 'ping' command to test network connectivity, or more advanced tools & protocols like MQTT, HTTP, etc to interact with their Arduino devices or other Raspberry Pi/Linux devices wireless, over RF24Gateway & the radio network.

Linux users, for example, can connect to other Linux devices over the RF24 based network by using SSH, SSL or other secure protocols.

A good general understanding of standard networking and protocols is probably going to be beneficial to anyone trying to make use of RF24Gateway/RF24Ethernet. The system uses the proprietary ESB protocol by Nordic for communication, and although it operates on similar frequencies to WiFi, it is not a WiFi based system. However, it does allow your RF24Ethernet & RF24Gateway devices to interact with WiFi devices on your network, because RF24Gateway can link the WiFi & RF24 networks together using standard networking techniques like forwarding and routing, or by using an intermediary like an MQTT server.

 This means advanced users can also do things like link RF24 based sensor networks together by using VPNs, SSH tunneling, etc and create a centralized environment to collect and display data, control devices, or otherwise interact with the networks. I've experimented with automation systems like Home Assistant and Node Red to create these types of networks myself, and there are a number of alternatives that can be easily integrated into the system. Networks can work over very wide geographical areas, covering many KM locally, via radio link, and spanning the globe using VPNs or SSH tunneling.

 RF24Ethernet is companion software to RF24Gateway, which is meant to run on smaller devices like Arduino Uno, Nano, etc. while RF24Gateway is designed solely for Linux devices. RF24Gateway typically requires little to no programming, while RF24Ethernet lets users build web servers, sensor nodes, interactive nodes with on/off switches, lighting controllers, etc. and customize things to the nth degree, or simply build off the included examples.

 ESP32 based devices have become extremely popular, but in a large scale sensor network, WiFi is a very power hungry alternative to nRF24 based devices, which can offer similar features, wireless protocols & connectivity. 

To sum it up, RF24Gateway & RF24Ethernet provide connectivity & functionality similar to a WiFi network, but using much less power, and are able to support the smallest devices like ATTiny all the way to larger, much more powerful computers like Raspberry Pi and other Linux based devices that support GPIO & SPI capabilities. By default, the system uses the RF24Mesh and RF24Network libraries, so all of the internal networking and routing is handled automatically.

 

Creating IoT Sensor Networks & Systems using nRF24 & nRF52840 - An Analysis

 Creating IoT Sensor Networks & Systems using nRF24 & nRF52840

A Brief Analysis

In playing around with the nRF24 and nRF52x devices, I've been running and monitoring a couple large IoT networks of them over the past few years. In that time, I've developed a system that works very well, with nodes staying connected over long periods of time, despite the mesh protocols and need to re-establish connectivity periodically.

The system is based on TCP/IP & Node-Red, and makes heavy use of HTTP & MQTT protocols. Nodes either interact with an MQTT server, or are polled periodically via HTTP requests. Devices range from Raspberry Pi running RF24Gateway & Node-Red, to a wide range of Arduino devices utilizing RF24 and nrf_to_nrf libraries.

A graph showing connection times for various network nodes

 The graph above shows connection times for nodes connecting via MQTT. They report in periodically, showing how long they have been connected to the MQTT server. By default, in the MQTT library I use, the keep-alive interval is 10 seconds. This means each node must communicate with the MQTT server successfully every 10 seconds or be disconnected. Given the mesh nature of the system, this works out very well, with nodes staying connected for hours and days at a time.

With the default logic in place, it is very interesting to see how nodes assemble themselves around the master node over time. At first, it can be very chaotic, with many nodes on a network, each grabbing a unique identifier and place in the network. Over time, the nodes with the best connectivity find their way to a direct connection with the master node, or at least a secondary connection, with only one routing node between them and the master node.
One would almost think there was a more intelligent algorithm handling placement of nodes in the networks, but it is a relatively simply methodology, with nodes contacting the master node first, then secondary (children) nodes, then children of those nodes, etc. in order to establish a connection to the mesh. Nodes will also contact their parent node periodically to verify connectivity to the mesh.

 

 Ping Response Times for a Connected Node

 The above picture shows ping response times for a connected Arduino node. At the left of the top graph one can see that the node was able to re-establish its connection with better connectivity to the master node, not having its traffic routed through another node. Response times vary, with the node typically responding in about 8mS, up to around 275ms in some cases when there is a direct connection to the master node.

"External Network Activity"

 

The above picture shows the results while "external activity" interferes with the network. During this time, due to the nature of radio and jamming techniques, most nodes on the network lose connectivity to the mesh, but the self-healing nature of the network ensures that they only go offline while the jamming is taking place. Nodes quickly re-converge around the master node as the mesh re-establishes itself, and every sensor or device resumes normal operations. 

HTTP based nodes reporting in during the jamming

 

I recently updated many of my nodes, due to bug fixes and testing, per the above graph. In it, as can be seen, some of my HTTP nodes lost connectivity during the jamming. Others stayed online with only small blips in connectivity. Depending on proximity to other nodes and the master node, as well as the location of the jamming device, different nodes are affected differently, but again, they all resume operation after the interference stops.

Temperature & Humidity Readings from a Connected Node

This system works extremely well, with a distant node reporting in its temperature and humidity over time. You can see in the chart above, it has been reporting in consistently.

 

Ping Statistics over a Number of Days

 In the above graph, you can see the success rate of pinging one of my sensor nodes over the course of a few days. In this picture, the success rate is down a bit from the norm, due to ongoing "external network activity" beyond my control. This has been the case for years and years, with one of my neighbors continuously interacting with my network on an almost daily basis. I can't understand the obsession, but I've witnessed it on many occasions.

Conclusion:

The end results kind of speak for themselves. In a given networking scenario where reliability and consistency is key, making use of the advanced features of TCP/IP networking in combination with HTTP & MQTT allows users to construct networks to suit many different scenarios. This system is highly reliable and can be customized to the nth degree. 

The combination of nRF24L01 & nRF52x radios works quite well, and allows even more advanced functionality in a given network. With the 52x devices supporting things like encryption, Bluetooth & I2S, users can make great use of these features, with everything being done wireless.

Since we are using a mesh based system, and TCP/IP was designed to operate even on partially damaged networks, the combination works very well. Add that to recent fixes and updates in the RF24 communication stack, and you have a system that works extremely well over long periods of time. I measure uptime on the network, and nodes tend to only go offline when there is a power outage, hardware failure or they are manually taken offline.

Considering that there are great quality nRF24 based radios available with range anywhere from 2.5-5 KM (E-01 ML01DP5 or 2G4M27D), networks can be constructed that cover wide distances and manage themselves automatically. The network will support the smallest of devices, ranging from ATTiny all the way up to Raspberry Pi and other Linux based devices that have GPIO & SPI capabilities.

Libraries & Software Used:

1.  RF24, RF24Network, RF24Mesh, RF24Ethernet (Arduino) & RF24Gateway (Linux/Raspberry Pi)

2. nrf_to_nrf radio library (nRF52x devices)

3. Node-Red (Linux/Raspberry Pi)

4. Mosquitto (MQTT broker)

 

All of the software and systems used are open source, so anybody can create similar systems. Any software that is MQTT compatible like Home Assistant or Node Red will work.

Recreating nRF24 greater than 32 byte payload issue with nRF52840

 Recreating nRF24 greater than 32-byte payload issue with nRF52840

A D.O.S. attack against nRF24L01P that do not validate Dynamic Payload Sizes

 I made a video demonstrating how to send 33 byte payloads to the nRF24L01P and either cause the device to flush its RX buffer, or create a denial of service attack for any devices that do not validate Dynamic Payload sizes.

 

I find it interesting that the nRF24L01P devices accept these payloads at all, as well as sending an ACK to the transmitting device.