Creating IoT Sensor Networks & Systems using nRF24 & nRF52840 - An Analysis

 Creating IoT Sensor Networks & Systems using nRF24 & nRF52840

A Brief Analysis

In playing around with the nRF24 and nRF52x devices, I've been running and monitoring a couple large IoT networks of them over the past few years. In that time, I've developed a system that works very well, with nodes staying connected over long periods of time, despite the mesh protocols and need to re-establish connectivity periodically.

The system is based on TCP/IP & Node-Red, and makes heavy use of HTTP & MQTT protocols. Nodes either interact with an MQTT server, or are polled periodically via HTTP requests. Devices range from Raspberry Pi running RF24Gateway & Node-Red, to a wide range of Arduino devices utilizing RF24 and nrf_to_nrf libraries.

A graph showing connection times for various network nodes

 The graph above shows connection times for nodes connecting via MQTT. They report in periodically, showing how long they have been connected to the MQTT server. By default, in the MQTT library I use, the keep-alive interval is 10 seconds. This means each node must communicate with the MQTT server successfully every 10 seconds or be disconnected. Given the mesh nature of the system, this works out very well, with nodes staying connected for hours and days at a time.

With the default logic in place, it is very interesting to see how nodes assemble themselves around the master node over time. At first, it can be very chaotic, with many nodes on a network, each grabbing a unique identifier and place in the network. Over time, the nodes with the best connectivity find their way to a direct connection with the master node, or at least a secondary connection, with only one routing node between them and the master node.
One would almost think there was a more intelligent algorithm handling placement of nodes in the networks, but it is a relatively simply methodology, with nodes contacting the master node first, then secondary (children) nodes, then children of those nodes, etc. in order to establish a connection to the mesh. Nodes will also contact their parent node periodically to verify connectivity to the mesh.

 

 Ping Response Times for a Connected Node

 The above picture shows ping response times for a connected Arduino node. At the left of the top graph one can see that the node was able to re-establish its connection with better connectivity to the master node, not having its traffic routed through another node. Response times vary, with the node typically responding in about 8mS, up to around 275ms in some cases when there is a direct connection to the master node.

"External Network Activity"

 

The above picture shows the results while "external activity" interferes with the network. During this time, due to the nature of radio and jamming techniques, most nodes on the network lose connectivity to the mesh, but the self-healing nature of the network ensures that they only go offline while the jamming is taking place. Nodes quickly re-converge around the master node as the mesh re-establishes itself, and every sensor or device resumes normal operations. 

HTTP based nodes reporting in during the jamming

 

I recently updated many of my nodes, due to bug fixes and testing, per the above graph. In it, as can be seen, some of my HTTP nodes lost connectivity during the jamming. Others stayed online with only small blips in connectivity. Depending on proximity to other nodes and the master node, as well as the location of the jamming device, different nodes are affected differently, but again, they all resume operation after the interference stops.

Temperature & Humidity Readings from a Connected Node

This system works extremely well, with a distant node reporting in its temperature and humidity over time. You can see in the chart above, it has been reporting in consistently.

 

Ping Statistics over a Number of Days

 In the above graph, you can see the success rate of pinging one of my sensor nodes over the course of a few days. In this picture, the success rate is down a bit from the norm, due to ongoing "external network activity" beyond my control. This has been the case for years and years, with one of my neighbors continuously interacting with my network on an almost daily basis. I can't understand the obsession, but I've witnessed it on many occasions.

Conclusion:

The end results kind of speak for themselves. In a given networking scenario where reliability and consistency is key, making use of the advanced features of TCP/IP networking in combination with HTTP & MQTT allows users to construct networks to suit many different scenarios. This system is highly reliable and can be customized to the nth degree. 

The combination of nRF24L01 & nRF52x radios works quite well, and allows even more advanced functionality in a given network. With the 52x devices supporting things like encryption, Bluetooth & I2S, users can make great use of these features, with everything being done wireless.

Since we are using a mesh based system, and TCP/IP was designed to operate even on partially damaged networks, the combination works very well. Add that to recent fixes and updates in the RF24 communication stack, and you have a system that works extremely well over long periods of time. I measure uptime on the network, and nodes tend to only go offline when there is a power outage, hardware failure or they are manually taken offline.

Considering that there are great quality nRF24 based radios available with range anywhere from 2.5-5 KM (E-01 ML01DP5 or 2G4M27D), networks can be constructed that cover wide distances and manage themselves automatically. The network will support the smallest of devices, ranging from ATTiny all the way up to Raspberry Pi and other Linux based devices that have GPIO & SPI capabilities.

Libraries & Software Used:

1.  RF24, RF24Network, RF24Mesh, RF24Ethernet (Arduino) & RF24Gateway (Linux/Raspberry Pi)

2. nrf_to_nrf radio library (nRF52x devices)

3. Node-Red (Linux/Raspberry Pi)

4. Mosquitto (MQTT broker)

 

All of the software and systems used are open source, so anybody can create similar systems. Any software that is MQTT compatible like Home Assistant or Node Red will work.

Maximizing Throughput on Linux Devices using the RF24 communication Stack

 Maximizing Throughput on Linux Devices using the RF24 communication Stack

How to achieve peak performance via TCP/IP connections

 With the recent changes to the core RF24 driver improving stability, I've begun more thorough investigation & testing regarding max throughput and speed on Linux devices.

 What I've found is that the typical TCP/IP stack for Linux is designed to increase throughput for high speed, bidirectional communication devices, whereas the RF24 Comm Stack is built around the RF24 and RF52 radio devices, which can only either send or receive at a given time. To work around this problem, users are encouraged to modify the TCP/IP window sizes. This allows the system to send smaller payloads, one at a time, thus increasing throughput and overall speed of communication when using the RF24Gateway layers for Linux to Linux device communication.

To configure the window sizes for RF24Gateway, just run the following commands:

sudo sysctl net.ipv4.tcp_wmem="1500 1500 1500"
sudo sysctl net.ipv4.tcp_rmem="1500 1500 1500"

 This sets the window sizes to 1500, which is right around the MAX_PAYLOAD_SIZE configured in the RF24Network layer for Linux devices.

 Results can be tested by running the following commands before and after making this change:

iperf3 -c 10.1.3.134 -4 -t 60 - For no window limit

iperf3 -c 10.1.3.134 -4 -t 60 -w 1500 - With a 1500 byte TCP/IP window

 These changes are only temporary, but users can edit /etc/sysctl.conf to make them permanent.

 Note: These changes can severely impact or disable other network systems & services. Users are advised to put these commands into a script, to enable/disable enhanced RF24 throughput.

 With the RF24Gateway ncurses interrupt example, I'm achieving speeds up to 150-175Kbps or 20-25KB/S over TCP/IP. Up to around 30KB/s using UDP. In order to do this, one needs to modify the line gw.poll(2); and change it to gw.poll(1);. This reduces the delay in the handling of data, allowing maximum throughput.

 

 IPerf3 results over TCP/IP

 

This works out to around 100-150Kbps on average or 12.5-19KB/s over TCP/IP, which includes the RF24Mesh node renewing its address periodically, interrupting communication slightly. 

 

IPerf3 results over UDP

 

The results with UDP (shown above) vary depending on what bit-rate you set, in this case 130Kbps or about 16.25KB/s was the chosen bit-rate.

 

 

More IPerf3 results over UDP

Here, a slightly higher speed, 145Kbps was chosen, however there was a slight bit of loss. 

(1/14500 Datagrams)

 

Summary: A Personal Experiment & A New Model for Open, Open-source, Anonymous & Secure Wireless Communication

Communication, Network, Protocol Design & Implementation Using Arduino & Raspberry Pi

Updated Nov 23, 2016

Over the course of the last few years, I have been working on a standardized, open-source communication stack that can be scaled from the biggest, most powerful devices, to tiny inexpensive devices like Arduino micro-controllers.

This looks familiar!

In the course of this time, I've made great progress in development of the core RF24 radio driver for nrf24l01+ radio modules. The overall goal of this project is independent of the radio devices, but these devices were chosen due to their relative capabilities vs cost, availability and ease of use. They are simply 2Mbps half-duplex wireless communication devices. Virtually any type of wireless devices can utilize the same theory of operation, including the mesh (RF24Mesh) style protocols.

What is it ? This system/communication stack allows nrf24l01+ radio/wireless devices to be utilized as a standard Network Interface Card (NIC) using TCP/IP & Mesh protocols. This is a summary of the experiment, system, general operation, and some of what I've learned.

Why? The programming, design and implementation of this system is part of a personal experiment I've been conducting involving the nature of wireless communication systems and protocols in relation to privacy, security, the Internet and the Internet of Things.

Theory of Operation & Design:

Consideration A: Open Networks + Anonymity + Automation

In general operation the basic principle of the communication stack is thus:
1. Each device utilizes a unique identifier while connected to a given access point.
2. This 'unique' identifier essentially represents an IP address in this implementation
3. For reasons of simplicity, this implementation also requires a 'static' identifier or IP address, but using more powerful micro-controllers and radio devices, the identifier can be assigned randomly or via DHCP
4. Devices simply attach, at any point in the network, using any available node(s) as an access point.

In a large scale model incorporating this type of design, with proper randomization of identifiers, individual devices and users are (potentially) indistinguishable from one-another initially and generally anonymous.

The next stage of course involves communication to enable the IoT/automation & allowing any capable device to connect. In this case there appear to be some options:

Option A: Leave the network open, and devices can choose to encrypt/authenticate their communications.This may require additional protective measures to prevent abuse in a large or publicly accessible system.

Option B: Assign selected devices a shared "network key" of sorts and incorporate encryption protocols into the network

Either way, it seems that wireless devices & users can inherently maintain their relative anonymity in a system that is designed a little differently.


Consideration B: Mesh Networks

The Network Layer: 
In studying the different layers involved in a capable communication stack, it seemed that many of the boundaries between layers were blurry, with many different systems 'breaking the rules' as it were. A good example of this is TCP/IP which does not strictly adhere to the OSI model.

In my design, a 'static' network architecture is used, which resembles a tree configuration. The simplicity of the RF24Network layer means that each node can handle routing of any type of information, and only needs to know which nodes it is directly connected to. 

Some of the features/functionality involving the RF24Mesh layer have been placed into the RF24Network layer directly. The reasons are mainly due to efficiency and limitations of small devices and this is under review.

Due to the limitations of the specific device used ( nrf24l01+ ) radio modules, each device is limited to 5 individual connections in the mesh, but this could be easily extended for use with more powerful radio devices.

The Mesh Layer:

The mesh layer utilizes its knowledge of the 'static' network architecture to allow individual nodes to attach themselves to the network at any given point where another node is active.

As mentioned above, each device is currently pre-assigned a unique identifier from 0-255 due to the 8-bit nature of very small & low power devices. In a large scale or commercial implementation, these identifiers can be assigned automatically & randomly, with devices then registering themselves in DNS if required, or simply acting as an anonymous client.

The fundamental benefit of a mesh style network is that nodes will automatically adjust their physical address and connection as required as they move around the network or lose connectivity. Any other nodes will provide connectivity by allowing other nodes to attach, and routing or accepting their traffic according to specifications.

Per RF24Network design, there is very little overhead for individual nodes, since they only need to keep track of which node they are connected to, and which nodes are using them as an access point.

Consideration C: Limitations

There are currently a number of fundamental imitations and odd 'quirks' that have to be dealt with due to the nature of really small devices like 16Mhz AVR devices.

AVR Devices: The chosen limitations in this case include some of the smallest and least power hungry devices available to any consumer. ATTiny devices can operate as low layer nodes & relays using RF24Mesh, and slightly faster 328 based AVR devices can run an actual IP stack.

uIP TCP/IP Stack:  One of main issues with small devices involves certain limitations of the IP stack. As soon as slightly larger devices are utilized, more robust IP stacks are available that make operation faster and more reliable. uIP has provided a very interesting learning opportunity and is quite amazing given what it can do with extremely limited resources.

Radio Devices: nrf24l01+ radio modules are max 2Mbps half-duplex communication devices with a specific 5-byte addressing scheme.


Consideration D: Fully Automated Networks

Software and wireless development capabilities have come to the point of enabling fully autonomous networks that essentially manage themselves, devoid of human interaction. This goal has been partly realized or demonstrated in theory and action with the RF24Mesh layer, and could easily be built upon and implemented by large scale commercial interests.

The concept of independent networks that configure and maintain themselves may seem to some as a wild idea or a bit of a pipe dream, but we have all of the tools and capabilities within our grasp. It is simply a matter of engineering and development.

Connectivity of individual users can be managed entirely via authenticated and encrypted connections, so there is clearly no need for individual identification of users or user management beyond the application layer.

This type of system could allow communities to simply put up hardware and provide access. The system would be automated to the point that if a device fails, nearby devices send out an alert and the equipment is replaced. That is all.

Consideration E: Wireless Technology and Potential Breakthroughs

Communications technology, aka the Internet, has become ubiquitous in many ways to the daily life of millions and millions of individuals and businesses. Unfortunately, this type of connectivity has not been available to consumers in poor, rural or remote areas in a real, usable way.

Even with existing satellite technologies showing great promise towards providing quality communications capabilities to new places, it seems that additional measures will need to be taken to provide for the age of automation. This is the kind of automation that will see real-time control scenarios, with vehicles communicating among themselves, drawing in information from traffic control systems, news, weather, and countless other systems.  This is the kind of automation that will see drone based grocery and package delivery, combined with, at some point, flying vehicles and high speed transportation systems.

The implementation of automation on a massive scale is going to require something better than the kind of service we get from existing cellular networks and current technology. This is going to require a lot more than what public cellular, 5G and fiber technologies can provide. What we need moving forward is not just a standard step forward in communications technology, but a giant leap forward, while keeping in mind all the things that make the internet and technology beneficial to everybody.


General Summary:

It now seems entirely possible and practical to design and implement fully featured, autonomous. large scale, wireless network architecture that fundamentally provides anonymity and promotes privacy among its users, while providing full mesh networking capabilities and automated network management.

With the emerging nature of communications devices, potential breakthroughs in communication technology, and wireless/IoT/automation technology emerging as a major component of our future progress, it seems very important to study, understand and develop networking and communication technologies with us, the users in mind.

It also seems that wired communications systems have already become a thing of the past, with more and more devices embedding chips and features to support Wireless and WiFi protocols and interfaces. The importance of privacy and security is fundamental to the operation of any large or public communication system, and any large, public & worthwhile communication platform will incorporate these basic ideals as inherent to its operation.


http://nrf24.github.io/RF24Ethernet

http://nrf24.github.io/RF24Gateway

Raspberry Pi/Linux with the nrf24l01+ & RF24Gateway

 A guide/example of advanced RF24/nrf24l01+ usage and monitoring on Linux devices

Overview:

There are many examples of basic RF24 usage, but not so many demonstrating complex communication scenarios and networking. The RF24 communication stack is free, with all development information, examples, code and documentation available on GitHub, designed with Arduino and RaspberryPi/Linux devices in mind.

The RF24 stack generally follows the OSI model, and provides a separate library for each layer, with efforts toward proper documentation for each layer. This allows any type of data/communication to pass over the radios, including TCP, UDP or any other protocol, and users can write their own code to link devices together using any layer(s).

So, what can we really do with these little radios?

The nature and low-cost of the system allows literally anybody with interest in wireless communication to setup, deploy, and experiment with small scale, self-healing wireless networks, or start writing their own code with the core RF24 driver, in a very short period of time.

The stack currently does not include sensor specific code or functionality, it is simply a communication stack which enables all kinds of different functionality.


Requirements, Hardware Setup & Installation:

1. A Raspberry Pi or Linux device a compatible radio device connected
    RPi Install Script

2. An Arduino device with a compatible radio device connected
    Arduino Install (Select RF24, RF24Network, RF24Mesh, RF24Ethernet libraries)

See the RF24 docs for hardware setup info & installation
Note: Please report any issues at https://github.com/TMRh20/RF24/issues

Demonstration and Information:

The video below demonstrates how to use RF24Gateway in general - Setting up nodes and communicating over RF24 using RF24Network, RF24Mesh, and standard protocols (ICMP, HTTP, SSH), along with monitoring

1. Config on RPi & Arduino devices
2. Complex mesh networks: How the different layers (RF24Network,RF24Mesh & RF24Ethernet) can be used with RF24Gateway
3. RF24Ethernet: Simple TCP/IP communication and web server running on Arduino over RF24
4. RF24Mesh: Node config & interaction with RF24Gateway
5. RF24Network: Node config & interaction with RF24Gateway
6. How to use WireShark to monitor user payloads with RF24Gateway and LUA Wireshark dissector script (using laptop w/SSH & X11 forwarding in this example)
7. How to configure Raspberry Pi as a standard node in an existing RF24Mesh
8. How to sniff RF24Network, RF24Mesh, and RF24Ethernet traffic using RF24Gateway & Wireshark(optional)
9. Tacos... oops they were already eaten, maybe next time

As shown in the video, any nodes running RF24Ethernet, RF24Mesh or RF24Network can be used with a master node running RF24Gateway, allowing users to try out or utilize the different layers quite easily.

RF24Ethernet: Provides TCP/IP communication. Very simple and reliable way to control on/off devices via a web browser or scripting. Has all functionality of lower layers.

RF24Mesh: Provides automatic (mesh) networking for RF24Network. RF24Mesh networks are self-healing and dynamic in nature.

RF24Network: Provides addressing, routing, fragmentation/reassembly of payloads via manual/statically defined wireless networks.

See the detailed overview and related pages at http://nrf24.github.io/RF24Ethernet

RF24Gateway - Creating Hybrid IoT Networks using TCP/IP and RF24Network

 RF24Gateway User Interface Example

RF24Gateway is a new complimentary library to RF24Ethernet, designed for RPi and Linux based devices with SPI and GPIO capabilities (BBB, Intel Edison, Galileo, etc).

The RF24Gateway library acts as a network/internet gateway, and allows Arduino/AVR based sensor nodes to interact with each other, the environment, and the world using TCP/IP and/or lower level RF24Network messages.

In short, it allows hybrid IoT networks to be created using Arduino and nrf24l01 modules. Some nodes are capable of internet or local network communication using standard protocols (TCP/IP) and/or RF24Network messages, while other nodes can communicate only at the network level, using only the lower layer RF24Network messaging.

Once RF24Gateway is configured, nodes can be controlled completely using standardized tools and protocols. Additional client examples have been added to demonstrate how to control and/or receive information from sensor nodes using standard tools and scripting methods via Bash, NodeJS and Python scripts. Any method of establishing a TCP connection can be used, with the examples generally using HTTP to some degree as well.

Libraries and code that utilizes RF24Network can utilize RF24Gateway in much the same way, since external data types (TCP/IP) are handled out of sight from the user. RF24Network is used exactly the same way, except that the gateway.update() function is called instead of network.update() or mesh.update(). The included examples demonstrate usage.

Note: RF24Gateway defaults to using a TUN interface with RF24Mesh enabled. RF24Ethernet examples have been updated to accommodate this.

Current Status:

April 2015: Near completion. Integrated fully with RF24Ethernet. Graphical (NCurses) interface example finalized.

Source Code:
https://github.com/TMRh20/RF24Gateway

Documentation:
http://nrf24.github.io/RF24Gateway

Download (See documentation for installation info):
https://github.com/TMRh20/RF24Gateway/archive/master.zip

ENC28J60 Network Modules, Arduino and the Internet of Things (IoT):

An experiment, overview and analysis of TCP/IP networking with Arduino Ethernet

The module in question was sent to me by the friendly people at ICStation.com , in exchange for a review of the module itself.

Video Review:

ENC28J60 Mini Network Module:

• 3.3V operation
• Create a mini Arduino Web Server or Client
• Plugs into your home router or modem just like a computer

The uIP TCP/IP stack is software driven, while the ENC28J60 module handles the data mainly at the network (Ethernet) level. One main difference between UIPEthernet and the official Arduino Ethernet library is the need to keep the TCP/IP stack updated via software with the ENC28J60.

Hardware: 

The ENC28J60 was very easy to wire up, using the default SPI pins shown at http://arduino.cc/en/Reference/SPI along with the VCC and Ground pins.

The device seems to work ok connected to the 3.3v output of the Arduino, but it seems best to use an external power supply or battery.

Software:

Two main libraries appear to be available: Ethercard and the UIPEthernet libraries. This post will focus on usage via UIPEthernet, since it shares a common API with the official Arduino Ethernet library.

I am using the UIPEthernet library and examples found at: https://github.com/TMRh20/arduino_uip/tree/fix_errata12

In attempting to utilize the modules and UIPEthernet for creating a simple webserver, I ran into a number of issues with data being sent or received incorrectly. Over a number of days, and benefiting from the information gained in building the RF24Ethernet library, I have been working to improve UIPEthernet, and also created some examples to demonstrate the issues and current usage.

(A pull request has been made regarding these suggested changes to the main library)

Overview of Library Changes (At the time of this post):
Changes:

• Introduce client.waitAvailable(); function  - Ensures the IP stack is processing incoming data while waiting for incoming data for a maximum defined duration in milliseconds: client.waitAvailable(3000);
• Introduce Ethernet.update(); function - Allows user applications to ensure the IP stack is processing data while delaying or performing other tasks: Ethernet.update();
• Perform additional processing of data while calling client.available();
• Per testing, reduce max segment size (MSS) and the receive window to 511 bytes from 512 due to unidentified issue.
• Reduce number of packets per socket to prevent data loss
• Re-open the TCP window at timed intervals when data has not been sent or received on an open connection to prevent stalling during large downloads
• Fix for outgoing data corruption
• Force application to wait while writing data 
• Better TCP window handling
• Other minor changes, add examples

Changes - Technical Summary:

The first changes introduced added some new functionality, geared towards a software driven IP stack.  With the addition of additional processing and the Ethernet.update() function, the IP stack is more responsive to ICMP requests etc.

The new examples included with my branch of the library clearly demonstrate some of the issues I encountered when attempting to create a simple web client and/or web server. The first issue I noticed involved the device crashing or stalling, and then failing to reconnect or respond to pings. Testing indicated that the hardware itself was still responding and receving packets, so the issue seemed to be software based. These issues mostly seem to be addressed in the fix_errata12 branch, but testing via large dowloads from a web server indicated there were still some remaining issues, mostly surrounding data corruption and flow control.

Most of the remaining changes attempt to address issues with corrupted or out of sequence data, and to manage the flow of data better. In some situations, for example, web servers will send TCP payloads equal in size to 1/2 of the TCP Max Segment Size (MSS) which causes problems with flow control. The IP stack will now only close the TCP window if data is being received fast enought to do so, which appears to reduce issues with these transfers. The main issues come when a sending device sends only 1/2 MSS, and the recipient closes the window. The library will now count individual bytes in memory vs packets with data in them, and only close the TCP window if the assigned data buffers are completely full.

Testing has been very successful, with the device demonstrating full functionality for extended periods of time, downloading 70+MB of data without any errors detected in the data or hardware. The new web server examples and HttpClientTest.ino example seem to be good displays of the current capabilities and error free operation, and can be used to demonstrate the issues, when used with the original library code, or when changes are reverted.

Summary:
 
The mini ENC28J60 modules are an inexpensive and effective way to add internet connectivity to any Arduino project, and their size make them very suitable for use with Arduino.

I found the hardware itself to be very reliable and easy to use, although there seemed to be a few minor issues with the related Arduino library (UIPEthernet) that was tested.
As noted, I have submitted some workarounds/fixes for review and possible integration into the main library at the time of this post, so users are encouraged to use the library linked above.

ICStation.com:

A big thank you to www.ICStation.com for sending me this product, as well as being very patient.

It has been a number of months since I actually received it, but their small contribution has lead to the creation of the RF24Ethernet TCP/IP Radio Network library, and some potential improvements to the UIPEthernet library. A wealth of Arduino and Raspberry Pi related products, gizmos and devices are available on their website, and their staff seem very friendly.