nRF24l01 radios: Long Term Stability Issues & Error Handling Enhancements
Creating robust, reliable networks
We recently identified and addressed some long-term reliability issues affecting the RF24 radio library for nRF24l01 radios. The issues affect any users that make use of the write() or txStandBy() functions of the library. About a year ago, we found some issues with the radios returning "impossible" values, with one register indicating a payload is available, another saying there is not, and another saying the payload length is 0. This and similar issues appear to have affected the radio in transmitting mode as well, a fact we missed upon initial review of the previous issue. Now, a year later, we've addressed issues affecting transmission as well.
As of this writing, we are just completing testing and the changes have been pushed to the master branch of RF24. A new release should be available in a few days (version higher than 1.5.0) for the Arduino and PlatformIO platforms. For now, users can install from ZIP or clone the RF24 repo directly from the master branch.
This should be the final fix for stability in the RF24 platform, as we seem to have identified and implemented handling for all failure modes now.
Setting up direct TCP/IP connectivity between Arduinos using a nRF24 or nRF52 radio link w/RF24Ethernet
Utilizing the new functionality of the RF24Ethernet library
With some recent experimentation and prototyping involving the lwIP IP Stack, I was able to modify the RF24Ethenet library to function standalone, without the need for a Linux/Raspberry Pi device running RF24Gateway. This allows users to directly connect multiple Arduino devices using the RF24Ethernet library, utilizing TCP or UDP protocols to communicate between devices.
The RF24Ethernet library API is based on the official Arduino Ethernet API, so users utilize the same coding style to communicate over nRF24 or nRF52 radio links.
Setting things up:
1. The first thing to do is verify you have working radios. With nRF52 devices, they are built-in so, there is not much to worry about, but with nRF24 radios, users need to keep in mind power supply and wiring issues, so testing using the official gettingStarted sketch included with the RF24 library is recommended before attempting this.
2. Install RF24Mesh and its dependencies from Arduino Library Manager
3. Install the RF24Ethernet library from ZIP from https://github.com/nRF24/RF24Ethernet/tree/lwIP Note: Once deployed, the updated library will be available from the Arduino Library Manager. Users may need to uninstall/reinstall to get the latest updates at that time.
4. Install the Arduino lwIP library using the Arduino Library Manager as required for non-ESP32 & non-ESP8266 devices which already include lwIP.
5. Run the included examples from the Headless directory in the RF24Ethernet examples on two devices. I've tested so far on Arduino Due, ESP8266, ESP32 and nRF52 based devices. The RPi Pico still utilizes the uIP stack due to technical issues using lwIP with the Arduino MBED based core.
What to expect:
The main server example sets up a RF24Mesh 'Master Node' which handles addressing and address look-ups for all other nodes. Nodes not in range of the Master Node will attempt to connect automatically via routing traffic through other connected nodes.
The client examples simply connect to the Master Node and request an HTML based web-page.
In a real life deployment this activity may be reversed. With the Master Node running a modified Client example, and sensor or other nodes running modified Server examples, the Master Node could then connect to each device in turn and retrieve data as required.
To designate a master node, simply call the following before calling mesh.begin()
mesh.setNodeID(0);
then from the main loop()
mesh.DHCP();
Things to Note:
RF24Ethernet makes use of two separate IP Stacks, the older, unmaintained uIP Stack works on smaller devices like Uno, Nano, Mega, etc, while the newer, lwIP stack is used automatically for devices >50MHz CPU speed, including the Arduino Due, ESP32, ESP8266, and nRF52 based devices using the nrf_to_nrf radio library.
To ensure you are using the lwIP stack, users can #define USE_LWIP 1 & #define RF24ETHERNET_USE_UDP 1 in the RF24Ethernet.h file or prior to compilation.
In testing out the RF24 library and related features, I managed to get a few different things working together on the Arduino Uno Q to provide a more advanced example of RF24 interaction.
The main thing the sketch does is transfer data over RF24 radio link along with a standard Arduino device running the RF24 GettingStarted sketch included with the RF24 library.
Then, any data received is displayed in the Serial Monitor, Python Console and an associated webpage to demonstrate usage of incoming data etc.
A button is included on the webpage that allows users to toggle the LED matrix on/off, which demonstrates how to pass data from Webpage to Python to Microcontroller.
All in all its a simple concept, but it did take some figuring to work out how the functions and stuff all relate through JavaScript, HTML, Python, C++ etc.
Trying out the Arduino Uno Q - A quick and dirty overview
Love it or hate it, it... exists
With the release of the new Arduino Uno Q board, I decided to get one and try it out. After a few days of playing around and experimentation I have to say I'm less than impressed.
The concept seems to be a solid one, combining a standard Micro-Processor Unit (MPU) with a Microcontroller Unit (MCU), but the execution has been a little rough.
The first and main thing to probably address is the Bridge concept, which is a way for users to make the MCU 'talk' to the MPU. It is driven by a serial connection running at 115200 baud, using RPC, which altogether is far too slow to pass copious amounts of data, or do it really quickly. On the other hand, it can be used to pass information between programs, be it Python to Python or C++ (Arduino) to Python.
One of the first things I tried, of course, is running RF24 radio based applications. It appears to work OK initially with the core RF24 driver and RF24Network, but once I got into the RF24Mesh layers, I started to notice problems. I thought it was basic speed issues, but it turned out to be a radio power supply related issue it seems. After changing out the radio and calling 'radio.setPALevel(RF24_PA_MIN,0);', it worked.
Next, I attempted to create a basic TUN network interface in Linux using Python, to see about creating a RF24Gateway like interaction, but realized quickly that this too was outside the general capabilities of the device. It runs a Python virtual environment within a Docker container, with no apparent way to configure things, so I can't seem to get it working. That means an RF24Gateway style library is out of the question for now.
It seems that so much of what I've tried do to so far just doesn't work. There was also a forced, breaking update that happened recently, to version 3.0, where users have to manually download a new version of App Lab in order to program their devices.
One essential thing to note, is that when the App Lab stops responding, or the Serial/Monitor output gets corrupted, I need to close App Lab, and run `arduino-app-cli system cleanup` from the command line to get it working again.
Some of the features still seem promising, the ability to easily incorporate "Bricks" and features into your programs, and the underlying software for the MPU is Debian/Linux along with Arduino, so you can still do a lot with it.
I don't think I'll get another one, but I will definitely be playing around with the one I've got for a while.
In the meantime, I've included a new example in the RF24 core library that users can try with the Arduino Uno Q that allows users to configure the radio and print debug information.
I've also added an example sketch to go along with the stock RF24 GettingStarted example here:
The nrf_to_nrf Radio Library for nRF52x Devices and nRF24L01 Compatible Communication.
A brief tutorial on payload sizes & handling
Max Payload Sizes: The nrf_to_nrf Core Library
The nrf_to_nrf library allows many configuration options, but here I want to talk about payload sizes and communication using the core nrf_to_nrf library, as well as that in combination with RF24Network and higher libraries.
The max payload size of the nRF52840s is 127 bytes. This is with CRC disabled and static payload sizes. The radios support 0, 8, 16 or 24-bit CRC, so that extra data takes up 0, 1, 2 or 3 bytes depending on your chosen options. nRF24 devices only support 0, 8 or 16-bit CRC, so users are limited in their selection when communicating with nRF24 devices.
Similarly, the nRF24 will only handle 32-bytes maximum payload size, with any CRC value and either static or dynamic payloads. The nRF52x devices are slightly different, so one needs to be aware how changing these options affect the max payload size.
The above is with the core nrf_to_nrf driver. Once we get into libraries like the RF24Network library, another 8-bytes of data are required for the network header, which contains information like the destination address, sender address, and a counter.
This means that when communicating solely between nRF52 devices, we almost always want to configure the max payload size to 123, since RF24Network will figure out the max payload size based on this information, and fragment payloads accordingly. By default RF24Network is nRF24 compatible, so only is configured to handle 32-byte payloads.
To configure RF24Network, before calling network.begin(); we want to call radio.begin(); followed by radio.enableDynamicPayloads(123); This will allow maximum payload sizes prior to fragmentation.
The same is true when using the RF24Mesh library, call radio.begin(); followed by radio.enableDynamicPayloads(123); prior to calling mesh.begin();
If using encryption, the RF24Network layer will take this into account, so the max payload sizes with RF24Network are as follows:
Max Payload Sizes in RF24Network:
1. RF24Network, default: 24 bytes before fragmentation, 144 bytes with fragmentation
2. RF24Network, w/enableDynamicPayloads(123): 115 bytes before fragmentation, 144 bytes with fragmentation
3. RF24Network, w/enableDynamicPayloads(123) + enableEncryption = true: 103 bytes before fragmentation, 144 bytes with fragmentation
Maximizing Throughput on Linux Devices using the RF24 communication Stack
How to achieve peak performance via TCP/IP connections
With the recent changes to the core RF24 driver improving stability, I've begun more thorough investigation & testing regarding max throughput and speed on Linux devices.
What I've found is that the typical TCP/IP stack for Linux is designed to increase throughput for high speed, bidirectional communication devices, whereas the RF24 Comm Stack is built around the RF24 and RF52 radio devices, which can only either send or receive at a given time. To work around this problem, users are encouraged to modify the TCP/IP window sizes. This allows the system to send smaller payloads, one at a time, thus increasing throughput and overall speed of communication when using the RF24Gateway layers for Linux to Linux device communication.
To configure the window sizes for RF24Gateway, just run the following commands:
This sets the window sizes to 1500, which is right around the MAX_PAYLOAD_SIZE configured in the RF24Network layer for Linux devices.
Results can be tested by running the following commands before and after making this change:
iperf3 -c 10.1.3.134 -4 -t 60 - For no window limit
iperf3 -c 10.1.3.134 -4 -t 60 -w 1500 - With a 1500 byte TCP/IP window
These changes are only temporary, but users can edit /etc/sysctl.conf to make them permanent.
Note: These changes can severely impact or disable other network systems & services. Users are advised to put these commands into a script, to enable/disable enhanced RF24 throughput.
With the RF24Gateway ncurses interrupt example, I'm achieving speeds up to 150-175Kbps or 20-25KB/S over TCP/IP. Up to around 30KB/s using UDP. In order to do this, one needs to modify the line gw.poll(2); and change it to gw.poll(1);. This reduces the delay in the handling of data, allowing maximum throughput.
IPerf3 results over TCP/IP
This works out to around 100-150Kbps on average or 12.5-19KB/s over TCP/IP,
which includes the RF24Mesh node renewing its address periodically,
interrupting communication slightly.
IPerf3 results over UDP
The
results with UDP (shown above) vary depending on what bit-rate you set,
in this case 130Kbps or about 16.25KB/s was the chosen bit-rate.
More IPerf3 results over UDP
Here, a slightly higher speed, 145Kbps was chosen, however there was a slight bit of loss.
So after all this time developing and maintaining the RF24 core library, I found yet another bug affecting the Auto-Acknowledgement functionality of the radios. An issue had long ago been identified and fixed regarding pipe 0, where the assigned reading address would be overwritten when transmitting, since the radios use pipe 0 exclusively for transmission. This affects the RX_ADDR_P0 register.
Somehow, we never realized that the receiving address, when written to the radios after switching from TX to RX mode would interfere with the reception of Acknowledgement packets in TX mode, since it would overwrite the RX_ADDR_P0 register on the switch to RX. Now the radio library caches BOTH the RX & TX address for RX_ADDR_P0, and writes to it when switching between modes as required.
This enables full functionality of the radios on all pipes, since previously, auto-ack would not work properly on pipe 0 in some cases. The changes have been committed to the source code and will be included in the next release, after v1.4.11.
There are minor impacts to throughput, but after careful consideration, these changes were included to fully enable the radios capabilities. We are working on a more efficient resolution.
This bug was mainly discovered due to my work on the nrf_to_nrf driver for nRF52x radios, which I already had caching both the TX & RX addresses. I realized the RF24 driver didn't do that.
***
I also found a bug that appeared to affect SPI functionality on Linux devices, but it turns out it affects all devices using the RF24 core driver with Dynamic Payloads. This includes the entire RF24 Comm Stack.
I first thought it affected SPI, assumed I was getting bad data, but nothing that I tried to adjust worked. Eventually I narrowed it down to the available() function consistently returning true. Once that was identified, I then discovered it had to do with 0 length payloads, and also discovered that doing a radio.read() had no effect when this happens, the RX buffers need to be flushed. So that's what is being implemented in the RF24 Core layer. When using Dynamic Payloads, and the payload size returns either >32 or 0, the buffers need to be flushed, so the RF24 layer will now do that for both sizes.
After all this time and searching for the problem, it was one line of code that had to be changed.
I'd included failure handling in all of the RF24Gateway examples due to this bug, which would intermittently cause the radios to become unresponsive, requiring them to be restarted/reconfigured. I searched high and low for a long time to find it, but at one point I thought it came down to an issue with the network layer not being able to process information fast enough.
The current approach is for the update() function in RF24Network to return a new system
type, indicating there has been corruption, and the RX buffers have
been flushed. The core RF24 layer will simply return 0 for Dynamic Payload Length.
This is being patched, will be available in the source code very soon, available in the next release. The Linux installer downloads from the source code.
RF24Gateway now displays a count of corrupted payloads
With these changes, I've begun keeping track of how often this happens. On faster devices like RP2040 or Raspberry Pi, it seems to be more prevalent. Of course, it happens way more on a device that is doing more reception than transmitting. In the picture above, the network has detected and flushed 32 corrupt payloads over a short period of time. This was how I was able to replicate the bug, by utilizing RF24Gateway as a testing tool, and hammering it with data from another RPi and from Arduino. Over time, the bug and its workings became clearer, so I was able to narrow it down to the network.update() function in RF24Network, and then further down the stack to the getDynamicPayloadSize() function of the RF24 core library.
I am also now logging the data on Arduino devices, via MQTT and NodeRed. I'm testing on an Arduino Nano, Due and RP2040, to see just how often this affects slower devices. As of writing this, no data is available yet, but I am running the tests long-term, so data will come in eventually, and will report back here, on this blog post.
Update: The issue affected the RP2040 and AVR devices doing standard communication in my "production" environment after a few days. I've filed a ticket with Nordic in hopes of identifying if this is a known issue, new issue or other.
So we've been putting in a bunch of work lately on the RF24/NRF52x communication stack for Arduino, and surprisingly, given the age of the libraries, we were able to make a number of changes that significantly impact the performance of the network and mesh.
RF24Mesh:
1. Utilize all poll requests.
The prior behavior of the mesh was to send out a poll request via multicast and gather the responses. Then the mesh layer would attempt to request an address via one of the nodes that responded to the poll. If this failed however, the node would continue on and send out another poll. The current behavior now utilizes up to 4 received poll responses and loops through them all until either failure or a successful address request is made before moving to the next multicast level and sending out another poll request.
2. Enhance the ability to verify connectivity
The prior behavior of the mesh was to call `getAddress(_nodeID);` and verify its own address with the master node. The call would fail if anything but a successful address response was received. Now there is some logic added to return immediately if the address is not found in the master list or is default. It will also return immediately if the nodes address has been zeroed out. Retries will take place if the request failed or timed out. Finally, the node actually verifies that the received address matches its own address.
Update: I am currently testing a new method of connection checking, where each node only contacts their parent node instead of the master node. This appears to reduce traffic, increase connection times, and improve the overall stability of the mesh.
It may not seem like much, but the changes should tend to reduce network traffic, while providing more seamless mesh operation.
RF24Network:
1. Fix error sending data
Per the RF24 documentation and behavior, when using some of the advanced write functions, there is a need to call `radio.txStandby()` if a write fails, otherwise the radio will essentially hang and all subsequent writes will fail. This was partially missed in RF24Network, so a call has been added when an auto-ack payload fails. This should improve reliability and consistency in sending data.
RF24:
1. Interrupt enhancements.
The use of interrupts on Linux platforms like the Raspberry Pi has been greatly improved and no longer requires root access to operate. Unlike the RF24 library, the interrupt functionality uses GPIO pin numbers instead of BCM pin numbers. ie: Use GPIO 24 instead of pin 18.
RF24Ethernet
1. Fix re-transmit mechanism
The associated UIP IP stack incorporated with RF24Ethernet uses a retransmit mechanism when acks are not received etc, but it was not working. This is now fixed and results in much more reliable, longer connectivity with MQTT, HTTP and other protocols & applications that transmit data.
All in all its been a good year for the RF24/NRF52x communication stack, with support also recently added for Nordics NRF52x SOCs, there are so many potential ways these libraries can be configured! Encryption and authentication features are also available with the newer radio modules, which will easily handle audio streaming and more advanced functionality.
I've been playing around with the NRF52840 boards I have, and now that I have basic audio functionality working via PDM microphone and PWM audio output, the next thing to do is test out an encrypted wireless communication scenario, so that's what I've done.
The devices use the AutoAnalogAudio and nrf_to_nrf libraries created by me to communicate wirelessly. I've also enabled encryption at the nrf_to_nrf layer, and the related code is an example of implementation, with the device randomly generating a new key and re-keying every 30 seconds. This will prevent others from decrypting the payloads or performing replay attacks outside the 30-second windows.
This means that users would need to start the devices at the same time (within a 30-second window), so they start-up using the same key. After 30-seconds, a new key is generated and configured, so both devices will then use the same new key. The receiving device ensures that the old key matches before accepting the new key, so unauthorized users can't simply send a new key or replay an old key. Any power down of either device will result in lost communication unless the other device is restarted within the same 30-second window.
To use the sketches (posted lower down), users would need a XIAO 52840 Sense board and both of the aforementioned libraries installed, then just run the code. No wires to connect, no power supply issues to worry about, no counterfeit nrf24 devices to worry about, just a straightforward device that works.
There are still a few issues with synchronization of the audio vs how the device handles PWM data. Essentially, you can have the PWM pin default to a LOW or HIGH state when the current PWM sequence ends, which happens during radio communication when packets are lost or slow to transmit. Compared to other devices I've worked with, which maintain the current state of the pin when PWM or DAC data ends, this device will revert the pin to either HIGH or LOW which causes a popping or clicking sound.
Comes with preconfigured devices, just plug and play!
I've finally decided to put together some electronics kits with preconfigured devices, using the RF24 communication stack. As the system is now very stable and reliable, I think it is about time I started putting devices together and selling them to support future development and designs.
The current kit will come with a RPi4B and all the attachments like HDMI cable, a case, and power supply along with 3 Arduino Uno R3, 2 temperature & humidity sensors and 1 RBG LED ring, plus of course the RF24 radios. I've also designed some custom PCB shields for everything to just plug into, with power for the radios and extra capacitors, so its super simple to put it all together.
The setup uses Node-Red and RF24Gateway to provide a mesh network that seamlessly links all your devices together, and allows you to add custom devices to the network and/or expand the network by purchasing more devices or setting them up yourself with the Arduino platform.
If they sell, I intend to design more devices to go with the currently available products.
What are the different layers and which one do I use?
Over the past number of years, I've been developing a communication stack for nrf24l01+ radios, generally to be used with Arduino and Raspberry Pi devices. The comm. stack is organized into separate layers, per the OSI model, which allows users to establish communication between any number of devices depending on their needs.
1.The RF24 layer: This layer is generally used for simple device-to-device communication, where speed and simplicity are key. One example is for radio control of a single device, where a stream of small data packets are communicated from one device directly to another in rapid succession. Streaming of audio in real-time is another example where the RF24 layer would be used directly. Using the RF24 layer can be a bit complex, and some knowledge of radio communication is beneficial.
2. The RF24Network layer: This layer expands on the RF24 layer by providing a number of features, both to enhance radio communication and provide users with the features of an OSI layer 3 (network) layer. RF24Network nodes are arranged in a static tree topology, and this layer provides all the features to manage them. RF24Network handles routing, fragmentation/reassembly of large packets, and communication to any other device is straight-forward. Users don't require knowledge of the RF24 layer to operate a network of nodes at the RF24Network layer, and it can be much simpler than using the RF24 layer directly. Recommended when not able to use the RF24Mesh library, when there is a need for static nodes, or to simplify one-to-one communication.
3. The RF24Mesh layer: This layer expands on all underlying layers by providing an automated, self-healing network that allows nodes to move around and or re-establish connectivity as required. This layer is generally recommended when creating a network of devices, as it automates addressing for RF24Network, and provides a more seamless interaction. Users generally communicate using the RF24Network API, with RF24Mesh providing address lookups and automation for the network.
4. The RF24Ethernet/RF24Gateway layer: This layer allows communication using standard networking protocols, such as TCP, UDP, ICMP. It is generally recommended to run RF24Gateway on a Raspberry Pi as the master node, with smaller devices running RF24Ethernet. Allows very simple or very complex communication scenarios, with users requiring little to no knowledge of radio or RF24 programming APIs.
The RF24Ethernet API is very similar to the Arduino Ethernet API, and RF24Gateway allows users to use standard networking tools to communicate with devices running RF24Ethernet. Recommended for communication scenarios where speed is not essential, but reliability and consistency is, such as a home automation system using MQTT and Node-Red and/or a sensor network reporting data. Users can run a wireless, Arduino based webserver, or interact with nodes using their mobile device over MQTT, HTTP, etc.
As can be demonstrated, each layer has its place, providing communication capabilities for a very wide range of scenarios, and allowing users to benefit from the built-in features or to dig right in and customize things to the nth degree. Generally, the higher you go up in the stack, the simpler the interaction, with complex underlying code providing the simplest and most advanced user interaction.
A simple guide to managing memory and variables in C++
One of the most common issues that seems to be coming up more and more recently relates to transferring data between devices and how different devices store data in memory. With more devices using 32 and 64 bit processors, integrating things between the newer devices and 8-bit MCUs can require a little bit of forethought and planning.
Memory:
What it all comes down to is memory and how different devices store data in memory. This is very important to understand when transferring data between systems and ensuring that code is portable between various devices.
When using an 8-bit device, such as an ATMega 328 based MCU (Arduino Uno, Nano, etc) the smallest unit for storage in memory is 8bits or 1-byte. This means that even a boolean (true/false) variable will take up 8-bits of memory space. On a 32 bit system, (ESP32, ESP8266) the same boolean value will use 32-bits of memory space. Why does this matter? When transferring data between systems or creating code, certain variables will take up different amounts of space in memory depending on the architecture (8-bit, 32-bit etc) thus introducing inconsistencies and misinterpretation of data.
Examples:
char is equivalent to int8_t byte is equivalent to uint8_t int is equivalent to int16_t on an Arduino Uno (8-bit device) and int32_t on ESP32 (32-bit device) unsigned int is equivalent to uint16_t on Arduino Uno (8-bit device) and uint32_t on ESP32 (32-bit device)
Using variables like int, long etc can cause confusion because they are interpreted differently on different devices. Sending a 2-byte integer from an 8-bit system to a 32-bit system works fine, as long as the size of the variable is specified on both systems such as int16_t, rather than trying to use 'int' on both systems.
Take for example the following data structure:
struct myStruct{
bool variable1;
uint32_t var2;
uint8_t var3;
int16_t var4;
}
On an 8-bit system, this data structure will use 8 bytes of memory space, but on a 32-bit system it will take up 12-bytes. This is because the minimum unit for storage on a 32-bit system is 32-bits, so 32-bit devices will need to add padding wherever the data does not align in chunks of 4-bytes (32-bits). When using data structures, devices will typically attempt to pack the data as efficiently as possible, so re-arranging the data structure properly will prevent the 32-bit system from adding padding and making the data structures incompatible.
struct myStruct{
uint32_t var2;
int16_t var4;
uint8_t var3;
bool variable1;
}
In the above example, the data is arranged in chunks of 4-bytes, starting with a 32-bit unsigned integer (4-bytes), followed by a 16-bit signed integer (2-bytes), an unsigned 8-bit integer (1-byte) and a boolean variable (1-byte), so the 32-bit system will pack the data into memory exactly the same as the 8-bit system, using only 8-bytes of memory space.
This is important, because when sending data between devices, users can just send the data structure as-is and if packed properly, the data will align regardless of whether it is a 32-bit, 64-bit or 8-bit system.
When developing code, specifying the size of all your variables and aligning data properly in your data structures will ensure that the code and communication is portable between devices.
RF24Mesh - Dynamic Mesh Networking for NRF24L01+ Radio Modules
An Overview of Progress so Far
Overview:
RF24Mesh is a C++/Arduino/Linux/Raspberry Pi library allowing many devices to simultaneously connect wirelessly and remain connected in a mesh style network using nrf24l01+ radio modules. Nodes can communicate with any other node on the network, including a master node that acts as a gateway for the mesh, providing addressing and lookups.
Once assigned a unique ID, everything is automated, and nodes can move around and rejoin the mesh at any point where another node is in range. RF24Mesh supports some of the smallest and most power efficient devices available.
The RF24 Communication Stack is based on the OSI model, and RF24Mesh makes use of the features of the lower layers (RF24 (Layer 2) , RF24Network (Layer 3), and the overall capabilities can be extended via RF24Ethernet (Layers 4&5 TCP/IP)
Setting it Up:
The main thing when setting up a mesh network is to configure and test your devices accordingly, to make sure they are in a fully operational state. Most if not all of the available radio chips being sold are clones, so I've found it necessary to modify the radio devices slightly in some cases. In many cases, a capacitor (10-100uF) is added directly to the VCC/GND of the module itself to help with common power supply issues.
In the case of these modules, I've found that scraping down the last bit of the antenna, and soldering on a small 3-4" piece of wire greatly improves their performance:
In the case of the high powered PA + LNA modules, I've found it necessary to add shielding as in the post found here. Placing them close to a ground plane seems to help as well. Operating them at a lower power level may be required if the power supply is not adequate.
The modules that are low powered with an antenna generally work out of the box.
I generally recommend using an adapter, to provide a stable power source, even for the low powered modules. I've been using them with many of my modules.
From there it is a matter of installing the Arduino libraries etc. which is covered in the documentation and in earlier blog posts and the official documentation below.
How it works:
RF24Mesh leverages the features provided in the supporting libraries, RF24 and RF24Network to provide connectivity between many devices, even if they are far away from the central node or moving around. Each node is assigned a unique identifier (1-255) and uses that high level-address to request a dynamic address at the network layer, according to the structure of the mesh and proximity to other devices in the mesh. Nodes typically communicate using the RF24Network API, with RF24Mesh working at a higher layer to provide address lookups, dynamic addressing and connectivity.
Nodes arrange themselves using a tree topology around the master node, which provides address lookups etc. for the mesh. At any given time, each node has a path of communication through the network, and can re-establish and verify a new link if the current path fails. In each network there is the master node (the base node), children of the master node (relay nodes) with children nodes themselves, and leaf nodes, (children of the master or its children) with no child nodes attached.
The protocol is fairly simple for each node. On power up or reset, nodes attempt to find a connection by using multicast. Multicasting is arranged into 5 levels, so the master node is contacted first, then its direct children, then their direct children and so on, making the way nodes attach themselves to the network tend towards close proximity to the master node in a tree-like structure. Once contact is made the node requests an address from the master node, either directly or through other nodes. Once an address has been assigned and verified, the node is considered connected to the mesh and can then communicate with all other nodes in the mesh, and beyond to other systems etc.
Nodes have the option of managing their connection however desired, and quite often, a simple timer is suitable to verify connectivity after a set period of time. Transmitting nodes can also detect when communication is failing and re-establish their connectivity to the mesh as required. That is, at any given time, nodes will have a single path to communicate over the network, but if that link fails, they can re-establish their communication path through the network.
The communication layers are interoperable, meaning a network can combine devices running RF24Network and RF24Mesh libraries together, with static nodes running RF24Network, and dynamic nodes running RF24Mesh. Devices running RF24Ethernet (TCP/IP) can also communicate using the other layers of the libraries, interacting directly with RF24Mesh and/or RF24Network nodes. This allows an array of devices to operate on the same wireless mesh network, from ATTiny all the way to the more powerful devices like ESP32 and Raspberry Pi, using the RF24Network, RF24Mesh APIs and/or standard networking tools and protocols. (ICMP/TCP/UDP, etc.)
Current status, testing etc.
RF24Mesh is now past the development/beta testing stage and considered (by me) to be stable and well tested. The mesh can handle a large number of nodes, theoretically up to 255, but in practice, more like 15-25 nodes sending or receiving data every few seconds is a reasonable bet. It generally depends on the type of traffic and frequency with which it is generated, as a much larger number of nodes reporting data every minute presents only a small load for the network.
The progress made with RF24Mesh is very encouraging, supporting simple communication scenarios, small household sensor networks, home automation systems, and larger expanded networks stretching over relatively long distances with the more powerful RF24 modules.
The RF24 communication stack is generally complete, with bug reports and issues dropping off quite nicely, but development is still ongoing. As always, please report any potential bugs or issues using the GitHub link below.
Or see the following for a video based description:
This is a quick tutorial on how to wirelessly control or monitor Arduino devices using nothing more than NRF24L01+ radio modules, a Raspberry Pi and a mobile device. I have found MQTT to be one of the best methods of controlling these devices wirelessly, in the case of simple input/output scenarios like monitoring sensor values or sending RGB values to wireless lighting systems.
This assumes users are familiar with the core RF24 library as well as installing libraries from the Arduino IDE, and building and running programs on the RPi.
Requirements:
Raspberry Pi with NRF24l01+ device attached
One or more Arduino devices with NRF24L01+ device attached
The RPi should be connected to the same network as your mobile device
Pre-configuration:
Ensure all devices are functional with RF24 core examples before attempting.
Install RF24 libraries on RPi from your home directory:
4. Follow the command prompts and install the RF24, RF24Network, RF24Mesh, and RF24Gateway libraries
Install RF24 libraries on Arduino:
1. From Arduino IDE select Sketch > Include Libraries > Library Manager
2. Install the RF24, RF24Network, RF24Mesh and RF24Ethernet libraries
RPi Setup:
Now that the correct libraries are installed, we need to install an MQTT broker on the Raspberry Pi. Setup is as easy as running the following command:
sudo apt-get install mosquitto
Note: As of mosquitto 2.0 you need to add listener 1883 andallow_anonymous true in mosquitto.conf.
Then we need to start the Gateway on the RPi, so it will pass traffic from the connected sensor nodes etc, to the MQTT server.
The RF24Gateway library is installed in ~/rf24libs/RF24Gateway/
In this example, we will use the ncurses example, located at ~/rf24libs/RF24Gateway/examples/ncurses
Edit the file RF24Gateway_ncurses.cpp and modify the radio constructor to suit your pin connections: RF24 radio(22,0); is the default, then type 'make -B' to build the example.
To run the example, type ' sudo ./RF24Gateway_ncurses ', and it should pop up asking for an IP and subnet mask to use. This is mainly arbitrary, and any suitable private IP/Mask can be selected.
Once the gateway is running, we can switch our attention to the Arduino.
Arduino Setup:
1. Open the MQTT example: File > Examples > RF24Ethernet > MQTT > mqtt_basic
2. Edit the radio constructor to suit your chosen CE/CS pins: RF24 radio(7,8); is default.
3. Edit the IP address of the device to match the range chosen when configuring the RPi.
4. Edit the IP of the gateway to match exactly the IP of the RPi
5. Edit the IP of the server to match exactly the IP of the RPI
6. Upload the sketch to the Arduino.
Note: If using an external MQTT server, the RPI must be configured to forward packets and perform NAT, see the forwarding and routing section here.
You should see serial output like the following:
Attempting MQTT connection...connected
If not, something has gone wrong. Ensure your device is connecting to the mesh and showing up in the address list. If not, troubleshoot using examples from RF24 or RF24Mesh to diagnose the connectivity issues.
Nodes running the MQTT example will publish their NodeID every second to the MQTT server, at outTopic and will receive all messages published to the inTopic.
Setup on iPhone etc:
On my iPhone, I chose to use the program MQTTTerminal (MQTTool is also good with no ads). Setup is easy, just input the external IP address of the MQTT server (the ip of your RPi) and the port (1883). Then set the Publish Topic to inTopic and the Subscribe Topic to outTopic.
You should see messages incoming from the device(s) and can send data to the devices from the app.
From here it is a simple matter of customizing the topics and messages to suit your needs, whether controlling a simple LED on the Arduino, or reporting temperature and humidity from sensors.
This can be expanded using programs like Node Red to create an open-source home automation system. See my more recent blog post.
New Sampling/Audio library for Arduino : AutoAnalogAudio
Easy access to the internal DAC(or PWM), ADC, Timers & DMA for sound generation
I've been playing around with the Arduino Due for a while now, and finally got to looking over the datasheet and playing around with some of the internals because of ongoing requests regarding audio functionality and advanced timer usage etc.
As a result of my dabbling, I've created a sampling library to simplify access to the onboard Analog-to-Digital (ADC), Digital-to-Analog (DAC), Timer and DMA peripherals.
It is a simplified API that allows users to create a wide range of audio or related applications in short order.
Auto Analog Audio (Automatic DAC, ADC & Timer) library
Goals:
Extremely low-latency digital audio recording, playback, communication and relaying using a simple API
Features:
New: Now supports AVR devices (Uno,Nano,Mega,etc)
Designed with low-latency radio/wireless communication in mind
Very simple user interface/API to Arduino DUE DAC, ADC, Timer and DMA
PCM/WAV Audio/Analog Data playback using Arduino Due DAC
PCM/WAV Audio/Analog Data recording using Arduino Due ADC
Onboard timers drive the DAC & ADC automatically
Automatic sample rate/timer adjustment based on rate of user-driven data requests/input
Uses DMA (Direct Memory Access) to buffer DAC & ADC data
ADC & DAC: 8, 10 or 12-bit sampling
Single channel or stereo output
Multi-channel ADC sampling
AVR devices with no DAC or DMA use pseudo DAC(PWM) & DMA(Timer + Memory Buffer)
Testing:
The results so far have been very good. Using the nrf24l01+ radio modules, I was able to stream a standard format *.wav file of high quality (48khz, 16-bit, Stereo) from a Raspberry Pi to the Arduino Due using a slightly modified version of the included Wireless Speaker example.
To put that in perspective: Data Rate = Sample Rate * Channels * BytesPerSample
Streaming over Radio (RF24): Data Rate = 48,000 * 2 * 2 = 192.0KB/s
Streaming from SD Card: Data Rate = 44,100 * 2 * 1 = 88.2KB/s (maximum rate with noticeable slowdown)
Using an SD card is a bit less exciting, as the audio starts to slow down noticeably with audio of 44.1khz, 8-bit, Stereo. Unfortunately, the SD speed is currently a bit limited, since the HSMCI is not available on stock Arduino Due, and I haven't found a faster working library like sdFat for Due.
SdFat Lib:
The sdFat library works with the due, I just had to initialize it at SPI_DIV6_SPEED (10.25Mhz SPI), and it worked using a value of 5 (17mhz SPI) as well. Initial tests show the max read speeds with the stock SD library at about 113KB/s, and 182KB/s with sdFat lib. Adjusting the SdFatConfig.h file to set ARDUINO_FILE_USES_STREAM = 0 results in speeds of 204KB/s with my current card & module.
These numbers indicate that the Due can handle quite a bit of punishment, and is easily up to the task at hand. The main limitation seems to be the throughput of whatever device is providing the audio or recording it etc.
Initial testing on AVR devices seems to indicate functionality similar to my RF24Audio and TMRpcm libraries.
If using a faster device, or generating audio from tables stored in memory, it seems that the Due with the AAAudio library will most likely handle it.
Notes:
If using the RF24 library in combination with an SD card, I highly recommend using a separate SPI BUS for the radio and SD card. I experienced a number of problems with high speed transfers etc when attempting to extend the length of wires used or connect an SD card module as well.
See the SPI_UART library section of the RF24 docs to enable a secondary SPI BUS for the radio. Due Pins - TX1: MOSI, RX1: MISO, SDA1: SCK, CS&CE: User selected Installation: AutoAnalogAudio is available via the Arduino Library Manager
I've also included a number of examples that demonstrate usage: Click: File -> Examples -> AutoAnalogAudio in the Arduino IDE
SDAudio Examples: Demonstrate how to play back and record WAV/PCM format audio from SD card using the AAAudio library
Wireless Examples: Demonstrate receiving and sending streams of audio data via nrf24l01+ radio modules and are easily modified to work with other radios or devices. Some matching examples and audio samples are included for Raspberry Pi. Audio Generation Examples: Demonstrate playback of simple audio tones or synthesized audio. ADC/Audio Capture Examples: Demonstrate how to capture data/audio streams from the ADC on one or more pins/channels.
Raspberry Pi/Linux with the nrf24l01+ & RF24Gateway
A guide/example of advanced RF24/nrf24l01+ usage and monitoring on Linux devices
Overview:
There are many examples of basic RF24 usage, but not so many demonstrating complex communication scenarios and networking. The RF24 communication stack is free, with all development information, examples, code and documentation available on GitHub, designed with Arduino and RaspberryPi/Linux devices in mind.
The RF24 stack generally follows the OSI model, and provides a separate library for each layer, with efforts toward proper documentation for each layer. This allows any type of data/communication to pass over the radios, including TCP, UDP or any other protocol, and users can write their own code to link devices together using any layer(s).
So, what can we really do with these little radios?
The nature and low-cost of the system allows literally anybody with interest in wireless communication to setup, deploy, and experiment with small scale, self-healing wireless networks, or start writing their own code with the core RF24 driver, in a very short period of time.
The stack currently does not include sensor specific code or functionality, it is simply a communication stack which enables all kinds of different functionality.
Requirements, Hardware Setup & Installation:
1. A Raspberry Pi or Linux device a compatible radio device connected RPi Install Script
2. An Arduino device with a compatible radio device connected Arduino Install(Select RF24, RF24Network, RF24Mesh, RF24Ethernet libraries)
See the RF24 docs for hardware setup info & installation Note: Please report any issues at https://github.com/TMRh20/RF24/issues Demonstration and Information:
The video below demonstrates how to use RF24Gateway in general - Setting up nodes and communicating over RF24 using RF24Network, RF24Mesh, and standard protocols (ICMP, HTTP, SSH), along with monitoring
1. Config on RPi & Arduino devices
2. Complex mesh networks: How the different layers (RF24Network,RF24Mesh & RF24Ethernet) can be used with RF24Gateway
3. RF24Ethernet: Simple TCP/IP communication and web server running on Arduino over RF24
4. RF24Mesh: Node config & interaction with RF24Gateway
5. RF24Network: Node config & interaction with RF24Gateway
6. How to use WireShark to monitor user payloads with RF24Gateway and LUA Wireshark dissector script (using laptop w/SSH & X11 forwarding in this example)
7. How to configure Raspberry Pi as a standard node in an existing RF24Mesh
8. How to sniff RF24Network, RF24Mesh, and RF24Ethernet traffic using RF24Gateway & Wireshark(optional)
9. Tacos... oops they were already eaten, maybe next time
As shown in the video, any nodes running RF24Ethernet, RF24Mesh or RF24Network can be used with a master node running RF24Gateway, allowing users to try out or utilize the different layers quite easily.
RF24Ethernet: Provides TCP/IP communication. Very simple and reliable way to control on/off devices via a web browser or scripting. Has all functionality of lower layers.
RF24Mesh: Provides automatic (mesh) networking for RF24Network. RF24Mesh networks are self-healing and dynamic in nature.
RF24Network: Provides addressing, routing, fragmentation/reassembly of payloads via manual/statically defined wireless networks.
RF24Gateway - Creating Hybrid IoT Networks using TCP/IP and RF24Network
RF24Gateway User Interface Example
RF24Gateway is a new complimentary library to RF24Ethernet, designed for RPi and Linux based devices with SPI and GPIO capabilities (BBB, Intel Edison, Galileo, etc).
The RF24Gateway library acts as a network/internet gateway, and allows Arduino/AVR based sensor nodes to interact with each other, the environment, and the world using TCP/IP and/or lower level RF24Network messages.
In short, it allows hybrid IoT networks to be created using Arduino and nrf24l01 modules. Some nodes are capable of internet or local network communication using standard protocols (TCP/IP) and/or RF24Network messages, while other nodes can communicate only at the network level, using only the lower layer RF24Network messaging.
Once RF24Gateway is configured, nodes can be controlled completely using standardized tools and protocols. Additional client examples have been added to demonstrate how to control and/or receive
information from sensor nodes using standard tools and scripting methods
via Bash, NodeJS and Python scripts. Any method of establishing a TCP connection can be used, with the examples generally using HTTP to some degree as well.
Libraries and code that utilizes RF24Network can utilize RF24Gateway in
much the same way, since external data types (TCP/IP) are handled out of
sight from the user. RF24Network is used exactly the same way, except
that the gateway.update() function is called instead of network.update()
or mesh.update(). The included examples demonstrate usage.
Note: RF24Gateway defaults to using a TUN interface with RF24Mesh enabled. RF24Ethernet examples have been updated to accommodate this. Current Status:
April 2015: Near completion. Integrated fully with RF24Ethernet. Graphical (NCurses) interface example finalized.
RF24Ethernet - For DIY Internet of Things and Home Sensor Networks
Adding internet connectivity to RF24Networks using nrf24l01+ modules as ethernet cards
RF24Ethernet, What is it and how does it work?
For anybody not familiar with my blog, I have done a lot of recent work in improving the radio driver and related network library for NRF24L01+ radio modules, using Arduino and Raspberry Pi or using two or more Arduinos. These very inexpensive and feature-rich devices can be used to create your own home sensor network, with very little cost when compared to most available solutions.
RF24Ethernet will allow you to use a Raspberry Pi or Arduino as the 'gateway' machine to your network, and lets you connect to your sensors directly, using any device that has a web-browser, whether it is an iPod, PC, etc. or your sensors can connect out to the internet for information.
Current Libraries:
RF24 - OSI Layer 2 radio driver for nrf24l01+ modules
RF24Network - OSI Layer 3 network driver for RF24
RF24Ethernet - OSI Layer 4 Protocol (TCP/IP) driver for RF24Network
RF24Mesh - Mesh networking layer for RF24Network (Dynamic config and topology)
How it works: (Updated)
The RF24Ethernet library is currently in the testing phase, and uses the UIP TCP/IP stack.
The library has been modelled after the Arduino Ethernet library, allowing users to create web enabled, wireless devices, without having to learn the RF24 or RF24Network APIs. The addition of a real protocol (TCP) on top of RF24Network provides a level of simplicity, consistency and reliability beyond what was previously possible.
The IP address of each node needs to be statically assigned. Translation between RF24/RF24Network (MAC) addresses and IP addresses is handled in one of two ways, depending on the configuration. The default configuration uses ARP requests to find the correct node when TCP data is incoming, and users can optionally utilize RF24Mesh to provide MAC/IP translation and/or dynamic address assignment at the network layer.
Configuration/Testing: (Updated)
Configuration is now integrated directly into RF24Ethernet, with the RF24Network address specified as the MAC address, and this can be further automated by utilizing the RF24Mesh layer along with RF24Ethernet. Users have the option of creating a static network, or utilizing RF24Mesh to create a dynamic network with nodes capable of moving around physically or utilizing fail-over nodes. The RF24toTUN application running on a Raspberry Pi automatically performs discovery and routing for incoming TCP/IP data, so users only have to configure static IP addresses, and a unique identifier for each node when used with RF24Mesh.
Setup (Arduino & RPi):
RPi:
Install RF24, RF24Network, RF24Mesh & RF24toTUN libraries Note: RF24toTUN requires the boost libraries. Run sudo apt-get install libboost1.50-all a: wget http://tmrh20.github.io/RF24Installer/RPi/install.sh
b: chmod +x install.sh
c: ./install.sh (Note: This should be run without sudo or all your base files will belong to root) Arduino:
c: Install RF24Ethernet library
d: If using with a RPi, run any of the Getting_Started examples. You should be able to ping and connect to the remote Arduino.
e: If using with a SLIP interface (non-RPi) :
1. Run the SLIP_Gateway.ino example on the Arduino connected to the PC/MAC etc
2. Run the SLIP_InteractiveServer.ino example on another Arduino.
3. Once the SLIP interface is configured as per the examples, you should be able to ping and connect to the remote Arduino.
RF24Ethernet - Setup, config and demo
The initial proof of concept interface borrowed directly from the SerialIP library, using the uIP TCP/IP stack and uIP proto-sockets/proto-threads. Since testing was very succesful, the library has been developed to provide a simpler user interface, very similar to the Arduino Ethernet library. In its current form, a node running RF24Ethernet can act as a standard TCP server, and can function as a telnet or web server, by modifying the included example.
How can this be used?
The possibilities from here are virtually endless, with very inexpensive sensor networks being provided with direct TCP/IP connectivity. One of the more obvious applications involves simple retrieval of information from network or internet sources, such as the time or the current weather. Controlling a connected LED or other attached device can be as simple as opening a bookmark in your browser.
To take things a whole lot further, this could even allow RF24Network to be utilized as a bridge betwen LANs, using sensor networks for emergency commuinications etc, when main connections are down. Testing shows that speeds across the network are slightly better than or equivalent to a dial-up connection, so it seems to provide a practical solution for low-speed data transfers like sensor data, text-based email or chat.
Results of Testing:
The above picture presents an idea of the latency, etc when sending ping requests of varying sizes from the RPi master node to a connected sensor node.
Working with UIP TCP/IP stack has been a bit of a challenge to say the least, but things seem to be coming along nicely. The RF24toTUN library has been updated to route payloads according to the radio MAC address, so integration with RF24Network is coming along nicely. Support for the RPi to send fragmented multicast payloads has been added, so ARP requests are working nicely within RF24Network. This will allow full integration with RF24Mesh, to create a dynamic TCP/IP mesh sensor network if desired, with very little to no configuration needed by the user. I've commited the current code to a new branch (master_dev) to hold the new code until it is cleaned up and more functionality is added. Currently, it only supports incoming data connections as a simple server, and ICMP packets.
Update Nov30:
The Server and Client API to match the official Ethernet library is now in place and working for the most part. DHCP, DNS lookups and UDP in generall is not developed yet. There are still some bugs and oddities to work out, but overall, the TCP server and client seem to perform fairly well given the circumstances. Web client and server examples have been included.
Update Dec 7:
The library seems to be working very well now, and the API is about 95% complete. Most bugs have been addressed, with only a few minor issues left, generally surrounding timeouts during very large data transfers. The existing examples have been updated, and some new examples have been added, demonstrating a simple web server and interaction with a sensor node via a web browser.Some documentation has also been created and is linked below.
Update Dec9:
RF24Mesh has been integrated with RF24toTUN, to provide automated addressing and mesh support for RF24Ethernet. When building RF24toTUN, use 'sudo make install MESH=1' to compile with RF24Mesh support. If using with RF24Ethernet, see the new SimpleServer_Mesh.ino example for usage. An install script has also been created for RPi, to simplify installation of the various libraries.
Update Dec 14:
Updated RF24Ethernet to support SLIP or TUN devices as well. This requires RF24Mesh to perform MAC/IP translation. Two new examples have been added, SLIP_Gateway.ino demonstrates how an Arduino can act as a simple USB interface to any device that supports SLIP. SLIP_InteractiveServer demonstrates how to use RF24Ethernet and RF24Mesh with a SLIP or TUN interface.
Update Dec 28:
Modified the MAC address format due to issues with standards when using it on RPi/Linux. Modified the timing of uip restarts. Updated RF24toTUN to provide cmd line configuration for all major options including node addressing. Fixes and updates to RF24Network and RF24toTUN seem to have addressed some buggy behaviour.
Update Dec 30:
Todays latest updates should be applied along with the latest updates to RF24toTUN and RF24Network. The main change fixes corrupt client requests, which would have been noticed when establishing outgoing connections. The remaining changes revolve around reliability, the order of operations, and the timing of things, and seem to really bring RF24Ethernet closer to a stable, reliable library. Update Jan 2, 2015
This round of updates comes after a very long round of testing and coding. The RF24 library has been updated to provide even better timing, reliability and throughput. Changes to RF24Network improve fragmentation/reassembly. RF24toTUN has been updated to fully support all 3 datarates, and throughput has been improved for all speeds. RF24Ethernet has been updated to provide a timeout for connections, part of better handling for TCP window reopening and failures, and allows users to configure the periodic timer via uip-conf.h to provide faster transfers. These changes bring RF24Ethernet a lot closer to a stable release. Testing shows that these updates provide easily noticeable increases in reliability and speed at all levels, and updating all the mentioned libraries is recommended.
Update Jan 4, 2015
Added connection timeouts to recover from hangs during failed client donwloads, which adds improved reliability to the library. Added better TCP window management to prevent those hangs during client downloads, along with configuration options. Documentation updated to include new features and options.
Update Jan 16, 2015
Updated to v1.2b - Adds UDP and DNS support along with a pile of fixes and updates. Users should update RF24Network and RF24toTUN along with RF24Ethernet. This will be the final update to this blog post. See the documenation below or check back here at tmrh20.blogspot.com for additional posts.
