Using RF24Ethernet & RF24Gateway with the Arduino Cloud

 Using RF24Ethernet & RF24Gateway with the Arduino Cloud

 Pushing automation even further

 Recently, I was offered an Arduino Uno Q, Arduino Uno R4 Wifi and 1-year of Arduino Cloud access for free! This came as quite the surprise, because I was contacted out of nowhere by Per Tillisch, an Arduino Support Engineer, who offered me the devices and cloud access with no strings attached! I was quick to take him up on his offer. He stated it was due to my work contributing to the Arduino ecosystem.

In looking at the cloud access, it looks like it will be very simple to create dashboards etc, and integrate everything I want from my existing systems into the cloud! No code changes, no updates, no nothing. Just some simple changes in the NodeRed system I use.

With the current wireless IoT systems I am using, based on RF24Ethernet and RF24Gateway + wireless radios + MQTT, I also use NodeRed for managing all the data and interactions locally. This means that all I had to do was install the node-red-contrib-arduino-iot-cloud nodes, setup an API key, and use the newly installed nodes to communicate with the cloud!

 My system is setup so the RF24Ethernet nodes use MQTT to communicate with the RPi running RF24Gateway & NodeRed. NodeRed then handles the communication with the cloud.

See  https://docs.arduino.cc/arduino-cloud/guides/node-red/ for more information.

Right off the bat, I was able to create a simple dashboard that displays incoming Temperature, Humidity & Barometric pressure data on some charts!

 From here, it is very simple to add/remove & create new parts of the dashboard.

All of the existing Arduino devices in my network now have potential cloud access via NodeRed! I didn't have to modify any code, upload new firmware or really do much of anything! This all works with my existing systems!

The beauty of this is users can keep all their data and integrations local, logging to a database if desired etc, and still make use of the Arduino Cloud at the same time. Some data can go to the cloud, some to a local repository, and everything just seems to work in sync!

I currently have all my integrations set up via NodeRed + NodeRed Dashboard, so it is very similar already to the Arduino Cloud Dashboard, though only available locally. Using the cloud, I can now make some parts of it accessible from anywhere.

This is all very cool, I can't wait to get the new hardware! 

 

Direct TCP/IP connectivity between Arduinos using a nRF24 or nRF52 radio link w/RF24Ethernet

 Setting up direct TCP/IP connectivity between Arduinos using a nRF24 or nRF52 radio link w/RF24Ethernet

 Utilizing the new functionality of the RF24Ethernet library

 With some recent experimentation and prototyping involving the lwIP IP Stack, I was able to modify the RF24Ethenet library to function standalone, without the need for a Linux/Raspberry Pi device running RF24Gateway. This allows users to directly connect multiple Arduino devices using the RF24Ethernet library, utilizing TCP or UDP protocols to communicate between devices.

The RF24Ethernet library API is based on the official Arduino Ethernet API, so users utilize the same coding style to communicate over nRF24 or nRF52 radio links.  

Setting things up:

1. The first thing to do is verify you have working radios. With nRF52 devices, they are built-in so, there is not much to worry about, but with nRF24 radios, users need to keep in mind power supply and wiring issues, so testing using the official gettingStarted sketch included with the RF24 library is recommended before attempting this.

 2. Install RF24Mesh and its dependencies from Arduino Library Manager

3. Install the RF24Ethernet library from ZIP from https://github.com/nRF24/RF24Ethernet/tree/lwIP  Note: Once deployed, the updated library will be available from the Arduino Library Manager. Users may need to uninstall/reinstall to get the latest updates at that time.

 4. Install the Arduino lwIP library using the Arduino Library Manager as required for non-ESP32 & non-ESP8266 devices which already include lwIP.

5. Run the included examples from the Headless directory in the RF24Ethernet examples on two devices. I've tested so far on Arduino Due, ESP8266, ESP32 and nRF52 based devices. The RPi Pico still utilizes the uIP stack due to technical issues using lwIP with the Arduino MBED based core.

 

What to expect:

The main server example sets up a RF24Mesh 'Master Node' which handles addressing and address look-ups for all other nodes. Nodes not in range of the Master Node will attempt to connect automatically via routing traffic through other connected nodes. 

The client examples simply connect to the Master Node and request an HTML based web-page. 

 In a real life deployment this activity may be reversed. With the Master Node running a modified Client example, and sensor or other nodes running modified Server examples, the Master Node could then connect to each device in turn and retrieve data as required.

To designate a master node, simply call the following before calling mesh.begin()

mesh.setNodeID(0);

then from the main loop()

 mesh.DHCP();

 

 Things to Note: 

 RF24Ethernet makes use of two separate IP Stacks, the older, unmaintained uIP Stack works on smaller devices like Uno, Nano, Mega, etc, while the newer, lwIP stack is used automatically for devices >50MHz CPU speed, including the Arduino Due, ESP32, ESP8266, and nRF52 based devices using the nrf_to_nrf radio library.

To ensure you are using the lwIP stack, users can #define USE_LWIP 1 & #define RF24ETHERNET_USE_UDP 1 in the RF24Ethernet.h file or prior to compilation. 

 

Creating IoT Sensor Networks & Systems using nRF24 & nRF52840 - An Analysis

 Creating IoT Sensor Networks & Systems using nRF24 & nRF52840

A Brief Analysis

In playing around with the nRF24 and nRF52x devices, I've been running and monitoring a couple large IoT networks of them over the past few years. In that time, I've developed a system that works very well, with nodes staying connected over long periods of time, despite the mesh protocols and need to re-establish connectivity periodically.

The system is based on TCP/IP & Node-Red, and makes heavy use of HTTP & MQTT protocols. Nodes either interact with an MQTT server, or are polled periodically via HTTP requests. Devices range from Raspberry Pi running RF24Gateway & Node-Red, to a wide range of Arduino devices utilizing RF24 and nrf_to_nrf libraries.

A graph showing connection times for various network nodes

 The graph above shows connection times for nodes connecting via MQTT. They report in periodically, showing how long they have been connected to the MQTT server. By default, in the MQTT library I use, the keep-alive interval is 10 seconds. This means each node must communicate with the MQTT server successfully every 10 seconds or be disconnected. Given the mesh nature of the system, this works out very well, with nodes staying connected for hours and days at a time.

With the default logic in place, it is very interesting to see how nodes assemble themselves around the master node over time. At first, it can be very chaotic, with many nodes on a network, each grabbing a unique identifier and place in the network. Over time, the nodes with the best connectivity find their way to a direct connection with the master node, or at least a secondary connection, with only one routing node between them and the master node.
One would almost think there was a more intelligent algorithm handling placement of nodes in the networks, but it is a relatively simply methodology, with nodes contacting the master node first, then secondary (children) nodes, then children of those nodes, etc. in order to establish a connection to the mesh. Nodes will also contact their parent node periodically to verify connectivity to the mesh.

 

 Ping Response Times for a Connected Node

 The above picture shows ping response times for a connected Arduino node. At the left of the top graph one can see that the node was able to re-establish its connection with better connectivity to the master node, not having its traffic routed through another node. Response times vary, with the node typically responding in about 8mS, up to around 275ms in some cases when there is a direct connection to the master node.

"External Network Activity"

 

The above picture shows the results while "external activity" interferes with the network. During this time, due to the nature of radio and jamming techniques, most nodes on the network lose connectivity to the mesh, but the self-healing nature of the network ensures that they only go offline while the jamming is taking place. Nodes quickly re-converge around the master node as the mesh re-establishes itself, and every sensor or device resumes normal operations. 

HTTP based nodes reporting in during the jamming

 

I recently updated many of my nodes, due to bug fixes and testing, per the above graph. In it, as can be seen, some of my HTTP nodes lost connectivity during the jamming. Others stayed online with only small blips in connectivity. Depending on proximity to other nodes and the master node, as well as the location of the jamming device, different nodes are affected differently, but again, they all resume operation after the interference stops.

Temperature & Humidity Readings from a Connected Node

This system works extremely well, with a distant node reporting in its temperature and humidity over time. You can see in the chart above, it has been reporting in consistently.

 

Ping Statistics over a Number of Days

 In the above graph, you can see the success rate of pinging one of my sensor nodes over the course of a few days. In this picture, the success rate is down a bit from the norm, due to ongoing "external network activity" beyond my control. This has been the case for years and years, with one of my neighbors continuously interacting with my network on an almost daily basis. I can't understand the obsession, but I've witnessed it on many occasions.

Conclusion:

The end results kind of speak for themselves. In a given networking scenario where reliability and consistency is key, making use of the advanced features of TCP/IP networking in combination with HTTP & MQTT allows users to construct networks to suit many different scenarios. This system is highly reliable and can be customized to the nth degree. 

The combination of nRF24L01 & nRF52x radios works quite well, and allows even more advanced functionality in a given network. With the 52x devices supporting things like encryption, Bluetooth & I2S, users can make great use of these features, with everything being done wireless.

Since we are using a mesh based system, and TCP/IP was designed to operate even on partially damaged networks, the combination works very well. Add that to recent fixes and updates in the RF24 communication stack, and you have a system that works extremely well over long periods of time. I measure uptime on the network, and nodes tend to only go offline when there is a power outage, hardware failure or they are manually taken offline.

Considering that there are great quality nRF24 based radios available with range anywhere from 2.5-5 KM (E-01 ML01DP5 or 2G4M27D), networks can be constructed that cover wide distances and manage themselves automatically. The network will support the smallest of devices, ranging from ATTiny all the way up to Raspberry Pi and other Linux based devices that have GPIO & SPI capabilities.

Libraries & Software Used:

1.  RF24, RF24Network, RF24Mesh, RF24Ethernet (Arduino) & RF24Gateway (Linux/Raspberry Pi)

2. nrf_to_nrf radio library (nRF52x devices)

3. Node-Red (Linux/Raspberry Pi)

4. Mosquitto (MQTT broker)

 

All of the software and systems used are open source, so anybody can create similar systems. Any software that is MQTT compatible like Home Assistant or Node Red will work.

RF24 Home Automation Kits now Available

 RF24 Home Automation Kits now Available

Comes with preconfigured devices, just plug and play!

I've finally decided to put together some electronics kits with preconfigured devices, using the RF24 communication stack. As the system is now very stable and reliable, I think it is about time I started putting devices together and selling them to support future development and designs. 

So with that, I am announcing the opening of my online store at https://tmrautomation.myshopify.com

The current kit will come with a RPi4B and all the attachments like HDMI cable, a case, and power supply along with 3 Arduino Uno R3, 2 temperature & humidity sensors and 1 RBG LED ring, plus of course the RF24 radios. I've also designed some custom PCB shields for everything to just plug into, with power for the radios and extra capacitors, so its super simple to put it all together.

The setup uses Node-Red and RF24Gateway to provide a mesh network that seamlessly links all your devices together, and allows you to add custom devices to the network and/or expand the network by purchasing more devices or setting them up yourself with the Arduino platform.

If they sell, I intend to design more devices to go with the currently available products.

Support is available at https://github.com/TMRh20/RF24-Automation-Systems/wiki

General documentation is available at: https://github.com/nRF24 and more documentation is being created to specifically support the kits.

Sensors & IoT Stuff: Home Automation with NodeRed, RF24Ethernet/RF24Gateway & MQTT

Sensors & IoT Stuff: Home Automation with NodeRed, RF24Ethernet/RF24Gateway & MQTT 

Control a homemade, Arduino-based RGB light, other devices or display data using NodeRed

I made a video demonstrating how to configure NodeRed with RF24Ethernet/RF24Gateway to control a homemade, Arduino-based RGB light. The process is fairly straightforward, with the Arduino node subscribing to the MQTT server, and the NodeRed device publishing some RGB values. The same system can be used to send or receive data from a wide range of devices, thus enabling an nrf24l01 based home automation/IoT and/or monitoring system.

The video assumes users are already somewhat familiar with the RF24 communication stack, having the radios installed and ready to go. The sketch used in the video is very closely based on the existing RF24Ethernet MQTT examples, with some extra code to control some NeoPixel based lighting.

Note: As of mosquitto 2.0 you need to add  listener 1883 and  allow_anonymous true in /etc/mosquitto/mosquitto.conf.

With this setup, users can add simple input/output toggles, sliders and charts to manage devices and display incoming sensor data. It is just a matter of configuring the Arduino devices to handle the input/output messages to and from MQTT. NodeRed can also log incoming MQTT data to a database, to store and display historical data.

This system has the benefit of requiring no apps, so everything can be controlled from a mobile device or PC etc. with some simple button clicks. Everything used is also completely open-source and free, so setup costs are minimal, and there is a host of information available for Arduino and Raspberry Pi.

Upcoming Projects: ESP32 - WiFi Kit 32 OLED/BLE

Playing around with ESP32 - WiFi Kit 32 OLED/BLE

New Toys to Play With!

I was recently provided some nice new toys to play with by the people at DigitSpace and will be documenting some upcoming projects involving these devices. The main player will be an ESP32 with built in OLED, which is very nice to have, not just because of the ESP32 capabilities and processing power, but the built-in OLED provides a nice way to get output and information without having to watch a serial monitor or send data to another device etc.

ESP32 WiFi Kit32 OLED from www.digitspace.com

I initially had some problems connecting the devices directly to computers via USB cable. I used different cables that work fine with my ESP8266 modules, but these would not work with any Windows, Linux or Mac devices I tried. Response, communication and professionalism was great, but neither I nor the supplier could seem to find a solution, even after contacting the manufacturer.

By chance, I ordered another module of the same model from a different supplier, and it came with a CH340G based USB->TTL module! It is starting to look like this is a known issue. So I tried it with these devices and BAM! Working nicely. Upload speed needed to be 256000 or lower.

The 5v pin on my TTL modules is connected directly to the USB 5v, so it can power the ESP32 via the 5v pin, even when configured to operate at 3.3v. If the RTS and DTR pins are not exposed, you only need to connect TX & RX, then press RST then PGM, then release RST then PGM buttons to put it into upload mode. RST again after complete. If exposed, connect DTR to GPIO0 and RTS to RST per the pinout below, and uploading will be automatic.

ESP32 WiFi Kit32 OLED Pinout

Now with the capabilities here, including a 240Mhz processor, BLE, WiFi and built in OLED, you might not think this module would be cheaper than an AVR based Arduino, but in some cases it is! Per the DigitSpace store these modules offer a great platform at a great price. Beyond the USB->TTL requirement, installation is easy. The Arduino IDE includes this device in the default ESP32 boards, and there are many examples etc. to test with.

I now have some working ESP32 devices and some other hardware that was given to use in upcoming projects! Some will relate to existing libraries and projects, but as usual I'm not sure exactly how or if I am going to be able to pull it off. Its always a challenge to understand the internal workings of newer MCUs and related code to produce the results I am expecting. It will be a challenge, but it seemed like a good idea at the time, so why not?

In any case, stay tuned for more to come with development & projects on ESP32 modules!

nRF24l01+ Library Roundup

Overview and Status of RF24 Arduino/Linux Comm. Stack

Six, seven years? Has it really been that long? According to GitHub that is in fact the case, as my first commits to my own fork of the RF24 library took place in early 2014. It all started by identifying existing limitations in the available libraries and working to achieve the highest level of performance and reliability possible. The main fork for RF24 at the time was written and maintained by ManiacBug, but he dropped off the scene shortly after publishing this very nicely designed library. Other users attempted to add support for various devices and address certain bugs, but nobody had really taken a thorough look at the capabilities of the hardware vs what was achievable at the time. Only through a lot of reading, testing and learning to program was I able to make those initial changes to begin work improving the RF24 core library. Looking back now, I am very grateful that I picked ManiacBugs code base as a place to get back into programming in C/C++, as it was well thought out and I discovered ways of doing things that I previously did not understand or know about.

And with that began a long and arduous journey into the internal workings of nrf24l01+, Arduino (AVR) and RPi (Linux) devices with the stated goal of optimizing the library to whatever extent possible. Once the RF24 library began to take shape, I began looking at the RF24Network library, another very nicely designed bit of code by ManiacBug, but it had its limitations and problems. Many issues were addressed, features and functionality added in order to push the nrf24 devices to the test in a multi-device scenario. It performed much better than expected using this OSI Layer 3 (network) library.

At that point, I began to really understand the OSI model, how the different layers actually work together and what they do down the last bit. Based on a number of conversations and user input, I came up with the idea for RF24Mesh, which is basically another layer on top of RF24Network -> RF24 that handles addressing, similar to DHCP, but able to verify connectivity and reconnect nodes at any point in the mesh. This allows nodes to move around and reconnect quickly as required while maintaining the mesh structure. The Network & Mesh layers use a 'master' node that acts like the gateway in a standard IP network, and it provides addressing and resolution for RF24Mesh.

Based on input and discussions with users and the performance of the radio modules when used in a multi-device mesh/network scenario along with the emergence of IoT and sensor networks, it seemed like the thing to do would be to work my way up the OSI model, so that is exactly what I did. Upon discovering the uIP stack for Arduino, I realized it would be possible to add ip4 support to some of the smallest devices like ATmega328 AVRs, and larger devices like RPi could just use their own IP stack and encryption etc. The uIP stack and related software was quite a challenge to understand and implement, as uIP is written in C and is designed to be as small and lightweight as possible as opposed to being human readable and easy to understand. The UIPEthernet library was also essential in helping me to understand and implement RF24Ethernet.

With that, an OSI RF24 comm. stack was established, with RF24, RF24Network, RF24Mesh, RF24Ethernet and RF24Gateway libraries all working together and inter-operational at any of the layers. This means that on a RF24Gateway/RF24Ethernet mesh network incorporating RPi and Arduino nodes, devices can choose to operate using OSI layer 2,3,4,5 and/or 7 (Data Link, Network, Transport, Session, Application). When using the higher layer libraries (RF24Gateway & RF24Ethernet) standard IP connectivity is established and devices can utilize standard encryption & authentication protocols (SSH, HTTPS etc.) to secure their traffic as per the OSI model.

The RF24 Communication Stack vs The OSI Model

With the associated libraries and enhanced functionality, the RF24 comm. stack is able to provide IoT and sensor network connectivity and capabilities to suit many different scenarios at a very low price point with much lower power consumption than WiFi networks. With the production of many devices like RPi, ESP8266 and ESP32 which support WiFi, users can construct and deploy IoT networks very easily, extending them as required.

Even now, it is really kind of cool to witness the system in action, using these little radios on tiny little computers we refer to as 'Arduinos'. The methodology and processes in place to manage a network of devices, fragmentation & reassembly, mitigate wireless data collisions and ensure delivery of data are quite effective, from the radio hardware and auto-ack functionality, all the way up to the TCP/IP and Mesh levels. With TCP/IP for example, using Raspberry Pi devices, the MTU is 1500-bytes, so each packet can require up to 63 sequential, successful data transmissions at the core RF24 layer, but this still works relatively well in real-world testing/usage.

It has been a lot of fun and a great learning experience so far. The creation of RF24Gateway really allowed the limits of the radios to be tested, and provided an excuse to play around with different network/IP configurations, routing scenarios and traffic handling at any level of the OSI model. A full and complete understanding of how devices operate on the internet or similar networks helps dramatically in troubleshooting, testing and development of systems and software that operate using these protocols.

Going forward, the development of the RF24 libraries has slowed significantly along with bug reports and issues. My focus has recently been on cleaning things up, finding and fixing bugs, mostly in the higher layers, and improving the user interface via examples and documentation. I had previously almost given up on the RF24Gateway and Ethernet layers as being too difficult to fix, but with the rise of more powerful MCUs and the potential for expanding their capabilities, it seems worthwhile to take another crack at working out bugs and issues.

 It would be nice to find another similar radio device with lower power consumption and/or more advanced capabilities and throughput to create more robust networks and mesh capabilities. The current design allows for speeds up to about 20KB/s over IP (RF24Gateway) using two Linux devices, but it would be nice to have a higher throughput to allow for more nodes, longer range and more advanced communication scenarios.

In conclusion I want to extend a big thank you to everybody who contributed ideas, analysis, information and programming skills along the way. An extra thank you to Avamander (GitHub) who has played a big part in ongoing maintenance, support and development!

Update: Aug 2020

Wow, I mentioned that I would be working on finding and fixing bugs due to the slowdown involving support, dev and maintenance etc, but did not expect to spend so much time, or that I would be able to identify so many issues, their root cause and a solution. Some pretty significant issues have been fixed throughout the RF24 stack, including 1 major bug in RF24, affecting all libraries, a number of bugs/issues affecting functionality and reliability in RF24Network and RF24Mesh, and a memory issue in RF24Ethernet affecting stability.
RF24Gateway has been updated with better handling of interrupts, a few corrections and the first release made, v1.0 due to all the issues that have been addressed. Again, a portion of these improvements are due to assistance, testing and input from a select few users. Users should notice a marked improvement in stability, functionality and the ability to recover from significant errors/hardware issues.

RF24 Communication Stack:
https://github.com/nRF24
https://nrf24.github.io/RF24Gateway
https://nrf24.github.io/RF24Ethernet

RF24Mesh - Dynamic Mesh Networking for NRF24L01+ Radio Modules

An Overview of Progress so Far

Overview:

RF24Mesh is a C++/Arduino/Linux/Raspberry Pi library allowing many devices to simultaneously connect wirelessly and remain connected in a mesh style network using nrf24l01+ radio modules. Nodes can communicate with any other node on the network, including a master node that acts as a gateway for the mesh, providing addressing and lookups.

Once assigned a unique ID, everything is automated, and nodes can move around and rejoin the mesh at any point where another node is in range. RF24Mesh supports some of the smallest and most power efficient devices available.

The RF24 Communication Stack is based on the OSI model, and RF24Mesh makes use of the features of the lower layers (RF24 (Layer 2) , RF24Network (Layer 3), and the overall capabilities can be extended via RF24Ethernet (Layers 4&5 TCP/IP)

Setting it Up:

The main thing when setting up a mesh network is to configure and test your devices accordingly, to make sure they are in a fully operational state. Most if not all of the available radio chips being sold are clones, so I've found it necessary to modify the radio devices slightly in some cases. In many cases, a capacitor (10-100uF) is added directly to the VCC/GND of the module itself to help with common power supply issues.

In the case of these modules, I've found that scraping down the last bit of the antenna, and soldering on a small 3-4" piece of wire greatly improves their performance:

In the case of the high powered PA + LNA modules, I've found it necessary to add shielding as in the post found here. Placing them close to a ground plane seems to help as well. Operating them at a lower power level may be required if the power supply is not adequate.


The modules that are low powered with an antenna generally work out of the box.

I generally recommend using an adapter, to provide a stable power source, even for the low powered modules. I've been using them with many of my modules.

From there it is a matter of installing the Arduino libraries etc. which is covered in the documentation and in earlier blog posts and the official documentation below.

How it works:

RF24Mesh leverages the features provided in the supporting libraries, RF24 and RF24Network to provide connectivity between many devices, even if they are far away from the central node or moving around. Each node is assigned a unique identifier (1-255) and uses that high level-address to request a dynamic address at the network layer, according to the structure of the mesh and proximity to other devices in the mesh. Nodes typically communicate using the RF24Network API, with RF24Mesh working at a higher layer to provide address lookups, dynamic addressing and connectivity.

Nodes arrange themselves using a tree topology around the master node, which provides address lookups etc. for the mesh. At any given time, each node has a path of communication through the network, and can re-establish and verify a new link if the current path fails. In each network there is the master node (the base node), children of the master node (relay nodes) with children nodes themselves, and leaf nodes, (children of the master or its children) with no child nodes attached.

The protocol is fairly simple for each node. On power up or reset, nodes attempt to find a connection by using multicast. Multicasting is arranged into 5 levels, so the master node is contacted first, then its direct children, then their direct children and so on, making the way nodes attach themselves to the network tend towards close proximity to the master node in a tree-like structure. Once contact is made the node requests an address from the master node, either directly or through other nodes. Once an address has been assigned and verified, the node is considered connected to the mesh and can then communicate with all other nodes in the mesh, and beyond to other systems etc.

Nodes have the option of managing their connection however desired, and quite often, a simple timer is suitable to verify connectivity after a set period of time. Transmitting nodes can also detect when communication is failing and re-establish their connectivity to the mesh as required. That is, at any given time, nodes will have a single path to communicate over the network, but if that link fails, they can re-establish their communication path through the network.

The communication layers are interoperable, meaning a network can combine devices running RF24Network and RF24Mesh libraries together, with static nodes running RF24Network, and dynamic nodes running RF24Mesh. Devices running RF24Ethernet (TCP/IP) can also communicate using the other layers of the libraries, interacting directly with RF24Mesh and/or RF24Network nodes. This allows an array of devices to operate on the same wireless mesh network, from ATTiny all the way to the more powerful devices like ESP32 and Raspberry Pi, using the RF24Network, RF24Mesh APIs and/or standard networking tools and protocols. (ICMP/TCP/UDP, etc.)

Current status, testing etc. 

RF24Mesh is now past the development/beta testing stage and considered (by me) to be stable and well tested. The mesh can handle a large number of nodes, theoretically up to 255, but in practice, more like 15-25 nodes sending or receiving data every few seconds is a reasonable bet. It generally depends on the type of traffic and frequency with which it is generated, as a much larger number of nodes reporting data every minute presents only a small load for the network.

The progress made with RF24Mesh is very encouraging, supporting simple communication scenarios, small household sensor networks, home automation systems, and larger expanded networks stretching over relatively long distances with the more powerful RF24 modules.

The RF24 communication stack is generally complete, with bug reports and issues dropping off quite nicely, but development is still ongoing. As always, please report any potential bugs or issues using the GitHub link below. 

Documentation:
http://nrf24.github.io/RF24Mesh

Source Code/Issues:
https://github.com/nrf24/RF24Mesh

RF24 Addressing: A review of NRF24L01+ radio addressing

Pipes, Addresses and Acknowledegments

There still seems to be a fair bit of questions, misconceptions and misunderstandings regarding the nature of RF24 radio addressing and how to set up communication scenarios. This is a review of RF24 pipes, addressing and acknowledgements that should provide any user with enough information to set up any supported configuration.



Acknowledgements (ACKs):

The radios support an automatic acknowledgement feature, enabled by default, in which the receiving radio switches into transmit mode after reception, and acknowledges reception of data.

1. A sends to B
2. B responds to A
3. A is now aware that B received the payload successfully

The acknowledgment process is very important to understand, because correct addressing is required for this feature to function properly.




Simplest Explanation/Rules of Addressing (Using ESB/Auto-Ack):

1. When using Auto-Ack, every radio can utilize a maximum of 6 unique addresses at once, which other radios can send data to.

2. Each address is assigned to a logical "pipe", and each radio has only 6 pipes that can be assigned addresses.

3. Since auto-ack implies a bi-directional communication channel, each address/pipe can only support communication to a single radio using acknowledgements. If two or more devices send data to a single pipe/address, auto-ack should be disabled to prevent ACK spamming and erroneous acknowledgements.
   Exceptions:
   a: Disabling Auto-Ack - This prevents excessive spamming of acknowledgements and errant acknowledgement.
   b: Timing of Transmissions - Using a token-ring style network or other method of preventing simultaneous transmissions would allow multiple radios to send data to the same pipe/address while maintaining the reliability of acknowledgements.

Note: At 1MBPS, a transmission can take up to 85ms or so, and overlapping transmissions are likely with any significant traffic

4. Per manufacturer design, the addresses assigned to pipes 1-5 must be the same, except for a single byte.



Basic Scenario - One to One Communication with AutoAck:

For purposes of standardization and to allow the system to be scaled, it is generally recommended to use two radio addresses. With auto-ack disabled, a single address can be used for all radios, or groups of radios.


Code:
Radio #1:
byte address_1[] = {0xCC,0xCC,0xCC,0xCC,0xCC};
radio.openReadingPipe(1,address_1);
Radio #2:
byte address_2[] = {0xC3,0xCC,0xCC,0xCC,0xCC};
radio.openReadingPipe(1,address_2);


Process - Radio #1 sending to Radio #2:

1. Radio #1 opens pipe 0 for writing (and internally, reading) using address of Radio #2 (CCCCCCCCC3)
2. Radio #2 receives the data on the first attempt, and (internally) opens a writing pipe using its own address (CCCCCCCCC3) and sends an acknowledgement back to Radio #1
3. Radio #1 receives the ACK, so does not engage in a series of timed retries





Basic Scenario - One to One or Many to Many Communication with Multicast:

With AutoAck disabled, radios can share the same address, and the system can still be scaled to support any number of devices.

Code:
byte address_1[] = {0xCC,0xCC,0xCC,0xCC,0xCC};
radio.setAutoAck(0);
radio.openReadingPipe(1,address_1);

In this situation, any radio can send a message to all other radios sharing the same address:

Code:

radio.stopListening();
radio.openWritingPipe(address_1);
radio.write(&data,sizeof(data));
radio.startListening();





Simple Mesh Scenario - Five-to-Five Communication with AutoAck:




In the above scenario any radio can send or receive data from any other radio in the mesh, thus creating a 'true mesh' of 5 nodes.

Radio #1 will always open a writing pipe using the address assigned to pipe#1 of the recipient.
Radio #2 will always open a writing pipe using the address assigned to pipe#2 of the recipient.
Radio #3 will always open a writing pipe using the address assigned to pipe#3 of the recipient.
... and so on

Process Confusion:
In this example of addressing, if Radio#2 and Radio#3 were to both open a writing pipe to Radio#1 using the same pipe/address, there would be confusion with acknowledgements. Radio#1 will respond to both devices with acknowledgements, since they have both opened a writing and reading pipe using the same address.




Complex Network/Mesh Scenario - Many-to-Many Communication with AutoAck:


As can be seen, this format of addressing is very similar to creating sub-nets with IPV4 addressing, except here we are using HEX instead of decimals, and there are 5 bytes per address instead of 4. Each radio is linked as shown in the image below, and each radio has a unique communication channel (pipe/address combination) to use when communicating with any of its parent or child nodes.

Due to the correlation of addresses and associated topology, each radio only needs to be aware of its parent and child nodes. Any traffic sent to a given node will either be a: Accepted b: Routed to parent c: Routed to a child

Address configuration, routing etc, can be managed very simply by masking and bit-shifting.





In the above example, pipe0 is NOT assigned a reading address, leaving it available to be used for broadcast or multicast. With AutoAck disabled on a single pipe, or by using selective acknowledgements, all radios or groups of radios can share an address, thus all radios will receive all messages sent to that address.

See the RF24Network Documentation and Source Code for more information regarding this type of configuration.



Summary:

In many low-throughput situations, the radios will appear to function correctly even with improper addressing, and may not cause any obvious problems, however, performance and functionality will most certainly be affected.

In any situation where reliability and acknowledged payloads are important, proper addressing and an understanding of the underlying processes behind acknowledged payloads is very important.

HB-100 / GH1420 10.25GHz Microwave/Doppler Radar Motion Sensor w/Arduino

I picked up some of these modules out of curiosity, as these modules can detect the motion and speed of objects through walls or other non-metal obstacles.


Overview:

The modules themselves are fairly simple in use, since they output a signal with the frequency/pulse width indicating the speed, and the amplitude/voltage indicating the strength of the reflected signal.

The speed is very simple to calculate once the frequency of the incoming signal is detected:

km/h = hz / 19.49
mph = hz / 31.36

These HB-100 modules require a pre-amp circuit to boost the IF signal, so I used an LM358N and the circuit at https://hackaday.io/project/371-hb100-radar-shield to boost the intermediate signal.

The output of the circuit sits at about 2.5v when quiet, and moves up & down to represent the received AC waveform.

In pure digital form, this would be very easy to detect using interrupts, but with an amplified analog signal, weaker signals may be missed.

Problem:

Tests with this radar shield using standard frequency counters/libraries seemed to show that interrupts and edge-detection have limited sensitivity to weak signals. They also provide no means for detecting the amplitude of the signal. It seemed that using the on-board ADC may provide better results in this frequency range. (0 to 4000 Hz or 0 to 205km/h)

Goal: 

Detect the frequency and amplitude of low frequency (0-4khz) signals from HB-100 modules, while potentially increasing the sensitivity of waveform detection using the HB-100 & circuit as linked above.

Result:

The resulting code uses the Arduino ADC to sample incoming signals using very simple peak-to-peak detection.

Signals are detected when the waveform transitions above or below a set "midPoint" which represents the 0v value of the AC waveform. Typically, it will be ADC_RESOLUTION / 2.

This is very similar to the edge detection used for interrupts and digital signal detection, but using the ADC to detect weaker transitions.

ADC Resolution / VCC == Volts per Increment ( 1023/5v = 0.005v )
Sensitivity of 10 == peak-to-peak detection of +/- 0.05v signals


Technical Info:

The ADC is driven in free running mode, taking samples at about 38.5khz while looking for specific deviations in the measured voltage that correlate with a peak-to-trough waveform.

The results are summed in a single variable, and returned either as a single result, or an average, depending on how often the results are read in relation to the received signal.

Information about the signal, including frequency, amplitude and number of samples allows more complex analysis.

Conclusion:

The code was developed in a very short time, and can still be considered a work in progress, although it seems good as-is. It tests well up to about 2-4khz with another Arduino producing square wave output. When used with the radar module, it is very sensitive, and allows the sensitivity of the device to be defined in software, and/or results to be filtered.

Code & Examples:

https://github.com/TMRh20/AnalogFrequency

RF24Gateway - Creating Hybrid IoT Networks using TCP/IP and RF24Network

 RF24Gateway User Interface Example

RF24Gateway is a new complimentary library to RF24Ethernet, designed for RPi and Linux based devices with SPI and GPIO capabilities (BBB, Intel Edison, Galileo, etc).

The RF24Gateway library acts as a network/internet gateway, and allows Arduino/AVR based sensor nodes to interact with each other, the environment, and the world using TCP/IP and/or lower level RF24Network messages.

In short, it allows hybrid IoT networks to be created using Arduino and nrf24l01 modules. Some nodes are capable of internet or local network communication using standard protocols (TCP/IP) and/or RF24Network messages, while other nodes can communicate only at the network level, using only the lower layer RF24Network messaging.

Once RF24Gateway is configured, nodes can be controlled completely using standardized tools and protocols. Additional client examples have been added to demonstrate how to control and/or receive information from sensor nodes using standard tools and scripting methods via Bash, NodeJS and Python scripts. Any method of establishing a TCP connection can be used, with the examples generally using HTTP to some degree as well.

Libraries and code that utilizes RF24Network can utilize RF24Gateway in much the same way, since external data types (TCP/IP) are handled out of sight from the user. RF24Network is used exactly the same way, except that the gateway.update() function is called instead of network.update() or mesh.update(). The included examples demonstrate usage.

Note: RF24Gateway defaults to using a TUN interface with RF24Mesh enabled. RF24Ethernet examples have been updated to accommodate this.

Current Status:

April 2015: Near completion. Integrated fully with RF24Ethernet. Graphical (NCurses) interface example finalized.

Source Code:
https://github.com/TMRh20/RF24Gateway

Documentation:
http://nrf24.github.io/RF24Gateway

Download (See documentation for installation info):
https://github.com/TMRh20/RF24Gateway/archive/master.zip

ENC28J60 Network Modules, Arduino and the Internet of Things (IoT):

An experiment, overview and analysis of TCP/IP networking with Arduino Ethernet

The module in question was sent to me by the friendly people at ICStation.com , in exchange for a review of the module itself.

Video Review:

ENC28J60 Mini Network Module:

• 3.3V operation
• Create a mini Arduino Web Server or Client
• Plugs into your home router or modem just like a computer

The uIP TCP/IP stack is software driven, while the ENC28J60 module handles the data mainly at the network (Ethernet) level. One main difference between UIPEthernet and the official Arduino Ethernet library is the need to keep the TCP/IP stack updated via software with the ENC28J60.

Hardware: 

The ENC28J60 was very easy to wire up, using the default SPI pins shown at http://arduino.cc/en/Reference/SPI along with the VCC and Ground pins.

The device seems to work ok connected to the 3.3v output of the Arduino, but it seems best to use an external power supply or battery.

Software:

Two main libraries appear to be available: Ethercard and the UIPEthernet libraries. This post will focus on usage via UIPEthernet, since it shares a common API with the official Arduino Ethernet library.

I am using the UIPEthernet library and examples found at: https://github.com/TMRh20/arduino_uip/tree/fix_errata12

In attempting to utilize the modules and UIPEthernet for creating a simple webserver, I ran into a number of issues with data being sent or received incorrectly. Over a number of days, and benefiting from the information gained in building the RF24Ethernet library, I have been working to improve UIPEthernet, and also created some examples to demonstrate the issues and current usage.

(A pull request has been made regarding these suggested changes to the main library)

Overview of Library Changes (At the time of this post):
Changes:

• Introduce client.waitAvailable(); function  - Ensures the IP stack is processing incoming data while waiting for incoming data for a maximum defined duration in milliseconds: client.waitAvailable(3000);
• Introduce Ethernet.update(); function - Allows user applications to ensure the IP stack is processing data while delaying or performing other tasks: Ethernet.update();
• Perform additional processing of data while calling client.available();
• Per testing, reduce max segment size (MSS) and the receive window to 511 bytes from 512 due to unidentified issue.
• Reduce number of packets per socket to prevent data loss
• Re-open the TCP window at timed intervals when data has not been sent or received on an open connection to prevent stalling during large downloads
• Fix for outgoing data corruption
• Force application to wait while writing data 
• Better TCP window handling
• Other minor changes, add examples

Changes - Technical Summary:

The first changes introduced added some new functionality, geared towards a software driven IP stack.  With the addition of additional processing and the Ethernet.update() function, the IP stack is more responsive to ICMP requests etc.

The new examples included with my branch of the library clearly demonstrate some of the issues I encountered when attempting to create a simple web client and/or web server. The first issue I noticed involved the device crashing or stalling, and then failing to reconnect or respond to pings. Testing indicated that the hardware itself was still responding and receving packets, so the issue seemed to be software based. These issues mostly seem to be addressed in the fix_errata12 branch, but testing via large dowloads from a web server indicated there were still some remaining issues, mostly surrounding data corruption and flow control.

Most of the remaining changes attempt to address issues with corrupted or out of sequence data, and to manage the flow of data better. In some situations, for example, web servers will send TCP payloads equal in size to 1/2 of the TCP Max Segment Size (MSS) which causes problems with flow control. The IP stack will now only close the TCP window if data is being received fast enought to do so, which appears to reduce issues with these transfers. The main issues come when a sending device sends only 1/2 MSS, and the recipient closes the window. The library will now count individual bytes in memory vs packets with data in them, and only close the TCP window if the assigned data buffers are completely full.

Testing has been very successful, with the device demonstrating full functionality for extended periods of time, downloading 70+MB of data without any errors detected in the data or hardware. The new web server examples and HttpClientTest.ino example seem to be good displays of the current capabilities and error free operation, and can be used to demonstrate the issues, when used with the original library code, or when changes are reverted.

Summary:
 
The mini ENC28J60 modules are an inexpensive and effective way to add internet connectivity to any Arduino project, and their size make them very suitable for use with Arduino.

I found the hardware itself to be very reliable and easy to use, although there seemed to be a few minor issues with the related Arduino library (UIPEthernet) that was tested.
As noted, I have submitted some workarounds/fixes for review and possible integration into the main library at the time of this post, so users are encouraged to use the library linked above.

ICStation.com:

A big thank you to www.ICStation.com for sending me this product, as well as being very patient.

It has been a number of months since I actually received it, but their small contribution has lead to the creation of the RF24Ethernet TCP/IP Radio Network library, and some potential improvements to the UIPEthernet library. A wealth of Arduino and Raspberry Pi related products, gizmos and devices are available on their website, and their staff seem very friendly.

RF24Ethernet - For DIY Internet of Things and Home Sensor Networks

Adding internet connectivity to RF24Networks using nrf24l01+ modules as ethernet cards

RF24Ethernet, What is it and how does it work?

For anybody not familiar with my blog, I have done a lot of recent work in improving the radio driver and related network library for NRF24L01+ radio modules, using Arduino and Raspberry Pi or using two or more Arduinos. These very inexpensive and feature-rich devices can be used to create your own home sensor network, with very little cost when compared to most available solutions.

RF24Ethernet will allow you to use a Raspberry Pi or Arduino as the 'gateway' machine to your network, and lets you connect to your sensors directly, using any device that has a web-browser, whether it is an iPod, PC, etc. or your sensors can connect out to the internet for information.

Current Libraries:

RF24 - OSI Layer 2 radio driver for nrf24l01+ modules

RF24Network - OSI Layer 3 network driver for RF24

RF24Ethernet - OSI Layer 4 Protocol (TCP/IP) driver for RF24Network

RF24Mesh - Mesh networking layer for RF24Network (Dynamic config and topology)

How it works: (Updated)

The RF24Ethernet library is currently in the testing phase, and uses the UIP TCP/IP stack.

The library has been modelled after the Arduino Ethernet library, allowing users to create web enabled, wireless devices, without having to learn the RF24 or RF24Network APIs. The addition of a real protocol (TCP) on top of RF24Network provides a level of simplicity, consistency and reliability beyond what was previously possible.

The IP address of each node needs to be statically assigned. Translation between RF24/RF24Network (MAC) addresses and IP addresses is handled in one of two ways, depending on the configuration. The default configuration uses ARP requests to find the correct node when TCP data is incoming, and users can optionally utilize RF24Mesh to provide MAC/IP translation and/or dynamic address assignment at the network layer.

Configuration/Testing: (Updated)

Configuration is now integrated directly into RF24Ethernet, with the RF24Network address specified as the MAC address, and this can be further automated by utilizing the RF24Mesh layer along with RF24Ethernet. Users have the option of creating a static network, or utilizing RF24Mesh to create a dynamic network with nodes capable of moving around physically or utilizing fail-over nodes. The RF24toTUN application running on a Raspberry Pi automatically performs discovery and routing for incoming TCP/IP data, so users only have to configure static IP addresses, and a unique identifier for each node when used with RF24Mesh.

Setup (Arduino & RPi):

 RPi:

  Install RF24, RF24Network, RF24Mesh & RF24toTUN libraries
  Note: RF24toTUN requires the boost libraries. Run sudo apt-get install libboost1.50-all 
  a: wget http://tmrh20.github.io/RF24Installer/RPi/install.sh
  b: chmod +x install.sh
  c: ./install.sh  (Note: This should be run without sudo or all your base files will belong to root)
Arduino:

 a:  Install RF24 lib

  b: Install RF24Network (Development Fork)

  c: Install RF24Ethernet library
  d: If using with a RPi, run any of the Getting_Started examples. You should be able to ping and connect to the remote Arduino.
  e: If using with a SLIP interface (non-RPi) :
      1. Run the  SLIP_Gateway.ino example on the Arduino connected to the PC/MAC etc
      2. Run the SLIP_InteractiveServer.ino example on another Arduino.
      3. Once the SLIP interface is configured as per the examples, you should be able to ping and connect to the remote Arduino.

 RF24Ethernet - Setup, config and demo

The initial proof of concept interface borrowed directly from the SerialIP library, using the uIP TCP/IP stack and uIP proto-sockets/proto-threads. Since testing was very succesful, the library has been developed to provide a simpler user interface, very similar to the Arduino Ethernet library. In its current form, a node running RF24Ethernet can act as a standard TCP server, and can function as a telnet or web server, by modifying the included example.

How can this be used?

The possibilities from here are virtually endless, with very inexpensive sensor networks being provided with direct TCP/IP connectivity. One of the more obvious applications involves simple retrieval of information from network or internet sources, such as the time or the current weather. Controlling a connected LED or other attached device can be as simple as opening a bookmark in your browser.
 
To take things a whole lot further, this could even allow RF24Network to be utilized as a bridge betwen LANs, using sensor networks for emergency commuinications etc, when main connections are down. Testing shows that speeds  across the network are slightly better than or equivalent to a dial-up connection, so it seems to provide a practical solution for low-speed data transfers like sensor data, text-based email or chat.

Results of Testing:

The above picture presents an idea of the latency, etc when sending ping requests of varying sizes from the RPi master node to a connected sensor node.

Documentation/Further Development?

See http://tmrh20.github.io/RF24Ethernet for available documentation.

Update Nov24:

Working with UIP TCP/IP stack has been a bit of a challenge to say the least, but things seem to be coming along nicely. The RF24toTUN library has been updated to route payloads according to the radio MAC address, so integration with RF24Network is coming along nicely. Support for the RPi to send fragmented multicast payloads has been added, so ARP requests are working nicely within RF24Network. This will allow full integration with RF24Mesh, to create a dynamic TCP/IP mesh sensor network if desired, with very little to no configuration needed by the user. I've commited the current code to a new branch (master_dev) to hold the new code until it is cleaned up and more functionality is added. Currently, it only supports incoming data connections as a simple server, and ICMP packets.

Update Nov30:
The Server and Client API to match the official Ethernet library is now in place and working for the most part. DHCP, DNS lookups and UDP in generall is not developed yet. There are still some bugs and oddities to work out, but overall, the TCP server and client seem to perform fairly well given the circumstances. Web client and server examples have been included.

Update Dec 7:
The library seems to be working very well now, and the API is about 95% complete. Most bugs have been addressed, with only a few minor issues left, generally surrounding timeouts during very large data transfers. The existing examples have been updated, and some new examples have been added, demonstrating a simple web server and interaction with a sensor node via a web browser. Some documentation has also been created and is linked below.

Update Dec9:
RF24Mesh has been integrated with RF24toTUN, to provide automated addressing and mesh support for RF24Ethernet. When building RF24toTUN, use 'sudo make install MESH=1' to compile with RF24Mesh support. If using with RF24Ethernet, see the new SimpleServer_Mesh.ino example for usage. An install script has also been created for RPi, to simplify installation of the various libraries.

Update Dec 14:
Updated RF24Ethernet to support SLIP or TUN devices as well. This requires RF24Mesh to perform MAC/IP translation. Two new examples have been added, SLIP_Gateway.ino demonstrates how an Arduino can act as a simple USB interface to any device that supports SLIP. SLIP_InteractiveServer demonstrates how to use RF24Ethernet and RF24Mesh with a SLIP or TUN interface.

Update Dec 28:
Modified the MAC address format due to issues with standards when using it on RPi/Linux. Modified the timing of uip restarts. Updated RF24toTUN to provide cmd line configuration for all major options including node addressing. Fixes and updates to RF24Network and RF24toTUN seem to have addressed some buggy behaviour.

Update Dec 30:
Todays latest updates should be applied along with the latest updates to RF24toTUN and RF24Network. The main change fixes corrupt client requests, which would have been noticed when establishing outgoing connections. The remaining changes revolve around reliability, the order of operations, and the timing of things, and seem to really bring RF24Ethernet closer to a stable, reliable library. 
 
Update Jan 2, 2015
This round of updates comes after a very long round of testing and coding. The RF24 library has been updated to provide even better timing, reliability and throughput. Changes to RF24Network improve fragmentation/reassembly. RF24toTUN has been updated to fully support all 3 datarates, and throughput has been improved for all speeds. RF24Ethernet has been updated to provide a timeout for connections, part of better handling for TCP window reopening and failures, and allows users to configure the periodic timer via uip-conf.h to provide faster transfers. These changes bring RF24Ethernet a lot closer to a stable release. Testing shows that these updates provide easily noticeable increases in reliability and speed at all levels, and updating all the mentioned libraries is recommended.

Update Jan 4, 2015
Added connection timeouts to recover from hangs during failed client donwloads, which adds improved reliability to the library. Added better TCP window management to prevent those hangs during client downloads, along with configuration options. Documentation updated to include new features and options.

Update Jan 16, 2015
Updated to v1.2b - Adds UDP and DNS support along with a pile of fixes and updates. Users should update RF24Network and RF24toTUN along with RF24Ethernet. This will be the final update to this blog post. See the documenation below or check back here at tmrh20.blogspot.com for additional posts.

See https://github.com/TMRh20 for all related code/library downloads, and http://nrf24.github.io/RF24Ethernet for available documentation.