Showing posts with label wifi. Show all posts
Showing posts with label wifi. Show all posts

Thursday

RF24Gateway - What Does It Do And How Do I Use It?

 RF24Gateway - What Does It Do And How Do I Use It?

 A simplified explanation...

 So maybe you've taken a look at the RF24 communication stack and have seen RF24Gateway, along with the RF24Ethernet libraries, and are wondering what they do? I've noticed some confusion among users who aren't sure what the libraries do exactly.

 The simplest explanation, is that it allows users to use nRF24L01 radios as standard Network Interface Cards (NICs). Once RF24Gateway is installed, its like having a standard NIC linked into the wireless RF24 radio network.

This means users don't need to understand programming, RF24 APIs, or anything much about how the RF24 comm. stack operates in order to use RF24Gateway &/or RF24Ethernet. It means users can use standard networking tools like the 'ping' command to test network connectivity, or more advanced tools & protocols like MQTT, HTTP, etc to interact with their Arduino devices or other Raspberry Pi/Linux devices wireless, over RF24Gateway & the radio network.

Linux users, for example, can connect to other Linux devices over the RF24 based network by using SSH, SSL or other secure protocols.

A good general understanding of standard networking and protocols is probably going to be beneficial to anyone trying to make use of RF24Gateway/RF24Ethernet. The system uses the proprietary ESB protocol by Nordic for communication, and although it operates on similar frequencies to WiFi, it is not a WiFi based system. However, it does allow your RF24Ethernet & RF24Gateway devices to interact with WiFi devices on your network, because RF24Gateway can link the WiFi & RF24 networks together using standard networking techniques like forwarding and routing, or by using an intermediary like an MQTT server.



 This means advanced users can also do things like link RF24 based sensor networks together by using VPNs, SSH tunneling, etc and create a centralized environment to collect and display data, control devices, or otherwise interact with the networks. I've experimented with automation systems like Home Assistant and Node Red to create these types of networks myself, and there are a number of alternatives that can be easily integrated into the system. Networks can work over very wide geographical areas, covering many KM locally, via radio link, and spanning the globe using VPNs or SSH tunneling.

 RF24Ethernet is companion software to RF24Gateway, which is meant to run on smaller devices like Arduino Uno, Nano, etc. while RF24Gateway is designed solely for Linux devices. RF24Gateway typically requires little to no programming, while RF24Ethernet lets users build web servers, sensor nodes, interactive nodes with on/off switches, lighting controllers, etc. and customize things to the nth degree, or simply build off the included examples.

 ESP32 based devices have become extremely popular, but in a large scale sensor network, WiFi is a very power hungry alternative to nRF24 based devices, which can offer similar features, wireless protocols & connectivity. 

To sum it up, RF24Gateway & RF24Ethernet provide connectivity & functionality similar to a WiFi network, but using much less power, and are able to support the smallest devices like ATTiny all the way to larger, much more powerful computers like Raspberry Pi and other Linux based devices that support GPIO & SPI capabilities. By default, the system uses the RF24Mesh and RF24Network libraries, so all of the internal networking and routing is handled automatically.


 

Friday

First Little Project with ESP32 - WiFi Kit 32 OLED/BLE

Taking the First Steps - ESP32 - WiFi Kit 32 OLED/BLE
Attempting to build something useful

As per my previous post I have an opportunity to utilize these ESP32 modules in projects, so I thought I'd start with something pretty straightforward. In this case, I'll be using both ESP32 modules and a DHT22 temperature and humidity sensor, all provided by DigitSpace, to manage a few tasks:

1. Get and display the current date/time
2. ESP #1 get and display the current household temperature and humidity
3. Pass the Temp/Humidity info to ESP#2
4. ESP#1 Download current weather information and display it using the OLED
5. ESP#1 Download local daily Covid19 stats and display it using the OLED
6. ESP#2 receive the info from ESP#1 and alert via LED and OLED if temp/humidity out of range

The overall goal here is to create something useful while running the devices through some general tasks that test out the operation and programming for the things I am most interested in here, the WiFi and the OLED display.

Hardware:

2xESP32 WiFi Kit 32 w/OLED
1xDHT22 Temperature/Humidity sensor

Note: The ESP modules come with the pins separate, you will need a soldering iron to connect them.

Setting Up The IDE & Software:

1. Install the ESP32 Arduino Core per the installation instructions for Arduino Boards Manager
2. Select Heltec WiFi Kit 32 for the board
3. I chose the U8g2lib for the OLED display. It is available via Arduino Library Manager

Getting it All Up and Running

Graphics Library:

I started out with just the U8g2lib for the OLED, choosing the font group, font size etc, and figuring out basically how to use it. In this case, with the font I chose, I can fit five lines of 20 characters onto the screen, so I created 5 character arrays, and have them drawn on screen every loop. Then it is just a matter of loading the buffers with whatever information I want displayed.

For that, I created a simple function to take in a char array along with its length and the intended line number to display it on. Then all I have to do is put data in the buffers at any given time, and it gets displayed on screen. This may be over-complicating things a bit, but there is no messing about with what to refresh, what lines to draw etc, just put data in the buffers and it gets drawn.

Current Date/Time:

This part was surprisingly easy. It is a matter of specifying an NTP server or pool of servers, the GMT and DST offsets, then calling a standard time function to retrieve the information as desired.

WiFi Connectivity, HTTPS/SSL:

There are a number of examples and tutorials regarding getting connected to WiFi itself, so I'll leave that part out.

The client and SSL/HTTPS connectivity is nice to have, but seems to be a bit painful, since you apparently need to grab the certificate for the root authority of the site in question and put it in your code in order to establish a connection.

In order to grab the certificate, I found the easiest way is with a simple Linux command:
openssl s_client -showcerts -connect <hostname>:443

Beyond that, I pretty much followed the tutorial found here for setting up SSL/HTTPS connections.

It seems that once connected to a host via SSL, the ESP32 will continue to use the certificate as specified, since in order to connect to a second website for information,  I had to use SSL and specify the root_ca cert for the second site, even though I could have connected without SSL. I'm still looking into how to clear that from memory and allow normal connections etc.

The DHT22 Sensor:

This step was pretty simple also. The sensor itself connects to 3.3v, GND and any suitable GPIO pin. There is a DHT library specifically for the ESP32 available in the Arduino IDE with included examples.

The Modules in Operation:

The main module will grab information from the internet on startup, time/date, weather and covid19 stats for the local area.
The date/time and household temperature & humidity are constantly displayed on screen and updated regularly.
The bottom portion of the screen alternates between weather and covid19 information every 10 seconds.
The weather is updated every hour, and daily covid19 data refreshed every 6 hours.

The secondary module will receive the humidity and temperature sensor data from the main module as it is refreshed via http get request. If out of range a message will display on screen, and the onboard LED will flash. Users can disable the flashing LED by querying http://<IP-OF-ESP32>/s
An 's' character will stop the alerting, any other character will resume.

A video of the modules in operation is below:



The onboard LED does not show up well in the video because of the light on the camera. The OLED text was too bright and blurry on screen without the light, so it was a bit of  a compromise.

Overview/Result:

This was a nice project to work on with some opportunities to try a few new things, learn some things and mess around with some new hardware. The code itself could be cleaned up a bit and fine tuned, since I didn't pay much attention to conserving memory or program space, but there is more than enough to handle the tasks at hand.

I think I'll keep the main device in operation, and maybe look at updating the code to handle additional tasks & processing. I think the secondary device could easily be an AVR based device with an audible alerting system, using RF24 for data transfer. It is nice to have the ESP32, but it seems a bit overkill for the situation.

This was a good way to test and get used to having the OLED display and WiFi capabilities on hand to do with whatever is needed. All in all I'm really liking these devices. More to come with ESP32 projects and development!

Code/Sketches:

https://github.com/TMRh20/Sketches/tree/master/ESP32_WiFiKit32



Sunday

nRF24l01+ Library Roundup
Overview and Status of RF24 Arduino/Linux Comm. Stack

Six, seven years? Has it really been that long? According to GitHub that is in fact the case, as my first commits to my own fork of the RF24 library took place in early 2014. It all started by identifying existing limitations in the available libraries and working to achieve the highest level of performance and reliability possible. The main fork for RF24 at the time was written and maintained by ManiacBug, but he dropped off the scene shortly after publishing this very nicely designed library. Other users attempted to add support for various devices and address certain bugs, but nobody had really taken a thorough look at the capabilities of the hardware vs what was achievable at the time. Only through a lot of reading, testing and learning to program was I able to make those initial changes to begin work improving the RF24 core library. Looking back now, I am very grateful that I picked ManiacBugs code base as a place to get back into programming in C/C++, as it was well thought out and I discovered ways of doing things that I previously did not understand or know about.

And with that began a long and arduous journey into the internal workings of nrf24l01+, Arduino (AVR) and RPi (Linux) devices with the stated goal of optimizing the library to whatever extent possible. Once the RF24 library began to take shape, I began looking at the RF24Network library, another very nicely designed bit of code by ManiacBug, but it had its limitations and problems. Many issues were addressed, features and functionality added in order to push the nrf24 devices to the test in a multi-device scenario. It performed much better than expected using this OSI Layer 3 (network) library.

At that point, I began to really understand the OSI model, how the different layers actually work together and what they do down the last bit. Based on a number of conversations and user input, I came up with the idea for RF24Mesh, which is basically another layer on top of RF24Network -> RF24 that handles addressing, similar to DHCP, but able to verify connectivity and reconnect nodes at any point in the mesh. This allows nodes to move around and reconnect quickly as required while maintaining the mesh structure. The Network & Mesh layers use a 'master' node that acts like the gateway in a standard IP network, and it provides addressing and resolution for RF24Mesh.

Based on input and discussions with users and the performance of the radio modules when used in a multi-device mesh/network scenario along with the emergence of IoT and sensor networks, it seemed like the thing to do would be to work my way up the OSI model, so that is exactly what I did. Upon discovering the uIP stack for Arduino, I realized it would be possible to add ip4 support to some of the smallest devices like ATmega328 AVRs, and larger devices like RPi could just use their own IP stack and encryption etc. The uIP stack and related software was quite a challenge to understand and implement, as uIP is written in C and is designed to be as small and lightweight as possible as opposed to being human readable and easy to understand. The UIPEthernet library was also essential in helping me to understand and implement RF24Ethernet.

With that, an OSI RF24 comm. stack was established, with RF24, RF24Network, RF24Mesh, RF24Ethernet and RF24Gateway libraries all working together and inter-operational at any of the layers. This means that on a RF24Gateway/RF24Ethernet mesh network incorporating RPi and Arduino nodes, devices can choose to operate using OSI layer 2,3,4,5 and/or 7 (Data Link, Network, Transport, Session, Application). When using the higher layer libraries (RF24Gateway & RF24Ethernet) standard IP connectivity is established and devices can utilize standard encryption & authentication protocols (SSH, HTTPS etc.) to secure their traffic as per the OSI model.

The RF24 Communication Stack vs The OSI Model

With the associated libraries and enhanced functionality, the RF24 comm. stack is able to provide IoT and sensor network connectivity and capabilities to suit many different scenarios at a very low price point with much lower power consumption than WiFi networks. With the production of many devices like RPi, ESP8266 and ESP32 which support WiFi, users can construct and deploy IoT networks very easily, extending them as required.

Even now, it is really kind of cool to witness the system in action, using these little radios on tiny little computers we refer to as 'Arduinos'. The methodology and processes in place to manage a network of devices, fragmentation & reassembly, mitigate wireless data collisions and ensure delivery of data are quite effective, from the radio hardware and auto-ack functionality, all the way up to the TCP/IP and Mesh levels. With TCP/IP for example, using Raspberry Pi devices, the MTU is 1500-bytes, so each packet can require up to 63 sequential, successful data transmissions at the core RF24 layer, but this still works relatively well in real-world testing/usage.

It has been a lot of fun and a great learning experience so far. The creation of RF24Gateway really allowed the limits of the radios to be tested, and provided an excuse to play around with different network/IP configurations, routing scenarios and traffic handling at any level of the OSI model. A full and complete understanding of how devices operate on the internet or similar networks helps dramatically in troubleshooting, testing and development of systems and software that operate using these protocols.

Going forward, the development of the RF24 libraries has slowed significantly along with bug reports and issues. My focus has recently been on cleaning things up, finding and fixing bugs, mostly in the higher layers, and improving the user interface via examples and documentation. I had previously almost given up on the RF24Gateway and Ethernet layers as being too difficult to fix, but with the rise of more powerful MCUs and the potential for expanding their capabilities, it seems worthwhile to take another crack at working out bugs and issues.

 It would be nice to find another similar radio device with lower power consumption and/or more advanced capabilities and throughput to create more robust networks and mesh capabilities. The current design allows for speeds up to about 20KB/s over IP (RF24Gateway) using two Linux devices, but it would be nice to have a higher throughput to allow for more nodes, longer range and more advanced communication scenarios.

In conclusion I want to extend a big thank you to everybody who contributed ideas, analysis, information and programming skills along the way. An extra thank you to Avamander (GitHub) who has played a big part in ongoing maintenance, support and development!

Update: Aug 2020

Wow, I mentioned that I would be working on finding and fixing bugs due to the slowdown involving support, dev and maintenance etc, but did not expect to spend so much time, or that I would be able to identify so many issues, their root cause and a solution. Some pretty significant issues have been fixed throughout the RF24 stack, including 1 major bug in RF24, affecting all libraries, a number of bugs/issues affecting functionality and reliability in RF24Network and RF24Mesh, and a memory issue in RF24Ethernet affecting stability.
RF24Gateway has been updated with better handling of interrupts, a few corrections and the first release made, v1.0 due to all the issues that have been addressed. Again, a portion of these improvements are due to assistance, testing and input from a select few users. Users should notice a marked improvement in stability, functionality and the ability to recover from significant errors/hardware issues.

RF24 Communication Stack:
https://github.com/nRF24
https://nrf24.github.io/RF24Gateway
https://nrf24.github.io/RF24Ethernet



















Friday

Summary: A Personal Experiment & A New Model for Open, Open-source, Anonymous & Secure Wireless Communication
Communication, Network, Protocol Design & Implementation Using Arduino & Raspberry Pi


Updated Nov 23, 2016

Over the course of the last few years, I have been working on a standardized, open-source communication stack that can be scaled from the biggest, most powerful devices, to tiny inexpensive devices like Arduino micro-controllers.

This looks familiar!

In the course of this time, I've made great progress in development of the core RF24 radio driver for nrf24l01+ radio modules. The overall goal of this project is independent of the radio devices, but these devices were chosen due to their relative capabilities vs cost, availability and ease of use. They are simply 2Mbps half-duplex wireless communication devices. Virtually any type of wireless devices can utilize the same theory of operation, including the mesh (RF24Mesh) style protocols.

What is it ? This system/communication stack allows nrf24l01+ radio/wireless devices to be utilized as a standard Network Interface Card (NIC) using TCP/IP & Mesh protocols. This is a summary of the experiment, system, general operation, and some of what I've learned.

Why? The programming, design and implementation of this system is part of a personal experiment I've been conducting involving the nature of wireless communication systems and protocols in relation to privacy, security, the Internet and the Internet of Things.

Theory of Operation & Design:


Consideration A: Open Networks + Anonymity + Automation


In general operation the basic principle of the communication stack is thus:
1. Each device utilizes a unique identifier while connected to a given access point.
2. This 'unique' identifier essentially represents an IP address in this implementation
3. For reasons of simplicity, this implementation also requires a 'static' identifier or IP address, but using more powerful micro-controllers and radio devices, the identifier can be assigned randomly or via DHCP
4. Devices simply attach, at any point in the network, using any available node(s) as an access point.

In a large scale model incorporating this type of design, with proper randomization of identifiers, individual devices and users are (potentially) indistinguishable from one-another initially and generally anonymous.

The next stage of course involves communication to enable the IoT/automation & allowing any capable device to connect. In this case there appear to be some options:

Option A: Leave the network open, and devices can choose to encrypt/authenticate their communications.This may require additional protective measures to prevent abuse in a large or publicly accessible system.

Option B: Assign selected devices a shared "network key" of sorts and incorporate encryption protocols into the network

Either way, it seems that wireless devices & users can inherently maintain their relative anonymity in a system that is designed a little differently.


Consideration B: Mesh Networks

The Network Layer: 
In studying the different layers involved in a capable communication stack, it seemed that many of the boundaries between layers were blurry, with many different systems 'breaking the rules' as it were. A good example of this is TCP/IP which does not strictly adhere to the OSI model.


In my design, a 'static' network architecture is used, which resembles a tree configuration. The simplicity of the RF24Network layer means that each node can handle routing of any type of information, and only needs to know which nodes it is directly connected to. 

Some of the features/functionality involving the RF24Mesh layer have been placed into the RF24Network layer directly. The reasons are mainly due to efficiency and limitations of small devices and this is under review.

Due to the limitations of the specific device used ( nrf24l01+ ) radio modules, each device is limited to 5 individual connections in the mesh, but this could be easily extended for use with more powerful radio devices.

The Mesh Layer:


The mesh layer utilizes its knowledge of the 'static' network architecture to allow individual nodes to attach themselves to the network at any given point where another node is active.

As mentioned above, each device is currently pre-assigned a unique identifier from 0-255 due to the 8-bit nature of very small & low power devices. In a large scale or commercial implementation, these identifiers can be assigned automatically & randomly, with devices then registering themselves in DNS if required, or simply acting as an anonymous client.

The fundamental benefit of a mesh style network is that nodes will automatically adjust their physical address and connection as required as they move around the network or lose connectivity. Any other nodes will provide connectivity by allowing other nodes to attach, and routing or accepting their traffic according to specifications.

Per RF24Network design, there is very little overhead for individual nodes, since they only need to keep track of which node they are connected to, and which nodes are using them as an access point.

Consideration CLimitations

There are currently a number of fundamental imitations and odd 'quirks' that have to be dealt with due to the nature of really small devices like 16Mhz AVR devices.

AVR Devices: The chosen limitations in this case include some of the smallest and least power hungry devices available to any consumer. ATTiny devices can operate as low layer nodes & relays using RF24Mesh, and slightly faster 328 based AVR devices can run an actual IP stack.

uIP TCP/IP Stack:  One of main issues with small devices involves certain limitations of the IP stack. As soon as slightly larger devices are utilized, more robust IP stacks are available that make operation faster and more reliable. uIP has provided a very interesting learning opportunity and is quite amazing given what it can do with extremely limited resources.

Radio Devices: nrf24l01+ radio modules are max 2Mbps half-duplex communication devices with a specific 5-byte addressing scheme.


Consideration D: Fully Automated Networks



Software and wireless development capabilities have come to the point of enabling fully autonomous networks that essentially manage themselves, devoid of human interaction. This goal has been partly realized or demonstrated in theory and action with the RF24Mesh layer, and could easily be built upon and implemented by large scale commercial interests.

The concept of independent networks that configure and maintain themselves may seem to some as a wild idea or a bit of a pipe dream, but we have all of the tools and capabilities within our grasp. It is simply a matter of engineering and development.

Connectivity of individual users can be managed entirely via authenticated and encrypted connections, so there is clearly no need for individual identification of users or user management beyond the application layer.

This type of system could allow communities to simply put up hardware and provide access. The system would be automated to the point that if a device fails, nearby devices send out an alert and the equipment is replaced. That is all.

Consideration E: Wireless Technology and Potential Breakthroughs



Communications technology, aka the Internet, has become ubiquitous in many ways to the daily life of millions and millions of individuals and businesses. Unfortunately, this type of connectivity has not been available to consumers in poor, rural or remote areas in a real, usable way.

Even with existing satellite technologies showing great promise towards providing quality communications capabilities to new places, it seems that additional measures will need to be taken to provide for the age of automation. This is the kind of automation that will see real-time control scenarios, with vehicles communicating among themselves, drawing in information from traffic control systems, news, weather, and countless other systems.  This is the kind of automation that will see drone based grocery and package delivery, combined with, at some point, flying vehicles and high speed transportation systems.

The implementation of automation on a massive scale is going to require something better than the kind of service we get from existing cellular networks and current technology. This is going to require a lot more than what public cellular, 5G and fiber technologies can provide. What we need moving forward is not just a standard step forward in communications technology, but a giant leap forward, while keeping in mind all the things that make the internet and technology beneficial to everybody.


General Summary:

It now seems entirely possible and practical to design and implement fully featured, autonomous. large scale, wireless network architecture that fundamentally provides anonymity and promotes privacy among its users, while providing full mesh networking capabilities and automated network management.

With the emerging nature of communications devices, potential breakthroughs in communication technology, and wireless/IoT/automation technology emerging as a major component of our future progress, it seems very important to study, understand and develop networking and communication technologies with us, the users in mind.

It also seems that wired communications systems have already become a thing of the past, with more and more devices embedding chips and features to support Wireless and WiFi protocols and interfaces. The importance of privacy and security is fundamental to the operation of any large or public communication system, and any large, public & worthwhile communication platform will incorporate these basic ideals as inherent to its operation.


http://nrf24.github.io/RF24Ethernet

http://nrf24.github.io/RF24Gateway


Wednesday

Raspberry Pi/Linux with the nrf24l01+ & RF24Gateway
 A guide/example of advanced RF24/nrf24l01+ usage and monitoring on Linux devices

Overview:

There are many examples of basic RF24 usage, but not so many demonstrating complex communication scenarios and networking. The RF24 communication stack is free, with all development information, examples, code and documentation available on GitHub, designed with Arduino and RaspberryPi/Linux devices in mind.

The RF24 stack generally follows the OSI model, and provides a separate library for each layer, with efforts toward proper documentation for each layer. This allows any type of data/communication to pass over the radios, including TCP, UDP or any other protocol, and users can write their own code to link devices together using any layer(s).

So, what can we really do with these little radios?

The nature and low-cost of the system allows literally anybody with interest in wireless communication to setup, deploy, and experiment with small scale, self-healing wireless networks, or start writing their own code with the core RF24 driver, in a very short period of time.

The stack currently does not include sensor specific code or functionality, it is simply a communication stack which enables all kinds of different functionality.


Requirements, Hardware Setup & Installation:

1. A Raspberry Pi or Linux device a compatible radio device connected
    RPi Install Script

2. An Arduino device with a compatible radio device connected
    Arduino Install (Select RF24, RF24Network, RF24Mesh, RF24Ethernet libraries)

See the RF24 docs for hardware setup info & installation
Note: Please report any issues at https://github.com/TMRh20/RF24/issues

Demonstration and Information:

The video below demonstrates how to use RF24Gateway in general - Setting up nodes and communicating over RF24 using RF24Network, RF24Mesh, and standard protocols (ICMP, HTTP, SSH), along with monitoring

1. Config on RPi & Arduino devices
2. Complex mesh networks: How the different layers (RF24Network,RF24Mesh & RF24Ethernet) can be used with RF24Gateway
3. RF24Ethernet: Simple TCP/IP communication and web server running on Arduino over RF24
4. RF24Mesh: Node config & interaction with RF24Gateway
5. RF24Network: Node config & interaction with RF24Gateway
6. How to use WireShark to monitor user payloads with RF24Gateway and LUA Wireshark dissector script (using laptop w/SSH & X11 forwarding in this example)
7. How to configure Raspberry Pi as a standard node in an existing RF24Mesh
8. How to sniff RF24Network, RF24Mesh, and RF24Ethernet traffic using RF24Gateway & Wireshark(optional)
9. Tacos... oops they were already eaten, maybe next time



As shown in the video, any nodes running RF24Ethernet, RF24Mesh or RF24Network can be used with a master node running RF24Gateway, allowing users to try out or utilize the different layers quite easily.

RF24Ethernet: Provides TCP/IP communication. Very simple and reliable way to control on/off devices via a web browser or scripting. Has all functionality of lower layers.

RF24Mesh: Provides automatic (mesh) networking for RF24Network. RF24Mesh networks are self-healing and dynamic in nature.

RF24Network: Provides addressing, routing, fragmentation/reassembly of payloads via manual/statically defined wireless networks.

See the detailed overview and related pages at http://nrf24.github.io/RF24Ethernet


Playing around with the Arduino Uno R4 Minima, Uno R4 Wifi and Uno Q

Playing around with the Arduino Uno R4 Minima, Uno R4 Wifi and Uno Q Stepping up to a new level of micro-controller     Thanks to the fine p...