Monday

A simple guide to managing memory and variables in C++

Transferring Data between Systems Using RF24
A simple guide to managing memory and variables in C++

One of the most common issues that seems to be coming up more and more recently relates to transferring data between devices and how different devices store data in memory. With more devices using 32 and 64 bit processors, integrating things between the newer devices and 8-bit MCUs can require a little bit of forethought and planning.

Memory:

What it all comes down to is memory and how different devices store data in memory. This is very important to understand when transferring data between systems and ensuring that code is portable between various devices.

When using an 8-bit device, such as an ATMega 328 based MCU (Arduino Uno, Nano, etc) the smallest unit for storage in memory is 8bits or 1-byte. This means that even a boolean (true/false) variable will take up 8-bits of memory space. On a 32 bit system, (ESP32, ESP8266) the same boolean value will use 32-bits of memory space. Why does this matter? When transferring data between systems or creating code, certain variables will take up different amounts of space in memory depending on the architecture (8-bit, 32-bit etc) thus introducing inconsistencies and misinterpretation of data.

As an example, see https://www.arduino.cc/reference/en/language/variables/data-types/int/

Using variables like int, long etc can cause confusion because they are interpreted differently on different devices. Sending a 2-byte integer from an 8-bit system to a 32-bit system works fine, as long as the size of the variable is specified on both systems such as int16_t, rather than trying to use 'int' on both systems.

Take for example the following data structure:

struct myStruct{
  bool variable1;
  uint32_t var2;
  uint8_t var3;
  int16_t var4; 
}

On an 8-bit system, this data structure will use 8 bytes of memory space, but on a 32-bit system it will take up 12-bytes. This is because the minimum unit for storage on a 32-bit system is 32-bits, so 32-bit devices will need to add padding wherever the data does not align in chunks of 4-bytes (32-bits). When using data structures, devices will typically attempt to pack the data as efficiently as possible, so re-arranging the data structure properly will prevent the 32-bit system from adding padding and making the data structures incompatible.

struct myStruct{  
  uint32_t var2;
  int16_t var4;
  uint8_t var3;  
  bool variable1; 
}

In the above example, the data is arranged in chunks of 4-bytes, starting with a 32-bit unsigned integer (4-bytes), followed by a 16-bit signed integer (2-bytes), an unsigned 8-bit integer (1-byte) and a boolean variable (1-byte), so the 32-bit system will pack the data into memory exactly the same as the 8-bit system, using only 8-bytes of memory space.

This is important, because when sending data between devices, users can just send the data structure as-is and if packed properly, the data will align regardless of whether it is a 32-bit, 64-bit or 8-bit system.

When developing code, specifying the size of all your variables and aligning data properly in your data structures will ensure that the code and communication is portable between devices.

https://www.gnu.org/software/libc/manual/html_node/Integers.html



Friday

First Little Project with ESP32 - WiFi Kit 32 OLED/BLE

Taking the First Steps - ESP32 - WiFi Kit 32 OLED/BLE
Attempting to build something useful

As per my previous post I have an opportunity to utilize these ESP32 modules in projects, so I thought I'd start with something pretty straightforward. In this case, I'll be using both ESP32 modules and a DHT22 temperature and humidity sensor, all provided by DigitSpace, to manage a few tasks:

1. Get and display the current date/time
2. ESP #1 get and display the current household temperature and humidity
3. Pass the Temp/Humidity info to ESP#2
4. ESP#1 Download current weather information and display it using the OLED
5. ESP#1 Download local daily Covid19 stats and display it using the OLED
6. ESP#2 receive the info from ESP#1 and alert via LED and OLED if temp/humidity out of range

The overall goal here is to create something useful while running the devices through some general tasks that test out the operation and programming for the things I am most interested in here, the WiFi and the OLED display.

Hardware:

2xESP32 WiFi Kit 32 w/OLED
1xDHT22 Temperature/Humidity sensor

Note: The ESP modules come with the pins separate, you will need a soldering iron to connect them.

Setting Up The IDE & Software:

1. Install the ESP32 Arduino Core per the installation instructions for Arduino Boards Manager
2. Select Heltec WiFi Kit 32 for the board
3. I chose the U8g2lib for the OLED display. It is available via Arduino Library Manager

Getting it All Up and Running

Graphics Library:

I started out with just the U8g2lib for the OLED, choosing the font group, font size etc, and figuring out basically how to use it. In this case, with the font I chose, I can fit five lines of 20 characters onto the screen, so I created 5 character arrays, and have them drawn on screen every loop. Then it is just a matter of loading the buffers with whatever information I want displayed.

For that, I created a simple function to take in a char array along with its length and the intended line number to display it on. Then all I have to do is put data in the buffers at any given time, and it gets displayed on screen. This may be over-complicating things a bit, but there is no messing about with what to refresh, what lines to draw etc, just put data in the buffers and it gets drawn.

Current Date/Time:

This part was surprisingly easy. It is a matter of specifying an NTP server or pool of servers, the GMT and DST offsets, then calling a standard time function to retrieve the information as desired.

WiFi Connectivity, HTTPS/SSL:

There are a number of examples and tutorials regarding getting connected to WiFi itself, so I'll leave that part out.

The client and SSL/HTTPS connectivity is nice to have, but seems to be a bit painful, since you apparently need to grab the certificate for the root authority of the site in question and put it in your code in order to establish a connection.

In order to grab the certificate, I found the easiest way is with a simple Linux command:
openssl s_client -showcerts -connect <hostname>:443

Beyond that, I pretty much followed the tutorial found here for setting up SSL/HTTPS connections.

It seems that once connected to a host via SSL, the ESP32 will continue to use the certificate as specified, since in order to connect to a second website for information,  I had to use SSL and specify the root_ca cert for the second site, even though I could have connected without SSL. I'm still looking into how to clear that from memory and allow normal connections etc.

The DHT22 Sensor:

This step was pretty simple also. The sensor itself connects to 3.3v, GND and any suitable GPIO pin. There is a DHT library specifically for the ESP32 available in the Arduino IDE with included examples.

The Modules in Operation:

The main module will grab information from the internet on startup, time/date, weather and covid19 stats for the local area.
The date/time and household temperature & humidity are constantly displayed on screen and updated regularly.
The bottom portion of the screen alternates between weather and covid19 information every 10 seconds.
The weather is updated every hour, and daily covid19 data refreshed every 6 hours.

The secondary module will receive the humidity and temperature sensor data from the main module as it is refreshed via http get request. If out of range a message will display on screen, and the onboard LED will flash. Users can disable the flashing LED by querying http://<IP-OF-ESP32>/s
An 's' character will stop the alerting, any other character will resume.

A video of the modules in operation is below:



The onboard LED does not show up well in the video because of the light on the camera. The OLED text was too bright and blurry on screen without the light, so it was a bit of  a compromise.

Overview/Result:

This was a nice project to work on with some opportunities to try a few new things, learn some things and mess around with some new hardware. The code itself could be cleaned up a bit and fine tuned, since I didn't pay much attention to conserving memory or program space, but there is more than enough to handle the tasks at hand.

I think I'll keep the main device in operation, and maybe look at updating the code to handle additional tasks & processing. I think the secondary device could easily be an AVR based device with an audible alerting system, using RF24 for data transfer. It is nice to have the ESP32, but it seems a bit overkill for the situation.

This was a good way to test and get used to having the OLED display and WiFi capabilities on hand to do with whatever is needed. All in all I'm really liking these devices. More to come with ESP32 projects and development!

Code/Sketches:

https://github.com/TMRh20/Sketches/tree/master/ESP32_WiFiKit32



Saturday

Upcoming Projects: ESP32 - WiFi Kit 32 OLED/BLE

Playing around with ESP32 - WiFi Kit 32 OLED/BLE
New Toys to Play With!

I was recently provided some nice new toys to play with by the people at DigitSpace and will be documenting some upcoming projects involving these devices. The main player will be an ESP32 with built in OLED, which is very nice to have, not just because of the ESP32 capabilities and processing power, but the built-in OLED provides a nice way to get output and information without having to watch a serial monitor or send data to another device etc.

ESP32 WiFi Kit32 OLED from www.digitspace.com

I initially had some problems connecting the devices directly to computers via USB cable. I used different cables that work fine with my ESP8266 modules, but these would not work with any Windows, Linux or Mac devices I tried. Response, communication and professionalism was great, but neither I nor the supplier could seem to find a solution, even after contacting the manufacturer.

By chance, I ordered another module of the same model from a different supplier, and it came with a CH340G based USB->TTL module! It is starting to look like this is a known issue. So I tried it with these devices and BAM! Working nicely. Upload speed needed to be 256000 or lower.

The 5v pin on my TTL modules is connected directly to the USB 5v, so it can power the ESP32 via the 5v pin, even when configured to operate at 3.3v. If the RTS and DTR pins are not exposed, you only need to connect TX & RX, then press RST then PGM, then release RST then PGM buttons to put it into upload mode. RST again after complete. If exposed, connect DTR to GPIO0 and RTS to RST per the pinout below, and uploading will be automatic.

ESP32 WiFi Kit32 OLED Pinout

Now with the capabilities here, including a 240Mhz processor, BLE, WiFi and built in OLED, you might not think this module would be cheaper than an AVR based Arduino, but in some cases it is! Per the DigitSpace store these modules offer a great platform at a great price. Beyond the USB->TTL requirement, installation is easy. The Arduino IDE includes this device in the default ESP32 boards, and there are many examples etc. to test with.

I now have some working ESP32 devices and some other hardware that was given to use in upcoming projects! Some will relate to existing libraries and projects, but as usual I'm not sure exactly how or if I am going to be able to pull it off. Its always a challenge to understand the internal workings of newer MCUs and related code to produce the results I am expecting. It will be a challenge, but it seemed like a good idea at the time, so why not?

In any case, stay tuned for more to come with development & projects on ESP32 modules!


Sunday

nRF24l01+ Library Roundup
Overview and Status of RF24 Arduino/Linux Comm. Stack

Six, seven years? Has it really been that long? According to GitHub that is in fact the case, as my first commits to my own fork of the RF24 library took place in early 2014. It all started by identifying existing limitations in the available libraries and working to achieve the highest level of performance and reliability possible. The main fork for RF24 at the time was written and maintained by ManiacBug, but he dropped off the scene shortly after publishing this very nicely designed library. Other users attempted to add support for various devices and address certain bugs, but nobody had really taken a thorough look at the capabilities of the hardware vs what was achievable at the time. Only through a lot of reading, testing and learning to program was I able to make those initial changes to begin work improving the RF24 core library. Looking back now, I am very grateful that I picked ManiacBugs code base as a place to get back into programming in C/C++, as it was well thought out and I discovered ways of doing things that I previously did not understand or know about.

And with that began a long and arduous journey into the internal workings of nrf24l01+, Arduino (AVR) and RPi (Linux) devices with the stated goal of optimizing the library to whatever extent possible. Once the RF24 library began to take shape, I began looking at the RF24Network library, another very nicely designed bit of code by ManiacBug, but it had its limitations and problems. Many issues were addressed, features and functionality added in order to push the nrf24 devices to the test in a multi-device scenario. It performed much better than expected using this OSI Layer 3 (network) library.

At that point, I began to really understand the OSI model, how the different layers actually work together and what they do down the last bit. Based on a number of conversations and user input, I came up with the idea for RF24Mesh, which is basically another layer on top of RF24Network -> RF24 that handles addressing, similar to DHCP, but able to verify connectivity and reconnect nodes at any point in the mesh. This allows nodes to move around and reconnect quickly as required while maintaining the mesh structure. The Network & Mesh layers use a 'master' node that acts like the gateway in a standard IP network, and it provides addressing and resolution for RF24Mesh.

Based on input and discussions with users and the performance of the radio modules when used in a multi-device mesh/network scenario along with the emergence of IoT and sensor networks, it seemed like the thing to do would be to work my way up the OSI model, so that is exactly what I did. Upon discovering the uIP stack for Arduino, I realized it would be possible to add ip4 support to some of the smallest devices like ATmega328 AVRs, and larger devices like RPi could just use their own IP stack and encryption etc. The uIP stack and related software was quite a challenge to understand and implement, as uIP is written in C and is designed to be as small and lightweight as possible as opposed to being human readable and easy to understand. The UIPEthernet library was also essential in helping me to understand and implement RF24Ethernet.

With that, an OSI RF24 comm. stack was established, with RF24, RF24Network, RF24Mesh, RF24Ethernet and RF24Gateway libraries all working together and inter-operational at any of the layers. This means that on a RF24Gateway/RF24Ethernet mesh network incorporating RPi and Arduino nodes, devices can choose to operate using OSI layer 2,3,4,5 and/or 7 (Data Link, Network, Transport, Session, Application). When using the higher layer libraries (RF24Gateway & RF24Ethernet) standard IP connectivity is established and devices can utilize standard encryption & authentication protocols (SSH, HTTPS etc.) to secure their traffic as per the OSI model.

The RF24 Communication Stack vs The OSI Model

With the associated libraries and enhanced functionality, the RF24 comm. stack is able to provide IoT and sensor network connectivity and capabilities to suit many different scenarios at a very low price point with much lower power consumption than WiFi networks. With the production of many devices like RPi, ESP8266 and ESP32 which support WiFi, users can construct and deploy IoT networks very easily, extending them as required.

Even now, it is really kind of cool to witness the system in action, using these little radios on tiny little computers we refer to as 'Arduinos'. The methodology and processes in place to manage a network of devices, fragmentation & reassembly, mitigate wireless data collisions and ensure delivery of data are quite effective, from the radio hardware and auto-ack functionality, all the way up to the TCP/IP and Mesh levels. With TCP/IP for example, using Raspberry Pi devices, the MTU is 1500-bytes, so each packet can require up to 63 sequential, successful data transmissions at the core RF24 layer, but this still works relatively well in real-world testing/usage.

It has been a lot of fun and a great learning experience so far. The creation of RF24Gateway really allowed the limits of the radios to be tested, and provided an excuse to play around with different network/IP configurations, routing scenarios and traffic handling at any level of the OSI model. A full and complete understanding of how devices operate on the internet or similar networks helps dramatically in troubleshooting, testing and development of systems and software that operate using these protocols.

Going forward, the development of the RF24 libraries has slowed significantly along with bug reports and issues. My focus has recently been on cleaning things up, finding and fixing bugs, mostly in the higher layers, and improving the user interface via examples and documentation. I had previously almost given up on the RF24Gateway and Ethernet layers as being too difficult to fix, but with the rise of more powerful MCUs and the potential for expanding their capabilities, it seems worthwhile to take another crack at working out bugs and issues.

 It would be nice to find another similar radio device with lower power consumption and/or more advanced capabilities and throughput to create more robust networks and mesh capabilities. The current design allows for speeds up to about 20KB/s over IP (RF24Gateway) using two Linux devices, but it would be nice to have a higher throughput to allow for more nodes, longer range and more advanced communication scenarios.

In conclusion I want to extend a big thank you to everybody who contributed ideas, analysis, information and programming skills along the way. An extra thank you to Avamander (GitHub) who has played a big part in ongoing maintenance, support and development!

Update: Aug 2020

Wow, I mentioned that I would be working on finding and fixing bugs due to the slowdown involving support, dev and maintenance etc, but did not expect to spend so much time, or that I would be able to identify so many issues, their root cause and a solution. Some pretty significant issues have been fixed throughout the RF24 stack, including 1 major bug in RF24, affecting all libraries, a number of bugs/issues affecting functionality and reliability in RF24Network and RF24Mesh, and a memory issue in RF24Ethernet affecting stability.
RF24Gateway has been updated with better handling of interrupts, a few corrections and the first release made, v1.0 due to all the issues that have been addressed. Again, a portion of these improvements are due to assistance, testing and input from a select few users. Users should notice a marked improvement in stability, functionality and the ability to recover from significant errors/hardware issues.

RF24 Communication Stack:
https://github.com/nRF24
https://tmrh20.github.io/RF24Gateway
https://tmrh20.github.io/RF24Ethernet

Wednesday

RF24Mesh - Dynamic Mesh Networking for NRF24L01+ Radio Modules
An Overview of Progress so Far


Overview:

RF24Mesh is a C++/Arduino/Linux/Raspberry Pi library allowing many devices to simultaneously connect wirelessly and remain connected in a mesh style network using nrf24l01+ radio modules. Nodes can communicate with any other node on the network, including a master node that acts as a gateway for the mesh, providing addressing and lookups.

Once assigned a unique ID, everything is automated, and nodes can move around, go to sleep, and rejoin the mesh at any point where another node is in range. RF24Mesh supports some of the smallest and most power efficient devices available.

The RF24 Communication Stack is based on the OSI model, and RF24Mesh makes use of the features of the lower layers (RF24 (Layer 2) , RF24Network (Layer 3), and the overall capabilities can be extended via RF24Ethernet (Layers 4&5 TCP/IP)

Setting it Up:

The main thing when setting up a mesh network is to configure and test your devices accordingly, to make sure they are in a fully operational state. Most if not all of the available radio chips being sold are clones, so I've found it necessary to modify the radio devices slightly in some cases. In many cases, a capacitor (10-100uF) is added directly to the VCC/GND of the module itself to help with common power supply issues.

In the case of these modules, I've found that scraping off the last bit of the antenna, and soldering on a small 3-4" piece of wire greatly improves their performance:
In the case of the high powered PA + LNA modules, I've found it necessary to add shielding as in the post found here. Placing them close to a ground plane seems to help as well. Operating them at a lower power level may be required if the power supply is not adequate.


The modules that are low powered with an antenna generally work out of the box.

I generally recommend using an adapter, to provide a stable power source, even for the low powered modules. I've been using them with many of my modules.

From there it is a matter of installing the Arduino libraries etc. which is covered in the documentation and in earlier blog posts and the official documentation below.

Current status, testing etc.

RF24Mesh is now past the development/beta testing stage and considered (by me) to be stable, and well tested. The mesh can handle a large number of nodes, theoretically up to 255, but in practice, more like 15-25 nodes is a reasonable bet. It generally depends on the type of traffic and frequency with which it is generated, as a much larger number of nodes reporting data every minute presents only a small load for the network.

Documentation:
https://tmrh20.github.io/RF24Mesh

Source Code/Issues:
https://github.com/nrf24/RF24Mesh





Sunday

Comparative Performance Analysis between nRF24L01+ and XBEE ZB Module Based Wireless Ad-hoc Networks

Comparative Performance Analysis between nRF24L01+ and XBEE ZB Module Based Wireless Ad-hoc Networks


A paper was written comparing Zigbee vs nRF24L01+ based networks. For those of you who are interested, here is the paper:



Abstract—Among the common wireless communication modules like Bluetooth and Wi-Fi, XBee modules are embedded solutions providing wireless communication standard with self-healing mesh networks, which has longer range than Bluetooth and lower power consumption than Wi-Fi. 

An alternative to the XBee radio modules is nRF24L01+ radio modules which are cheap and powerful, highly integrated, ultra-low power (ULP) 2Mbps RF transceiver ICs for the 2.4GHz ISM (Industrial, Scientific, and Medical) band. In this paper, performances of nRF24L01+ modules have been analyzed and compared with that of XBee ZB modules in wireless ad-hoc networks. The performance metrics for the analytical study are - 

1) Throughput measurement, 
2) Mesh routing recovery time and 
3) Power consumption. 

This work has revolved around an open source library released by the developer, tmrh20 which builds a complete TCP/IP suite on top of the nRF24L01+ modules.

ConclusionIn this study, we have analyzed all the parameters mentioned in the introduction, focusing on the stark differences in the performance of nRF24L01+ against XBee. nRF24L01+ could provide a better throughput compared to XBee in almost all scenarios especially in point-to-point communication where it outshines XBee by a country mile. For multihop networks as well, the rate of transmission remains considerably higher...

...The radio, as well as well its supporting libraries have been under focus for quite some time now, however, in terms of development, it is still in its nascent phase. Nevertheless, its impact in the field has been nothing short of revolutionary because of the raw range and bandwidth it provides on paper at affordable prices. However, it remains to be seen whether with improvements in the sleep cycle implementation in the RF24 library or with a better alternative to the RF24 family, the nRF may be indeed considered a cheap viable replacement for XBee in wireless ad-hoc networks

TMRh20 Opinion: 

The study in itself is somewhat fair, but the researchers did make a number of errors, both in hardware configuration of the devices and in not taking the time to understand the functionality of the devices in testing and before making a formal assessment.

It is very clear that the radios were not fully operational due to the hardware configuration, with power supply issues. ( Although an official Arduino Uno should be capable of driving the radios at 3.3v with no major issues ) It doesn't really make a lot of sense to conduct a range of testing using a known faulty hardware configuration, as this limited the range to 1m between nodes, which is laughable even for a test setup, and throughput suffered greatly.

As far as power consumption and implementation, the devices use approx. 15mA in active RX and slightly less during transmission. As active RF24Mesh nodes are always listening, they can choose to use interrupts to sleep the MCU during periods of inactivity (wake on RX), or remove themselves from the network and sleep completely. There are really not many options for managing the power usage/sleep cycles of these devices.


 Why me?