Markdown with Yada Wiki and Jetpack

In the office we have recently decide to migrate our team wiki to WordPress and Yada Wiki has been selected.

The team is also quite comfortable with Markdown, even if not everybody is ready to adopt it as the main editor, so I struggled a bit to find a solution, until I stumbled upon Jetpack.

I installed Jetpack directly from the WordPress admin console (which I just love!), but I had to connect to the server console to force Jetpack into developer mode, which is required if your server isn’t going to be publicly accessible.
To do so you need to open your wp-config.php, search for define('WP_DEBUG', false); and add the following line

add_filter( 'jetpack_development_mode', '__return_true' );

That brings you one step forward, but after enabling the Markdown feature in the Jetpack installation page you will be able to use Markdown in pages and posts only: your wiki pages will not be affected. That’s because Yada Wiki uses it’s own custom content type to distinguish wiki pages from other contents, which is a good thing.
So you need to extend the Markdown support to this additional content type, which is easily achievable adding the following lines at the very end of the functions.php file of your theme of choice:

add_action('init', 'my_custom_init');
function my_custom_init() {
    add_post_type_support( 'yada_wiki', 'wpcom-markdown' );
}

Now your wiki editors can decide to use the WYSIWYG editor or switch to the text editor and start typing their contents in Markdown syntax and preview their edits by just hitting the Preview button.

Win 10 FTDIgate solved!

My recent upgrade to Windows 10 has brought back to me an old issue: FTDI drivers.

Yes, I’ve bought cheap Chinese Arduino clones, mostly small boards to use in my projects as the official ones are quite expensive.

I never had problems on my previous Windows 7 install: even if I knew I was probably using one or more of those infamous FTDI counterfeited usb-to-ttl chips, I was lucky enough to dodge the FTDIgate.

When today, while setting up again my Eclipse + Arduino Eclipse Plugin, I stumbled into a weird message on the serial console my first thought was they did it again!

So, if your cheap Arduino clone has suddenly stopped responding and your OS is Windows 8, 8.1 or 10 you are probably stumbling on my exact same issue and you might want to carefully read this post to solve your troubles.

1. Assess

First of all you want to verify you are affected by the new FTDIgate, so connect via a serial terminal of your choice to your problematic Arduino: if you see a repeated message stating NON GENUINE DEVICE FOUND! then welcome aboard!

2. The tools

We will download all the tools we need upfront because this process implies we will have to disconnect from the Internet at a certain point to prevent Windows 10 to kick our butts with its automatic updates.

What you need is:

  • an older version of the FTDI drivers, latest working ones are 2.10.00, which you can get from FTDI in the No Longer Supported section (get the zip version)
  • a tool called CDMUninstaller which completely removes the existing drivers and it’s distributed by FTDI itself

3. Clean up

First we need to remove the drivers causing us all these troubles. If you believe you can do that via the Windows Device manager then youa re wrong: these drivers come in two separate parts, only one of which can be managed via the default driver management tool. You need the CDMUninstaller tool to ensure the drivers are completely removed:

remove

I’m not 100% it is needed, but I suggest you restart your system now, just to be sure there are no leftovers.

4. Prepare

It’s now time to load the new drivers, but be sure to do it with your Arduino board unconnected and while being offline.

Start decompressing the drivers package, then install the drivers right clicking on the two .inf files and selecting Install from the context menu.

Now it’s a good time to verify you have solved your problem, so open a serial terminal and check you can now communicate again with your Arduino.

The drivers we have just installed are going to be replaced at first opportunity by Windows with the trouble-making ones. In order to prevent Windows to find a more recent drivers version online, which will occur as soon as you’ll go back online, you have to open the two .inf files in any text editor, change the DriverVer line with a date very far in the future (I sat mine to 01/27/2034) and save.

If you now right click on the two .inf files and select Install from the context menu you’ll receive an error message informing you the files do not match their signature: don’t worry, we are going to solve this little issue.

We need to restart our OS, but allowing for unsigned drivers install: this can be achieved only with a system restart so open the Settings app and navigate to Update & security > Recovery.

Since the next step will actually restart your computer you have to read the next paragraph carefully before selecting the Restart Now button in the Advanced startup section.

5. Advanced Restart

Once in the Advanced restart section you’ll have to browse the advanced settings to Disable drivers signature verification, something a little tricky to achieve, but look at the following pictures and prepare yourself:

656x307ximage49-pagespeed-gpjpjwpjjsrjrprwricpmd-ic-cwzcnv7v-_

Select option Troubleshoot

656x307ximage50-pagespeed-gpjpjwpjjsrjrprwricpmd-ic-s7_14-sqwn

Head into Advanced Options

656x307ximage51-pagespeed-gpjpjwpjjsrjrprwricpmd-ic-izel4o6a5

Pick the Startup Settings

656x307ximage52-pagespeed-gpjpjwpjjsrjrprwricpmd-ic-mfl_b-m4mc

And finally Restart

656x473ximage53-pagespeed-gpjpjwpjjsrjrprwricpmd-ic-m7oxusdlqr

Activate the Disable driver signature enforcement setting

During the restart process you will be asked to select an option to activate during the next boot: we need to disable drivers signature verification, which was option number 7 for me.

6. Permanent Install

After reboot you will be able to right click on the two .inf files and select Install from the context menu: you will still receive a warning, but this time you will be able to get around the issue and proceed ignoring the verification error.

You can now go back online on the Internet, this time Windows shouldn’t be able to replace your working drivers!

7. Verify

It’s now time to restart your computer and verify everything still works. After a reboot open the Device Manager, select the problematic Arduino COM port and check the drivers version: if it’s anything above the version you selected…. well, something went wrong.

If I were you would try restarting from step 1, but this time try to read more carefully the instructions!

Cucumber is NOT a testing framework!

I have been repeating this statement over and over on the Cucumber forum, but apparently with no good result. To me this is a simple statement and I fail to understand what’s difficult to understand in its simplicity: Cucumber (JVM, Ruby, C, the flavour doesn’t matter) is not a testing framework.

I’ll try to further explain the statement, not because I believe it needs to be explained, but hoping an explanation will solve doubts. To do that I’ll try to state the facts in way as much practical as I can.

JUnit, TestNG and DBUnit are examples of a testing framework: they provide facilities to put a software in a certain condition, leverage some parts of the software and assert the outcome responds to the expected parameters.

The goal of those frameworks is to provide the tools to achieve the result (testing the software) with the least possible effort and the maximum outcome. As such, they have programming structures for asserting conditions are met, for setting up the test rig and tear down the test rig as well, ensuring the system is always in a clean condition before a new test is run.

Cucumber doesn’t have any of those facilities, simply because it is not a testing framework.

Cucumber is a specifications tool, something completely different. It uses Gherkin (the language) to describe in a natural language the software specifications.

Does anyone of you remember the Use Case Template? use-case-template-visio

Well a Cucumber feature is much more like one of those than a bunch of tests, would you agree?

There was once a time when we were describing software systems by collecting a few of those use case templates in a Word document, right?

Now, it wasn’t much of a trouble to describe a system like that, at least at first instance, but after a couple of maintenance cycles the above documents were not describing the system any more: I haven’t found one single situation where the above documents were maintained and kept in sync with the implementation. So the system documentation was just obsolete after a few months of operation.

Why was that? Because the cost of maintaining such documentation was not justified by the short term benefit of doing so: why should I spend 30 minutes to find the correct place where to apply the patch in the system documentation if the fix itself costs me 15 minutes?

So, Cucumber steps in and tries to transform the above specifications into something valuable and alive. How?

The specification format (or structure, if you prefer) is almost free, so to leave the writer the freedom to express himself freely. This is even less structured than the template above, but there are good practices telling us how to maximize the result, like providing the list of actors and the general goal at the very beginning.

The file is plain text, so to avoid any requirement on the tool used to open and modify the document and doesn’t add any formatting to avoid reader to be distracted or writer being dragged into endless make it prettier sessions.

A file represents a use case (feature in the Gherkin language), so you end up having multiple files, each one representing a piece of the software. This greatly simplifies version management, collaboration and merging, enabling multiple writers and revisors working on a single system. It’s not uncommon to use the same version control system in use for the system to store the documentation.

Files can be structured in a hierarchical fashion via folder structure, so big systems with lots of features can organize their specifications.

Files can be annotated via tags so to create parallel organizational structures, with the folder structure still being the prominent one: this enables additional categorizations, useful to track other associations between use cases (features), as much as what the introductory Use Case Diagram was doing in the Word document.

What makes the biggest confusion, though, is Gherkin files can be executed.

That is what Cucumber provides: the support software structures and an execution environment for Gherkin files.

Why? To create a connection between the system documentation and the documented system, so to ensure the documentation is aligned to the system it describes.

How? By mapping each statement in each scenario with some lines of code in the language you like (Ruby? C++? Java? Scala?). If the code ensures somehow the system does what it is expected to do, than the documentation is in sync with the implementation.

Only this last part resembles a test. Only at this stage there is some code involved. And usually testing frameworks and automation libraries are used to do such verification, like JUnit and Selenium.

So, if you need to test your system, please use a testing framework! If, instead, you want to document your system, you are welcome to use Gherkin and Cucumber!

nRF24 walk through – Sensors’ firmware

Let’s move to the 3rd part of the nRF24 walk through series before the excitement cools off, shall we?

Now that I have three boards wired and I’m certain only my code can break things up, it is time to create the sensors’ firmware.

It should be quite simple: all we need to do is getting the transceiver set up at the beginning and send the current sensor identifier every time the button is pressed.

As this series is intended to be a tutorial, I’ll expand a little on the basic requirement. To add a bit of salt on our sensors’ network communication, we will expect the hub to send back to the nodes 2 pieces of information in response to their click events:

  • the total count of clicks received from all nodes is going to be part of this article and we’ll achieve that using the ack packet payload
  • the number of clicks received from the specific node will be instead added in another post, so not to push too much info in one shot

Each device is going to require a unique identifier: as radio network address (think ofit like an IP address) and for the software to print out (like a computer name). I will use one byte stored in EEPROM for both: this will limit the maximum amount of sensors for this network to 255 (0 is going to be reserved as the hub identifier/address).

That’s not such a low number after all, but we want our hub to print out the node identifier as a letter, so I will further limit the range to [1, 26] to simplify the hub code.

NOTE The above limits to the number of nodes is very soft and has been introduced into this tutorial to simplify the code. The real hard limit to the number of unique nodes is much higher and in the order of 1 thousand billions (1 followed by 12 zeros)!

Setting things up

The radio transceiver configuration occurs, as you might expect, in the setup() function, plus a global variable declaration and a few defines:

// Creates a new instance using SPI plus pins 9 and 10
RF24 radio(9, 10);

// nRF24 address family: all addresses will be in the format 0xFACEC0DE## with the last two
// bytes determined by the node identifier.
#define ADDR_FAMILY 0xFACEC0DE00LL
#define MAX_ID_VALUE 26

#define BUTTON_PIN 3

// This node unique identifier: 0 is used for the hub, anything above MAX_ID_VALUE is considered
// not valid
byte nodeId = 255;

void setup() {
  SERIAL_DEBUG_SETUP(57600);

  // Setup the push button
  pinMode(BUTTON_PIN, INPUT_PULLUP);

  // Read the address from EEPROM
  byte reading = EEPROM.read(EEPROM_ADDR);

  // If it is in a valid range for node addresses, it is our address
  if (reading > 0 && reading <= MAX_ID_VALUE) {
    nodeId = reading;
    DEBUG("Node identifier is %c", nodeId + 64);

    // Initialize the transceiver
    radio.begin();
    radio.setAutoAck(true); // Enables auto ack: this is true by default, but better to be explicit
    radio.enableAckPayload(); // Enables payload in ack packets
    radio.setRetries(1, 15); // Sets 15 retries, each every 0.5ms, useful with ack payload
    radio.setPayloadSize(2); // Sets the payload size to 2 bytes
    radio.setDataRate(RF24_2MBPS); // Sets fastest data rate, useful with ack payload

    // Opens the first input pipe on this node address
    radio.openReadingPipe(1, ADDR_FAMILY + nodeId);

    // Opens the first input pipe on this node address
    radio.openReadingPipe(2, ADDR_FAMILY + 254);
#if (SERIAL_DEBUG)
    // Prints current configuration on serial
    radio.printDetails();
#endif
  } else {
    DEBUG("Invalid node id %u found: use S## (a number between 1 and %u) to configure the board", reading, MAX_ID_VALUE);
  }
}

There are a few details in the code related to the nRF24 module which might be valuable to explain:

  • when I instantiate the module driver class I’m specifying only two out of the five pins used by module (not including the power line) because the MISO, MOSI and SCK pins are defined by the board SPI interface
  • I’m defining a network address class 40 bits long (0xFACEC0DE00), not very different from IP addressing when you specify the network class (192.168.0.0 is the network class C of an host having IP address 192.168.0.*)
  • I’m enabling auto acknowledge packets (useful to know if your transmission was successful) capable to carry payloads (more on this later)
  • I’m instructing the library to auto retry packet transmission every half millisecond (500 us) up to 15 times, so to reduce transmission errors
  • I’m setting a fixed payload size, which improves transmission reliability, of 2 bytes
  • the data rate is set to highest possible value (2Mb per second), so to have the ack packets coming back in time for being recognized (more on this in the next article)
  • one of the input pipes (the second if we count in pipe 0) is set to this node address. This is not used to receive ack packets (pipe 0 is used for that), but it will come into play later on, when I’ll show how the sensors can receive data (other than ack packets) from the hub (or whoever wants to talk to us, for what it matters!)

IMPORTANT Reading pipes are not associated to addresses you talk to, but to addresses you want to listen for. If that sounds confusing, consider reading pipes like mailboxes: you can have multiple mailboxes, but that does not limit who can send you mail, actually anybody knowing at least one of them can.

This means you are not limited to receive data from maximum five nodes (as commonly and mistakenly known) and this walk through will demonstrate it!

NOTE If you want to know more about pipes and addressing, the best source of information is the datasheet, but to summarize:

  • you have one writing pipe only, but you can change it’s value to whatever address you want to talk to
  • there are six reading pipes, only two (0 and 1) can be associated with 40 bits addresses, with pipe 0 automatically associated to the writing address (to receive acks); this means pipes 1 to 5 share the same upper 32 bits, but you can change the reading pipes addresses
  • almost every address is valid, only a few are excluded (like 0x0000000000, 0xFFFFFFFFFF and similar)
  • address length is configurable to 40 (default), 32 or 24 bits, but the limitation on pipes 1 to 5 still applies as they keep sharing the upper 24 or 16 bits

For the sensors’ setup() function to complete correctly we need to read a valid node id from EEPROM which means we have to push something in the correct location: this is responsibility of another function called config(). I’m not going to analyze that function in detail, but it should be clear its purpose is to allow to set the node id using a serial connection.

A looping life

Moving into the sensor loop() function, the plan is even easier, if possible: check the button state and, if pressed, send the sensor node identifier to the hub.

To add value to this tutorial we will expect the hub to send back the total amount of received clicks and we will use ack packet as the carrier of this information.

NOTE Acknowledge (here abbreviated into ack) packets are used to verify data transmission has been successful. This is a common technique used in TCP/IP protocol as well: if you don’t feel comfortable with the concept just consider those as return receipts.

The protocol used by this chip allows to optionally attach some information to the return receipts: we refer to this info as the ack packet payload.

Ack packets maximum payload size depends on data rate and re-transmission speed and is explained in details in the nRF24 datasheet at paragraph 7.5.2.

I’m going to use some macros to perform a simple software debouncing on the push button, but apart from that, the code should be very easy to read:

// Button debouncing macros
#define DEBOUNCE 15
#define DMASK ((uint16_t)(1<<DEBOUNCE)-1)
#define DF (1<<(uint16_t)(DEBOUNCE-1))

// Macro for detection of falling edge and debouncing
#define DFE(signal, state) ((state=((state<<1)|(signal&1))&DMASK)==DF)

// Button debouncing status store
unsigned int buttonStatus;

void loop() {
  // Checks if we are trying to configure the node identifier
  config();

  delay(1); // Slow down a bit the MCU

  // Checks the push button: if we enter here the button has been pressed
  if (DFE(digitalRead(BUTTON_PIN), buttonStatus)) {

    // Sets the destination address for transmitted packets to the hub address
    radio.openWritingPipe(ADDR_FAMILY);

    // Put transceiver in transmit mode
    radio.stopListening();

    // Transmit this node id to the hub
    bool write = radio.write(&nodeId, 1);

    DEBUG("Send attempt from node %c was %s", nodeId + 64, write ? "successful" : "UNSUCCESSFUL");

    // Get acknowledge packet payload from the hub
    while (write && radio.available()) {
      unsigned int count;
      radio.read(&count, 2);

      // This function shows how a node can receive data from the hub
      // without using ack packets payload
      receiveNodeCount();

      DEBUG("Got response from hub: total click count is %u", count);
    }
  }
}

Once again, let’s analyze in detail the parts specific to the nRF24 module, which are activated whenever the push button state has gone LOW for enough time:

  • I set the transmission destination address using the openWritingPipe() function, providing the hub node identifier (that is the node we want to send data to)
  • to put the module into transmit mode we need to call the stopListening() function
  • since ack packets have been enabled, we can transmit reliably, so the result of the write() function (a boolean value) is going to be stored and checked
  • the data I’m going to transmit (the sensor node identifier, stored into the nodeId global variable) is provided to the write() function along with the number of bytes (only one) we are going to send. The actual payload sent on the air is going to be 2 bytes long, because we have set the packet size to a fixed value of 2, which means the packet will be zero filled
  • if the transmission was successful we expect to have an ack packet available. We have enabled ack packets payload, so we can pull out an unsigned int (2 bytes) into the count variable: this represents the total amount of clicks the hub has received so far

This is a complete example of data exchange, but the fact we are using the ack packet payload has some caveats we will analyze when we will talk about the hub.

The whole picture

The complete sensor’s firmware is available on Github and its compilation requires the two libraries mentioned in the previous post: the RF24 lib from TMRh20 and my own MicroDebug lib.

Please ignore the receiveNodeCount() function call and its implementation for now: we’ll get into that in another post. If you prefer, comment out line 105 to completely exclude that part of the firmware.

Naming the newborns

To avoid packet transmission conflicts we will have to initialize each sensor board with its own unique identifier: that’s the part handled into the config() function.

Internally I will use numeric identifiers, but they will be translated into one character when printing, hence the limit to numerical values between 1 and 26: that’s the number of letters in the latin charset.

Each board will initially have 255 as identifier, which is to be considered invalid, but you can change it via serial.

Once connected, type the S character followed by the board identifier you want to assign (like S2 or s5): the board will print the corresponding identifier as a character, store it in EEPROM and self reconfigure.

IMPORTANT The communication relies entirely on assigned node identifiers. The hub has no other way than the node identifier to distinguish between the nodes.

If you configure two nodes with the same identifier they will be considered as one and their clicks will not be distinguished. You can leverage this in some situations and have two sensors appear as one, if that’s what you want.

Sensor node id configuration is very simple and can be done via serial console in a handful of seconds

Sensor node id configuration is very simple and can be done via serial console in a handful of seconds

Repeat the procedure on each board and you have your sensors setup!

From this point forward your sensors does not require to be hooked up to your computer, they will just need power. If you have enough USB ports available and decide to leave the sensors connected to your computer though, you will be able to receive some debug messages on the serial ports and enjoy the occurring communication… once we have the hub running!

eclipse-console

The Arduino Eclipse Plugin supports multiple consoles and distinguishes among them using different colors: black text is from the hub, red, blue and green from the sensors!

nRF24 walk through – Building the sensors

It’s now time to physically build the sensor devices part of the nRF24 sensor network we are planning to build, so move to your electronics bench and pick some breadboards!

From an electronics point of view each sensor node consists of:

  • 1 Arduino (any low-end model will do, I will be using two nano and one pro micro as this is what I have at hand)
  • 1 push button
  • 1 nRF24L01P module
  • 1 1-100uF capacitor rated for 5V or more (usually, but not necessarily, electrolytic – any value in the range will do)
  • 1 voltage source for 3.3V capable to deliver at least 20mA; viable options include:
    • the 3V3 pin of your Arduino board (available only on my two nano)
    • a voltage divider made by two resistors (I will use 100Ω and 220Ω allowing 50mA to pass through)

WARNING While the module I/O pins are 5V tolerant (which makes this module compatible with practically every microcontroller), the module requires a 3.3V power source.

For absolutely no reason you should power the nRF24L01 module from a 5V source, unless you want to permanently damage your radio transceiver!

Please note I’m compromising here to keep things simple as the 3.3V source is definitely non optimal if you want to save on power consumption: for battery powered sensors you should be using 3.3V Arduinos, like the 8MHz pro mini, or an efficient voltage regulator like the AMS1117-3.3.

To keep the bill of materials at it’s bare minimum I’ll use the Arduino internal pull-up feature: this eliminates one resistor, but implies that pin 3 (the one attached to the button along with GND) will be HIGH when not pressed and turns LOW when pressed, just keep this in mind when we’ll start developing the sensors’ firmware.

NOTE These modules are quite sensitive to power line fluctuations and it should not be surprising considering they are radio transceivers operating in the 2.4GHz frequency: when transmitting data they require short but intense energy bursts to generate the electromagnetic field carrying the data. The capacitor we are adding should be as close as possible to the module in order to provide a local source of energy and avoid power fluctuations: it’s not an optional component!

I will build 3 of those using, more or less, the following wiring:

The breadboard view for a Pro Mini

The breadboard view for a Pro Mini

The associated schematics for Pro Mini

The associated schematics for a Pro Mini

Breadboard view for a Nano

Breadboard view for a Nano

The associated schematics for a Nano

The associated schematics for a Nano

If you wonder where I found the nRF24L01 Fritzing part, the answer is I made one myself (you can get it here).

Believe it or not, this is how these drawings turned into reality on my desk:

I’m going to build three adapters to plug the nRF24 modules onto my breadboards.

I’ve used a few bits of wire to ensure connection on these single side protoboards

IMG_6344

The components on a half size breadboard, wiring still missing

The nRF24 module plugged into its socket: isn’t it a beauty?

And here they are in all their splendor! Two are Arduino Nano clones, the rightmost one is an Arduino Pro Micro clone.

Now, the important thing to remember when wiring the modules, is they use a Serial Peripheral Interface, also known as SPI. What that means? Well, whatever board you are using to communicate to your modules, you need to identify the pins associated with the MISOMOSI and SCK functions. Luckily for us their location is pretty well documented for Arduino and Arduino-like boards.

The modules use two additional I/O pins, called CE and CSN, but they are configurable so you can pick your own: I went for pins 9 and 10 respectively. Bear in mind though, if you’ll ever decide to change those pins, you will have to change the software accordingly.

NOTE Most microcontrollers have one single SPI interface, this doesn’t mean you can have only one SPI device attached though! Most SPI devices, including nRF24, have a CSN (Chip Select Not) pin which instructs the device to temporarily ignore SPI communication, thus allowing for communicating with other SPI devices.

Testing

Now that you have your sensor boards ready, let’s move onto the software part.

You might want to use the Arduino IDE or, as I will, a more advanced solution like the Eclipse Arduino Plugin, an open source project I’m proud to contribute whenever I get the opportunity to. In case you decide to go for the Eclipse based solution, please follow the website instructions as not every configuration is supported.

I will use a few open source libraries trying to reduce the code to the real interesting parts:

  • the RF24 from TMRh20 library is a fork of the original library from Maniacbug, who is no longer active
  • my MicroDebug library, which I’ll use to provide some development feedback on serial console

Before getting your hands dirty though, it’s preferable to check your wiring with a known working firmware.

Among the many examples available in the RF24 library, the one I will use is RF24/examples/pingpair_ack.ino: this Arduino sketch is pretty simple and provides enough info to verify your setup.

There are just a couple of changes I’m going to make, to adapt to the above wiring, and I’ve marked them with comments in this Gist:

  1. the pins used as CE and CSN are pins 9 and 10, so line 24 has been changed from RF24 radio(7,8) to RF24 radio(9,10) in order to reflect that
  2. there’s no need to run the serial connection at 57600bps, so I slowed it down to 9600bps in line 41
  3. one of my boards uses the ATmega 32u4 chip (Leonardo and Micro use it), which means a board reset does not reset the serial connection; I need to wait for serial connection in the setup() function to be able to check the module config, this required the addition of line 43

Once the changes are in place, upload the sketch onto the first sensor board and connect using the serial terminal. Don’t worry if you don’t see anything printed out, the addition of line 43 requires you to send something to unlock the setup phase.

As soon as you send a character, you should see the following appear onto your screen:

An example of invalid setup.

The serial output generated by a successful initialization of the nRF24 control library

The serial output generated by a successful initialization of the nRF24 control library

As you can see the radio.printDetails() function call at line 59 provides tons of information: most will be blank (0xff) in case your module has not been setup correctly (99.9% as a consequence of bad wiring or false connections). If everything is setup correctly, you should see the addresses used at line 27 printed in the serial console.

Once you have verified the first module, leave it powered and move on to another one, but this time we will go a step further verifying the two modules are actually talking each other: if the second module reports a correct configuration send the character T on the serial and watch the two talking each other.

IMPORTANT When moving to verify the next board you must ensure to remove power to one of the other two, because this sketch uses and forces a linear network topology of two elements only: if you try to power up another one you will be unable to distinguish who is replying to whom.

In the next article we will look at the sensors’ firmware and how we can wireless notify the hub about the sensor’s button state change.

nRF24 walk through – Introduction

The Nordic nRF24 is a family of silicon integrated radio transceivers operating in the 2.4GHz band, the most popular one being the nRF24L01. This is the core element of some extremely cheap module boards available in online stores like eBay, Aliexpress and Banggood.

These boards do not provide WiFi (801.11) or Bluetooth connectivity (both in the 2.5GHz band), but they can be used to establish custom wireless networks between small electronic devices, including Arduino, RaspberryPi and Particle (formerly known as Spark).

Whenever we talk about networks you must take in account a few key aspects of networking, one of the most important being the network topology.

NetworkTopologies.svg

During this series we will aim to establish a star network between a series of Arduino based peripheral nodes and a central hub node, being either a RaspberryPi or a Particle Core/Photon: this represents a basic but invaluable configuration allowing for complex elaborations on remotely collected information, either on premise (RPi) or in the cloud (Particle).

If this series gets enough attention I might invest some more time and extend it to cover more complex topologies like tree and mesh, with the latter being my favorite and, IMHO, most valuable for inexpensive IoT projects.

The project

To keep things simple our peripheral nodes will be only collecting button presses, communicating to the central hub whenever a button gets pressed: the central hub will periodically (once every 30 seconds) print out the amount of button clicks it has received with a breakdown for each node; something like:

Received 14 clicks in the past 1 minute(s)
* 5 click(s) from node A
* 2 click(s) from node B
* 7 click(s) from node F

This will obviously represent just an example of what you will be able to do from the hub node; nothing prevents you, as an example, from pushing data into a database and generating graphs. You could aggregate the data differently or, more likely, collect other types of data from your sensor nodes: I’m not here to place constraints to your imagination!

Keep in mind though, the little radio transceivers we are using have a few limitations that are commonly misunderstood, which will be analyzed when we get there during the project.

The steps

This walk through will be split into the following posts:

Arduino analogRead pitfalls

I’ve just published a small, open source, Arduino library to sense battery level in Arduino projects and, while writing the readme file, I found myself trapped explaining the reasons behind some decisions. While I believe those might be valuable, I believe they deserve a post as they have a broader range of application than the library itself.

I’ll try to keep the explanations as simple as possible, mainly for two reasons: keep the thing simple for beginners and avoid saying stupid things myself.

Resolution

The Arduino microcontroller (commonly an Atmega328 integrated circuit) is capable to convert analog readings into a digital value through its internal ADC (Analog to Digital Converter). This conversion is never perfect and it’s precision is determined by the ADC resolution, in our case being 10 bits: we get a value between 0 and 1023 representing usually a voltage between 0V and 5V.

This means our readings have a resolution of about 5mV ad we will not be able to determine the difference between 1.000V and 1.004V.

This is not bad for most projects, but definitely something to be aware of. You can’t get around the 10 bit resolution limit (well, you can using averaging techniques, but that’s out of scope for now), but you can increase your resolution by reducing the reference voltage: a scale of 1024 applied to 1V means an increment every approximately 1mV, five times better.

Impedance and capacitance

The internal conversion circuit stores the voltage to be measured into a small capacitor (14pF in most cases) through a resistor, in the order of 10k Ohm. This means the ADC is not reading the voltage at the pin, but the voltage at the capacitor plates! This might be higher or lower than the voltage you are trying to measure, due to ADC multiplexing (more on this later).

If you try to measure the voltage of a low impedance circuit this has no side effects: the capacitor will charge/discharge very quickly due to it’s very small capacitance and your readings will be accurate.

But if you try to measure a high impedance circuit (like an LM35 temperature sensor) , you must take into account the time it requires to allow the capacitor to charge/discharge. A common workaround for this is to perform two or more readings in rapid succession: each reading will give the capacitor a chance to equalize to the pin voltage, eventually bringing it to the desired voltage.

Multiplexing

You have many analog pins on your board, but there is only 1 ADC in your microcontroller.

This means your analog readings will be multiplexed or, if you prefer, they will share the same ADC circuit. This condition limits the speed of the analogRead function and, in conjunction to the circuit capacitance and impedance, can lead to wrong readings: if you try to measure a 5V signal on pin A0 and then try to measure a 0V signal on pin A1, the internal ADC capacitor needs to fully discharge before an accurate conversion can be performed.

If you need to perform multiple, fast, accurate readings, you might want to consider using an external ADC.

Reference

It is very often underestimated, but it’s of the highest importance: all the values returned by the ADC are determined by comparing the internal capacitor voltage to a reference voltage.

Unless you provide a specific voltage to the AREF pin, the ADC uses Vcc or 5V as reference voltage, meaning a value of 1023 means approximately 5V. Yes, you read it right: approximately.

The voltage at the Vcc or 5V pin is not guaranteed to be exactly 5V: it will depend on your voltage source circuit precision and calibration. For a battery powered project using a voltage booster to generate 5V out of a LiPo/LiIon battery, it will probably change with time, while an Arduino powered from a 9V battery through the VIN pin or the barrel jack, it will depend on the on-board voltage regulator.

As an example, using my medium priced multimeter, using a completely charged 9V battery, I measure 4.9V at the 5V pin on one board and 5.2V on another one. Those are cheap chinese Arduino clones.

Fortunately  the microcontroller provides an internal reference voltage which is independent from Vcc, but it is not the default reference voltage and it is not as precise as we wish. If we want to get accurate, voltage source independent analog readings we need to compensate the internal reference voltage.

Without trying to get into the details, I wrote a nice, simple and open source library to do it: please refer to VoltageReference Github project.

The reference voltage can be used to improve your readings accuracy, if you can trade off on the voltage range: if you provide exactly 1V to the AREF pin each reading you will make will have about 1mV precision (977uV, to be precise), but it can only range between 0V and 1V. If you are dealing with low voltage measurements it’s best to use the internal 1.1V voltage reference, possibly having it calibrated for best accuracy.

The “Maven and Github on Windows” hell!

I’m sure this is not the first time I prepare a post like this, but I might have decided to last minute drop it: this time it’s not going to happen.

Sadly I have to use Windows at work and when I have some spare time I do contribute to Open Source Software and I have once again came face to face with the hell caused by trying to run a Maven Release Plugin on a Github hosted project. In this particular case it was a project I started myself called SmartUnit, but you can bet the same applies to any other Github hosted project using Maven and willing to perform releases on the Maven Central Repository.

Where do my problems start? Well, when I run the infamous mvn release:prepare obviously.

First I started encountering failures with the missing commands:

  • gpg.exe must be installed and available on the PATH to be able to sign the packages for the Maven Central Repository
  • git.exe must be installed and available on the PATH to be able to commit the tag and updated pom to Github

Installing the above though does solve only a fraction of the problem as you need the corresponding keys:

  • your GPG key should go into %APPDATA%\gnupg and it’s easier if it is the first key in your GPG key store
  • your SSH key should go into %USERPROFILE%\.ssh and, to avoid further complications, just name the file as rsa_id

As if it wasn’t enough hassle, you have to manually start an SSH agent to provide the SSH key, so don’t forget to (commands should be already on the PATH at this stage):

  1. run the Git’s bash shell with bash
  2. run the SSH agent with eval$(ssh-agent)
  3. add your SSH key with ssh-add ~\.ssh\id_rsa and input the key passphrase

If your key wasn’t already listed among your Github SSH Keys, don’t forget to add it.

Now the Maven release:prepare goal should complete successfully, but problems might arise during the release:prepare one as:

  • the PGP key must be publicly verifiable, so it must be published like on http://pgp.mit.edu/ (you can use gpg --armour --export to obtain the key signature to submit)
  • your Sonatype nexus credentials must be in your Maven settings.xml, under a server directive using an id matching the distribution repository (it should be sonatype-nexus-staging, but it might change over time)

Do you believe it’s all? Well, it’s not! Now you need to:

  1. log into your Sonatype Nexus account
  2. search your newly created Staging Repostory
  3. select the repository and close
  4. wait and refresh the view until it is reported as closed
  5. select the repository and release it!

Only at this point your artifacts will be available on the Maven Central Repository…

This is what I’ve twice learnt about the release process and what I hope I will be able to retrieve here the next time I’ll have to set this thing up again.

Might the force be with you!

Windows 7 disk space hunger

My recently formatted desktop computer just started to show those nice notifications regarding system disk drive exhaustion…

Really? I gave you 60GB of my brand new SSD and I’ve just installed one third of the software I want to run! What are you Windows doing with my precious hard disk?

So yes, I admit I use Windows along with GNU/Linux. Don’t kill me for that, please.

The weird thing was the disk space run low so quickly I couldn’t believe my eyes…

I installed a nice piece of free software called WinDirStat to highlight the disk space usage on my SSD and I realized I did forget about a few little optimizations.

My little beast has 16GB of RAM and I didn’t tune anything on my Windows 7 installation, which meant I had 16GB of hibernation file and another 12GB of paging file!

C:\> powercfg -h off

Just immediately freed up a nice few gigs and some manual settings on the system page file location and size gave me back another slice of storage.

I then refreshed the WinDirStat view just to realize the Windows folder was now leading the league taking 18GB of space, half of which was allocated under a folder called winsxs: what’s all that space for? I can’t exactly tell what’s under that folder (I’m not a Windows expert by all means!) but I can tell you I was able to regain another gigabyte by just running the system disk clean tool as Administrator and remove what the tool reports as Windows Update Cleanup.

Now a little question: for which misterious reason a set of resources identified as Windows Update Cleanup should occupy a gigabyte of space and a No Free Disk Space warning should pop up instead of just freeing up that wasted space silently trying to cover up your sins?

I then realized I could save another large amount of space by moving off the SSD my iTunes folder, but to achieve that I had to create an hard link (or directory junction in Windows terminology)

mklink /J "%APPDATA%\Apple Computer\MobileSync\Backup" "D:\iTunes\Backup"

This, along with the iTunes Media folder relocation, performed through the application settings, saved me another 12GB of space!

My disk is now breathing again!

XBian from USB with SD fallback

After the big mess I’ve previously posted about I’ve decided to get the opportunity to switch over to a better and more performing setup for my Raspberry Pi, not to mention the addition of multiple backups…

The goal I set up was to have mi Pi running off a root filesystem on my external USB hard drive while still being able to be portable, so running off the SD when needed.

I took the opportunity to switch from RaspBMC to XBian, mostly because the latter is based on Raspbian which seems to have a better support on the forums.

I started by flashing an XBian RC2 image onto my SD card from a Windows laptop using the tool provided by the community (the Linux version didn’t work for me).

After being surprised by the fact this distro doesn’t use ext3/4 but btrfs (I didn’t even know it existed) and it already includes the kernel module for my WiFi dongle I started messing with the boot sequence: I wanted to be able to drop to command line before the init sequence started and manipulate the cmdline.txt to change the rootfs location.

Because my keyboard seems to use a weird, non standard, firmware, I suddenly discover I was unable to use it to drop into rescue mode so I had to revert to a somewhat automated way to switch to SD card root fs.

I ended up with a little more complicated solution which performs fully automated fallback: if the USB disk isn’t connected at boot time then it reverts to SD card and reboots. Once the USB disk gets connected back though, you need to execute a command to restore the USB root.

To achieve this I had to create a little shell script:

#!/bin/sh

fallback() {
	$mount_bin /dev/mmcblk0p1 /mnt &amp;amp;amp;&amp;amp;amp; echo &amp;quot;Boot filesystem mounted&amp;quot;
	if [ ! -f /mnt/cmdline.usb.txt ] &amp;amp;amp;&amp;amp;amp; [ -f /mnt/cmdline.sd.txt ]; then
		echo &amp;quot;Switching back to SD card root filesystem...&amp;quot;
		mv /mnt/cmdline.txt /mnt/cmdline.usb.txt
		cp /mnt/cmdline.sd.txt /mnt/cmdline.txt
		umount /mnt
		exit 0
	else
		umount /mnt
		echo &amp;quot;Fallback procedure unavailable&amp;quot;
		exit 1
	fi 
}

restore() {
	if [ -f /boot/cmdline.usb.txt ]; then
		echo &amp;quot;Re-enabling USB root filesystem...&amp;quot;
		mv -f /boot/cmdline.usb.txt /boot/cmdline.txt
	else
		echo &amp;quot;Nothing to do&amp;quot;
	fi
	exit 0
}

case $1 in
	fallback) 
		fallback;;
	restore) 
		restore;;
	*)
		echo &amp;quot;Usage: `basename $0` ( fallback | restore )&amp;quot;;;
esac

As you can see this script provides two functions: you can either try to fallback from USB to SD or to restore your boot sequence to use a USB root filesystem.
You can obviously use this from command line, something you will have to if the fallback procedure occurs, so I created a symbolic link to it into /usr/sbin to have it on my superuser execution path.
Because I want this file to be available at boot time I need to have it within the initramfs, that’s why I created this file as /etc/xbian-initramfs/usb-boot.sh.
But this is not enough as the initramfs image is created by a script which doesn’t include the file unless you explicitly tell it to: to do so I edited the update-initramfs.sh script adding the following highlighted lines to include my new creation:

cp /etc/xbian-initramfs/cnvres-code.sh ./
cp /etc/xbian-initramfs/splash_updater.sh ./
# Add root fs USB to SD fallback/restore script
cp /etc/xbian-initramfs/usb-rootfs.sh ./

copy_with_libs /usr/bin/stdbuf
copy_with_libs /usr/lib/coreutils/libstdbuf.so
copy_with_libs /usr/bin/setterm

Now you need to have a copy of the /boot/cmdline.txt file which runs nicely from SD card and name it /boot/cmdline.sd.txt: I performed the copy at this stage as so far everything was working fine for me.

At this stage you might want to move your root filesystem over your USB hard drive and this is going to be the easiest task: you can do it through the xbian-config command by simply choosing option number 6 and specifying the partition on your USB hard disk. With XBian RC2 this does practically everything for you!

Next step is to enable the initramfs at boot as it is disabled by default to save a little on boot time: edit /boot/config.txt uncommenting the initramfs line.

Before creating the new initramfs image let’s tweak the init process a little to automate the fallback procedure: remember, my keyboard is not recognized during the boot sequence so I will not be able to hold down the shift key…

The file to alter is /etc/xbian-initramfs and the following highlighted lines are the contributions which automate the fallback check and reboot if needed:

if [ &amp;quot;$CONFIG_rootfstype&amp;quot; != &amp;quot;nfs&amp;quot; ]; then
    while ! get_root &amp;amp;amp;&amp;amp;amp; [ -n &amp;quot;$CONFIG_rootwait&amp;quot;  -a  &amp;quot;$X&amp;quot; -lt &amp;quot;$CONFIG_rootwait&amp;quot; ]; do
	/sbin/modprobe -q usb_storage
	X=$(($X+1))
	echo &amp;quot;$X Waiting for root...&amp;quot;
        sleep 1
    done
    if [ ! -b &amp;quot;${CONFIG_root}&amp;quot; ]; then
	echo &amp;quot;Root partition ${CONFIG_root} missing&amp;quot;
	eval ./usb-rootfs.sh fallback
	if [ &amp;quot;$?&amp;quot; -eq '0' ]; then
	    echo &amp;quot;Fallback root filesystem applied: rebooting now!&amp;quot;
	    sync
	    reboot -f -d2
	fi
	drop_shell
    fi
    ln -s ${CONFIG_root} /dev/root
    up &amp;quot;root block device found&amp;quot;

    export FSCHECK=$(blkid -s TYPE -o value -p ${CONFIG_root})
    resize_part; resize_ext4; convert_btrfs
    up &amp;quot;after resize, convert&amp;quot;
fi

Practically the added section steps into the root partition identification and right after the declaration of a missing root partition tries to perform a fallback procedure: if it succeeds it reboots the system, otherwise the process continues as normal and the user is dropped into the rescue shell.