Skip to main content

Data Persistence In Electric Imp Applications

Preserve Your Data In The impCloud™ Or Within The imp

Developers of Electric Imp applications often need to save state information to ensure that when their products are restarted those devices continue to operate as they had before the reboot. Batteries run flat; power cables may be accidentally pulled out of sockets; end-users may inadvertently catch and press power buttons; and out-of-specification circumstances, such a high environmental temperatures, may trigger a device reset.

To cater for such eventualities, the imp API provides a number of tools to allow developers to cache small volumes of low-level data in permanent storage located within the Electric Imp impCloud™ or within the imp’s own non-volatile memory.

Saving Data In The impCloud

Storage space in the impCloud is made available for preserving small volumes of device and agent data across restarts. Some 64KB of space is provided, a quantity that is clearly insufficient to be used for storing very large volumes of complex data — readings generated by a massive sensor array, say. Such readings should instead be passed from the agent to the developer’s own online storage or to a third-party storage or data management service, such as Dropbox or Firebase.

However, the impCloud storage facility can be used for small volumes of data you wish to persist across power-cycles, such as device settings applied by the end-user. Data preserved this way cannot be returned to the device until the imp reconnects to the impCloud, but the agent can access it at any time.

Data is saved by the agent using two API methods: server.save() and server.load().

The first of these methods, server.save(), takes a Squirrel table as its argument; you place all the data you want to be preserved into this table as key-value pairs (‘slots’ in Squirrel terminology).

The second method, server.load(), reads backs that data and returns it as a new table. Because only a single, unique table is stored, server.load() has no parameters.

The agent can call server.load() when it starts and, if it has previously called server.save(), the cached data will now be ready to use. If no data had been previously saved, server.load() will just return an empty table. This provides the agent with a way to test whether a previous server.save() operation has taken place. For example:

// Create a global variable to access application settings
settings <- null;

// Load in any previously saved data table...
local savedSettings = server.load();

// ...and check whether the returned table has any slots
if (savedSetting.len() != 0) {
    // Data was saved during the previous agent run, so point 'settings' at it
    settings = savedSettings;
} else {
    // No data persisted - this may be the first agent run - so 
    // point 'settings' at a default table of values
    settings = setDefaults();
}

Should the agent need to clear the saved data, all it has to do is write an empty table to the permanent storage:

function clearSavedSettings() {
    // Clear the settings by writing an empty table
    server.save({});
}

Data Availability

This server-side data is available as long as the agent is running. The impCloud management system may move an agent from one host to another, and this migration will inevitably cause the agent to go offline for a brief period. During that time, the persisted data is not available, but is retained.

Agents are shut down if the device disconnects and then does not re-connect within 31 days. But any persisted data is retained. If the device subsequently reconnects, its agent will be restarted and will have access to all data previously preserved using server.save().

However, persisted data will be lost if the agent is removed. This may happen if the end-user re-configures their device with your BlinkUp SDK-based mobile app: if your app is coded to request a new plan ID at BlinkUp, this will cause a new agent to be instantiated for the device. This agent will have a different ID than its predecessor, and will not be able to access the old agent’s data. If you wish to avoid this for your application, code your mobile app to request a new plan ID only at the end-user’s first BlinkUp. This plan ID should be stored locally and re-used if the same end-user ever re-configures their device. This will ensure the existing agent is retained and any preserved data continues to be accessible.

If the end-user configures a second device, the unit will always get its own agent and therefore have no access to the first device-agent pair’s persisted data. Similarly, if the end-user gives the first device to a second end-user, when that subsequent user configures the device, a new plan ID will be requested (because BlinkUp is being performed on a different mobile device) and so a new agent will be instantiated.

Squirrel Serialization

It’s important to remember that there a limits on what may and may not not be included in the agent’s permanent storage table because the data is serialized before it is stored. Not all Squirrel entities can be serialized. Integers, floats, blobs and Boolean values may be serialized, as may arrays containing these data types. Nested arrays can be serialized only if they too contain serializable data.

Tables, and nested tables, can be serialized only if string-type slot keys are themselves serializable: they are encoded in Ascii or UTF-8 and contain no embedded NUL ("\0") characters. Such strings are said to be ‘safe’. Value strings are serializable, but those which are considered ‘unsafe’ are first converted to blobs. This conversion will be performed automatically but your application will not be notified that this has taken place.

Classes, class instances and functions are not serializable so must not be included in tables that you expect to pass into server.save(); if they are, an exception will be thrown.

Saving Data In The Device

Saving device data by relaying it to the agent is only one way of persisting important information. There are two ways of saving data on the device itself. These methods have the advantage over the use of impCloud storage that they can be accessed even when the imp is offline.

nv

The first of these makes use of the imp API’s nv object. This is a table which is made available to you as a global variable. It does not exist until you make use of it, but once you have, the keys and values added to nv are retained under most circumstances. For example, data added to nv can be read back after the imp has been through a deep sleep cycle. But if the imp has been power-cycled, the imp’s Squirrel has been refreshed (for example, if a new version of the application has been deployed, or a Squirrel error occurred), or the imp underwent an impOS update, the contents of nv will be lost.

For more details of tye circumstances in which nv will or will not be retained, please see the nv documentation.

The space available for nv, just under 4KB, is also much lower than that offered by the impCloud, which offers 64KB of persisted storage capacity. As such, nv is primarily intended to cache data across deep sleeps, rather than a true data persistence mechanism.

Flash Storage

From impOS 36, a portion of the imp’s Flash storage, whether internal to the imp or connected externally, is available for storing application data permanently. The saved data will persist across all types of device restarts (unlike the nv table) and is not dependent on an Internet connection (unlike server.save()). The data is retained across application changes and the blessing process. In fact, the data will be retained until it is actively overwritten.

Up to 4KB of data may be stored in the imp’s Flash, using the imp API method imp.setuserconfiguration(). The data is retrieved using the companion method imp.getuserconfiguration(). To clear the data, pass null into imp.setuserconfiguration(). In fact, this is the only way to clear the data.

Unlike nv, imp.setuserconfiguration() takes either a string or a blob as its argument, so your application will need to organize the data itself: for example, converting tables to serial data.

Another limitation to consider is that the imp’s Flash storage has a finite write capacity, placing a limit on the number of times user data can be written to the imp using imp.setuserconfiguration(). impOS’ implementation of the Flash code provides a minimum lifespan of 60,000 writes, and users should have significantly more writes than this.

As such, this storage is primarily intended to be used to to save data for use by the device during its lifetime. For example, you might write image data to the device’s Flash via your factory firmware so that it is ready to be displayed when an end-user starts the device up. You might also use it to store end-user settings which need to be available to the device upon a cold start.

Custom Storage

If the 4KB limit of the storage available to imp.setuserconfiguration(), don’t forget that the agent’s server.save() mechanism has up to 64KB available to it, but at the cost of requiring an Internet connection. Depending on the needs of your application, you may be able to use both of these mechanisms for different classes of data.

Alternatively, you may choose to add your own non-volatile storage to your product, such as Flash or Ferroelectric RAM (FRAM), but these will introduce extra bill-of-materials cost to your design.