Skip to main content

Data Persistence In Electric Imp Applications

Preserve Your Data In The impCloud™ Or Within The imp

Developers of Electric Imp applications often need to save state information to ensure that when their products are restarted those devices continue to operate as they had before the reboot. Batteries run flat; power cables may be accidentally pulled out of sockets; end-users may inadvertently catch and press power buttons; and out-of-specification circumstances, such a high environmental temperatures, may trigger a device reset.

To cater for such eventualities, the Electric Imp API provides a number of tools to allow developers to cache small volumes of low-level data in permanent storage located within the Electric Imp impCloud. From impOS™ 36, a limited amount of storage is also available within an imp’s own Flash.

Saving Data in the impCloud

Storage space in the impCloud is made available solely for preserving small volumes of device and agent data across restarts. Only 64KB of space is provided, a quantity that is clearly insufficient to be used for storing large volumes of complex data — readings generated by a massive sensor array, say. Such information should instead be cached within the developer’s own online storage, or passed from the agent to a third-party storage or data management service, such as Dropbox or Firebase.

Nor is impCloud permanent storage intended to be used to preserve device data when the imp is goes into deep sleep. When the imp enters this state, its memory is cleared. However, temporary data may be preserved on the device itself using its nv facility, which provides faster access to state data than the process outlined below. Note that nv is not currently available on the imp005 or the imp004m. Please see the API documentation for more information about using nv table in your application.

For impCloud storage, every imp-enabled device can save a single Squirrel table to the impCloud and read it back at a later time. Both processes — write and read — should be mediated by the imp’s agent, which has at its disposal two API methods, server.save() and server.load(), specifically for these tasks.

The first, server.save(), takes a single parameter: a Squirrel table into which the data you want to be preserved has been packaged as a series of key/value pairs. Each pair is called a ‘slot’ in Squirrel terminology.

The second method, server.load(), simply reads backs that data and returns it as a new table. Because only a single, unique table is stored, server.load() has no parameters.

The agent can call server.load() when it starts up and, if it has previously called server.save(), the cached data will now be ready to use. If no data had been previously saved, server.load() will still create a table, but it will be empty. This provides the agent with a way to test whether a previous server.save() operation has taken place or not.

Similarly, should the agent need to clear the saved data, all it has to do is write an empty table to the permanent storage.

Data Availability

This server-side data is available as long as the agent is running. The impCloud management system may move an agent from one host to another, and this migration will inevitably cause the agent to go offline for a brief period. During that time, the persisted data is not available, but is retained.

Agents are shut down if the device disconnects and then does not re-connect within 31 days. Persisted data is retained. If the device subsequently reconnects, its agent will be restarted and will have access to all data previously preserved using server.save().

However, persisted data will be lost if the agent is removed. This may happen if the end-user re-configures their device with your BlinkUp SDK-based mobile app: if your app is code to request a new plan ID at BlinkUp, this will cause a new agent to be instantiated for the device. This agent will have a different ID than its predecessor, and will not be able to access the old agent’s data. If you wish to avoid this for your application, code your mobile app to request a new plan ID only at the end-user’s first BlinkUp. This plan ID should be stored locally and re-used if the same end-user ever reconfigures their device. This will ensure the existing agent is retained and any preserved data continues to be accessible.

If the end-user configures a second device, the unit will always get its own agent and therefore have no access to the first device-agent pair’s persisted data. Similarly, if the end-user gives the first device to a second end-user, when that subsequent user configures the device, a new plan ID will be requested (BlinkUp is being performed on a different mobile device) and so a new agent will be instantiated.

Serializable Squirrel

It’s important to remember that there a limits on what may and may not not be included in the agent’s permanent storage table because the data is serialized before it is stored. Not all Squirrel entities can be serialized. Integers, floats, bools and blobs may be serialized, as may arrays containing these values. Nested arrays can be serialized only if they too contain serializable data.

Tables, and nested tables, can be serialized only if string-type slot keys are themselves serializable: they are encoded in Ascii or UTF-8 and contain no embedded NUL ("\0") characters. Non-key data strings are serializable, but those which are considered ‘unsafe’ — ie. are not encoded in Ascii or UTF-8, or contain embedded NUL ("\0") characters — are first converted to blobs.

Classes, class instances and functions are not serializable so must not be included in server.save()/server.load() tables.

Saving Data in the impCloud: An Example

A typical use for permanent storage is to cache a device’s settings so that they are retained even if the power should fail or the gadget be restarted for some other reason. The device’s developer might well incorporate a small Flash memory chip (or the imp’s own — see below) and write the data there, but the Electric Imp impCloud’s permanent storage facility offers a considerably less expensive alternative — though it requires Internet access, of course.

How might an agent-imp combination manage the caching of settings?

The logical approach is to make the agent custodian of the ‘master’ settings, not only because it can manage the archiving process on behalf of the agent but also because it mediates communications between the device’s control app and the device itself. Mobile phones get lost or broken, so they are not necessarily a good place to retain settings data; better that they read that information back from the agent. This will also ensure that if the device is being controlled by more than one phone or tablet, the user interface always presents the correct, current settings.

Let’s say you are working on a digital clock project. You might have a number of settings for the clock, as follows:

  • Whether the clock’s display is presented in AM/PM or 24-hour mode.
  • Whether the user wants to see the current time, including Daylight Savings, or the base UTC time.
  • The brightness of the clock’s LED display.

These settings might be held by the agent in a global table variable, clockPrefs, and provisioned with some default values:

clockPrefs <- {};
clockPrefs.hr24mode <- true;
clockPrefs.dst <- false;
clockPrefs.brightness <- 8;

This happens very early in the agent’s current runtime, but it needs to check for any settings that were cached earlier:

local table = server.load();
if (table.len() != 0) clockPrefs = table;

We use the server.load() call to read any persisted data. If there is, the loaded table is used to re-populate clockPrefs. If no data has been saved, then the result of server.load() is an empty table. We use the table object’s len() method to find out if this is the case. If the table is empty, we retain the default settings.

The first time the agent has run, the loaded table will indeed be empty. As soon as the end-user adjusts one of the controls in the accompanying mobile app, the agent is notified of the change through an HTTP request sent to its unique URL by the app. The agent can now update its record of the current settings using server.save().

The method will replace a current saved table with a new one or, in this case, with the first table to be saved. server.save() returns an integer success-or-failure code. If it is 0, the table was written to permanent storage successfully. The API documentation for the server object lists the values that indicate failure and what those values mean.

For an end-user, this approach means that behind-the-scenes agent restarts are entirely transparent. If the agent was briefly disabled during the night because of a server migration, for instance, the end-user will have no sign that this took place: the agent, running once more, has settings that match what the users expects to see when he or she checks their phone app the following morning.

Saving Data in the imp Flash

From impOS 36, a portion of the imp’s Flash storage, whether internal to the imp or connected externally, can be used to stored data permanently. The data will persist across both warm and cold starts (unlike the nv table) and is not dependent on an Internet connection (unlike server.save()). The data is retained across application changes and the blessing process. In fact, the data will be retained until it is actively overwritten.

As such, this storage is primarily intended to be used to to save data for use by the device during its lifetime. Such information can be loaded by the factory firmware and will be available to the application code when it is loaded after blessing.

Up to 4KB of data may be stored, using the imp API method imp.setuserconfiguration(). The data is retrieved using the companion method imp.getuserconfiguration().

To clear the user data, pass null into imp.setuserconfiguration(). This is the only way to clear the data.

The imp’s Flash storage has a finite write capacity, placing a limit on the number of times user data can be written to the imp using imp.setuserconfiguration(). impOS’ implementation of the Flash code provides a minimum lifespan of 60,000 writes, and users should have significantly more writes than this.