SPI Flash File System

The SPIFlashFileSystem library implements a basic wear leveling file system intended for use with SPI Flash devices, such as the built-in hardware.spiflash object available on the imp003 and above, or an external SPI Flash plus the SPIFlash library on the imp001 and imp002.

You can view the library’s source code on GitHub. Click here to see information on other versions of this library.

To include this library in your project, add #require "SPIFlashFileSystem.device.lib.nut:3.0.1" at the top of your device code.

Library Classes

The SPIFlashFileSystem consists of three classes:

SPIFlashFileSystem — The main programming interface for the file system.
SPIFlashFileSystem.File — An object representing an open file that you can use to read, write, etc.
SPIFlashFileSystem.FAT — The File Allocation Table (not used by the application developer).

Overview Of The File System

The SPIFlashFileSystem (SFFS) divides the flash into 64KB blocks and 4KB sectors. The SPI flash must have at least one block allocated for the file system, and start and end bytes must be on block boundaries.

Files are written to pages that are one sector is size (4KB). Pages include a six-byte header containing a two-byte unique file ID (indicating the file that the page belongs to), a two-byte span ID (what order the pages of a particular file should be read in) and a two-byte length.

If the page is the first span of a file, it will contain an additional byte to denote the length of the filename, followed by the filename itself. It will also contain four bytes which record the file creation timestamp as a 32-bit integer.

Pages can’t be shared by files, meaning that the smallest amount of space a file can occupy is 4KB.

An Example File

Let’s look at how the SFFS stores file information and data. In this example we are going to create a file called "test.txt" that contains 6232 bytes of assorted data, though we don’t actually care about what the data looks like.

The file will be broken into two pages, each 4096 bytes is size. Both pages will contain six bytes of header information:

Two bytes for the file ID.
Two bytes for the span ID.
Two bytes for the length of the data in the span/page.

The first span will also contain:

Four bytes to record the creation timestamp.
One byte to denote the length of the filename.
The filename.

After we’ve written the header information, we fill the remainder of the span with the file’s data. We will use all of the available storage in the first span, but only the first 2150 bytes of the second span. The remaining 1946 bytes in the second span will be unusable (as pages cannot be shared by multiple files).

Because each sector contains a file ID and a span ID, the sectors can be located anywhere in the allocated SPI Flash, and likely will not be adjacent to one another (as part of the effort to wear-level the flash).

First Sector

Byte	Data	Notes
`0x0000`	`0x01`	Low Byte of FileID word
`0x0001`	`0x00`	High Byte of FileID word
`0x0002`	`0x01`	Low Byte of SpanID word
`0x0003`	`0x00`	High Byte of SpanID word
`0x0004`	`0xF0`	Low Byte of length of data in this span/sector
`0x0005`	`0x0F`	High Byte of length of data in this span/sector
`0x0006`	`0x56`	First Byte of Creation time
`0x0007`	`0xA8`	Second Byte of Creation time
`0x0008`	`0x76`	Third Byte of Creation time
`0x0009`	`0x47`	Fourth Byte of Creation time
`0x000A`	`0x08`	Length of file name
`0x000B`	`0x74`	t
`0x000C`	`0x65`	e
`0x000D`	`0x73`	s
`0x000E`	`0x74`	t
`0x000F`	`0x2e`	.
`0x0010`	`0x74`	t
`0x0011`	`0x78`	x
`0x0012`	`0x74`	t
`0x0013`	data	File Data (byte 1)
...	data	File Data (bytes 2 - 4079)
`0x1000`	data	File Data (byte 4080)

Second Sector

Byte	Data	Notes
`0x0000`	`0x01`	Low Byte of FileID word
`0x0001`	`0x00`	High Byte of FileID word
`0x0002`	`0x02`	Low Byte of SpanID word
`0x0003`	`0x00`	High Byte of SpanID word
`0x0004`	`0x66`	Low Byte of length of data in this span/sector
`0x0005`	`0x08`	High Byte of length of data in this span/sector
`0x0006`	data	File Data (byte 4081)
...	data	File Data (bytes 4082 - 6230)
`0x084e`	data	File Data (byte 6231)
`0x084f`	`0xFF`	Unusable
...	`0xFF`	Unusable
`0x1000`	`0xFF`	Unusable

Garbage Collection

When the SFFS deletes a file, it simply marks all of the pages the file was using as erased. In order to use those pages again in the future, we first need erase the sectors that the file used. This is done automatically through a process called garbage collection.

Each time a file is closed or erased, the SFFS determines whether or not it needs to run the garbage collector. It does this by comparing the number of free pages to the value of autoGcThreshold, which can be set with the setAutoGc() method.

SPIFlashFileSystem Usage

Constructor: SPIFlashFileSystem([start, end, spiflash])

The SPIFlashFileSystem constructor allows you to specify the start and end bytes of the file system in the SPIFlash, as well as an optional SPIFlash object (if you are not using the built in hardware.spiflash object).

Parameters

Parameter	Type	Required?	Description
start	Integer	No	The address of the file system’s first byte. Default: `0x0000`
end	Integer	No	The address of the file system’s last byte. Default: set according to the size of the flash
spiflash	Object	No	A class instance representing your SPI flash hardware. Default: the imp API hardware.spiflash object

Note The start and end values must be on block boundaries (0x010000, 0x020000 etc.), otherwise a SPIFLASHFILESYSTEM_ERROR.INVALID_SPIFLASH_ADDRESS error will be thrown.

Example: imp003 And Above

#require "SPIFlashFileSystem.device.lib.nut:3.0.1"

// Allocate the first 2MB to the file system
sffs <- SPIFlashFileSystem(0x000000, 0x200000);
sffs.init();

Example: imp001/imp002

#require "SPIFlash.class.nut:1.0.1"
#require "SPIFlashFileSystem.device.lib.nut:3.0.1"

// Configure the external SPIFlash
flash <- SPIFlash(hardware.spi257, hardware.pin8);
flash.configure(30000);

// Allocate the first 2 MB to the file system
sffs <- SPIFlashFileSystem(0x000000, 0x200000, flash);
sffs.init();

SPIFlashFileSystem Methods

init([callback])

This method initializes the file system, including the creation of its file allocation table, and so must be called before invoking any other SPIFlashFileSystem method. If it is called while the file system has files open, a SPIFLASHFILESYSTEM_ERROR.FILE_OPEN error will be thrown.

Parameters

Parameter	Type	Required?	Description
callback	Function	No	A function called upon completion of the initialization process. It has one parameter of its own which receives an array: a list of files currently stored within the SPI flash. See File Records for details of each record in the array

File Records

Each file record passed into the optional callback is a table with the following keys:

Key	Type	Description
id	Integer	The file’s identifier
fname	String	The file’s name
size	Integer	The file’s size in bytes
created	Integer	A timestamp indicating the date and time of the file’s creation

Return Value

Nothing.

Example

// Allocate the first 2MB to the file system
sffs <- SPIFlashFileSystem(0x000000, 0x200000);

// Initialize the file system FIRST
sffs.init(function(files) {
    // Log how many files we found
    server.log(format("Found %d files", files.len()));

    // Log all the information returned about each file:
    foreach(file in files) {
        server.log(format("  %d: %s (%d bytes)", file.id, file.fname, file.size));
    }
});

dimensions()

This method provides information about the file system.

Return Value

Table — File system information with the following keys:

Key	Description
size	The file system size in bytes
len	The size of the file system
start	The address of the first byte of SPI flash assigned to the file system
end	The address of the last byte of SPI flash assigned to the file system
pages	The number of pages available in the file system

Example

local d = sffs.dimensions();
server.log("The file system contains" + d.pages + " pages");

getFreeSpace()

This method provides an estimate of the free space available in the file system. Smaller files have more overhead than larger files so it is impossible to know exactly how much space is free.

Return Value

Table — The file system information with the following keys:

Key	Description
free	The estimated free space in bytes
freeable	The estimated space in bytes that may be freed through further garbage collection

getFileList([orderByDate])

This method provides file information identical to that passed into the init() callback.

Parameters

Parameter	Type	Required?	Description
orderByDate	Bool	No	Should the file list be sorted into date order. Default: `false`

Return Value

Array — A list of file records.

Example

local files = sffs.getFileList();
foreach(file in files) {
    server.log("id: " + file.id);
    server.log("fname: " + file.fname);
    server.log("size: " + file.size + " bytes");
}

fileExists(filename)

This method indicates whether the specified file is present in the file system.

Parameters

Parameter	Type	Required?	Description
filename	String	Yes	The name of a file

Return Value

Bool — Whether the named file exists in the file system (true) or not (false).

Example

if (!(sffs.fileExists("firstRun.txt")) {
    // Create the firstRun file
    sffs.open("firstRun.txt", "w").close();
    server.log("This is the first time running this code. \"firstRun.txt\" created.");
} else {
    server.log("Found \"firstRun.txt\"");
}

isFileOpen(filename)

This method indicates whether the specified file is currently open.

Parameters

Parameter	Type	Required?	Description
filename	String	Yes	The name of a file

Return Value

Bool — Whether the named file is open (true) or not (false).

fileSize(filename)

This method indicates the size of the specified file’s data in bytes. It does not include the file’s header information or the amount of unusable space at the end of a page/sector. For example, if we created a file and wrote hello! to it, fileSize() would return 6, but the file would actually take up 4096 bytes in our file system, as that is the smallest page we can write to.

Please see Overview Of The File System for more information.

Parameters

Parameter	Type	Required?	Description
filename	String	Yes	The name of a file

Return Value

Integer — the size of the named file’s data payload.

Example

local filename = "HelloWorld.txt";
server.log(filename + " is " + sffs.fileSize(filename) + " bytes long");

created(fileRef)

This method retrieves the creation timestamp for a specified file reference.

Parameters

Parameter	Type	Required?	Description
fileRef	String or integer	Yes	Either a filename (string) or a unique file ID (integer)

Return Value

Integer — the file’s 32-bit creation timestamp.

Example

// Get creation date by file name
sffs.created("file.txt");

// Get creation date by ID
sffs.created(5);

open(filename, mode)

This method opens the specified file with read permissions, or creates a new file.

If you attempt to open a non-existent file in reading mode, a SPIFLASHFILESYSTEM_ERROR.FILE_NOT_FOUND error will be thrown.

If you attempt to open an existing file in write mode (ie. create a file), a SPIFLASHFILESYSTEM_ERROR.FILE_EXISTS error will be thrown.

If you attempt to open a file with a value of mode other than "r" or "w", a SPIFLASHFILESYSTEM_ERROR.UNKNOWN_MODE error will be thrown.

If you create an empty file, it will be stored in cache only and will not be available after the next reboot.

// Create an empty file
sffs.open("filename.txt", "w").close();

// Note: The filename.txt cannot be reopened in "w" mode to be modified and 
// will not be persisted after the device reboots.

Parameters

Parameter	Type	Required?	Description
filename	String	Yes	The name of a file
mode	String	Yes	How the file should be opened: `"r"` (for reading) or `"w"` (for writing, ie. file creation)

Return Value

SPIFlashFileSystem.File instance — the opened file.

Example

// Create a file called HelloWorld.txt
local file = sffs.open("HelloWorld.txt", "w");
file.write("hello!");
file.close();

// Open HelloWorld.txt and log the contents:
file = sffs.open("HelloWorld.txt", "r");
local data = file.read();
server.log(data);
file.close();

eraseFile(filename)

This method marks a single file as erased. However, the file’s data will not be erased until the garbage collector is run.

If the method is called while the specified file is open, a SPIFLASHFILESYSTEM_ERROR.FILE_OPEN error will be thrown.

Parameters

Parameter	Type	Required?	Description
filename	String	Yes	The name of a file

Return Value

Nothing.

Example

// Delete testdata.txt
sffs.eraseFile("testdata.txt");

eraseFiles()

This method erases all of the files within the file system. It will return with an error if it is called when there are open files. The files are marked as erasable: their data will not be erased until the garbage collector is run.

Return Value

String — An error message, otherwise null.

eraseAll()

This method erases the portion of the SPI Flash allocated to the file system. Unlike eraseFiles(), eraseAll() will actually trigger an impOS spiflash erasesector operation.

If the method is called while the file system has files open, a SPIFLASHFILESYSTEM_ERROR.FILE_OPEN error will be thrown.

Return Value

Nothing.

Example

// Erase all information in the file system
sffs.eraseAll();

setAutoGc(numPages)

This method sets the autoGcThreshold property. The garbage collector will automatically run when the file system has fewer than autoGcThreshold pages. The default autoGcThreshold value is 4.

Setting numPages to 0 will turn off automatic garbage collection.

Parameters

Parameter	Type	Required?	Description
numPages	Integer	Yes	The minimum page count limit that will trigger garbage collection, or 0 to suspend garbage collection

Return Value

Nothing.

Example

// Set the filesystem to free pages marked as 'erased' whenever
// there are ten or fewer free pages left in the file system.
sffs.setAutoGc(10);

gc([numPages])

This method manually starts the garbage collection process. The SPIFlashFileSystem is designed in such a way that the auto garbage collection should be sufficient, and you should never need to call gc() manually.

Parameters

Parameter	Type	Required?	Description
numPages	Integer	No	A number of pages to be freed by garbage collection

If the numPages parameter is specified, the garbage collector will free up to numPages pages and return when it completes. This is what happens when the garbage collector runs because the file system needs a page and none are free.

If the numPages parameter is omitted, the garbage collector will run asynchronously in the background This is what happens when the garbage collector runs because free pages drops below the value of autoGcThreshold.

Return Value

Nothing.

SPIFlashFileSystem.File Usage

A SPIFlashFileSystem.File object is returned by the SPIFlashFileSystem instance each time a file is opened. Typically, you will not need to instantiate SPIFlashFileSystem.File objects yourself.

The SPIFlashFileSystem.File object acts as a stream, with an internal read/write pointer which can be manipulated with a variety of methods in the SPIFlashFileSystem.File class.

Constructor: SPIFlashFileSystem.File(filesystem, fileId, fileIndex, filename, mode)

The constructor creates a file record.

Parameters

Parameter	Type	Required?	Description
filesystem	Integer	Yes	The current SPIFlashFileSystem instance
fileId	Integer	Yes	The file’s unique ID
fileIndex	Integer	Yes	The index of the file in the internal record of open files
filename	String	Yes	The current file’s name
mode	String	Yes	The file’s access mode: `"r"` (read) or `"w"` (write)

SPIFlashFileSystem.File Methods

seek(position)

This method moves the file pointer to the specified location within the file.

Parameters

Parameter	Type	Required?	Description
position	Integer	Yes	An index within the file data

Return Value

Nothing.

Example

// Read the last byte of a file
file <- sffs.open("HelloWorld.txt", "r");
file.seek(file.len() - 1);
lastByte <- file.read(1);

tell()

This method gets the current location of the file pointer.

Return Value

Integer — The current pointer index within the file data.

eof()

This method indicates whether the file pointer is at the end of the file.

Return Value

Bool — true if the file pointer is at the end of the file, otherwise false.

Example

// Read a file 1-byte at a time and look for 0xFF
file <- sffs.open("HelloWorld.txt", "r");
while (!file.eof()) {
    local b = file.read(1);
    if (b == 0xFF) {
        server.log("Found 0xFF at " + b.tell());
        break;
    }
}

len()

This method gets the current size of the file data.

Return Value

Integer — the file size in bytes.

Example

// Read and log the length of a file
file <- sffs.open("HelloWorld.txt", "r");
server.log(file.len());

read([length])

This method gets data from the file, starting at the current file pointer, and returns it as a blob. If the optional length parameter is specified, that many bytes will be read, otherwise read() will read and return the remainder of the file.

Parameters

Parameter	Type	Required?	Description
length	Integer	No	The number of bytes to read

Return Value

Blob — The file data that was read.

Example

// Read and log the contents of a file
file <- sffs.open("HelloWorld.txt", "r");
server.log(file.read().tostring());

write(data)

This method writes a string or blob to the end of the target file’s data — provided you opened with mode "w".

If you attempt to write to a file opened with mode "r", a SPIFLASHFILESYSTEM_ERROR.FILE_WRITE_R error will be thrown.

Note The page header is not written to the SPI Flash until the entire page is written, or close() is called.

Parameters

Parameter	Type	Required?	Description
data	Blob or string	Yes	The data to be written

Return Value

Nothing.

Example

In the following example, we download a file in chunks from the agent:

file <- sffs.open("HelloWorld.txt", "w");

// Write some data to the file
// We can call write() as many times as desired
file.write("Hello");
file.write(" ");
file.write("World!");
file.close();

created()

This method yields the file’s creation timestamp.

Return Value

Integer — The target file’s creation timestamp.

close()

This method closes a file and writes data to the SPI Flash if required. All files that are opened should be closed, regardless of what mode they were opened in.

Please see write() for example usage.

Return Value

Nothing.

SPIFlashFileSystem.FAT Usage

A SPIFlashFileSystem.FAT object is automatically generated when the file system is initialized. It records the file allocation table (FAT). You should not instantiate SPIFlashFileSystem.FAT objects yourself.

Constructor: SPIFlashFileSystem.FAT(filesystem[, pages])

Parameters

Parameter	Type	Required?	Description
filesystem	Integer	Yes	A master SPIFlashFileSystem instance
pages	Integer	No	The number of 4KB pages the file system should contain. If no value is provided, the constructor will scan the file system for files and build an FAT from them

SPIFlashFileSystem.FAT Methods

scan()

This method scans the filesystem for files and uses them to construct a file allocation table.

Return Value

Nothing.

get(fileRef)

This method retrieves information about a specific file from the file allocation table.

Parameters

Parameter	Type	Required?	Description
fileRef	String or integer	Yes	Either a filename (string) or a unique file ID (integer)

Return Value

Table — File-specific information with the following keys:

Key	Type	Description
id	Integer	The file’s ID
fname	String	The file’s name
spans	Integer	The spans of the file
pages	Array	A list of pages in which the file is stored
pageCount	Integer	The number of pages in which the file is stored
sizes	Array	A list of the sizes (in bytes) of the file chunks each page contains
sizeTotal	Integer	The total size of the file
created	Integer	A timestamp indicating the date and time of the file’s creation

getFileList([orderByDate])

This method gets an array of file information, one entry per file in the file allocation table.

Parameters

Parameter	Type	Required?	Description
orderByDate	Bool	No	Whether the returned files should be pre-sorted into date order. Default: `false`

Return Value

Array — A set of file records.

getFileId(filename)

This method gets the unique integer by which the specified file is referenced in the file allocation table.

Parameters

Parameter	Type	Required?	Description
filename	String	Yes	The file’s name

Return Value

Integer — The file’s unique ID.

fileExists(fileRef)

This method indicates whether the specified file is present in the file system.

Parameters

Parameter	Type	Required?	Description
fileRef	String or integer	Yes	Either a filename (string) or a unique file ID (integer)

Return Value

Bool — Whether the named file exists in the file system (true) or not (false).

getFreePage()

This method provides the address of a random free page in the file system. It will return an error, SPIFLASHFILESYSTEM_ERROR.NO_FREE_SPACE, if it is unable to do so because there is no free space left.

Return Value

Integer — The address of a random free page in the file system, or an error.

markPage(address, status)

This method sets the status of the page at the specified address. The page’s status is set by providing one of the following values:

SPIFLASHFILESYSTEM_STATUS.FREE
SPIFLASHFILESYSTEM_STATUS.USED
SPIFLASHFILESYSTEM_STATUS.ERASED
SPIFLASHFILESYSTEM_STATUS.BAD

Parameters

Parameter	Type	Required?	Description
address	Integer	Yes	Either a filename (string) or a unique file ID (integer)
status	Constant	Yes	The page’s status (see above)

Return Value

Nothing.

addPage(fileId, page)

This method adds the specified page to the file allocation table for the specified file.

Parameters

Parameter	Type	Required?	Description
fileId	Integer	Yes	The file’s ID
page	Integer	Yes	The page number

Return Value

Nothing.

getPageCount(fileRef)

This method yields the number of pages that the specified file comprises.

Parameters

Parameter	Type	Required?	Description
fileRef	String or integer	Yes	Either a filename (string) or a unique file ID (integer)

Return Value

Integer — The number of pages holding the file.

forEachPage(fileRef, callback)

This method iterates over each page used to record the specified file.

Parameters

Parameter	Type	Required?	Description
fileRef	String or integer	Yes	Either a filename (string) or a unique file ID (integer)
callback	Function	Yes	The callback executed for each page. It has a single parameter which receives the current page in the iteration

Return Value

Nothing.

pagesOrderedBySpan(pages)

This method takes a set of pages and returns them in span order.

Parameters

Parameter	Type	Required?	Description
pages	Array of integers	Yes	The pages to be ordered

Return Value

Array — The pages in span order.

addSizeToLastSpan(fileId, bytes)

This method updates the size of the last span in a file.

Parameters

Parameter	Type	Required?	Description
fileId	Integer	Yes	The file’s ID
bytes	Integer	Yes	The new size in bytes

Return Value

Nothing.

set(fileId, file)

This method sets the span for the file specified by its ID.

Parameters

Parameter	Type	Required?	Description
fileId	Integer	Yes	The file’s ID
file	SPIFlashFileSystem.File instance	Yes	The object representing the file

Return Value

Nothing.

removeFile(filename)

This method removes the specified file from the file allocation table, ie. it deletes the file.

Parameters

Parameter	Type	Required?	Description
filename	String	Yes	The file’s name

Return Value

Nothing.

getSectorMap()

This method returns the file allocation table’s sector map.

Return Value

Blob — the map.

getStats()

This method provides the number of pages in the file system in each of the four status categories.

Return Value

Table — The information with the keys free, used, erased and bad. Each key’s value is an integer: the number of pages in the file system with that status.

describe()

This method is intended to assist with debugging: it provides a readout in the impCentral log of the current number of files in the file allocation table, and lists each file by name, the number of pages it spans and its total size.

Return Value

Nothing.

Release History

The Electric Imp Dev Center documents the latest version of the library. For past versions, please see the Electric Imp public GitHub repos listed below.

Version	Source Code	Notes
1.0.0	GitHub	Formal initial release
1.0.2	GitHub	Bug fixes
1.1.0	GitHub	Add dimensions(), created(), getFreeSpace(), eraseFiles() methods to SPIFlashFileSystem; add created() method to SPIFlashFileSystem.File
1.2.0	GitHub	Critical bug fix
2.0.0	GitHub	Update library name to new naming convention
3.0.0	GitHub	Moved constants and statics to enums; updated dimensions() method docs; flash enable limited to duration of flash operations; updated garbage collector async flash enable()/disable() method call placements, so flash is not enabled across imp.wakeup() calls; removed un-needed _enables flag; replaced _enable() and _disable() functions with flash.enable()/disable(); fixed example
3.0.1	GitHub	Bug fixes: flash no longer enabled through garbage collection imp.wakeup() calls

License

This library is licensed under the MIT License.

SPI Flash File System

Latest Version: 3.0.1

Library Classes

Overview Of The File System

An Example File

First Sector

Second Sector

Garbage Collection

SPIFlashFileSystem Usage

Constructor: SPIFlashFileSystem([start, end, spiflash])

Parameters

Example: imp003 And Above

Example: imp001/imp002

SPIFlashFileSystem Methods

init([callback])

Parameters

File Records

Return Value

Example

dimensions()

Return Value

Example

getFreeSpace()

Return Value

getFileList([orderByDate])

Parameters

Return Value

Example

fileExists(filename)

Parameters

Return Value

Example

isFileOpen(filename)

Parameters

Return Value

fileSize(filename)

Parameters

Return Value

Example

created(fileRef)

Parameters

Return Value

Example

open(filename, mode)

Parameters

Return Value

Example

eraseFile(filename)

Parameters

Return Value

Example

eraseFiles()

Return Value

eraseAll()

Return Value

Example

setAutoGc(numPages)

Parameters

Return Value

Example

gc([numPages])

Parameters

Return Value

SPIFlashFileSystem.File Usage

Constructor: SPIFlashFileSystem.File(filesystem, fileId, fileIndex, filename, mode)

Parameters

SPIFlashFileSystem.File Methods

seek(position)

Parameters

Return Value

Example

tell()

Return Value

eof()

Return Value

Example

len()

Return Value

Example

read([length])