Skip to main content

Squirrel Generator Functions

How To Add Parallel Processing Capabilities To Your Code

Squirrel programs, whether they are running on the device or in the impCloud™ as agents, have to be ‘event driven’. In this methodology, the system runs the main program loop on your behalf, and you write code that attaches to the main loop as handlers for the specific individual events you are interested in. This is because there are other, system-maintained event handlers that also hang off the same loop, for example to deal with network messages.

  • If you’re new to the event-driven approach, we suggest you take a look at the Developer Guide ‘Event-driven Programming’, which explores in detail how imp application code uses this technique.

While this event-driven approach is a well-known way of writing desktop programs, mobile apps and server software, it is less common in the embedded world, where programmers are more used to writing their own main loop. But if you attempt to do that on an imp, you’ll find that other important things, including network messages, don’t get handled.

Fortunately, there is a way to write what looks very much like a main loop yet still allows the system to do what it needs to do; it allows the system event handlers to continue running. This technique relies on an little-known feature of the Squirrel language: generators.

Yield

A generator is a special kind of function. When a generator is called, it doesn’t execute immediately and in its entirety in the the way an ordinary function would. Instead it is loaded into memory and its execution suspended until it is required.

It is the presence of Squirrel’s yield keyword in one or more statements within the code that marks the function as a generator. When the generator function is called in the customary way, Squirrel establishes the function’s data structure in memory and then checks for the presence of yield. Having found the keyword, Squirrel returns a reference to the generator. If there is no yield, the function executes there and then.

To activate the generator, the calling code must include a resume statement with the generator’s reference. When resume is encountered, Squirrel runs the generator code. When the Squirrel encounters yield, it completes the current line and then once again suspends the execution of the function. It preserves the function’s state, including all of its local variables.

If the generator is brought back into action by a further resume statement, the generator will continue to execute its code with the line that followed the suspending yield statement.

Think of a generator, then, as a set of instructions which can be executed in stages at arbitrary points in time without losing any of its data.

Emulating main()

To go back to our original example, here is some sample code — a typical ‘Hello World’ program that flashes an LED — which might be used to simulate an embedded device’s main loop:

local looper;

function mainloop() {
    while (true) {
        hardware.pin1.write(1);
        yield pause(1.0);
        hardware.pin1.write(0);
        yield pause(1.0);
    }
}

function pause(duration) {
    return imp.wakeup(duration, function(){ 
        resume looper; 
    });
}

looper = mainloop();
resume looper;

First, we establish a variable, looper, to which we will later (at line 19) assign a reference to the instance of the generator function mainloop() in memory.

mainloop() itself loops infinitely, writing a 1 to the imp’s pin 1, waits a second, writes a zero, waits another second, and then starts all over again. To do so, it contains a while(true) loop which you would normally expect to block the CPU and thus something to be avoided in an event-driven environment such as impOS. However, by including yield statements in lines 8 and 10, we establish points where we can pause the execution of mainloop() and step out of the loop to run another part of the program.

In this case, we do so by calling the second function, pause(). This simply tells the imp to idle for the passed period — one second — then wake and resume mainloop(), accessed through the variable, looper.

Line 19 is where the program starts: mainloop() is called for the first time. Squirrel examines the called function, spots the yield statements and so simply establishes the function data structure in memory then returns a reference to that structure. This reference is stored in looper. Note that we actually call mainloop() and store a returned value — we don’t simply load looper with a reference to mainloop() as we might if we intended mainloop() to be a callback function.

At this point mainloop() is suspended, so line 20 uses the resume command to start it up. The generator begins its operation by instigating the seemingly closed loop, writes a 1 to pin 1 to turn on the LED, and then yields its execution. The function pause() is called, and this queues up a trigger to resume mainloop() in one second’s time.

When this event fires, mainloop() resumes execution at line 9; it writes a 0 to pin 1, turning the LED off again. Line 10 now once again causes mainloop() to yield its runtime to the nominated function, pause().

This time when the timer queued by pause() fires, mainloop() begins to run again: the while loop checks its condition and re-iterates the loop: pin 1 goes to 1, and the yield statement at line 8 is run once again.

Crucially, although we have what appears to be an infinitely blocking loop structure — replicating, say, an Arduino loop() function, which is an implicit while(true) loop — we nonetheless literally yield some of the loop’s runtime to the system. This allows the system’s own event loop to be processed. As far as the rest of the code inside the generator is concerned, however, it is running on the CPU exclusively, with a couple of one-second delay loops built in.

Any generator comes to an end when it hits a return statement or returns implicitly by reaching the end of the sequence of instructions in its block. At this point, the generator is considered dead and can no longer be resumed. In our sample code, the generator never dies, and the loop runs forever, just like an Arduino loop(). Squirrel runtime errors will automatically cause generators to die, as will exceptions that are not caught and handled.

Return Values

Like return, yield can send back a value. This value may then be accessed as the result of the resume statement. In the example above, yield returns a reference to the function pause(), though the code makes no use of that reference, but an alternative example is:

function loopsteps(total) {
    for (local i = 1 ; i < total + 1 ; ++i) {
        yield i;
    }

    return null;
}

local looper = loopsteps(10);
local yieldResult;

while (yieldResult = resume looper) {
    server.log("Step through the loop: " + format("%u", yieldResult));
}

This will display ten entries in the IDE log, with the numbers 1 through 10, each value return by yield i and relayed by resume looper.

Retained State

It’s the generator’s ability to retain its state even though its execution is suspended that gives the generator power that alternative constructions lack. For instance, we might use an imp.wakeup() call to restart the function. We might have written:

function mainloop() {
    hardware.pin1.write(flag ? 1 : 0);
    flag = !flag;
    imp.wakeup(1.0, mainloop);
}

flag <- true;
mainloop();

This also switches pin 1 on and off every second, while also allowing the system access to processing resources in the meantime. However, this approach would not preserve the state of either function after the timer has been queued in each function’s last line. This is not a concern with such a simple example, but it a more complex program it might well be a requirement.

In our earlier generator example we make no specific use of this, but even simply adding a loop counter shows how values persist across iterations of the generator. Other applications may rely on the preservation of local variables through periods of suspension. We could, for instance, modify mainloop() this way:

function mainloop() {
    local lifespan = 0;

    while (true) {
        hardware.pin1.write(1);
        yield pause(1.0);
        hardware.pin1.write(0);
        yield pause(1.0);
        lifespan++;
        if (lifespan == 30) return;
    }
}

This loop will continue as before, but this time the variable lifespan will be incremented with each pass of the code. When lifespan’s value equals 30, the function returns and the generator dies.

Automatic Looping

Squirrel’s foreach command provides a convenient way of managing generator loop stages until the function has run its course:

function intGenerator() {
    server.log("Yielding 1");
    yield 1;
    server.log("Yielding 2");
    yield 2;
    server.log("Yielding 4");
    yield 4;
    server.log("RETURNing 8");
    return 8;
}

local looper = intGenerator();
foreach (stage in looper) {
    server.log(stage);
}

In this example, the foreach statement spots that looper is a reference to a generator and so issues all the appropriate resume commands implicitly: once to start it up and then again following each of the generator’s yield statements. The generator’s return statement triggers the exit from the foreach loop; stage will never take the value 8, only 1, 2 and 4.

Other Features

Calling a single generator function multiple times will create multiple, independent instances of the generator:

function intGenerator() {
    server.log("Yielding 1");
    yield 1;
    server.log("Yielding 2");
    yield 2;
    server.log("Yielding 4");
    yield 4;
    server.log("RETURNing 8");
    return 8;
}

local looper1 = intGenerator();
local looper2 = intGenerator();
local looper3 = intGenerator();

The three variables looper1, looper2 and looper3 are entirely separate entities, though of course they all perform exactly the same tasks. Passing a parameter can be used to distinguish one from the other:

function intGenerator(value) {
    server.log("Yielding 1");
    yield value + 1;
    server.log("Yielding 2");
    yield value + 2;
    server.log("Yielding 4");
    yield value + 4;
    server.log("RETURNing 8");
    return value + 8;
}

local looper1 = intGenerator(10);
local looper2 = intGenerator(20);
local looper3 = intGenerator(30);

Generators are first-class objects in Squirrel so they can be stored in arrays and tables:

local generators = [looper1, looper2, looper3];

and operate as the properties of other objects. However, because they are functions they can’t be serialized, neither can arrays or table containing them.

Storing generators in arrays and tables is particularly useful nonetheless: it allows you to maintain multiple, parallel loops. This provides a convenient way of storing the references to each generator, allowing you to iterate through all of them, triggering each with a resume command.

Multiple generators can therefore mimic a multitasking environment without any need to write complex code to manage threads. In this case, because all other generators are suspended when one is running — ie. the code is not truly parallel — there is no need to lock resources to prevent, say, global variables being changed by one generator mid-way through a calculation being performed on those variables by another.

Memory Considerations

There is no limit on the number of generators you can employ, either in device or agent code, except for the memory they take up. That memory consists of the stack frame of the generator function, which includes all the generator function’s local variables. However, you can easily keep track of available memory by using the imp API method imp.getmemoryfree().