Live fast, die young and leave a good-looking corpse

Any application that keeps state needs a plan for maintaining it as the application evolves. Successfully managing these upgrades is one of the more challenging aspects of application development. This document explains several patterns and techniques for managing upgrade in a Waterken application.

Persistence engine

Application objects hosted by the Waterken server can exploit the provided orthogonal persistence. At the end of every event loop turn, before any externally visible I/O is done, the Waterken server persists all application objects that are reachable from previously persisted objects. This graph of persistent objects is rooted in objects bound to an exported web-key. The programmer expresses different strategies for persistent state through patterns of extending, modifying and using this persistent object graph. This paper presents these patterns and shows how they can be expressed using the ref_send API.

Ephemeral pleasures

An effective strategy for dealing with the challenges of persistent state is avoiding it. When using the Waterken server, there are several ways in which persistence can be usefully avoided.

Any created Java object that is not referenced by a previously persisted Java object at the end of an event loop turn is not persisted. Since the object is never stored, the programmer is freed from the burden of planning for its upgrade.

The GUI for a Waterken application is often a web-application. The state of this GUI is rebuilt anew on each visit to the application, using state pulled into the browser via web-keys. It's a good strategy to express as much of the application logic as possible in the ephemeral web-application, rather than in persistent Java objects. Think of the state maintained by the Waterken server as the application's model, as in the Model-View-Controller (MVC) pattern. Your application's view and controller are then implemented in the ephemeral web-application. This design reduces the upgrade challenge to the design of the application's model. Constraining the scope of the upgrade challenge in this way is the most important step to take in the design of an application's persistence strategy.

Minor surgery

As an application evolves, its model may need to be extended. Objects are persisted by the Waterken server using Java Object Serialization, which supports a useful set of compatible upgrades, such as:

adding and removing methods
adding and removing interfaces
adding and removing classes in the inheritance chain
adding and removing fields
changing the declared type of a non-primitive field

When upgrading an object requires an algorithmic step, a class can declare a readObject method of the form:

private void
readObject(final ObjectInputStream in) throws IOException,
                                              ClassNotFoundException {
    // read in the old state
    in.defaultReadObject();

    // using the old state, upgrade the object to the new format
    …
}

Java Object Serialization only allows these upgrades if the serialized class declares them to be allowed. This is done by declaring a serialVersionUID field, such as:

    static private final long serialVersionUID = 1L;

This field should be declared as part of the first steps in creating a persistent class.

Pushing all state through a minor upgrade

Once the desired changes have been made to a set of upgraded classes, the upgrade can be applied using the touch.jar command provided by the Waterken server. For example, to upgrade a vat named test, as well as all of its child vats:

stop the Waterken server
replace the old class files with the upgraded ones
from the home folder of the Waterken installation, run the command:
java -jar touch.jar vat/test
restart the Waterken server

The touch.jar command sweeps through all of the persistent state for the named vat, pushing each object through the Java Object Serialization upgrade process and saving its new state to disk. The command runs as a normal vat transaction and so will either commit with all objects upgraded, or abort, leaving the vat in its original state.

Amputation

Sometimes, it's not feasible to express an algorithm for upgrading old state using the techniques described in the previous section. In this case, it's also sometimes the case that the old state is no longer needed in the upgraded application. In this situation, a viable upgrade strategy is to simply plan to abandon some objects. When the Waterken server is deserializing part of an object graph, it doesn't immediately follow an eventual reference, but instead delays loading of this part of the graph until the target object is accessed by the application. To plan for amputation of part of an application's persistent object graph, only refer to objects in that part of the graph using eventual references. Once bit rot takes hold in that part of the object graph, change the application code to no longer access it. If the stale objects are no longer accessed, they will never be loaded and so will never produce incompatible serialization errors.

Commuter pattern

To support delayed loading of part of an object graph, and so enable amputation of it, a class must use an eventual reference to refer to an object that instances may need immediate access to. To facilitate this pattern, the ref_send API provides some syntactic sugar:

import static org.ref_send.promise.eventual.Eventual.near;
…

public final class
Stable implements Serializable {
    static private final long serialVersionUID = 1L;

    private final Eventual _;
    private       Unstable dependency_;

    …

    public void
    setDependency(final Unstable dependency) {
        dependency_ = _._(dependency);
    }

    public void
    doSomething() {
        final Unstable dependency = near(dependency_);
        final String foo = dependency.getFoo();
        …
    }
}

If the type Unstable in the example above is a class, rather than an interface, implementation of the pattern is slightly different:

import static org.ref_send.promise.Fulfilled.detach;
import static org.ref_send.promise.eventual.Eventual.near;
…

public final class
Stable implements Serializable {
    static private final long serialVersionUID = 1L;

    private final Eventual _;
    private       Promise<Unstable> dependency_;

    …

    public void
    setDependency(final Unstable dependency) {
        dependency_ = detach(dependency);
    }

    public void
    doSomething() {
        final Unstable dependency = near(dependency_);
        final String foo = dependency.getFoo();
        …
    }
}

Implants

There are two major roles for an application's model: storing information and enforcing the rules that govern changes and access to this information. Evolving an application sometimes means supporting new ways of manipulating the same underlying information. To support this kind of upgrade, an application's model should segregate objects that hold the application's state from objects that define the application's behaviour. The goal of this segregation is to enable creation of a new layer of behaviour objects for the same set of state objects.

For example, the yurl.net DNS nameserver is a Waterken server application for managing DNS records. When the genkey.jar command is run, a new vat is created to manage the DNS resources for a newly created hostname. The application model stored in this vat segregates application state, the held DNS resources, from application behaviour, editing of those resources, and provides a hook for the introduction of new behaviour.

The nameserver's application state is held in an array of variables, each of which holds an arbitrary Resource.

The nameserver's behaviour is defined by the ResourceGuard, which constrains the supported DNS resource types, and the Menu, which supports editing of the variable array.

On creation, a DomainMaster object provides the client access to objects that define the application's current behaviour, as well as an object that may define future application behaviour. The extension member of DomainMaster refers to an object that holds references to all of the objects embodying the application's state. This extension object provides a hook for introducing new behaviour for the existing application state. The hook is used by adding a new method to the extension object that creates a new layer of behaviour objects.

Offload

If plans for upgrade don't work out, or the cumulative complexity from multiple upgrades becomes too much, it may be best to start over with a new design, bootstrapping new vats with application data pulled in from the old model. To enable this offloading of application state, the application model should support a query interface that can generate documents describing the current application state.

For example, the previously discussed DNS nameserver supports creation of such application state documents via the Menu's getSnapshot() method. This method returns a document describing all of the current DNS resources for a hostname.

Application design philosophy

In addition to its role in persistence, the vat is also the unit of concurrency. Each vat is serviced by its own thread, so spawning a new vat creates a new thread, enabling concurrent execution within an application. To exploit multiple processor cores, an application should be designed to use many vats.

Once an object is persisted, its state remains on disk until the hosting vat is destructed. If the object is not in use, it will never be loaded and so won't consume the computer's processing power or memory, but will consume filesystem space. To reclaim this space, an application should be designed around short-lived vats that are periodically destructed, giving way to new vats. Whenever an application is about to engage in an activity that will generate a bunch of new persistent objects, a new vat should be spawned for that activity. When the activity completes, the spawned vat should be destructed, after producing a document describing the outcome of the activity.

To best leverage the platform provided by the Waterken server, applications should be designed around a vat usage philosophy of: "Live fast, die young and leave a good-looking corpse". Spawn vats frequently to make an application faster by exploiting concurrency. Destruct vats soon to avoid the upgrade problems that come with long-lived state and to reclaim filesystem space. Produce state description documents to facilitate upgrade and interoperation with other applications.