Debugging a Waterken^TM application

Trace

When programming in the event loop paradigm, application logic is often spread across multiple event loop turns, as processing of one event triggers other events, which each may trigger further events. In a distributed application, this causal chaining of events also crosses the network, as an event is triggered on one machine and processed on another. To enable tracing of these execution paths, the Waterken server can be configured to emit a log of every event triggered or processed. To facilitate debugging of applications that run on multiple machines, these log events can be transmitted, as they occur, to a centralized viewing application.

Configure event logging

The Waterken server can both emit log events, and collect log events emitted locally or remotely.

Configure event storage

The default configuration files that come with the waterken-server distribution setup the server as a log event store, so the feature just needs to be turned on. To do so, you will first need to generate the web-key used to transmit log events. The key part of this web-key can come from a web-key created by a discarded test application. Configure your server with this key by putting it in the config/logKey.json file. For example, the contents of this file should look like:

{ "=" : "mySecretKey" }

Configure event transmission

For each server transmitting log events, add a file named log.json to its config/ folder. This file specifies the web-key used in sending log events. For example,

{ "@" : "http://localhost:8080/~/log/#s=mySecretKey" }

The above example sends log events to the previously configured log store on the local machine. The received events are stored in the file config/log/live.log. This file uses JSON syntax, but the closing ']' is missing. Manually add the closing bracket before using the file with any JSON parsing software.

You may also wish to output log comments to the console. If so, use the following content for log.json:

{
  "class" : [ "org.waterken.trace.Verbose" ],
  "out" : { "@" : "stdout" },
  "next" : { "@" : "http://localhost:8080/~/log/#s=mySecretKey" }
}

Log format

The format of each log event is defined by a class in the org.ref_send.log package. For example, each eventual invocation generates an event like:

  {
    "class" : [ "org.ref_send.log.Sent", "org.ref_send.log.Event" ],
    "anchor" : {
      "number" : 87,
      "turn" : {
        "loop" : "http://localhost:8080/-/bang/",
        "number" : 0
      }
    },
    "message" : "jeawtmckq2lm47-2-0",
    "trace" : {
      "calls" : [ {
          "name" : "Beat.make",
          "source" : "org/waterken/bang/Beat.java",
          "span" : [ [ 77 ] ]
        }, {
          "name" : "All.make",
          "source" : "org/waterken/all/All.java",
          "span" : [ [ 42 ] ]
        } ]
    }
  }

Comments

You can insert your own comments into the log by invoking comment() on the log member of the eventual operator. For example:

  {
    "class" : [ "org.ref_send.log.Comment", "org.ref_send.log.Event" ],
    "anchor" : {
      "number" : 93,
      "turn" : {
        "loop" : "http://localhost:8080/-/bang/",
        "number" : 0
      }
    },
    "text" : "all bang requests queued",
    "trace" : {
      "calls" : [ {
          "name" : "Beat.make",
          "source" : "org/waterken/bang/Beat.java",
          "span" : [ [ 97 ] ]
        }, {
          "name" : "All.make",
          "source" : "org/waterken/all/All.java",
          "span" : [ [ 42 ] ]
        } ]
    }
  }

Log analysis

The generated log file can be analyzed using Causeway.