SVN URL, Bang Tutorial, wsh

JSON shell for the browser

The web_send library provides a concise and expressive API for interacting with arbitrary JSON resources from the web browser. When used from the Firebug console, it acts like a command line for your web server; a great help during development of server-side code. The same API is also convenient for creating an AJAX user interface to JSON resources; so code born on the interactive command line migrates smoothly into application code. This tutorial also links to a live web page where you can try out the library against a simple server-side counter object.

Some quick introductory examples

For example, say you've got a brand new server-side object sitting at a URL like: <https://example.com/myApp/obj123>. All you want to do is invoke one of its methods, to see what happens. Using the web_send library from the Firebug console, you could write:

factory = lib.web.getLocation();    // grab the window.location URL
drum = lib.Q.post(factory, 'makeDrum', []);

That code generates the following HTTP request:

POST /myApp/obj123?q=makeDrum HTTP/1.1
Host: example.com
Content-Type: text/plain; charset=UTF-8
Content-Length: 2

[]

The arguments to Q.post() are:

remote reference for the target object
optional argument to add to the query string
optional JSON value for the request body

The makeDrum() method didn't take any arguments. Here's one that does:

lib.Q.post(drum, 'bang', [ 1 ]);

Causing the HTTP request:

POST /myApp/obj456?q=bang HTTP/1.1
Host: example.com
Content-Type: text/plain; charset=UTF-8
Content-Length: 5

[ 1 ]

The target URL in the above request was taken from the HTTP response to the previous request. For this to work, the web_send library introduces some conventions for HTTP requests and responses. These conventions are the least restrictive they can be, while still supporting the client-side remote reference API. This document explains these conventions and shows how to use legacy JSON resources that don't follow the conventions. Using the support for legacy JSON resources, GET, POST, PUT and DELETE requests can be sent to a resource at any URL, with any JSON request entity and any JSON response entity.

JSON conventions

The web_send library generates JSON for an HTTP request entity and extracts information from the JSON in an HTTP response entity. Doing so requires a few conventions.

JSONlink

In the introductory example, one of the requests was sent on a reference derived from a previous HTTP response. For this to work, the web_send library needs to know where to find the corresponding URL inside the HTTP response. To enable this, represent a URL as a JSON object having a single member named "@", whose value is the URL string. For example, the HTTP response in the introductory example was:

HTTP/1.1 200 OK
Content-Type: application/json
Content-Length: 19

{ "@" : "obj456" }

When constructing a remote reference, the web_send library will resolve any relative URL. Following the convention in HTTP, the Request-URI is the base URL for any relative URL in the response entity. In this case, the base URL was <https://example.com/myApp/obj123?q=makeDrum>, making the resolved URL <https://example.com/myApp/obj456>.

JSONerror

Sometimes, an invoked method will throw an exception. A thrown exception should be encoded in JSON as:

{ "!" : "LPT1 on fire" }

The value of the "!" member can be any JSON value. Use it to communicate the server method's error information. In the shown example, a simple string is used for error information. More information can be provided by using a JSON object:

{
  "!" : {
    "message" : "LPT1 on fire",
    "errno" : -1,
    "stack" : "print()@42"
  }
}

Following this convention for exceptional returns enables the following client-side error handling code:

hits = lib.Q.get(drum, 'hits');
lib.Q.when(hits, function (value) {
    // GET request successfully returned the given value
}, function (reason) {
    // GET request failed for the given reason
});

The reason passed to the error callback is the value of the "!" member.

JSONvalue

Some methods only need to return a simple value, like a string or a number; whereas JSON requires that a JSON text be a JSON object or JSON array. To return a simple value, return a JSON object with a member named "=", whose value is the simple value. For example, to return the number 42:

{ "=" : 42 }

URL conventions

When sending an HTTP request, it is often useful to include some arguments in the Request-URI. The web_send library supports the following conventions:

'q' query string argument

Both Q.post() and Q.get() take an optional string argument. If a value is provided, it will be prepended to the Request-URI's query component as the value of the 'q' parameter (using the application/x-www-form-urlencoded syntax). For example, the following transformations are made for the call:

lib.Q.get(drum, 'hits');

target URLref	Request-URI
/myApp/obj456	/myApp/obj456?q=hits
/myApp/?id=obj456	/myApp/?q=hits&id=obj456
/myApp/?q=42	/myApp/?q=hits&q=42

fragment arguments

Sometimes, it is useful to include information in a URL that won't show up in the HTTP protocol's Referer header, but can be made available to the server that issued the URL. To support this, the web_send library can move information in the URL fragment to the query component of the Request-URI. For example, for the call:

lib.Q.get(drum, 'hits');

target URLref	Request-URI
/myApp#s=obj456	/myApp?q=hits&s=obj456
/myApp?id=42#s=obj456	/myApp?q=hits&id=42&s=obj456
/myApp?s=42#s=obj456	/myApp?q=hits&s=42&s=obj456
/myApp?s=42#s=obj456&t=6&=label	/myApp?q=hits&s=42&s=obj456&t=6

In the last example, the character sequence '&=' is recognized as terminating the part of the fragment that should be copied to the Request-URI.

session identifier

Before sending any POST request, the web_send library will first send a request asking the server if a session should be created. This GET request is sent to the URL <?q=fresh&s=sessions>, resolved relative to the remote reference URL that is about to be used. If your server wishes to associate requests with a session, respond with JSON like:

{
  "sessionKey" : "session123"
}

The web_send library will then include this key as the 'x' parameter in all POST requests, as well as a number 'w', which is incremented on each request. For example, for the code:

lib.Q.post(drum, 'bang', [ 3 ]);

The Request-URI will be </myApp/obj456?q=bang&x=session123&w=2>.

It is crucial to ensure a success response to the session creation request is marked not cacheable, such as by including the response header: Cache-control: no-cache.

If your server does not wish to use a session, respond to the session creation request with a 404 response. Make the response cacheable to avoid receiving future session creation requests.

server-side promise

The reference returned by a call to Q.get() or Q.post() is a special kind of reference called a "promise". A promise is unlike a normal reference in that it can refer to an object that is yet-to-be-determined, such as the return value from an asynchronous request. If your server-side code also uses promises, a URLref that refers to a promise must be distinguished from one that refers to a direct object reference. Mark a URLref as referring to a promise by starting the fragment with the text 'o='. This marking is used by Q.when(), which registers callbacks to be notified when a promise is resolved. If the promise marking is present, Q.when() will fetch the resolved value from the server; otherwise, it will treat the client-side promise as the fulfilled value. For example:

var remoteProxy = … // @ https://example.com/myApp/#s=obj123
lib.Q.when(remoteProxy, function(value) {
    // value will be remoteProxy
});
var remotePromise = … // @ https://example.com/myApp/#o=&s=obj123
lib.Q.when(remotePromise, function(value) {
    // value will be JSON value returned by GET request to:
    // <https://example.com/myApp/?o=&s=obj123>
}, function(reason) {
    // or the reason the GET request failed
});

Server-side files

The code for the web_send library must be delivered to the browser by your server. Checkout the necessary files using the subversion command:

svn co https://waterken.svn.sourceforge.net/svnroot/waterken/server/trunk/waterken/config/file/site/

To bootstrap the web_send library, your server must return HTML code like that in the example/index.html file. This code links in the necessary JavaScript code. For example, if your application objects are at URLs like: <https://example.com/myApp/#s=obj123>, you should serve this HTML from a request to <https://example.com/myApp/>.

Core files

The example/index.html file includes lots of example code and is designed as an ADsafe widget. You don't have to use the ADsafe verifier, or include much of this code. The file minimal/index.html shows the minimum requirements:

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN"
   "http://www.w3.org/TR/html4/strict.dtd">
<html>
<head>
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<title></title>
<script type="text/javascript" src="/site/json2.js"></script>
<script type="text/javascript" src="/site/adsafe.js"></script>
</head>
<body>
<noscript>
<p>This page requires a Javascript enabled web browser.</p>
</noscript>

<div id="WIDGET_">
<script type="text/javascript">
"use strict";
ADSAFE.id('WIDGET_');
</script>
<script type="text/javascript" src="/site/ref_send.js"></script>
<script type="text/javascript" src="/site/web_send.js"></script>
<script type="text/javascript">
"use strict";
ADSAFE.go('WIDGET_', function (dom, lib) {
    /*
     * The ADsafe and web_send libraries are now initialized. Start your
     * application code from here.
     */
});
</script>
</div>

</body>
</html>

The key components are:

json2.js: JSON encoding and decoding
adsafe.js: HTML manipulation
ref_send.js: asynchronous operations
web_send.js: remote reference implementation
ADSAFE.go(): initializes all the libraries

Security model

If you do want to make use of the ADsafe verifier, you can use it and the web_send library to control the network access of widgets on your web pages. ADsafe is a subset of Caja, so any code you produce under ADsafe should also work in a Caja environment.

By default, the ADsafe verifier prevents any network access by a verified widget. If you allow such a widget to load the web_send library, it gets permission to send requests using the window.location URL, and any URL received in a request response. By controlling what URLs your server returns in its responses, you can control what URLs the verified widget can access. For example, your social networking site may host third-party applications running within your pages in the browser. Using ADsafe and web_send, you could give such an application access to a server-side user database and know that it cannot call home to its developer, nor leak the provided permission to another application.

The web_send library also allows clients to navigate the browser window using any received remote reference. If you wish to allow a widget to message with a particular server-side object, but not navigate the browser, load the web_send library in a separate widget which then passes a remote reference to the more constrained widget. Using this remote reference and the lib.Q library, the more constrained widget can message with the server-side object, but cannot navigate the browser, since it doesn't have access to the lib.web library.

API

The API is divided into two parts: lib.Q, the asynchronous operation API; and lib.web, the network access API.

`lib.Q` API

The ref_send.js script defines the ADsafe library: lib.Q. This API supports asynchronous operations; such as sending a GET request, and setting up a callback for a request response.

run(task)

Enqueues a task to be run in a future turn.

task: function to invoke later

reject(reason)

Constructs a rejected promise.

reason: object describing the failure

ref(value)

Constructs a promise for an immediate reference.

value: immediate reference

defer()

Constructs a ( promise, resolver ) pair.

The resolver is a callback to invoke with a more resolved value for the promise. To fulfill the promise, simply invoke the resolver with an immediate reference. To reject the promise, invoke the resolver with the return from a call to reject(). To put the promise in the same state as another promise, invoke the resolver with that other promise.

near(value)

Gets the current value of a promise.

value: promise or immediate reference to evaluate

when(value, fulfilled, rejected)

Registers an observer on a promise.

value: promise or immediate reference to observe
fulfilled: function to be called with the fulfilled value
rejected: function to be called with the rejection reason

promise for the return value from the invoked callback

get(target, q)

Sends a GET request.

target: request target reference
q: optional additional query string argument

promise for the response entity

post(target, q, entity)

Sends a POST request.

target: request target reference
q: optional additional query string argument
entity: optional JSON value for request entity

promise for the response entity

put(target, q, entity)

Sends a PUT request.

target: request target reference
q: optional additional query string argument
entity: optional JSON value for request entity

promise for the response entity

remove(target, q)

Sends a DELETE request.

target: request target reference
q: optional additional query string argument

promise for the response entity

`lib.web` API

The web_send.js script defines the ADsafe library: lib.web. This API provides a remote reference interface to the network; such as creating a remote reference from a URL, and creating a hyperlink from a remote reference.

getLocation()

Constructs a remote reference for the current window.location.

navigate(target)

Navigates the window.

target: remote reference for new location

true if navigation successful, else false

href(element, target)

Sets the 'href' attribute.

element: element to modify
target: remote reference

the element, or the ':rest' of the bunch if not modified

src(element, target)

Sets the 'src' attribute.

elements: element to modify
target: remote reference

the element, or the ':rest' of the bunch if not modified

fetch(field)

Constructs a remote reference from a URLref held in a password field.

field: bunch containing a single password field

getTitle()

Gets the document title.

title(text)

Sets the document title.

text: new title text

_ref(base, href, args)

Constructs a remote reference.

base: optional remote reference for base URL
href: URLref to wrap
args: optional query argument map

_url(arg, target)

Extracts the URLref contained within a remote reference.

arg: remote reference to extract URLref from
target: optional remote reference for base URL

the URLref, or null if not a remote reference

Legacy JSON resources

Though existing JSON resources may not follow the introduced conventions for URLs and entities, it may still be more convenient to use the web_send library to interact with them, than it is to use XMLHttpRequest, or even another wrapper library. This section explains how.

Construct a remote reference from a URL

If your JSON doesn't follow the JSONlink convention, you'll have to construct your remote references manually, instead of having the web_send library do it for you automatically. After extracting a URL from a JSON response, pass it and the request target reference to web._ref(). For example:

var page = lib.web.getLocation(); // @ <https://example.com/myApp/>
lib.Q.when(lib.Q.get(page), function (value) {
    var stuff = lib.web._ref(page, value.stuff.url);
    // if stuff.url was "stuff.php", then stuff is
    // <https://example.com/myApp/stuff.php>

    // make a request using the constructed remote reference
    var moreStuff = lib.Q.get(stuff);
    …
});

You can also augment the URL with query arguments:

var stuffArgs = lib.web._ref(page, value.stuff.url, {
    on: true,
    id: 'P123'
});

Extract the URL from a remote reference with a call to web._url(). For example:

var url = lib.web._url(stuffArgs);
// url is <https://example.com/myApp/stuff.php?on=true&id=P123>

The extracted URL is an absolute URL, unless you specify an optional base:

var relativeURL = lib.web._url(stuffArgs, page);
// relativeURL is <./stuff.php?on=true&id=P123>

Only HTTP error branch

If your JSON doesn't follow the JSONerror convention, only HTTP level errors will trigger the rejection callback provided to Q.when(). All successful HTTP requests are sent to the fulfilled callback. For example:

lib.Q.when(lib.Q.get(stuff), function (value) {
    // value is JSON entity from a success HTTP response
}, function (reason) {
    var status = reason.status; // HTTP status code
    var phrase = reason.phrase; // HTTP reason phrase
});

HTTP cookies

If your JSON resources require use of cookies, set these up as you normally would. The web_send library doesn't provide an API for doing this, but also doesn't prevent the browser from sending and receiving cookies with the produced HTTP requests.

Security considerations

For HTTP requests issued by the web_send library, the query component of the Request-URI contains data contributed by: optional arguments added by the web_send library ('q', 'x' and 'w'), the query component of the target URLref, and the fragment component of the target URLref. These elements are added to the Request-URI's query component in that order, which allows a server to disambiguate them. A request with duplicate query parameters may indicate an attack. For example, the following code:

var target = …   // @ <https://example.com/myApp?q=foo#q=bar>
lib.Q.post(target, 'baz', []);

Generates a POST request to: <https://example.com/myApp?q=baz&q=foo&q=bar>. In this case, the server-side code could either: treat only the first 'q' argument as the client specified query string; or reject the request. When designing the namespace of URLs issued by a web-application, ensure that the server can always either correctly disambiguate parameters in the query component of a Request-URI, or reject the request. Keep in mind that an attacker may send a user a specially crafted URLref that includes unexpected arguments in the query and fragment components.

Acknowledgments

Thanks to Alan Karp, Bill Frantz, Brian Warner, Charles Landau, Chip Morningstar, David-Sarah Hopwood, Douglas Crockford, Marc Stiegler, Mark Miller, Mike Samuel and Norm Hardy for comments and contributions.

Using the Firebug console

Try out the web_send library using the Bang Tutorial; or, if you're on Windows, try running the shell locally.