Setup

Workex is a solution to RPC in the JS world. You define interfaces in TypeScript, and Workex generates code so that calling a remote context uses the same interface you define.

Workex provides support for these bidirectional RPCs:

• Between Window and Worker
• Between 2 Windows, for both same- and cross-origin
  • For example, the main window and iframes, or popouts
• WebSocket support might be added in the future

The library works as follows:

• You define the messaging protocol as TypeScript interfaces
• Run the workex CLI to generate code in your project
• Use the generated code and the @pistonite/workex SDK library to have seamless RPC - Handshake, protocol agreement, error handling, same- and cross-origin… all handled internally

To get started, install the CLI tool with

cargo install --git https://github.com/Pistonite/workex

After installing, you can run workex --help to see the available options. See the tutorials for a full end-to-end walk-through of running some basic RPC stuff.

TypeScript SDK

The code generated by the workex CLI depends on the @pistonite/workex TypeScript SDK. You can install it with your favorite package manager. For example, for pnpm:

pnpm i @pistonite/workex

The tutorials shows basic examples for using the SDK. You can also refer to the SDK reference when needed.

Note that:

• The SDK version and CLI version should have the same minor version (the major version will always be 0)
  • i.e. 0.1.1 and 0.1.0 are compatible with each other, but not 0.1.1 and 0.2.0
• The SDK is TypeScript-only, so a bundler is needed to consume it.

Tutorial

This chapter is a basic tutorial for setting up a project with workex. Be sure to first install the workex tool before following the tutorial.

Project Setup

We will use a bare-bone node project as the starting point.

If you want to follow along from scratch, create an empty directory. Or, you can clone the repo and use packages/example-tutorial directory, which has all the code already there. (Code blocks on these tutorial pages actually pull code directly from those files)

Project Structure

The project should have package.json, tsconfig.json and a src directory that is currently empty:

- src/
- package.json
- tsconfig.json

Run the following to make sure you have typescript and @pistonite/workex installed.

You can use any package manager - I am using pnpm as an example

pnpm i -D typescript 
pnpm i @pistonite/workex

If you want to build and serve the example after the walk-through, also install serve, and make sure you have bun callable from the command line. The easiest way to install both is:

pnpm i -D serve
pnpm i -g bun

Your package.json should be similar to the following after installing those dependencies:

{
    "dependencies": {
        "@pistonite/workex": "workspace:*"
    },
    "devDependencies": {
        "serve": "^14.2.4",
        "typescript": "^5.7.2"
    }
}

Now, create tsconfig.json:

{
    "compilerOptions": {
        "noEmit": true,
        "lib": ["esnext", "dom"],
        "target": "esnext",
        "useDefineForClassFields": true,
        "moduleDetection": "force",
        "module": "esnext",
        "moduleResolution": "bundler",
        "allowImportingTsExtensions": true,
        "isolatedModules": true,
        "strict": true,
        "skipLibCheck": true
    },
    "include": [ "src" ]
}

Define Interfaces

The interfaces used for communication can be defined in one or more TypeScript files. Here, we will create one src/Interfaces.ts file that contains 2 interfaces AppSide and WorkerSide. These interfaces can have any names that doesn’t start with _ (to avoid name conflicts in generated code)

Create src/Interfaces.ts with the following content

// src/Interfaces.ts
import type { WxPromise } from "@pistonite/workex";

/**
 * Functions implemented by the main thread callable from the worker
 */
export interface AppSide {
    /**
     * Get some data from the main thread for the worker to process
     */
    getData(id: string): WxPromise<string>;
}

/**
 * Functions implemented by the worker callable from the main thread
 */
export interface WorkerSide {
    /**
     * Initialize the worker
     */
    initialize(): WxPromise<void>;

    /**
     * Process some data in the worker, and return the result
     */
    process(data: string): WxPromise<string>;
}

Generate Code

Once you have the interfaces defined in the input file(s), it’s time to run the CLI to generate some code!

There are 2 flags that will be most commonly used:

• -p/--protocol: A string that identifies your package/library. This is used to distinguish between multiple libraries generated with workex operating on the same messaging channel. It can also contain some versioning scheme to identify version mismatch, if your protocol is meant to be implemented by others. (For example, your webapp support custom UI widgets through iframes)
• -l/--link: This links 2 interfaces so that if one side of the connection implements one, the other side is assumed to implement the other. An interface can only be linked to one other interfaces. Unlinked interfaces are linked to a “stub” interface, i.e. the communication becomes one-direction.

In the example directory, run the following command, which generates code that refers to the testapp protocol and links our AppSide and WorkerSide

workex src/Interfaces.ts -p testapp -l AppSide,WorkerSide

workex -p myproto-1.0.0 --prefix myproto

This should generate the src/interfaces/ directory. Note:

• You can use --dir to change the name interfaces to something else, but you can’t change the output location otherwise.
• If there are multiple input files, they must be in the same directory

The directory structure should now look something like:

- src/
  - interfaces/
    - AppSide.ts
    - AppSide.bus.ts
    - WorkerSide.ts
    - WorkerSide.bus.ts
    - .gitignore
  - Interfaces.ts
- package.json
- tsconfig.json

The .gitignore file is automatically generated to ignore everything in the interfaces directory. You can turn it off with --no-gitignore.

Using Generated Code

Finally, we will write the code for the main thread and the worker that talks to each other using the generated code.

For simplicity, we will put the code for the main thread and the worker both in src directory, as src/App.ts and src/Worker.ts. In real projects, these can be put anywhere as long as they are able to import the generated code.

App Code

Create src/App.ts, and paste in the code below. See the comments in the code and the summary after for explanation.

// src/App.ts
import { wxWrapHandler, wxWorker } from "@pistonite/workex";

// this imports the generated code
import { testappWorkerSide } from "./interfaces/WorkerSide.bus.ts";
// this imports the original input interface
import type { AppSide } from "./Interfaces.ts";

// we will wrap the code in an async main function
// to avoid issues with top-level await in older browsers
// and be able to use early returns
const main = async () => {
    console.log("App: start");
    // define the handler on app side that responds
    // to calls from the worker
    const handler: AppSide = {
        getData: wxWrapHandler((id: string) => {
            if (id === "foo") {
                return "bar";
            }
            return "";
        })
    }

    console.log("App: creating worker");
    // create the worker
    // here we assume the bundled worker JS file will be served
    // as /worker.js
    const worker = new Worker("/worker.js");

    console.log("App: connectiong to worker");
    // because WorkerSide and AppSide are linked,
    // testappWorkerSide takes in an AppSide handler
    // and returns a WorkerSide interface
    const result = await wxWorker(worker)({
        workerApi: testappWorkerSide(handler)
    });

    // handle error in initialization of the connection
    if (result.err) {
        console.error("App got error:", result.err);
        // note if the connection fails, the worker will be terminated
        // automatically
        return;
    }
    console.log("App: worker connected!");
    // initialize the worker side
    // the `workerApi` name is derived from the wxWorker call
    // above, and is type-checked
    const { connection, protocols: { workerApi }} = result.val;

    // the returned connection handle can be used to control the connection.
    // for example, register a callback to be called when the connection is closed
    void connection.onClose(() => {
        console.log("App: connection closed");
    });
    // the onClose() function returns a function that can be called
    // to unregister the listener.

    console.log("App: calling worker.initialize()");
    // call the initialize() function defined on the WorkerSide
    // interface
    const ready = await workerApi.initialize();
    // Any RPC call would return WxResult<T>, and you must
    // handle the potential errors that happen during
    // communication
    if (ready.err) {
        console.error("App got error:", ready.err);
        // after connection is established, errors don't close
        // the connection. you can manually close the connection
        // with connection.close() or worker.terminate()
        connection.close();
        return;
    }

    console.log("App: calling worker.process()");

    // do some work!
    const output = await workerApi.process("hello foo");
    console.log("App: got response from worker:", output);
}

// call our main function
void main();

const handler: AppSide = {
    getData: async (id: string) => {
        if (id === "foo") {
            return { val: "bar" };
        }
        return { val: "" };
    }
}

Worker Code

Create src/Worker.ts, and paste in the code below. See the comments in the code and the summary after for explanation.

// src/Worker.ts

import { wxMakePromise, wxWorkerGlobal, type WxPromise } from "@pistonite/workex";

// this imports the generated code
import { testappAppSide } from "./interfaces/AppSide.bus.ts";
// this imports the original input interface
import type { WorkerSide, AppSide } from "./Interfaces.ts";


// we will wrap the code in an async main function
// to avoid issues with top-level await in older browsers
// and be able to use early returns
const main = async () => {
    console.log("Worker: start");
    // create the binding up here since the handler
    // needs to reference the app side, which may or may not be ready
    // when the handler is called
    const {
        promise: appApiPromise,
        resolve: resolveAppApi,
    } = wxMakePromise<AppSide>();

    // define the handler on worker side that responds
    // to calls from the app
    const handler: WorkerSide = {
        initialize: async () => {
            console.log("Worker: (fakely) initialized!");
            return {};
        },
        process: async (input: string): WxPromise<string> => {
            console.log("Worker: processing input:", input);
            // wait for the binding to be set
            const app = await appApiPromise;
            // some example logic
            const data = await app.getData("foo");
            if (data.err) {
                return data;
            }
            return { val: `${input} ${data.val}` };
        }
    };

    // connect to the app side that created this worker
    const result = await wxWorkerGlobal()({
        // this function can take a second parameter,
        // for a callback to be invoked when the binding is ready.
        // If the app side calls us after the connection is established,
        // but before the binding is set on our side, the handler
        // will wait until the binding is set before continuing.
        // thanks to the promise we created at the beginning
        appApi: testappAppSide(handler, resolveAppApi)
    });

    // handle error in initialization of the connection
    if (result.err) {
        console.error(result.err);
        return;
    }
    console.log("Worker: ready to be initialized!");
}

void main();

// casting is ok since we don't call this until it's ready
let appApi: AppSide = undefined as AppSide;
const { promise, resolve } = wxMakePromise();
const handler = {
    initialize: async () => {
        await promise;
        return {}
    }
    /* ... other functions ... */
}

/* ... setup the connection ... */

// assign the binding
({ appApi } = result.val);

// resolve the promise
resolve();

Running the Example

The example we walked through isn’t some sample code to show you the syntax. It’s fully runnable!

First, we will run tsc to do a sanity type-checking to make sure the example conforms to the API contracts:

pnpm exec tsc

To bundle the TS code we wrote and the SDK library, we will use bun, which offers zero-config bundling for TypeScript:

bun build src/App.ts > dist/index.js
bun build src/Worker.ts > dist/worker.js

We will also create a HTML file that does nothing but loads index.js:

<!-- dist/index.html -->
<DOCTYPE! html>
<html>
    <head>
    </head>
    <body>
        <script src="index.js"></script>
            You need to open the console to see the example output
    </body>
</html>

Now run pnpm exec serve to serve the example locally, and open it in your browser. You should see something like this in the console:

App: start
App: creating worker
App: connectiong to worker
Worker: start
App: worker connected!
App: calling worker.initialize()
Worker: ready to be initialized!
Worker: (fakely) initialized!
App: calling worker.process()
Worker: processing input: hello foo
App: got response from worker: {val: 'hello foo bar'}

SDK Reference

This section is the main documentation of the SDK library. It details the behavior of the API sets, as well as some designs and underlying concepts used internally (and externally).

Some of these concepts are:

• JS Messaging Channel
• An End of a Channel
• The BUS

Also, the generated API reference is available here. This can be used as quick reference and supplement to this documentation.

The SDK reference is organized bottom-up, meaning it starts with the lowest level of concepts and browser primitives, and then build up to the high level APIs that you saw in the tutorials.

Error Handling

The SDK uses the result pattern, powered by another of my in-house TypeScript library - pure. This is a purely type-based implementation, so it does not have any runtime overhead (other than having to check error).

In the SDK, the 2 types used for error handling is WxError and WxEc which is a string type-union enum that stands for “Error Code”.

Each WxError has a code: WxEc that will tell you what the error is, and optionally a message that might have more details about the error

const result = wxDoSomething();
if (result.err) {
    if (result.err.code === "Timeout") {
        // failed because of time out
    }

    // make this path diverge
    return;
}
// TypeScript can now infer the type of result.val
console.log(result.val);

JS Messaging Channel

Messaging in JS refers to passing JS objects between different contexts. Different contexts do not have access to each other directly, such as the main thread and worker threads, or windows with different origins. Messaging in JS is done by calling postMessage on an object, typically a Worker, WorkerGlobalScope, or a Window in the Web. The message will be delivered as a MessageEvent to event listeners listening for the "message" event.

Therefore, messaging is only relevant when there are multiple JS contexts. These are the usual ways how multiple JS contexts are usually involved or created:

• A Window object has a parent if it’s embedded in another document, for example as an iframe.
• A Window object has a opener if it’s opened by calling window.open.
• A Window object can open other Windows by calling window.open, or embed other documents (which can be same- or cross-origin).
• A Window object can spawn Workers.
• A Worker can spawn other Workers, but cannot open Windows.

This can be illustrated as a tree

Active and Passive Sides

This brings the lowest-level concept in Workex - Active and Passive sides. We call the context created by another context the active side, and the context creating the other the passive side.

This distinction is important in 2 places:

• When initiating the communication
• When terminating the communication

The terminology is derived from the relationship of the 2 sides when initiating the communication. Although major browsers have basically the same behavior, the web standard actually does not specify when a new context created by an existing context should start running. For example, when you call new Worker(), the worker JS code can start executing immediately in a different thread (since it’s a different context, this is multithreaded JS), or, like how most browser implemented it, does not start until the current JS task is done executing.

As a library, we do not want to rely on each browser implementing a specific behavior. Therefore, we will assume that the existing context does not know when the new context will start executing. This means to initiate the communication, the best way is to let the new context send the first message (referred to as the handshake message), and then the existing context will respond with a handshake message as well. The side that sends the first handshake message is the active side, and the other the passive side.

It’s also important when terminating the communication. Based on the diagram above, we can see the direction of the arrow as a “ownership” relation. For example, if a website creates a popup window, and the user closes the website, the popup window should probably close as well (note this is not the default behavior for popular browsers). We enforces this: the passive side is the owning side, and the active side is the owned side, and the differences are:

• When communication is to be terminated, the owner (passive) side will definitely be notified in order for any necessary clean-up to run. The owned (active) side may not be notified, as the context may simply get destroyed
• The owned (active) side will not close itself even though it’s technically possible. It will also “request” the owning side to close it, allowing the owner side to always be notified when the connection terminates, regardless of the implementation of the JS runtime.

An End of a Channel

The WxEnd is the first layer of abstraction over the JS messaging channels. It presents an established channel, meaning when an WxEnd object exists, the underlying channel it encapsulates must already exchanged the handshake and ready to send messages to each other.

Every channel has 2 ends, shown by the illustration below

At this level, there is no distinction between the active and passive side from the API perspective: calling close() on either end of the channel will close the connection. The underlying implementation may depend on if the side is active or passive, but this is encapsulated.

Finally, the end also owns managing the underlying resource, meaning if the messaging is done to a Worker or a popup Window, the will also be disposed of (terminated/closed). The only exception is embedded frames (iframes). Closing the connection for an embbeded frame with its embedder does not automatically remove the iframe element from the DOM.

Bidirectional Unicall System (BUS)

With the WxEnd abstraction, we have a uniform way of communicating between different contexts, regardless of if it’s between Windows, Workers, or any combination or nested combination of them. The Bidirectional Unicall System is the layer on top of WxEnd that implements the Remote Procedure Call, or RPC.

This layer turns messaging into async function call - the implementation is basically

// this is pseudo code and greatly simplified, 
// not how the BUS is actually implemented
function callRPC() {
    return new Promise(resolve => {
        end.postMessage({
            id: 1,
            call: "foo",
            args: ["bar"]
        });
        end.onmessage = ({data}) => {
            if (data.id === 1) {
                resolve(data.returnvalue);
            }
        };
    });
}

In reality, the implementation is slightly more complicated to handle potential errors, timeouts, catching exceptions on the other end and send it back, and multiple inflight messages, and perhaps mostly importantly, muxing different protocols, which will be explained in the next chapter.

Protocols

Protocols exist to solve 2 problems:

• The workex CLI tool ensures every function call can be uniquely identified by assigning a numeric ID to each function. However, it’s possible that an application uses multiple sets of code generated by workex. We need an external method to tell these messages apart.
• One great use case for this library is custom plugins loaded as frames, popups, or workers. We need a way to version the generated code, so an update doesn’t break the application because the 2 sides don’t have the same definition on what function 42 is, for example.

The following illustration shows 2 protocols operating on the same bus, using the same underlying connection.

For ergnomics and to simplify the implementation, the BUS enforces that each protocol can only register one connection. The protocols are registered at BUS-creation time, by passing in an implementation of one of the ends and calling a “binding function” generated by the CLI tool. The BUS will then automatically return an implementation of the linked interface. Calling functions on that interface will invoke the implementation on the other side through RPC, but it all feels like regular async function call.

// `wxCreateBus` is an internal function, so this is just
// pseudo code to demonstrate the concept - error handling is also omitted
const barAImpl: BarA = {
    /* ... implements the interface */
};
// wxCreateBus will handle all the handshake, protocol agreement, etc
const { fooA } = await wxCreateBus({
    // the bind config functions are generated by CLI
    fooA: bindBarA(barAImpl)
});
// use fooA, which implements FooA

To solve the second problem on the top, the BUS will send protocol query messages and agree on the protocols and interface types before accepting any other message. For example, if side A receives a bindBarA config, then it knows the other side must provide a FooA implementation in the same protocol. If the other side agrees, then the protocols are agreed, and RPCs can start.

When breaking changes to a library is made, the protocol can be changed (e.g. my-lib-v1 to my-lib-v2). This will cause outdated plugins to have a protocol disagreement, and prevent further breakage.

// the error is returned by the creator function in real code
const result = await wxCreateBus({...});
if (result.err) {
    if (result.err.code === "ProtocolDisagree") {
        // tell author of the plugin
        console.error("Hey! Please update the version of my-lib in your package.json!");
        return;
    }
}

Creator Functions

Finally, the creator functions are the entry points of this library. They encapsulate everything explained in the chapters before. They take in the config object to setup the BUS, establish the connection, agree on protocols, and return the linked interface implementation to call the other side.

There are five creator functions available, which can be divided into 2 groups:

• For workers:
  • wxWorker
  • wxWorkerGlobal
• For windows:
  • wxPopup
  • wxFrame
  • wxWindowOwner

Workers

wxWorker and wxWorkerGlobal are used for connecting to Workers. The passive side calls wxWorker(worker), and the active side calls wxWorkerGlobal to establish the connection using globalThis.

Windows

Windows are slightly more complicated. The global Window object can both receive messages from its owner and the windows it owns. The source of the MessageEvent tells where the message is sent. This is encapsulated in the WxWindow object, which is a handle/wrapper around the global Window. You don’t need to use this object directly.

For popups, the wxPopup creator function takes an URL and options, and will open the popup window for you. For frames, the wxFrame takes in an HTMLIFrameElement and only establish the connection (i.e. it doesn’t put the element in the DOM for you).

From the other side, the wxWindowOwner is used to connect to the owner side. Since a window can’t both be an embedded document and a popup at the same time, it will automatically detect what scenario it is.

What it does need to know is the origin of the owner. This is because of 2 reasons:

• same-origin Windows run in the same context, and is considered trusted. In this case, postMessage is quite expensive, and the SDK uses a simple implementation to directly call the handler on the other end.
• For cross-origin postMessage, the sender must indicate the origin of the recipient. This is to prevent sending message to an untrusted context. For example, if the other window navigates to another website, you don’t want to send your information to scripts on unknown websites. The browser will typically block these messages.

An easy way to do this, is to use a URL param when opening the popup or frame.

// === on the main window (passive) side 
const origin = window.location.origin;

const result = await wxPopup(`https://mywebsite.com/popup.html?ownerOrigin=${origin}`)({
    /* your config object */
});

// === on the popup (active) side 
const ownerOrigin = new URLSearchParams(window.location.search).get("ownerOrigin");

const result = await wxWindowOwner(ownerOrigin)({
    /* your config object */
});