Workflows

Processes that run for minutes or days, replay from a journal after restarts, and wait on signals.

What a workflow is

A workflow is a function that runs like ordinary code but can pause for days and resume where it stopped, including across crashes and deploys.

local sales = database "sales" { migrations = "migrations/sales" }
local Account = objects "Account"

local settle = workflow "settle-sale" (function(wf, sale)
    wf:step("notify", { retries = 3 }, function()
        Account(sale.winner):notify("pay-up", { price = sale.price })
    end)

    local paid, which = wf:race({ "paid", "cancelled" }, { timeout = "72h" })

    if which ~= "paid" then
        wf:step("void", { retries = 3 }, function()
            sales:exec("UPDATE sales SET state = 'void' WHERE id = ?", { sale.id })
        end)
        return { outcome = "void" }
    end

    wf:step("settle", { retries = 3 }, function()
        sales:exec("UPDATE sales SET state = 'settled' WHERE id = ?", { sale.id })
    end)

    return { outcome = "settled" }
end)

Start a run with an id you choose. The id is the run's identity, so a retried start joins the run instead of making a second one:

settle:start({ id = "lot-42", winner = "ada", price = 25 },
    { id = "sale-lot-42" })

Replay

stateDiagram-v2
    [*] --> running: start
    running --> sleeping: wf:sleep
    running --> awaiting: wf:await, race, all
    sleeping --> running: timer due
    awaiting --> running: signal or timeout
    running --> failed: a step exhausts its attempts
    failed --> running: resume from the console
    running --> completed: the function returns
    running --> cancelled: run:cancel
    completed --> [*]
    cancelled --> [*]

Each step, signal and timer is journaled before it takes effect. A resuming run replays the function from the top, and steps that already ran return their recorded results instead of running again.

Run code must take the same path given the same journal. The platform makes the usual sources of drift deterministic and refuses the rest:

In run code	Behaviour
`os.time()`, `os.clock()`, `math.random()`, `uuid.v4()`	journaled on first run, replayed identically
`secret "name"`	pinned to the version the run first resolved
`os.date`, `math.randomseed`	refused
kv, database and object calls outside a step	refused: effects belong inside `wf:step`
`http`, `crypto`, `Jwt`	absent, inside step bodies too

Reach the network by calling an object from a step. The step records what the object returned, so replay does not call again:

local receipt = wf:step("sign", function()
    return Registrar("desk"):sign(payload)
end)

Steps

wf:step("name", function() ... end)
wf:step("name", { retries = 3, backoff = "5s", timeout = "30s" }, function() ... end)

Option	Default	Means
`retries`	1	total attempts, not attempts after the first; 0 is read as 1
`backoff`	2s	wait before attempt 2, doubling after that, with no cap
`timeout`	none	how long one attempt may run; past it the attempt fails with `step 'name' timed out after Nms`

flowchart LR
    intent[journal: intent] --> body[run the body]
    body -- returns --> result[journal: result]
    body -- throws, attempts left --> park[journal: failed<br/>park for backoff]
    park --> replay[replay reaches the step again] --> intent
    body -- throws, last attempt --> failed[journal: failed, final<br/>the run parks as failed]

Step names must be stable across revisions, because replay matches recorded results to them by name. When a step exhausts its attempts the run parks as failed with step 'name' failed after N attempts, keeps its journal, and the console can resume it.

Making an effect safe to repeat

A step records its result after the body returns, so a crash in between leaves the platform unable to know whether the effect happened. wf.step_id names the effect the body is performing, so the outside world can recognise a repeat:

wf:step("charge", { retries = 3 }, function()
    return Payments("desk"):charge(sale.price, wf.step_id)
end)

Across	`wf.step_id`
a crash between the effect and its recorded result	stays the same, so the repeat is recognised
a retry after a recorded failure	changes, because that attempt is a new effect
the same step name reached again, in a loop	changes, because each pass is its own effect
outside a step body	an error; there is no effect to name

The same string works anywhere a repeat needs recognising: a row id, an object key, a filename. The reference has its shape.

Waiting

wf:sleep("24h")
wf:await("approved", { timeout = "48h" })
wf:race({ "paid", "cancelled" }, { timeout = "72h" })
wf:all({ "shipped", "received" }, { timeout = "7d" })

Verb	Returns
`sleep`	nothing; resumes after the duration
`await`	the signal payload, or nil on timeout
`race`	the payload and the name that arrived first; the others keep their signals consumable
`all`	payloads in argument order; a timed-out entry is nil. The timeout is per entry, waited in order

A parked run holds no resources. It is journal rows and a timer.

Signals

Signals come from outside the run, addressed by the run id:

local settle = workflows "settle-sale"

on "fetch" (function(request)
    if request.method == "POST" and request.path == "/api/sale/pay" then
        local body = json.parse(request.body)
        settle:get("sale-" .. body.id):signal("paid", { by = body.user })
        return { status_code = 200, body = "" }
    end
    return { status_code = 404, body = "" }
end)

A run id is a non-empty string without /. Choose it so a route can address the run without a lookup table.

Duplicate signals are safe. An unconsumed extra is set aside rather than breaking replay, which matters because buttons get clicked twice.

Inspecting

The console lists runs with status and current step, shows the journal as a timeline, and offers resume for failed runs and cancel for the rest. Cancelling a run cancels the runs it spawned.

Workflow or queue

If anyone will ask how one specific job is doing, use a workflow: runs are named, inspectable and signalable, and each carries a journal.

Otherwise use a queue: anonymous, cheap, retried in bulk. A queue handler can start a workflow when a message turns out to need one.

Gotchas

retries = 3 is three attempts in total, not one plus three.
Steps do not nest. One effect per step; a wf:step inside a step body is an error.
Renaming a step breaks runs already in flight. Replay compares the journal against the code it re-executes, and a mismatch fails the run with journal divergence: expected step, code reached step 'other' rather than guessing.
One script per project declares a given workflow. A second script publishing the same name is refused; reach it with workflows.

Queues: the anonymous alternative.
workflow reference: every method.