Guarantees and limits
What each surface promises, and what your code has to do about it.
Delivery
| Surface | Promise | What your code must do |
|---|---|---|
| Object method call | a call the platform answered ran exactly once, serialized per object | a call that failed with an unknown outcome may have run; retry it the way any timed-out request is retried |
| Queue message | at-least-once | handler safe to run twice |
| Object stream edge | at-least-once, in order per publisher; a handler that committed never re-runs for the same event | effects outside the follower's own transaction safe to run twice |
| Connection edge | at-most-once | client re-reads state on reconnect |
| Workflow step | recorded once, replayed after | keep run code deterministic |
| Alarm | fires once per set, survives restarts; its handler's writes are not gated, so a crash mid-handler loses them without a re-fire | an alarm handler's work that must not be lost re-arms itself |
| Directory row | every object that matched is present; a row may briefly still match after the object changed | plan for over-inclusion, never for omission |
At-least-once means a repeat is possible after a machine failure at
the wrong moment. Make repeated work harmless: check an id, use
INSERT OR IGNORE, or update conditionally on current state.
Stream edges narrow the window: the follower records the event in the same transaction as its handler's writes, so a redelivered event is skipped, not re-run. Effects that leave that transaction are the part to keep idempotent.
At-most-once means loss is possible. Only connection edges promise this, because a tab can vanish mid-frame. Anything that must survive belongs on the object one hop back.
Ordering
Events from one publisher reach one follower in publish order. There is no ordering promise across different publishers. If order matters across sources, put a sequence number in the data.
Calls to one object are ordered. Calls to different objects are not.
Transactions
flowchart LR
caller([caller]) --> chain{call chain:<br/>a cycle or 32 deep?}
chain -- refused --> caller
chain --> mb[mailbox, 128 deep,<br/>one call at a time]
mb --> txn[begin, method, directory row, commit]
txn -- threw --> rollback[roll back; answer the error now]
txn -- wrote --> ticket[take a ticket;<br/>the next call already runs]
txn -- read only --> answer
ticket --> flight[ship flight: frames to the replicas,<br/>then to the store]
flight -- Q replica acks --> answer([answer released])
flight -- no replicas: manifest written --> answer
flight -. 10 s without either .-> unknown([outcome unknown])
A method's writes to state.sql commit together when the method
returns. A state:publish inside the method is part of the same
commit, so an event never describes a write that did not happen. The
directory row, if the class declares one, is derived inside the same
transaction.
The answer waits for one thing more than the commit: the writes leaving the machine. Calls that arrive while an answer is waiting still run, so a busy object commits at its own speed and a burst of writes leaves together. A request that makes several calls to objects on its own node gets each answer at commit and waits once, before its response and before any outbound request, for all of them.
Migrations apply inside the transaction of the call that triggered them: a failing migration applies nothing and surfaces as that call's error.
Workflow steps commit their journal entry with their result. A crash between two steps replays from the last recorded one.
Limits
Defaults, all operator-tunable on a self-hosted worker.
| Limit | Value | What happens at it |
|---|---|---|
| Guest work budget | 5,000,000 work units (GUEST_WORK_LIMIT), about a second of solid computation, per request, object call and connection item | the run is cut off |
| Wall backstop | 10 seconds (GUEST_WALL_SECS), for code stuck outside the vm | the run is cut off |
| Request deadline | 30 seconds, waiting included | the request fails |
| Request body | 10 MB | 413 |
| Object database | 1 GB per instance (OBJECT_DB_MAX_MB) | writes fail, reads keep working |
| Object mailbox | 128 waiting calls | callers wait; nothing is dropped |
| Object call chain | 32 deep; an object calling itself, directly or through others, is a cycle | refused, naming the cycle or the depth |
| Writes from one request | answered at commit; the response waits once for all of them; a later read of the same object in that request sees them | a write that cannot be confirmed makes the whole response fail with an unknown outcome |
Database read anywhere | sees every write answered before it began, whichever node serves it | nothing |
| Any reply from an object | waits while the object has writes still in flight, whoever wrote them; a settled object answers at once | the same unknown outcome as a write, after the durability gate's ten seconds |
| Durability gate | 10 seconds for a write to reach a quorum of its replicas, or the store | the call fails with an unknown outcome; the write keeps trying to ship |
| Object idle | 300 seconds without a call, delivery or alarm | the instance goes cold; the next call wakes it |
state.store value | 65,536 bytes as stored text | refused, naming the key |
| kv value | no fixed limit | |
| Replica read wait | 50 ms for a node's copy to reach what the owner answered | the read goes to the owner instead |
| Queue attempts | 5, first retry 2s, doubling to 1h | dead-lettered |
| Queue file | 64 MB | send raises |
| Stream edge attempts | 8, first retry 500ms, doubling to 32s | the edge is dropped |
| Stream delivery | 10 seconds per attempt | counts as a failed attempt |
| Connection inbox | 256 items | deliveries: the edges are severed; client frames: the wire closes |
conn.state | 64 KB serialized | the handler raises and the connection ends; a bigger session belongs in an object |
| Connection idle | 300 seconds | the vm hibernates; the socket stays open |
| Directory row | 4 KB encoded, 5 ms to derive | the derivation fails; the last good row stands |
| Requests per second | unbounded by default; a project policy sets a rate, burst of the same size | 429 with Retry-After; the refusal costs no vm |
| Work units per second | unbounded by default; a project policy sets a rate, charged after each call | the project is in debt and refused until it refills |
| A node's pools | requests, blocking work, connections, resident objects, directory queries, shipping lanes, each split fairly among the projects using the node with a floor of 5% | over its share a project is refused with 429; under contention the floor is what it keeps |
| Egress hosts | a project's deny list, then its allow list (empty admits everything not denied), then the node's own policy: private and local addresses are refused unless the operator allows them | the request or dial is refused, naming the rule |
| Cross-region call | one hop to the object's region | one region's round trip added; a project being moved answers "moving", retryably |
Guest code is metered in work rather than seconds, so waiting on io, a database or another object costs nothing and only computation counts. The clock stays as a backstop for what the meter cannot see.
The connection inbox is not a buffer to fill. A connection whose handlers cannot keep up has chosen to fall behind, and the platform severs its edges rather than growing memory on its behalf.
An idle connection hibernates: its vm drops after 300 seconds and
rebuilds for the next item its program declares a handler for, with
conn.state intact. A class that declared event = "forward" does
not rebuild it for deliveries at all. A connection with a timer stays
warm.
Failure behaviour
A hard crash cannot take back a call the platform answered. A method that wrote is not answered until its writes have reached durable storage off the machine: a quorum of the object's replica nodes, or the blob store when there are none. Nor is any other reply from that object while those writes are still in flight: a read, a method that wrote nothing, a database read served from the object's own file, all wait the same few milliseconds, so nothing outside the object ever acts on state a crash could take back. Anything you were told succeeded survives the loss of the node that ran it, and the next node to hold the object takes it over from a replica without touching the store.
| Event | What survives | What does not |
|---|---|---|
| a machine leaves gracefully (deploys, scale-downs) | everything: the drain ships every dirty object it can within 50 ms each, and the epoch fence plus the next holder's restore cover the rest | nothing |
| a machine crashes | every answered call; alarms re-arm from storage, runs resume from their journal, queue messages redeliver, subscriptions follow their follower's rehoming | calls still waiting for an answer (the caller sees an unknown outcome) and platform work with no caller, such as an alarm handler's writes |
| durable storage is unreachable | reads keep answering | writes stop being acknowledged: the call fails with an unknown outcome after ten seconds while the platform keeps trying to store it, so a retry needs the same idempotence any timed-out request needs |
| a node goes away | objects, runs, queues | connections: clients reconnect and re-follow, which happens often enough (deploys, scaling) to be part of the client from the start |
| a region becomes unreachable | everything the region's bucket had replicated elsewhere; the operator rehomes the project and the next touch restores it there | writes inside the replication lag, stated as such; a region that returns cannot write, because an object's home is a fence on every flight |