Concepts
Memory snapshots
Because the runtime owns workflow memory, it can serialize the complete
execution state at any await point. In the Wasm tier a snapshot is
exactly linear memory plus exported mutable globals, with no code
transformation, because the
quiescence invariant
guarantees the native stack is empty at every await.
This one capability replaces several separate constructs from older orchestrators.
No more Continue-as-new
Older systems force you to periodically end a long-running workflow and
start a fresh one (ContinueAsNew), manually passing state across the
boundary. The abstraction leaks, and it breaks signal delivery.
Hopskip has no such construct. Core monitors each workflow’s log size. At an event horizon threshold it:
- Waits for the next
await. - Snapshots memory (a pause of under 5ms).
- Writes the snapshot as the first event of a fresh log partition.
- Seals the old partition to cold storage.
A while (true) workflow runs for years on a bounded hot log. Signals and
queries cross the horizon transparently because Core knows which partition
holds live state. Perpetual entity workflows are exactly this
mechanism plus background compaction.
Lazy replay: constant-cost resume
Core additionally snapshots every N events. A worker resuming a workflow at event 50,050 receives the snapshot from event 50,000 plus 50 events, not the full history.
hop replay wf_7b3a91 --from-snapshot 50000
# ▸ restored snapshot @ evt 50000 (0.8ms)
# ▸ replayed 50 events bit-identical ✓
Replay cost is bounded and constant regardless of workflow age. That is what makes millisecond resume possible for a workflow that has processed millions of events.
Time-travel debugging
History retains snapshots at await points. An engineer can scrub to any step, restore that snapshot locally, attach a native debugger (or a REPL for interpreted runtimes), and inspect variables as they were in memory, then fork execution from that point to test a fix.
hop debug wf_7b3a91 --at evt 50044
# ▸ restored snapshot, paused at charge_card
# ▸ attach: lldb / node --inspect / py-spy
Production forking
Reproducing a production failure requires no mocks. hop fork restores a
retained snapshot of a recorded run, optionally injects a synthetic result
for an activity the workflow was suspended on, and lands the fork under a
target-prefixed namespace. The fork and the injected value are both
durably recorded as principal-attributed override events on the fork’s own
history partition:
hop fork prod order-7b3a91 <run-hex> \
--boundary-ordinal 50031 \
--override 'charge_card=fail:CardDeclined/insufficient funds' \
--target staging \
--principal ian@example.com
# ▸ fork: staging/prod:order-7b3a91-fork-... run 9c1d02...
# ▸ recorded: fork provenance override event
# ▸ recorded: injected override (pending=3 (charge_card), 47 bytes)
Naming what you override
An override names what the system already prints for a suspension: an
activity name, or a reserved $hopskip. label for a timer, stream yield,
or cross-workflow call. That way you override exactly what you just read
in hop debug attach.
The name also says how far the override reaches:
- bare (
charge_card): the single call the fork is currently parked on #*: every matching call for the whole run#1..3or#0,2: anything in between
Values are literal bytes unless tagged (i64:, json:, hex:,
@file), and ok:/fail: wrap the envelope a fallible guest decodes,
which is how you make a workflow take its error path on demand. Repeat
--override as many times as you need, or keep a scenario in a file and
pass --override-file.
The forked instance observes the substituted value through the ordinary ABI path, indistinguishable from a real activity result, with its provenance on the permanent record.
Dispatching the fork forward
Adding --dispatch re-executes the fork forward to completion right
there. Each activity it suspends on is dispatched through the live server
(--addr) and executed by real workers, the fork’s partition accumulates
steps, snapshots, and timeline points as it goes, and the final result
lands on stdout:
hop fork prod order-7b3a91 <run-hex> \
--override-i64 "charge_card=0" \
--principal ian@example.com \
--dispatch --addr http://core.internal:7233
# ▸ re-executed forward to completion via http://core.internal:7233:
# 4 activities dispatched to live workers, 4 boundaries recorded
# ▸ result: 0
Without --dispatch, the fork is recorded and left for hop debug attach to drive or inspect. The drive is the same
fan-out/resolve-whichever-completes-first loop the production worker
runs, so concurrency is the baseline: a fork suspended mid-join2 with
several activities outstanding dispatches them all concurrently, exactly
as a live run would. When a module exports both run and
run_concurrent (two workflows in one module), --workflow-export says
which one the snapshot recorded.
Live state patching
For emergencies, the planned hop patch (planned interface: not in
the current CLI; the audited OverrideEvent machinery it builds on ships
today) pauses an instance, appends an audited ManualStateOverride event
(with the operator’s OIDC identity) to the Merkle log, mutates memory, and
resumes:
hop patch wf_7b3a91 --set "state.retryCount=0"
# ▸ appended ManualStateOverride (operator: ian@example.com)
# ▸ resumed
The intervention itself is on the permanent record: emergency action without a deploy, fully audited.
Zero-downtime failover assists
Paused-and-restored instances can be handed refreshed resource handles or migrated across hosts. Because a snapshot is self-contained, moving a running workflow between workers is a restore, not a re-run.
Six features, one mechanism. The runtime owns memory and upholds the quiescence invariant, so snapshotting falls out of the execution model instead of being bolted on.