Local deterministic sandbox

Workflows compiled to Wasm get determinism from the Wasm host, which runs anywhere. Workflows in a language that will not compile to Wasm (a JVM service, a Ruby worker, a threaded native binary) get it from the AD-VM tier: a forked Firecracker that controls the guest’s clock, entropy, and I/O. That fork needs KVM, so it does not run on a Mac.

hop sandbox is the same substrate for a machine that has a container runtime instead of a hypervisor. The determinism boundary moves from VM exits to syscalls: a supervisor process runs as PID 1 inside the container and answers the guest’s clock, entropy, and external reads from the same seam the hypervisor uses.

   On a KVM host                       On your machine
   ─────────────                       ───────────────
   forked Firecracker                  docker / podman / nerdctl
     └ patched device models             └ hopskip-advm-supervisor (ptrace)
         └ DeterminismController             └ DeterminismController

   same launch contract, same control frames, same recorded tapes

Those shared contracts are the point: a recording made locally replays on the production tier, and a workflow that is deterministic here is deterministic there for the same reasons.

Setting it up

The supervisor is a static Linux binary that gets bind-mounted into your image, so the image needs no changes at all: no base image to inherit from, nothing to install. Build it for the architecture your container runtime runs as (arm64 on Apple Silicon):

rustup target add aarch64-unknown-linux-musl
scripts/build-advm-supervisor.sh          # or: cargo build --release \
                                          #   -p hopskip-advm-supervisor \
                                          #   --target aarch64-unknown-linux-musl

Then check the machine:

hop sandbox doctor

Every way this can fail produces a run that looks fine and is not reproducible, so it is worth running before the first sandbox and after any change to Docker. It reports the runtime it found, whether the daemon is up, whether the supervisor matches the container’s architecture, whether your image is pinned, and, on Apple Silicon, the one source it cannot mediate.

Running something

hop sandbox run \
  --image ghcr.io/example/worker@sha256:… \
  --verify \
  -- /app/worker --once

--verify runs it twice and compares. That is the only way to observe determinism rather than assume it, and it is cheap: sleeps cost no wall-clock time in the sandbox.

Pin the image by digest. A tag can be repointed, and then two runs of “the same thing” are running different code, however well the clock is mediated.

--workflow-id seeds the entropy stream, exactly as the workflow instance id does in production: two runs with the same id see the same random bytes, two with different ids do not.

What the sandbox controls

The guest asks	It gets
`clock_gettime`, `gettimeofday`, `time`	logical time, see “The clock” below
`nanosleep`, `clock_nanosleep`	suspended, a durable timer, and the clock jumps to the deadline
`rdtsc` (x86)	the pinned guest counter
`getrandom`, `/dev/urandom`	a stream seeded from the workflow id
`/proc/sys/kernel/random/uuid`, `boot_id`	the same stream, formatted
a socket read	recorded on the first run, replayed afterwards
a `/proc` or `/sys` read	the same: recorded, then replayed
`uname`, `sysinfo`, `times`, `getrusage`, `getcpu`, `sched_getaffinity`	pinned values, not the host’s
a directory listing	sorted by name
file timestamps	logical time

Plus, at the container boundary: one CPU, no network, a fixed memory limit, a read-only root filesystem, a fixed hostname, address-space randomization and umask fixed, /etc/hosts and /etc/resolv.conf replaced with fixed copies (the runtime writes those itself, and they carry your machine’s DNS settings), and a replaced environment (TZ=UTC, LC_ALL=C.UTF-8, PYTHONHASHSEED=0, GODEBUG=randautoseed=0, MALLOC_ARENA_MAX=1, SOURCE_DATE_EPOCH).

The clock

Time does not pass in the sandbox unless something makes it pass.

Workflow-visible time is the timestamp of the event that resumed the workflow. It moves when your workflow calls an activity, starts a child workflow, or waits on a timer, and it stands still in between. Two consecutive clock_gettime calls return the same instant.

That is not a simplification of production; it is what production does. A clock that ticked a little on every read would make a workflow’s observed time depend on how often it looked at it, so adding a log line would change the replay.

Sleeping is one of the things that makes time pass, but not by the guest deciding so. A sleep is turned into the same thing a durable sleep() is in a Wasm workflow: the task is suspended, a timer is armed and reported to the engine, and the task wakes when the clock reaches the deadline. So it is durable (a crash mid-sleep resumes at the right instant, and a replay serves the wakeup from the log), and it is free: when everything in the guest is waiting on a timer, nothing can happen until the clock moves, so the clock simply moves.

$ hop sandbox run --image … -- sh -c 'date; sleep 2; date; sleep 60; date'
22:13:20
22:13:22
22:14:22
real  0m0.040s

Sixty-two seconds of guest time, forty milliseconds of yours. This is what makes a workflow with a one-hour retry backoff something you can actually test.

The practical consequence: a loop that waits for the clock to move waits forever. That is a real bug in a workflow, since the wait belongs in a durable timer, which is what makes it survive a crash. The sandbox tells you rather than hanging silently:

[hopskip-advm] determinism not enforced: the guest has read the clock 20000 times
in a row with nothing in between. Workflow time only advances when Core applies
an event…

If you are running something in the sandbox that is not a workflow and does poll the clock, hop sandbox run --clock-step-ns 1000 gives it a clock that ticks on reads. The run stays reproducible; it just stops matching what production would do, and the command says so.

What it does not control

Worth knowing before you trust a green run:

It is not an isolation boundary. The sandbox runs with SYS_PTRACE and an unconfined seccomp profile, because the supervisor needs them. Do not run untrusted code in it.
Thread interleaving is not deterministic. One CPU makes it far less varied, not reproducible.
On arm64, the raw cycle counter (cntvct_el0) cannot be trapped. A guest that reads it directly sees the host’s. This is rare outside profilers and some JIT warmup paths; hop sandbox doctor tells you about it up front.
A timerfd is a clock the sandbox does not close. It is detected and reported, not mediated.
Signal delivery and epoll/inotify timing are not mediated.
A guest that talks to itself over loopback has its own traffic taped, which is harmless while recording and can diverge on replay.
A guest whose every task blocks on something other than a timer deadlocks, exactly as it would outside the sandbox. Time can only jump forward to a deadline that something is actually waiting for.

When the supervisor cannot enforce something it says so and the run exits 121, even if the guest itself exited 0. A green run that was not actually deterministic is worse than a red one.

Reading a failure

[hopskip-advm] determinism not enforced: could not patch vDSO: …
[hopskip-advm] the guest exited 0 but this run is NOT reproducible; see the messages above.

The run happened, but its result is not evidence of anything. The usual causes are a supervisor built for the wrong architecture and a runtime whose seccomp profile blocks ptrace, both of which hop sandbox doctor catches.

[hopskip-advm] replay diverged from the recording: MissingRecord { ordinal: 3 }.

The guest read external input that this run’s tape does not contain: it took a different path than the recording did. This is the AD-VM tier’s form of the replay-divergence error, the same signal the Wasm tier raises on a nondeterministic replay.