Dispatch & resilience

Older orchestrators dispatch by polling: workers repeatedly ask “is there work?” That has a latency floor (your poll interval) and a scaling ceiling (the poll traffic itself).

Hopskip is push-based. Workers establish persistent bidirectional gRPC streams to Core. Core pushes tasks; workers signal readiness and flow control over the same stream.

Sub-5ms dispatch, native backpressure

Target dispatch latency is under 5ms, versus polling-interval latencies. Because readiness is explicit:

• A worker that stops signaling readiness stops receiving tasks. Tasks queue in Core (spilling to warm storage) at zero worker cost, so backpressure is built in rather than tuned.
• When Core’s own buffers for a workflow exceed a bound, it rejects external producers with RESOURCE_EXHAUSTED. The pushback lands at the edge of the system instead of a failure in the middle.

Topology-aware routing

Workers broadcast topology metadata (region, availability zone, hardware, attached volumes) on the stream. Dispatch supports:

• Affinity constraints, e.g. affinity: { region: "eu-*" }.
• Consistent-hash sticky routing for partitioned workloads.

This replaces string-matched task-queue names with real topology the scheduler can reason about.

Resilience is server-side policy

The control plane is heavier than a polling matching service, and this is what it buys. Core is on the path of every task dispatch and every yielded value, so it is the natural enforcement point for resilience patterns that older systems push onto application code:

Pattern	Older model	Hopskip
Bulkheads	App-level thread pools	Declarative Core policy
Rate limits	App-level limiters	Declarative Core policy
Circuit breakers	Library per service	Declarative Core policy
FIFO ordering	Manual sequencing	Server-side FIFO groups
Retry / backoff / jitter	Hand-written	Policy Core evaluates

These are declarative configuration evaluated by Core, so they apply uniformly and can be changed without redeploying workflows:

# A policy attached to a workflow type, evaluated by Core.
resilience:
  bulkhead:
    charge_card: { max_concurrent: 50 }
  rate_limit:
    ship_order: { per_second: 200 }
  circuit_breaker:
    reserve_inventory: { error_threshold: 0.5, cooldown: 30s }
  fifo:
    group_by: customer_id       # order preserved per customer

Change a limit and every in-flight and future workflow picks it up: no build, no deploy of workflow code.

Where scheduler state lives

Partition-scoped scheduler state (FIFO sequencers, bulkhead counters, channel buffers) lives on the owning shard’s leader, so it scales with shard count. Only truly global state (cluster-wide rate limits, the deployment registry) lives in the placement/metadata group.

Fencing tokens and leader leases

A small placement Raft group owns the shard map and issues epoch-numbered leadership leases that compose with worker fencing tokens. A deposed leader’s dispatches can be rejected end to end: a stale worker holding an old token cannot commit work against a shard that has moved on. This is what makes failover safe under live traffic.