Visibility & search

Visibility in Hopskip is a projection of the event log, not a separate database you keep in sync. Core continuously projects the log into an embedded store (SQLite) on Core nodes, and you read live workflow state back through a small set of typed queries.

There is no Elasticsearch, no separate visibility cluster, and no eventual-consistency lag beyond the projection pipeline itself.

Marking state searchable

Tag a workflow variable @searchable (or @indexed for deep state) and it is extracted at each checkpoint into structured rows. The value tracks the live workflow; mutate it and the next checkpoint re-extracts it.

from hopskip import durable, activity, Searchable, mark_searchable

@durable
async def order_processing(order):
    status = Searchable("status", "pending")       # re-extracted each checkpoint
    mark_searchable("customer_tier", order.tier)   # one-shot tag

    await activity("reserve_inventory", order.sku)
    status.set("reserved")

    await activity("charge_card", order.total)
    status.set("charged")

The same annotation exists in every SDK (@searchable in TypeScript, the Searchable type in Rust), reaching the same emit_searchable host import from idiomatic code in each language.

Declaring attribute types

Open Search attributes in the console and choose a namespace. Register an attribute key with one of these types:

Type	Accepted scalar value	Preview predicates
`keyword`	UTF-8 string	Equality, exists
`text`	UTF-8 string	Equality, exists, substring
`int`	Signed 64-bit decimal integer	Equality, exists, ordered comparisons
`double`	Finite floating-point number	Equality, exists, ordered comparisons
`bool`	Exact lowercase `true` or `false`	Equality, exists
`datetime`	Unsigned 64-bit decimal HLC `physical_ms`	Equality, exists, ordered comparisons

Declarations are namespace-scoped and journaled. Changing a type commits a new version with the caller’s identity, then reinterprets existing current values. It does not change workflow history or raw attribute bytes. Registering the same type again keeps the existing version.

The registry table shows the committed version, actor, and this node’s unparseable and stale counts. A value that cannot be parsed remains stored but is excluded from typed predicates, including exists. A type change marks old interpretations stale until reprojection succeeds. Reproject retries that work at the current declaration version.

The predicate preview queries one key on the connected node. It is not a cluster-wide workflow search. Its limit is 1 to 1,000 matches; truncated results say that more matches exist. An unregistered key supports raw byte equality and exists only. The browser accepts UTF-8 operands; the RPC accepts raw bytes. exists has no operand.

Built-in fields

Structured queries and the predicate preview accept these names without a declaration:

Field	Type	Value
`namespace`	`keyword`	Workflow namespace
`workflow`	`keyword`	Workflow instance name, not its workflow type
`event_type`	`keyword`	Kind of the latest projected event in this partition
`physical_ms`	`datetime`	HLC physical time of that event, when recorded

These names are reserved and cannot be redeclared as custom attributes. workflow_type and updated_at are not aliases. A missing event timestamp does not match timestamp predicates, including exists; it is not replaced with projection wall-clock time. Built-in query results report registry version zero because they have no registry declaration.

Current-state projection

The store materializes one row per workflow instance and history partition, with a separate typed column for each declared attribute. A multi-field predicate can read those columns from one row rather than joining the attribute table once per field. Structured queries combine fields; the console’s registry preview accepts one predicate at a time.

Each attribute keeps its highest-sequence value independently. An event that updates one attribute does not clear the others; an explicit unset does. Built-in event fields describe the latest projected event, while a custom attribute match retains the sequence of the event that last wrote that key. Raw values remain available for rebuilding the typed projection. Older databases without event timestamps report them as missing.

Ingestion boundary. Declarations operate on values already present in Core’s visibility database. Wasm workers currently persist emitted attributes in their own execution histories, which Core’s continuous projector does not consume. Registering a type does not import that history or establish that ingestion is caught up.

Registry API

The native gRPC service is hopskip.search_attributes.v1.SearchAttributeAdmin. The console exposes the same operations through its gRPC-Web ConsoleService:

RPC	Operation
`ListSearchAttributeTypes`	Read declarations and local degradation counts
`RegisterSearchAttributeType`	Commit a declaration and reproject the key
`ReprojectSearchAttribute`	Reparse current values at the existing version
`QuerySearchAttribute`	Run a bounded, node-local predicate

Reads require membership of the requested namespace. Registration and reprojection require hopskip-acl-admin; that capability alone does not grant access to another namespace’s values.

Reading it back

Every read is a typed query: the server, not the caller, decides which namespace it runs against. There is deliberately no endpoint that accepts a SQL string.

Structured queries

Send a query object to POST /api/workflows/query on the console API listener. The request names one namespace. Your token must permit that namespace; an OR filter cannot widen the request to another namespace.

Declare status as a keyword before using these examples. This request finds the workflow instance named orderProcessing whose current status is charged, before an HLC event-time boundary:

{
  "namespace": "orders",
  "query": {
    "filters": [
      {"column": "workflow", "op": "=", "value": "orderProcessing"},
      {"column": "status", "op": "=", "value": "charged"},
      {"column": "physical_ms", "op": "<", "value": "1735689600000"}
    ],
    "orders": [{"column": "physical_ms", "order": "ascending"}],
    "limit": 50
  }
}

Without calculations, the result contains workflow-instance rows. Add calculations and breakdowns to group the matching current-state rows:

{
  "namespace": "orders",
  "query": {
    "filters": [
      {"column": "workflow", "op": "=", "value": "orderProcessing"}
    ],
    "breakdowns": ["status"],
    "calculations": [{"op": "COUNT"}],
    "orders": [{"calculation": 0, "order": "descending"}],
    "havings": [{"calculation": 0, "op": ">=", "value": "1"}]
  }
}

Save either request as query.json. Set CONSOLE_API_ORIGIN to the server’s console API listener, not the frontend development server:

curl --fail-with-body "$CONSOLE_API_ORIGIN/api/workflows/query" \
  --header "Authorization: Bearer $HOPSKIP_TOKEN" \
  --header 'Content-Type: application/json' \
  --data-binary @query.json

Native gRPC and gRPC-Web clients can call hopskip.console.v1.ConsoleService.QueryWorkflows. Set namespace and encode the query object as UTF-8 JSON in query_json. The response’s result_json contains the same result as the HTTP response’s result field. Cluster responses also carry freshness.

Filters and calculations

filter_combination is "AND" by default; "OR" combines filters without changing namespace scope. Comparisons are =, !=, <, <=, >, and >=. String predicates are contains, starts-with, and ends-with, with does-not-contain, does-not-start-with, and does-not-end-with as their negations. Membership uses in or not-in with a nonempty array of scalar values. Operator/type combinations are validated against the registry.

Use exists or does-not-exist without a value to test presence. Other predicates, including negations, do not match missing values. Stale or unparseable values do not count as missing: they are excluded from those predicates and reported in the result’s per-column degradation metadata. Those counts cover the referenced columns before query predicates and time bounds. Unregistered keys support only raw equality and exists, not grouping, sorting, or calculations.

COUNT takes no column. COUNT_DISTINCT, MIN, MAX, SUM, and AVG require a column. SUM and AVG require a numeric declaration. Missing or invalid values do not contribute to column calculations. A missing breakdown value forms a null group; a stale or unparseable breakdown value is excluded instead.

keyword columns can contain any number of distinct values. Query budgets bound the work of grouping them; there is no per-attribute cardinality ceiling. text columns and unregistered keys cannot be breakdowns.

COUNT_DISTINCT is exact through 128 distinct values in a group. Larger sets use a fixed-size HyperLogLog sketch. An approximate calculation is identified by its zero-based index in the group’s estimates object: estimator is hyperloglog_p14_sha256_v1 and relative_standard_error is 0.008125 (0.8125%). This is expected standard error, not a guaranteed accuracy bound. An absent entry means the calculation is exact. Ordering and HAVING use the reported estimate.

Each orders entry selects a column or a zero-based calculation index and an order of "ascending" or "descending". A grouped query can order by a breakdown or calculation. havings selects calculation indexes, uses comparison operators, and combines its entries with AND. Grouping, having filters, ordering, and pagination are resolved across the answering shards, not independently truncated on each node.

Results use kind: "rows" or kind: "groups". Partition keys, sequences, timestamps, counts, and integer values are decimal strings. Typed values carry a tag, such as {"type":"int","value":"42"}. This preserves integers that a JavaScript number would round. AVG returns a floating-point value. Integer SUM overflow and non-finite arithmetic are query errors. Rows include built-in fields and the registered custom fields referenced by the query, not every attribute on the instance. The state endpoint below reads current @searchable values. Nulls sort first ascending and last descending.

Time bounds and pagination

start_time and end_time are inclusive HLC physical_ms bounds. time_range is a duration in milliseconds and requires an explicit end_time anchor; it cannot be combined with start_time. Neither form consults projection wall-clock time. Use decimal strings for bounds that must cross a JavaScript client without rounding.

limit defaults to 100 and accepts 1 to 1,000 rows or groups. Send the returned next_cursor in the next request’s query.cursor, keeping the namespace and query unchanged. Page size and resource budgets may change. Cursor validation also checks the referenced registry interpretations.

Pagination uses stable ordering for unchanged data, not a snapshot of live state. Concurrent updates can move rows across page boundaries. Partial cluster answers expose missing shards in freshness and do not issue a cursor that would advance past them.

Query budgets

Queries reject unknown fields and unsupported expressions rather than falling back to SQL. The fixed shape limits are 32 filters, 8 breakdowns, 16 calculations, 16 orders, 32 having predicates, and 256 values per membership predicate. Query JSON is bounded to 64 KiB and cursors to 32 KiB.

The optional query.budget object lowers or raises execution limits within the operator’s ceilings. limit controls output page size; it does not limit the work required to find, sort, or aggregate that page.

Budget field	Default	Hard maximum
`max_rows`	1,000,000	10,000,000
`max_groups`	10,000	100,000
`max_time_ms`	5,000	60,000
`max_state_bytes`	8,388,608	8,388,608
`max_vm_steps`	100,000,000	500,000,000

All limits must be positive. A smaller operator ceiling wins over a larger request. Exceeding a budget refuses the query without aggregate data or a continuation cursor; it never silently truncates aggregate input. Resource refusals return HTTP 429 or gRPC ResourceExhausted. Deadline refusals return HTTP 504 or gRPC DeadlineExceeded. The refusal JSON is also carried in gRPC status details.

Successful results and refusals include a cost object. rows_scanned counts candidate workflow rows before filtering, plus the attribute rows read for each referenced column’s degradation metadata. A workflow may therefore count more than once; a million workflows with one referenced custom column typically need about two million row scans. groups counts groups before HAVING and pagination in the current reduction. vm_steps counts SQLite instructions, including metadata SQL. These counters are decimal strings. limits reports effective ceilings; bound_hit is null on success or names the exhausted budget on refusal. unobserved_attempts counts executions whose cost was unavailable. When it is nonzero, the reported scan and instruction counts are lower bounds.

The execution deadline is monotonic process time, separate from HLC query bounds. Cancellation is cooperative: it is checked between candidate rows, during reduction and serialization, and every 1,000 SQLite instructions. It is not a hard-real-time scheduler guarantee.

Cluster queries divide resource allowances across their planned range requests and reserve bounded ownership retries. Uneven data placement or unused reservations can cause an explicit refusal even when another participant has spare allowance. Unknown work behind a missing reply is not treated as free. Freshness still identifies missing ranges.

Tenant query quotas

Set HOPSKIP_QUERY_QUOTA_CONFIG to an operator-owned JSON file, or use visibility.query_quota_config in the server configuration. For example:

{
  "max_concurrent": 64,
  "default": {"max_concurrent": 4},
  "tenants": [
    {
      "tenant": "merchant",
      "namespaces": ["orders", "billing"],
      "policy": {"max_concurrent": 1}
    }
  ]
}

orders and billing share one concurrency allowance in this example. Other namespaces each receive the default allowance. default and each tenant’s policy can also set a budget object using the fields above. Namespace authorization happens before admission; callers cannot choose a different tenant quota identity.

These concurrency limits are per server node, not fleet-wide reservations. They cover store-lock waits and queued or running blocking work across REST and RPC. Exhaustion refuses immediately instead of adding an admission queue. Disconnecting does not release a permit while its blocking work is still running. Invalid policies refuse server startup; restart to apply a policy change.

The console API

Route	Answers
`GET /api/workflows`	Distinct workflow identities with projected events, including events without attributes, in your namespaces
`POST /api/workflows/query`	Structured current-state filters, row pages, and grouped calculations in one namespace
`GET /api/workflows/{ns}/{wf}/state`	One instance’s current `@searchable` values
`GET /api/workflows/{ns}/{wf}/provenance`	One instance’s `@provenance` write history

Each read is scoped to the namespaces your token carries. A request for a namespace you are not a member of is refused (403) rather than returned empty. An empty result would be indistinguishable from “that workflow does not exist”, and you should be able to tell those apart.

GET /api/workflows/{ns}/{wf}/state returns each attribute’s current value. Every emission supersedes the last, so a workflow that passed through charged on its way to completed reads as completed. If you need a variable’s write history rather than its current value, that is what @provenance records (with its own envelope and sampling budget), and the log itself remains the authority for point-in-time reconstruction.

The CLI

hop logs reads a workflow’s projected log entries:

hop logs orders:order-1 --since-ms 1717000000000

physical_ms on each entry is HLC event time (the logical time workflows observe), not a projection wall-clock.

These results reflect the events this node has projected. Projection lag is not a claim that the workflow itself has stopped changing.

There is deliberately no raw-SQL endpoint. Every read is a typed query so the server, not the caller, decides the namespace it runs against: with caller-written SQL, the namespace scoping would live in the query where the server cannot enforce it, and in a cluster arbitrary SQL gives a shard no partition dimension to filter on, so a stale ex-owner could answer for ranges it no longer holds.

Cluster-wide visibility

With the sharded log-store backend, the server’s console listener queries the current owner of each hash range and merges the answers. Structured queries, log reads, attribute lookups, workflow listings, and SLO aggregates use this coordinator. The placement map and leader leases are read live; there is no separately configured peer list or central visibility index.

Configure a private query listener on every sharded server, including servers without a console listener:

export HOPSKIP_CLUSTER_QUERY_ADDR=core-1.internal:7002
export HOPSKIP_VISIBILITY_QUERY_BIND=0.0.0.0:7002
export HOPSKIP_VISIBILITY_QUERY_TOKEN="$(cat /run/secrets/visibility-query-token)"

The advertised address must reach this server from its peers. The bind override is optional when the advertised address is itself a bindable IP address. The token is one shared, private server credential, not a user or worker token. Missing query configuration refuses sharded startup. Keep the listener private. When Raft peer TLS is configured, visibility queries use the same certificate, client CA, and peer-name verification. There is no plaintext fallback.

The memory and disk backends keep node-local visibility and need no peer configuration. The raft and loglet backends do not supply this placement source: their visibility reads report unavailable, while other console operations remain accessible.

Each answer carries a per-shard freshness block, and a shard that was unreachable, whose epoch moved, or that could not show it had caught up is reported rather than quietly omitted, so a narrowed answer is never mistaken for a complete one. Because every request variant carries the hash range it is being asked about, a shard answers only for partitions it owns, which is what makes the merge safe to do at all.

Serving ownership and projection progress are separate. Each peer validates the requested range and current leadership epoch before and after reading. Freshness watermarks retain the epoch and time of the actual projection; an idle leader transfer does not invent newer projection progress. Unknown owners and missing query addresses remain missing ranges in the answer. An unavailable placement map is a query error, not an empty inventory. Partial structured-query results never include a continuation cursor.

Query peers must run the same internal protocol version. Upgrade them together; incompatible peers are reported as missing rather than queried through an older protocol.

Visibility queries read the current projection. They do not query archived execution summaries or reconstruct historical attribute values from object storage.

`@searchable` vs `@indexed`

Annotation	Extracts	Use for
`@searchable`	A scalar value at each checkpoint	Status fields, tiers, small keys you filter and group by
`@indexed`	Deeper structured state	Nested state you need to query into, at higher projection cost

Prefer @searchable for the fields you filter and aggregate on; reach for @indexed only when you must query into nested structure.

Retention

Retention’s purge operation removes a workflow’s visibility rows before deleting the source log. If projection eviction fails, the purge is refused: leaving rows whose log is gone would make the projection impossible to rebuild.

The production expiry scheduler still needs integration. Configuring a retention window does not currently establish automatic expiry.

Why this beats a separate index

No dual-write. The log is the source of truth; the projection can’t drift from it.
No operational dependency. There is no Elasticsearch cluster to run, scale, or recover.
Provenance preserved. Rows trace back to the Merkle-chained events that produced them.