Kubernetes - a hosted deployment

The repository ships a complete Kubernetes topology in deploy/k8s/: the shape of a hosted Hopskip region, as a Kustomize base with a component per cloud and an overlay per environment. Every cloud-specific choice in it is annotated with what it is for and what replacing it costs.

This is the multi-process, multi-node deployment. For one process, use hop dev; for one node with real network boundaries, use the compose stack.

Where it runs

Overlay	Cluster	Control plane	Authentication
local	k3s, k3d, kind, minikube	one node, disk log store	HS256 shared secret, tokens from hop token
staging	EKS	one-node Raft group	OIDC
production	EKS	three-node Raft group	OIDC
gke	GKE	three-node Raft group	OIDC
aks	AKS	three-node Raft group	OIDC

The base carries no cloud at all. Load balancers, cloud identity, the cold-tier object store, preemptible-capacity labels, and the metrics integration are the five things that differ, and each lives in components/<provider>/. Adding a cloud means writing a sixth component, not a second copy of the topology.

The local overlay is the step between docker compose up and a cloud cluster: the same manifests, the same network policies and security contexts, on a laptop. It comes up in three commands and mints its own credentials.

What it runs

Workload	Kind	Notes
hopskip-server	StatefulSet, 3	One Raft group (HOPSKIP_LOG_STORE_BACKEND=raft), one block volume each, spread across availability zones; resized with kubectl scale (the membership autopilot grows and shrinks the voter set to match)
hopskip-wasm-worker	Deployment, N	KEDA-scaled on dispatch backlog, spot by preference, one pool per tenant namespace
hopskip-console	Deployment	The console’s static build behind an oauth2-proxy that terminates an OIDC session
hopskip-www	Deployment	The marketing and documentation site: static files, no dependencies
hopskip-toolbox	Deployment, 1	An hop CLI with write access to the build registry

Two load balancers, because two protocols: a layer-7 one for the HTTP hostnames (site, console, SCIM) and a layer-4 one for gRPC, since workers hold a bidirectional stream open for their whole life and that is an L4 shape. Which products those are is the component’s business: an ALB and an NLB on AWS, a Google Application Load Balancer and a passthrough NLB on GCP, the app-routing nginx and an Azure Load Balancer on AKS, Traefik and ServiceLB on k3s.

That difference has one consequence worth knowing up front: AWS’s NLB can terminate TLS and Google’s and Azure’s cannot. On those two clouds the gRPC certificate goes on the server pods instead, a stricter posture, at the cost of a rollout whenever the certificate is renewed.

How workers authenticate

Every cloud overlay runs in-cluster gRPC over mTLS. hopskip-server opens a second listener (HOPSKIP_INTERNAL_BIND) that requires a client certificate; cert-manager creates a CA for the deployment and issues one to the worker fleet and one to the toolbox, and the NetworkPolicies stop admitting in-cluster traffic to the public gRPC port at all.

The public endpoint is untouched by this: customers keep connecting with a bearer token. That is the reason for the second socket. A client CA applies to a whole listener, so with one bind, requiring certificates from workers would have required one from every customer too.

The certificate authenticates the connection, not the principal: the server checks that it chains to the CA and nothing else, so namespace membership, ACL grants, and the audit trail still come from the bearer token. What it adds is that a stolen worker token is useless from outside the cluster.

How a person signs in

The console is a browser app that calls same-origin /api and only sends an Authorization header when an operator has pasted a token into its token box. In a hosted deployment that is not a login, so an oauth2-proxy sidecar supplies one: it runs the OIDC authorization-code flow against your identity provider, keeps the session in an encrypted cookie, and attaches the user’s own ID token upstream. hopskip-server validates it with HOPSKIP_OIDC_ISSUER and HOPSKIP_OIDC_AUDIENCE, and the subject that reaches namespace membership, ACL grants, and the audit trail is the person who signed in.

The proxy is a sidecar rather than its own Deployment so the console’s nginx can listen on loopback. There is then no path to the console API that skips the proxy, because there is no socket on the pod’s routable address.

The same identity provider pushes users over SCIM to its own hostname and its own listener, authenticated by HOPSKIP_SCIM_TOKEN.

Where the workers get their code

hop deploy does two things: it calls SetCurrentDeployment on the live server, which journals the pointer durably, and it writes the built module to a registry on disk. Workers resolve a dispatch’s component hash against that registry, both for the task types they register at startup and for a pinned hash they have not seen. So the registry has to be visible to every worker and writable by whatever runs the deploy: ReadWriteMany, which means EFS on AWS, Filestore on GCP, and Azure Files on AKS. The local overlay is the exception, and says so: one node makes ReadWriteOnce enough.

The toolbox pod is the only writer. Workers mount it read-only, so guest code cannot rewrite the modules another worker will execute.

Known limitations

Five are worth knowing before you deploy this:

• Admin writes must reach the Raft leader. The replicated backend has no forward-to-leader path yet, and nothing exposes a leadership signal a Service could select on. The console’s /api proxy works around it by retrying across replicas; the gRPC path does not, so a deploy that lands on a follower fails and must be retried.
• Worker credentials do not refresh. hopskip-wasm-worker reads its bearer token once at startup. With the server on OIDC that token is an identity-provider machine credential with an expiry, so rotation is a secret update plus a rollout.
• The cold tier speaks S3, and only S3. HOPSKIP_TIERED_OBJECT_STORE takes local or s3, so AWS works directly and GCP works through Cloud Storage’s S3-compatible API (with HMAC keys, since the S3 client cannot use Google IAM). Azure Blob has no S3 API, so the AKS overlay ships without a cold tier and sealed segments stay on the managed disks. The feature is also a compile-time one: the published image is built without s3-object-tier, and a server told to use s3 without it fails at startup rather than silently keeping everything local.
• Encrypted everywhere, mutually authenticated in most places. In-cluster gRPC and the Raft peer transport require certificates at both ends. The console API, SCIM, and the ops endpoint encrypt but do not demand a client certificate, because a kubelet, a Prometheus in another namespace, and an ingress controller cannot present one without changing infrastructure the manifests do not own. Certificate renewal needs a rollout, because every side reads its certificate once at startup, and turning peer mTLS on partitions a running cluster for the length of the rollout, since that transport has no plaintext-compatible mode.
• The scale numbers are unproven. Resource requests come from the instance shapes in the cloud unit-economics analysis, not from a load test of this topology.