Kubernetes Deployment

Scale Orca to production with Kubernetes StatefulSets for runners and Deployments for conductors.

This is a self-hosting guide. The hosted product at https://www.orcapods.ai needs none of this — sign in and go.

Architecture on Kubernetes

Key points:

• Conductors → Deployment + HPA (stateless, scale freely)
• Runners → StatefulSet + headless Service (stable pod DNS = stable RUNNER_BASE_URL)
• Sidecars → additional containers inside each runner pod
• Ingress → nginx with proxy_buffering off for SSE

---

Conductor Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: conductor
  namespace: orca
spec:
  replicas: 3
  selector:
    matchLabels:
      app: conductor
  template:
    metadata:
      labels:
        app: conductor
    spec:
      containers:
        - name: conductor
          image: orca/conductor:latest
          ports:
            - containerPort: 8080
          env:
            - name: CONDUCTOR_PORT
              value: "8080"
            - name: RUNNER_URLS
              value: >-
                http://runner-0.runners.orca.svc:7070,
                http://runner-1.runners.orca.svc:7070,
                http://runner-2.runners.orca.svc:7070
            - name: AGENT_ORC_RUNS_DIR
              value: /data/runs
            - name: AGENT_ORC_PLANS_DIR
              value: /data/plans.jsonl
            - name: S3_BUCKET
              value: agent-artifacts
            - name: AWS_REGION
              value: us-east-1
            - name: AWS_ENDPOINT_URL_S3
              value: "http://minio.orca.svc:9000"
            - name: AWS_ACCESS_KEY_ID
              valueFrom:
                secretKeyRef:
                  name: orca-secrets
                  key: s3-access-key-id
            - name: AWS_SECRET_ACCESS_KEY
              valueFrom:
                secretKeyRef:
                  name: orca-secrets
                  key: s3-secret-access-key
            - name: S3_FORCE_PATH_STYLE
              value: "true"
          volumeMounts:
            - name: runs-storage
              mountPath: /data/runs
          readinessProbe:
            httpGet:
              path: /healthz
              port: 8080
            initialDelaySeconds: 5
            periodSeconds: 10
          resources:
            requests:
              cpu: 250m
              memory: 256Mi
            limits:
              cpu: 1
              memory: 512Mi
      volumes:
        - name: runs-storage
          persistentVolumeClaim:
            claimName: conductor-runs-pvc
---
apiVersion: v1
kind: Service
metadata:
  name: conductor
  namespace: orca
spec:
  selector:
    app: conductor
  ports:
    - port: 8080
      targetPort: 8080
---
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: conductor-hpa
  namespace: orca
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: conductor
  minReplicas: 2
  maxReplicas: 10
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 60

---

Runner StatefulSet

Runners must have stable DNS names so conductors can route sessions to them.

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: runner
  namespace: orca
spec:
  serviceName: runners          # headless service name
  replicas: 3
  selector:
    matchLabels:
      app: runner
  template:
    metadata:
      labels:
        app: runner
    spec:
      containers:
        # --- Go Runner ---
        - name: runner
          image: orca/runner:latest
          ports:
            - containerPort: 7070
          env:
            - name: RUNNER_PORT
              value: "7070"
            - name: RUNNER_BASE_URL
              value: "http://$(HOSTNAME).runners.orca.svc:7070"
            - name: MCP_BASE_URL
              value: "http://localhost:7070"
            - name: RUNNER_CAPABILITIES
              value: "claude,general"
            - name: CLAUDE_SIDECAR_URL
              value: "http://localhost:7071"
            - name: GENERAL_SIDECAR_URL
              value: "http://localhost:7073"
            - name: ANTHROPIC_API_KEY
              valueFrom:
                secretKeyRef:
                  name: orca-secrets
                  key: anthropic-api-key
            - name: TAVILY_API_KEY
              valueFrom:
                secretKeyRef:
                  name: orca-secrets
                  key: tavily-api-key
            - name: S3_BUCKET
              value: agent-artifacts
            - name: AWS_REGION
              value: us-east-1
            - name: AWS_ENDPOINT_URL_S3
              value: "http://minio.orca.svc:9000"
            - name: AWS_ACCESS_KEY_ID
              valueFrom:
                secretKeyRef:
                  name: orca-secrets
                  key: s3-access-key-id
            - name: AWS_SECRET_ACCESS_KEY
              valueFrom:
                secretKeyRef:
                  name: orca-secrets
                  key: s3-secret-access-key
            - name: S3_FORCE_PATH_STYLE
              value: "true"
          readinessProbe:
            httpGet:
              path: /healthz
              port: 7070
            initialDelaySeconds: 10
            periodSeconds: 15
          resources:
            requests:
              cpu: 500m
              memory: 512Mi
            limits:
              cpu: 2
              memory: 2Gi

        # --- Claude Sidecar ---
        - name: sidecar-claude
          image: orca/agent-worker:latest
          env:
            - name: MODE
              value: claude
            - name: PORT
              value: "7071"
            - name: ANTHROPIC_API_KEY
              valueFrom:
                secretKeyRef:
                  name: orca-secrets
                  key: anthropic-api-key
          resources:
            requests:
              cpu: 250m
              memory: 256Mi
            limits:
              cpu: 1
              memory: 512Mi

        # --- General Sidecar ---
        - name: sidecar-general
          image: orca/agent-worker:latest
          env:
            - name: MODE
              value: general
            - name: PORT
              value: "7073"
            - name: ANTHROPIC_API_KEY
              valueFrom:
                secretKeyRef:
                  name: orca-secrets
                  key: anthropic-api-key
            - name: OPENAI_API_KEY
              valueFrom:
                secretKeyRef:
                  name: orca-secrets
                  key: openai-api-key
          resources:
            requests:
              cpu: 250m
              memory: 256Mi
            limits:
              cpu: 1
              memory: 512Mi
---
# Headless service for stable DNS
apiVersion: v1
kind: Service
metadata:
  name: runners
  namespace: orca
spec:
  clusterIP: None
  selector:
    app: runner
  ports:
    - port: 7070
      targetPort: 7070

With this StatefulSet and headless service, pods get stable DNS names:

• runner-0.runners.orca.svc:7070
• runner-1.runners.orca.svc:7070
• runner-2.runners.orca.svc:7070

Set these in RUNNER_URLS on the conductor.

---

Secrets

apiVersion: v1
kind: Secret
metadata:
  name: orca-secrets
  namespace: orca
type: Opaque
stringData:
  anthropic-api-key: "sk-ant-..."
  openai-api-key: "sk-..."
  tavily-api-key: "tvly-..."
  s3-access-key-id: "agent-orc"
  s3-secret-access-key: "replace-me"

---

VirtualFS Strict Mount Configuration

When deploying the standalone VirtualFS service (orca/vfs) to Kubernetes, the strict mount model requires explicit backend configuration. Do NOT use --profile dev in production — the dev profile uses in-process RAM and cannot persist data across pod restarts.

Required environment variables for the prod profile:

Variable	Description
VFS_S3_BUCKET	R2/S3 bucket name for the production root mount
VFS_S3_ENDPOINT	S3-compatible endpoint URL (e.g., https://<account>.r2.cloudflarestorage.com)
AWS_ACCESS_KEY_ID	S3/R2 access key
AWS_SECRET_ACCESS_KEY	S3/R2 secret key
VFS_AUTH_TOKEN	Bearer token for VirtualFS API authentication

If these are missing or the bucket is unreachable at pod startup, the affected mounts surface as unavailable in GET /vfs/mounts. The server still passes /healthz — liveness checks are independent of mount status. Use the VirtualFS smoke script (scripts/vfs-smoke.sh) in a post-deploy readiness check to assert all expected mounts are ready before traffic is sent.

# VirtualFS Deployment excerpt — add to your manifest
env:
  - name: VFS_S3_BUCKET
    valueFrom:
      secretKeyRef:
        name: orca-secrets
        key: vfs-s3-bucket
  - name: VFS_S3_ENDPOINT
    value: "https://<account>.r2.cloudflarestorage.com"
  - name: AWS_ACCESS_KEY_ID
    valueFrom:
      secretKeyRef:
        name: orca-secrets
        key: s3-access-key-id
  - name: AWS_SECRET_ACCESS_KEY
    valueFrom:
      secretKeyRef:
        name: orca-secrets
        key: s3-secret-access-key
  - name: VFS_AUTH_TOKEN
    valueFrom:
      secretKeyRef:
        name: orca-secrets
        key: vfs-auth-token

---

Ingress with SSE Support

Without proxy-buffering: "false", Server-Sent Events will not stream correctly. nginx-ingress will buffer the entire response.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: orca
  namespace: orca
  annotations:
    nginx.ingress.kubernetes.io/proxy-buffering: "off"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "3600"
    nginx.ingress.kubernetes.io/proxy-request-buffering: "off"
spec:
  ingressClassName: nginx
  rules:
    - host: orca.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: conductor
                port:
                  number: 8080

---

Updating Runner Count

When you scale the StatefulSet, update the conductor’s RUNNER_URLS to include the new pod DNS names. The safest approach:

1. Scale the StatefulSet: kubectl scale statefulset runner --replicas=4 -n orca
2. Wait for the new pod to be ready: kubectl rollout status statefulset runner -n orca
3. Update the conductor Deployment env to include runner-3.runners.orca.svc:7070
4. Roll out the conductor: kubectl rollout restart deployment conductor -n orca

New sessions will round-robin to include the new runner. Existing sessions remain pinned to their original runner.

---

Namespace and RBAC

apiVersion: v1
kind: Namespace
metadata:
  name: orca
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: orca
  namespace: orca

---

Monitoring Integration

Both conductor and runner expose /metrics in Prometheus text format. Create a ServiceMonitor if you’re using the Prometheus Operator:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: orca
  namespace: monitoring
spec:
  selector:
    matchLabels:
      app: conductor
  namespaceSelector:
    matchNames: [orca]
  endpoints:
    - port: http
      path: /metrics
      interval: 30s