Operations

Go-live checklist

The deployment, identity, data, observability, and recovery checks required before production traffic.

Use this as the release gate for a new environment. The linked pages contain the commands and detailed procedures.

Deployment

  • ENVIRONMENT=production is set.
  • PostgreSQL uses distinct migration, application, and system roles.
  • S3-compatible object storage, Weaviate, Redis, Docling, and required sidecars use durable production endpoints.
  • TLS terminates before the engine and PUBLIC_URL is the externally reachable HTTPS origin.
  • Every secret is injected by the deployment platform; none is in an image, repository, command line, or log.

Verify values and ranges against Configuration.

Identity and tenant onboarding

  • Keycloak and its database are backed up.
  • The platform realm contains operators only; each application has its own realm.
  • The engine's Keycloak service-account credential is configured and its rotation procedure is owned.
  • Every organization is bound, has a machine credential or human membership, has credits, and has at least one priced model grant.
  • A machine token completes one governed call and an end-user token completes one user-owned call.

Use Identity provider and Platform setup.

Health and observability

  • /healthz proves process liveness.
  • /readyz is green behind the production load balancer.
  • /v1/readiness is green for a representative organization.
  • Structured logs retain request_id and traces reach the configured OTLP collector.
  • Alerts cover readiness, authentication denials, billing reconciliation, failed ingestion, guardrail dependency failures, and exhausted capacity.

See Health and observability.

Security and recovery

  • The egress allowlist covers only the documented provider and sidecar destinations.
  • A PostgreSQL, object-store, vector-store, and Keycloak backup set has been taken together.
  • Restore has been rehearsed on disposable infrastructure.
  • SECRETS_KEY, AUDIT_ANCHOR_KEY, tenant machine credentials, and the Keycloak service-account credential each have an owner and rotation path.
  • Audit verification and billing reconciliation are green after the rehearsal.

See Security and recovery.

Release and rollback

  1. Record the engine image digest and database migration version.
  2. Back up before any schema or Keycloak version change.
  3. Deploy one environment or replica and run the readiness and representative tenant checks above.
  4. Promote traffic only after those checks pass.
  5. Roll back the image only when its database version is compatible. A destructive down migration or Keycloak schema rollback requires restoring the paired backup, not merely selecting an older image.

On this page