Operations
Go-live checklist
The deployment, identity, data, observability, and recovery checks required before production traffic.
Use this as the release gate for a new environment. The linked pages contain the commands and detailed procedures.
Deployment
ENVIRONMENT=productionis set.- PostgreSQL uses distinct migration, application, and system roles.
- S3-compatible object storage, Weaviate, Redis, Docling, and required sidecars use durable production endpoints.
- TLS terminates before the engine and
PUBLIC_URLis the externally reachable HTTPS origin. - Every secret is injected by the deployment platform; none is in an image, repository, command line, or log.
Verify values and ranges against Configuration.
Identity and tenant onboarding
- Keycloak and its database are backed up.
- The
platformrealm contains operators only; each application has its own realm. - The engine's Keycloak service-account credential is configured and its rotation procedure is owned.
- Every organization is bound, has a machine credential or human membership, has credits, and has at least one priced model grant.
- A machine token completes one governed call and an end-user token completes one user-owned call.
Use Identity provider and Platform setup.
Health and observability
/healthzproves process liveness./readyzis green behind the production load balancer./v1/readinessis green for a representative organization.- Structured logs retain
request_idand traces reach the configured OTLP collector. - Alerts cover readiness, authentication denials, billing reconciliation, failed ingestion, guardrail dependency failures, and exhausted capacity.
Security and recovery
- The egress allowlist covers only the documented provider and sidecar destinations.
- A PostgreSQL, object-store, vector-store, and Keycloak backup set has been taken together.
- Restore has been rehearsed on disposable infrastructure.
SECRETS_KEY,AUDIT_ANCHOR_KEY, tenant machine credentials, and the Keycloak service-account credential each have an owner and rotation path.- Audit verification and billing reconciliation are green after the rehearsal.
Release and rollback
- Record the engine image digest and database migration version.
- Back up before any schema or Keycloak version change.
- Deploy one environment or replica and run the readiness and representative tenant checks above.
- Promote traffic only after those checks pass.
- Roll back the image only when its database version is compatible. A destructive down migration or Keycloak schema rollback requires restoring the paired backup, not merely selecting an older image.