Security and recovery
Back up authoritative state, rotate keys safely, control egress, and restore service.
Backup scope
Different stores have different authority:
| Store | Authority |
|---|---|
| PostgreSQL | Organizations, credentials metadata, grants, ledger, audit, jobs, schedules, application metadata |
| S3-compatible storage | Original uploads and derived durable objects |
| Weaviate | Rebuildable retrieval vectors, but required for immediate service restoration |
| FalkorDB | Semantic memory graphs |
| Redis | Admin-session validity only; loss requires re-login |
Back up PostgreSQL and object storage as the minimum truth set. Include vector
and graph stores when recovery time objectives do not allow full rebuilding.
Protect SECRETS_KEY and AUDIT_ANCHOR_KEY separately from data backups.
Restore order
- stop tenant traffic and workers;
- restore PostgreSQL;
- restore object storage;
- restore Weaviate and FalkorDB, or begin controlled rebuilds;
- supply the exact required key versions;
- start one engine replica and let migrations complete;
- verify
/readyz; - verify audit chains and reconcile billing;
- exercise a non-destructive tenant read and retrieval;
- resume workers, then traffic.
Do not declare recovery complete based only on process health. Verify non-empty tenant, ledger, audit-anchor, object, vector, and graph records through logical interfaces.
Rotate a tenant machine credential
A tenant credential lives at the identity provider; the engine holds only a registry entry naming the machine an organization authenticates as. Each organization carries two rotation slots, so old and new overlap.
- provision the replacement into the organization's second slot (the secret is delivered once and is never retrievable afterwards);
- the tenant switches its backend to it and confirms traffic;
- retire the old slot;
- verify a token minted from the old credential is refused.
Offboard a person
There is nothing to revoke here and no key to hunt: disable or delete them at
the identity provider. The engine's cutoff sweep carries that across within one
sweep interval; for an immediate kill, bump the principal's valid_after and
every token minted before that instant is refused at the next request.
Rotate SECRETS_KEY
Add the new highest-version key while retaining the old key:
2:<new-key>,1:<old-key>Restart, then re-seal stored credentials:
POST /v1/admin/secrets/resealReview the checked and resealed counts. Remove the old version only after every sealed column is confirmed migrated. Dropping it early makes remaining blobs unrecoverable.
Changing SECRETS_KEY also changes the derived admin-cookie signing key and
invalidates existing admin sessions.
Rotate AUDIT_ANCHOR_KEY
Verify chains with the old key immediately before rotation. Deploy the new key, then re-sign provably intact anchors:
POST /v1/admin/audit/reanchorThe operation refuses to sign a chain whose rows do not reproduce its exact count and head hash.
Egress policy
Allow only destinations used by enabled capabilities:
- configured model, embedding, guardrail, router, and reranker providers;
- approved crawl targets;
- registered webhook endpoints;
- Gmail, Google OAuth, Microsoft identity, and Microsoft Graph when enabled;
- registered external MCP server URLs;
- internal sidecars and observability collector.
The crawler and webhook clients include SSRF-safe dialing, but production network policy remains a deployment responsibility.
Vulnerability response
Maintain a documented intake, severity classification, remediation SLA, release-block policy, and time-bounded exception process. Scan the Go module, container image, sidecar dependencies, and JavaScript documentation/admin assets. A known reachable high-severity vulnerability should block release.
Recovery drills
Run disposable-stack restore and key-rotation drills regularly. A drill is complete only when audit verification, billing reconciliation, tenant isolation, source-object access, retrieval, and memory behavior are proven through the restored system.