Identity provider

Run Keycloak for this engine — realms, the console theme, and the traps that cost real time.

The identity provider is Keycloak, deployed from the deeplinq-sso repository as a single image.

This page covers both the procedures you will need to run it and the behaviours that will surprise you. Procedures first.

For what the model means, read Identity model first.

Procedures

Every script lives in local/railway/ and is safe to read before running.

Read-only, and the first thing to run when something is odd. It asks whether the identity provider still matches what the engine believes: a realm per application, the console's clients carrying the mappers it reads, and every organization bound to a provider organization stamped as ours.

local/railway/keycloak-verify.sh

== organizations ==
  OK      GAHISSY — deeplinq/gahissy, binding active, stamp matches
no drift

Each drift line names its own fix. Almost all of them are "Sync identity" on the organization, which is idempotent and converges it.

One drift line says do not re-sync: a provider organization stamped for a different engine organization. That is a name collision or a reused alias, and re-syncing would bind a tenant to somebody else's organization. Investigate it by hand.

Back up and restore

local/railway/keycloak-backup.sh                    # → local/backups/keycloak-<stamp>.sql
local/railway/keycloak-backup.sh --verify <file>    # is this dump complete?

The dump holds everything Keycloak owns: realms, clients and mappers, users and their credentials, organizations and memberships, realm roles.

It does not hold the engine's own records — identity bindings, machine credentials, principal_roles. Those live in the engine's database. Restoring one without the other leaves the two disagreeing about which organization is which, so take and restore them together.

Restore into an empty database. Keycloak does not reconcile a dump against rows it already has, and a partial restore leaves realms half-formed:

railway ssh --service keycloak-db -- sh -c 'psql "$DATABASE_URL"' < <file>

Then restart the keycloak service and confirm the realms are back with local/railway/keycloak-admin.sh GET /realms.

The dump contains password hashes and client secrets. Treat the file as a credential — local/backups/ is gitignored for that reason.

Reset the console operator's password

When the platform operator cannot complete sign-in — the pending-required-action shape above is the usual cause — the console can issue them a fresh password without touching the identity provider by hand.

Sign in to /admin with the break-glass password, not SSO.
Go to SSO setup and use Reset password on the operator card.
Copy the password shown. It is displayed once and stored nowhere.
Sign in with SSO. The provider asks for a new password; it must satisfy the realm policy, and in particular cannot be one the account has used before.

Break-glass is required by design, not by accident: an identity-provider session must not be able to rewrite the credential that authorises it, or a stolen session becomes account takeover. The practical consequence is that ADMIN_USERNAME and ADMIN_PASSWORD must be set — a console with no break-glass credential has no in-console recovery, and the only route is the identity provider directly.

Two more limits worth knowing before you need this:

The password issued is temporary. That is deliberate: a password an operator can read and that keeps working is a shared credential.
Only the recorded operator can be reset — the login the SSO setup last granted console access to. Setup overwrites that record when a different operator is named while leaving the earlier operator's access intact, so after turnover an earlier operator is recoverable only at the identity provider.

The reset uses the engine's own standing service account, so no operator ever pastes an identity-provider credential — which means it needs TENANT_PROVISIONER_CLIENT_ID / TENANT_PROVISIONER_CLIENT_SECRET to be set. Without them the reset fails with a 401 from the provider.

Rotate the engine's own credential

The engine administers Keycloak as the deeplinq-engine service-account client in the master realm. To rotate its secret:

local/railway/keycloak-rotate-engine-credential.sh --check   # report only
local/railway/keycloak-rotate-engine-credential.sh           # rotate

It regenerates the secret, proves the new one can mint a token before handing it to the engine, then writes it straight into the engine's variables. The value is never printed and never passed as a command argument.

Keycloak replaces a client secret in place — there is no second slot. Between regeneration and the engine restarting, the engine holds a secret Keycloak no longer accepts:

tenant provisioning fails in that window — identity setup, machine credential issue and rotation, administrator management;
authentication is unaffected — tokens are verified offline against the realm's JWKS, and this credential is not on the request path.

The window is one engine restart. Run it when a provisioning failure is survivable, not mid-onboarding.

Upgrade Keycloak

The version is pinned in deeplinq-sso/keycloak/Dockerfile. Before moving it:

back up (above) — the upgrade migrates Keycloak's schema in place;
re-diff the two forked theme templates against the new stock ones. The command is in deeplinq-sso/keycloak/themes/README.md; both forks are additive, so an upgrade is a re-diff rather than a rewrite;
deploy, then run keycloak-verify.sh and sign in once.

Rolling back means restoring the backup and the previous image: Keycloak migrates its schema forward on start and will not run against a newer one.

There is no rehearsal environment for this today. An upgrade is exercised for the first time in production, which is why the backup step is not optional.

Start here; it resolves most reports.

Symptom	Almost always
"Unable to find factory for Required Action 'null'"	a realm import declared a required action with no `providerId`
Login page is unbranded	the theme is not mounted — Keycloak falls back silently
The page looks stale after a theme change	a browser holding the 30-day-cached stylesheet; the fingerprint fixes this going forward
`401` from the engine with a valid-looking token	the organization it names is not bound, or it names none — check `scope` includes `organization:<alias>`
`400` "carries no end user"	a machine credential on a user-owned route; expected, not a fault
Person authenticates but can do nothing	they hold no role — membership gives `org-member`; anything above it is granted in the console
Everything 403s right after creating a realm	a cached admin token minted before the realm existed
Sign-in redirects to the provider and never comes back — the engine logs `/admin/api/auth/oidc/start → 303` and then nothing	the account has a pending required action, almost always `UPDATE_PASSWORD`. The provider serves its change-password screen instead of completing the flow, so no callback ever reaches the engine. See below.
The change-password screen refuses every password you try	the realm's password policy. `passwordHistory(N)` refuses a password the account has used before, and the screen cannot be completed until you choose one it has never had

A pending required action stalls the callback

This is the shape to recognise: the browser sits at the identity provider, and the engine's log shows the redirect out with no callback back. The account is fine; it simply has an action to discharge first.

Read the account's state directly — requiredActions is the field that matters:

curl -s -H "Authorization: Bearer $ADMIN_TOKEN" \
  "$ISSUER_BASE/admin/realms/platform/users?email=someone%40example.com&exact=true" \
  | jq '.[0] | {username, enabled, requiredActions}'

If it carries UPDATE_PASSWORD, the person must set a new password to get through — and the realm's policy applies to that choice. A realm with passwordHistory(3) refuses the last three passwords, so reaching for a familiar one fails repeatedly and looks like a broken login rather than a policy working correctly.

The engine deliberately never clears required actions. Writing an empty list removes every registered action, which would silently cancel a pending VERIFY_EMAIL or a required CONFIGURE_TOTP enrolment on an account the engine did not create.

For the console's own operator, Reset the operator's password issues a fresh temporary one rather than needing any of this by hand.

The realms

master      Keycloak's own administration. The bootstrap admin and the engine's
            service account live here. NEVER product users — anyone in master
            can potentially administer the whole instance.
platform    This deployment's operators. The console signs in against it.
<product>   One realm per application: deeplinq, deepdesk, …
            Its organizations are customers; its users are their people.

master is for administering Keycloak, not for using the product. Putting product users there is a well-known anti-pattern and it hands them a foothold in the realm that administers every other realm.

How the engine administers it

The engine holds its own credential: a confidential service-account client named deeplinq-engine in the master realm, holding the admin realm role. It is configured through TENANT_PROVISIONER_CLIENT_ID / TENANT_PROVISIONER_CLIENT_SECRET.

That is why setting up console single sign-on asks for no credential: leave the field empty and the engine uses its own. A pasted one still wins when given, so a first-ever setup on an engine that has none still works.

Keycloak carries per-realm rights inside the access token. A token minted before a realm existed cannot administer it — every call answers 403 while the credential is, in fact, entitled. The adapter drops its cached token after creating a realm for exactly this reason. If you script against the admin API yourself, re-mint after creating a realm.

Which adapter the engine uses

Selected by the shape of the issuer, with nothing to configure:

issuer contains "/realms/"  → Keycloak
otherwise                   → refused, naming the issuer and the shape expected

A Keycloak issuer is necessarily <base>/realms/<realm> because the realm is the tenancy boundary, so the URL is a fact rather than a heuristic — and no second setting can disagree with it.

There is one integration, and an unrecognised issuer is an error, not a fallback. Until 2026-08-09 an issuer without /realms/ selected a second, older adapter. Once that provider was decommissioned the branch could only be reached by a typo or a misconfiguration, and it answered every one of them with a 404 from a management API nobody runs — a message about a product that is not there. The adapter is deleted; the refusal names the issuer you typed.

Pointing the console at a different provider

Setup refuses a mismatched issuer by default, because the overwhelmingly likely cause is a typo or a credential from the wrong instance, and silently repointing would strand every existing session.

To move deliberately, tick "Point the console at this identity provider instead" (replace_issuer over the API). The run discards the stored checkpoint first: its recorded ids name objects inside the instance being left behind.

The deeplinq theme lives in deeplinq-sso/keycloak/themes. It extends Keycloak's keycloak.v2 (PatternFly 5) — not the legacy keycloak theme — and matches the console's own sign-in screen value for value.

Two templates are forked, additively and marked DEEPLINQ FORK:

template.ftl adds the brand panel, because the stock DOM offers one brand node and CSS-generated content cannot be localised;
field.ftl resolves placeholders by convention from the field name (username → placeholder.username).

Keycloak falls back silently. A realm naming a loginTheme that is not mounted serves stock keycloak.v2 with no error and no warning. An unbranded login page means a missing mount, not a broken stylesheet.

Why the stylesheet filename carries a hash

Keycloak serves theme resources with cache-control: max-age=2592000 — thirty days — under a path whose version segment does not change when your theme files do. The same URL keeps returning a browser's cached copy across deploys.

The image build therefore fingerprints the stylesheet (deeplinq.<hash>.css) and rewrites theme.properties to match, so the URL changes exactly when the bytes do. Without it, a theme change is invisible for a month to anyone who has loaded the page, and the only cure is asking every user to hard-refresh.

Per-tenant branding

logoUri is a first-class Keycloak client attribute, and the login template renders it when present. So the mark is chosen by which client started the authorize request — the only thing Keycloak reliably knows before anyone has typed an email.

One browser client per organization therefore gives one logo per organization, with the realm's wordmark as the fallback.

logoUri is rendered into an <img src>. A tenant-supplied URL is a tenant-controlled outbound request from your login page. Store the asset yourself and serve it from your own origin; do not accept an arbitrary URL.

Schema changes after the consolidation

config/db/migrations/0001_init.up.sql is frozen. Every database that already exists is recorded at schema_migrations.version = 1, so the migration runner will never read that file again: an edit reaches new databases and no existing one.

This is not theoretical. principal_roles was added to 0001 after the consolidation, the engine shipped code that read it, and the first "Sync identity" in production answered relation "principal_roles" does not exist. The hermetic gate builds its database from scratch, so the file was right by construction everywhere except where it mattered.

After a consolidation, a schema change is a new migration — never an edit to 0001 — and it must be idempotent so it is a no-op on a database created from 0001. scripts/check-migrations.sh enforces this inside make gate; override it only for a deliberate re-consolidation, which must renumber existing databases too.

Traps measured against Keycloak 26

Each of these cost real debugging time. They are recorded so they cost it once.

What you do	What happens	Why
Create an organization with no domain	`400`	a domain is mandatory; the adapter synthesises one under `.deeplinq.invalid`
Read an organization from the list endpoint	`attributes` missing	attributes are returned by `GET /organizations/{id}` only — the reverse-adoption stamp must be read by id
Search an organization by its alias with `exact=true`	no match	`exact` compares the name; the substring search covers the alias
`POST /organizations/{id}/members` with `text/plain`	`415`	it declares `application/json` but the body is the bare user id, not a JSON string
Register a machine credential by the client's UUID	authenticates, then refused	a `client_credentials` token's `sub` is the service-account user id, a different UUID
Declare required actions in a realm import with no `providerId`	every login fails	Keycloak stores `null` and cannot find the factory: "Unable to find factory for Required Action 'null'"
Create a user with no `firstName`/`lastName`	every grant refused	`VERIFY_PROFILE` fires on an incomplete profile: "Account is not fully set up"

Verifying a change

The live contract tests exercise every adapter method against a real instance, in a throwaway realm they delete afterwards:

KEYCLOAK_BASE=https://identity.example.com \
TENANT_PROVISIONER_CLIENT_ID=deeplinq-engine \
TENANT_PROVISIONER_CLIENT_SECRET=… \
go test -tags=live ./internal/features/ssosetup/outbound/keycloakmgmt/ -run TestLive -v

They are not part of any gate — they need a credential and a network, and an operator invokes them deliberately. A fake written by the same hand as the client proves the two agree with each other, not that either speaks Keycloak.