design-proposal: per-tenant Keycloak realms for tenant Kubernetes cluster OIDC

[IvanHunters]

Summary

Design proposal companion to cozystack#3044.

The proposal answers the architectural decision raised in that PR's review: what is the identity unit for tenant Kubernetes cluster OIDC. This document picks per-tenant Keycloak realm (tenant-<ns>), with per-cluster public clients providing audience-bound token isolation, and explains why that is structurally different from using the flat cozy realm. It is intentionally written to compare both options head-on under Alternatives considered, so the trade-off is on the record before the implementation PR lands.

What the proposal covers

• Identity model. Per-tenant Keycloak realm tenant-<ns> owned by the tenant admin, separate from the platform-admin cozy realm. Provisioned declaratively via Tenant.spec.oidc=true.
• Per-cluster isolation. A KeycloakClient and a KeycloakClientScope per tenant cluster inside the tenant's realm; id_token.aud matches the per-cluster --oidc-client-id, so tokens are non-replayable across clusters.
• Default RBAC. Two-tier view + admin KeycloakRealmGroup per cluster, bound inside the tenant cluster to upstream ClusterRole/view and ClusterRole/cluster-admin respectively. No blanket cluster-admin grant; the static admin kubeconfig remains as the documented break-glass path.
• Realm propagation. Through cozystack-basics's static _namespace.oidc-realm emission, the same channel that already propagates DNS suffix and ingress class. Explicitly no Helm lookup is used.
• OIDC kubeconfig Secret. A kubernetes-<cluster>-oidc-kubeconfig Secret in the management cluster's tenant namespace, exposed via cozyrds spec.secrets.include, carrying a ready-to-use kubeconfig with a kubectl oidc-login exec auth block.
• Lifecycle. Create, inherit (child tenant), add a cluster, delete a cluster, disable OIDC on a tenant. All declarative.
• Open questions and follow-ups. BYO-OIDC via --authentication-config on KamajiControlPlane, RFC 8693 token exchange via a custom credential plugin, and a cozystack CLI helper. All called out as separate future proposals.
• Alternatives considered. Flat cozy realm + per-cluster client, BYO-OIDC only, EKS-style flat IAM + webhook, RFC 8693 token exchange plugin, embedding into the existing admin-kubeconfig Secret, single global client. Each with a short rationale for the rejection.

Status

Draft. Looking for review on the identity-unit decision and on the open questions before the cozystack#3044 implementation lands.

[coderabbitai]

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 976755f9-8adf-434c-a927-c1ca8b253cb1

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[gemini-code-assist]

Code Review

This pull request introduces a design proposal for implementing per-tenant Keycloak realms to back OIDC authentication for tenant Kubernetes clusters in Cozystack. The feedback suggests clarifying that the bootstrap Job runs in the management cluster rather than inside the tenant cluster itself. Additionally, it warns that attempting to delete ClusterRoleBindings inside the tenant cluster during a pre-delete hook is redundant and could block Helm release deletion if the control plane becomes unreachable.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[Andrei Kvapil (kvaps)]

Thanks for writing this up — the doc is thorough and the "Alternatives considered" section is genuinely useful as the record for the per-tenant-realm direction. The structured --authentication-config delivery mechanism (Layer 5) and its rationale (multi-issuer, private-CA, additive BYO) are the right call and match where we want to land.

My main concern is alignment with today's two OIDC syncs, which settled the phasing after this doc was written. The agreed shape was:

• Phase 1: authenticate tenant-cluster users via the existing platform cozy realm + per-cluster public client for audience isolation, plus the ability to pass a BYO OIDC/authentication config into the cluster. No groups, no auto-provisioning into Keycloak. API as a selector, e.g. oidc: System | CustomConfig | None.
• Phase 2 (deferred, may not be needed): per-tenant Keycloak realm / "managed IdP", as a separate, fully-designed product (oidc: ... | KeycloakRealm | ...).

As written, the proposal does the inverse of that split, so I'd ask to realign it before the #3044 implementation lands:

1. Identity unit (Overview / Rollout). The doc picks per-tenant realm tenant-<ns> as the unit to implement now. Per the syncs that's Phase 2. Phase 1 is the shared cozy realm + per-cluster client.
2. "Flat cozy realm + per-cluster public client" (Alternatives, L278) is rejected — but that's the option we chose for Phase 1. Note the rejection somewhat misframes it: Phase 1 does not put tenant end-users into cozy. It authenticates the platform users that already exist there, and BYO covers a company's external users. Could you re-evaluate it as the Phase-1 baseline rather than a rejected alternative?
3. BYO-OIDC is marked non-goal/deferred (L18, Non-goals) — it's part of Phase 1. Passing a custom OIDC/authn config into the cluster is one of the two Phase-1 deliverables, not a follow-up. Layer 5 already mounts a structured config, so this is mostly a framing/scope change.
4. Groups + auto-RBAC (Layer 4). Two KeycloakRealmGroups per cluster + auto ClusterRoleBindings is exactly the group/auto-provisioning machinery the syncs put out of Phase 1 ("no groups"; a users: map with a role is enough). Auto-creating clients/groups in the IdP at cluster-creation is the "deep integration = a product that needs serious design" we flagged — and it breaks the customer-Keycloak case where the tenant does not want us spawning groups in their realm. Suggest dropping it from Phase 1.
5. Billing / central-Keycloak load is unaddressed. All tenant-<ns> realms live on the platform master ClusterKeycloak (L72). That's the load/billing blocker raised on the call. Please add a section covering it and the alternative of running Keycloak as an External App inside the tenant namespace (billing becomes trivial, it's just API + DB pods), which was the proposed mitigation.

Factual fixes about the current codebase (verified against main):

• L81 — it's packages/system/cozystack-basics/, not packages/core/..., and the emission lives in templates/cozystack-values-secret.yaml; there is no _namespace.tpl.
• L85 — _namespace.dns-suffix / _namespace.kubelet-image-credential-provider-config are not propagated today. The actual _namespace.* keys are host, etcd, ingress, gateway, monitoring, seaweedfs, schedulingClass. The real precedent is host/ingress.
• L82 — there is no tenant-hierarchy walk-up today; propagation is single-level parent lookup (packages/apps/tenant/templates/namespace.yaml). Recursive inheritance (Layer 2) is new mechanism, not "the same channel that already exists" — worth calling out as new work.
• L46, L278 — the cozy groups are not <ns>-admins / <ns>-users. They are <tenant>-view, <tenant>-use, <tenant>-admin, <tenant>-super-admin (packages/apps/tenant/templates/keycloakgroups.yaml). This also affects the "group collision" argument.
• L35 — KamajiControlPlane in cozystack is controlplane.cluster.x-k8s.io/v1alpha1 (packages/apps/kubernetes/templates/cluster.yaml). tcp.kamaji.clastix.io is the TenantControlPlane group — a different CRD.
• L33 — apps/kubernetes has no oidc value at all today (not a "no-op field").
• L17 vs Layer 5 — Scope says --oidc-* wiring, Design uses --authentication-config. Please make these consistent.

None of this is a knock on the analysis — it's solid and I'd keep it as the Phase-2 design record. The ask is to re-frame the doc around the Phase-1/Phase-2 split we agreed on, fix the codebase references, and add the billing section. Happy to pair on the Phase-1 scope if useful.

[Timofei Larkin (lllamnyp)]

Strong +1 to the Phase-1 / Phase-2 split Andrei Kvapil (@kvaps) laid out. Adding the reasoning underneath it, because I think the split follows from a few distinctions the doc currently collapses, and naming them makes clear what belongs in Phase 1 and why Phase 2 "may not be needed."

1. Two orthogonal tasks, and "BYO for what"

Treat every consumer of identity as choosing its own issuer: the Cozystack platform (console/API) is one consumer; each managed service (managed Kubernetes, managed Grafana, …) is another. cozy is simply the issuer Cozystack ships by default. Pointing any consumer at an external issuer instead is "BYO" — and that is a different task from "use cozystack-provided identity," not a later phase of the same one.

BYO then splits by target, and the two are not the same mechanism:

• BYO for a managed service — the service trusts the external IdP directly. A tenant's Okta can be the issuer for their managed Kubernetes and managed Grafana with no relationship to cozy at all — cozy is never in the path. (This is exactly EKS associate-identity-provider-config: the cluster's apiserver trusts your Okta; your Okta never touches AWS IAM or Identity Center.)
• BYO for Cozystack itself — here, and only here, you federate: the external IdP is brokered into cozy so people sign into the Cozystack console with corporate credentials, and cozy stops being authoritative.

This also dissolves single-org-vs-multi-org as the deciding factor — it isn't one. A single enterprise running Cozystack next to an existing MS AD / LDAP may not want cozy authoritative for anything and will reach for BYO; a multi-org provider may have each tenant's managed clusters trust that tenant's Okta directly and never federate. Org count doesn't pick the model — which consumer, and whether a preexisting IdP should own it does.

For this proposal that means Phase 1 is precisely: let a managed cluster select its issuer — System (cozy + per-cluster client) or CustomConfig (BYO, direct trust) — which is exactly Andrei Kvapil (@kvaps)'s oidc: System | CustomConfig | None. Federating an external IdP into cozy, and Cozystack hosting an IdP for a tenant, are two further and separate things (below). It's also worth not trying to reproduce AWS-style cross-account IAM inside Keycloak — it doesn't carry those primitives, and keeping each consumer's issuer choice simple is the point.

2. The per-tenant hosted realm is a third, distinct thing: managed-IdP-as-a-service

Standing up a realm per tenant that the platform provisions, hosts, and that tenants self-administer means Cozystack is offering managed IdP-as-a-service — an entire product category (Cognito, Auth0, WorkOS exist solely to host an IdP for you). It is neither of the Phase-1 tasks above, and bundling it into the kube-apiserver OIDC PR under-scopes and under-designs it. That's the case for a separate, fully-specified Phase 2.

And the population it's meant to serve doesn't require it. The relevant split is within the tenant's workforce:

• Infra admins / ops — log into cozy, use the Cozystack API/console.
• Developers — need kubectl against the clusters their apps run on (kubectl logs, etc.) but are not allowed into the cloud console.

This is the everyday pattern: only ops has cloud-console access, while a larger slice of the workforce uses the systems hosted on top. Both are Phase-1 cases with no new realm. Ops are already in cozy. Developers either get cozy identities carrying no platform-console RBAC but a per-cluster kubectl grant (presence in the directory ≠ console access — authorization separates them), or the cluster trusts the company's own IdP directly (BYO CustomConfig, cozy not in the path). A per-tenant realm, by contrast, fragments one workforce across two platform-hosted directories (ops in cozy, devs in tenant-<ns>) or forces duplication — when the ops/dev distinction is one of authorization, not of directory.

3. cozy as the System default (reinforcing Andrei Kvapil (@kvaps)'s point 2)

The "flat cozy + per-cluster client" alternative is rejected in the doc on the grounds that it forces tenant end-users into the platform realm. It doesn't: the System path authenticates the platform users already in cozy, and CustomConfig covers everyone who should come from an external IdP. Two consequences:

• The "no IdP" shop cited as the batteries-included case already has a directory — cozy — because its people already log into Cozystack. A per-tenant realm hands those same humans a second, empty directory to populate. The batteries-included win is "your Cozystack login works on your cluster," not "here's a fresh realm to fill."
• The System path needs zero extra identity infrastructure — cozy already exists — so it's the natural zero-config default, with CustomConfig for anyone who already runs their own IdP. An opt-in, per-tenant-provisioned realm can't be that baseline; it has to be stood up first.

4. A design rule for Phase-1 coupling (generalizing the "no auto-provisioned groups" ask)

Dropping the Layer 4 machinery (auto-creating clients/groups + ClusterRoleBindings at cluster-creation) from Phase 1 is right not just on scope but because it couples in the wrong place. The durable rule: couple at provisioning, decouple at ownership. A cluster may request identity wiring — the System | CustomConfig | None selector is exactly that — but it shouldn't own the lifecycle of IdP objects. The doc's own lifecycle section shows the failure mode: Tenant.spec.oidc=false deletes the realm and all its users, so a directory gets destroyed by toggling a Kubernetes auth flag. And in the BYO case — where the tenant brings their own IdP — there's no business creating groups in someone else's realm at all. A users: map bound to a role keeps the cluster owning only disposable, cluster-scoped bindings — never the directory.

---

None of this is a knock on the write-up — like Andrei Kvapil (@kvaps) I'd keep it as the Phase-2 design record. The ask is the same: re-frame around the agreed Phase-1 (the System | CustomConfig | None issuer selector, no auto-provisioned groups), and keep the per-tenant realm as a separately-scoped Phase-2 "managed IdP" product. Happy to pair on the Phase-1 scope.

[mattia-eleuteri]

Sharing how we run multi-tenancy on our side, since the realm-per-tenant question is directly relevant to us.

Our tenancy shape

We run a multi-level tenant tree: tenant-root -> organization -> project. An organization sits directly under tenant-root and is our customer-facing unit; projects are Tenants nested under it. The meaningful identity boundary for us is the organization, not every tenant: projects are sub-units that should share their organization's identity rather than own a separate one.

We already rely on the shared cozy realm and the per-tenant RBAC groups Cozystack provisions (tenant-<name>-{admin,super-admin,use,view}), which map cleanly onto this tree.

On realm-per-tenant: what does it actually buy?

From our seat, a dedicated realm per tenant looks too heavy for the value it returns:

• It directly creates the central-Keycloak load concern raised in open question 1 (N realms vs. 1).
• It pushes realm lifecycle close to a Kubernetes toggle, the coupling the proposal explicitly warns against.
• It's the wrong granularity for a nested model: the boundary belongs at the organization level, which then forces solving "which tenant owns the realm" plus inheritance for every nested project.

If the real driver behind a dedicated realm is BYO-OIDC / per-customer IdP brokering, then Keycloak Organizations are more than sufficient and a much lighter fit:

• One shared cozy realm, one Organization per organization-tenant.
• Per-org domains, membership, and per-org IdP brokering, which is the BYO-OIDC story (open question 2), scoped to the customer instead of per-cluster.
• It honors "decouple at ownership" naturally: the platform owns the realm, the customer owns their Organization (members + brokered IdP). No realm proliferation, no per-tenant Keycloak hosting.

Honest caveat: Organizations are membership + brokering + attributes, not a hard security boundary (clients/roles stay realm-level). So this complements the RBAC-group model rather than replacing it. For our use case that trade-off is fine: we don't need realm-level config isolation, we need per-customer identity + IdP. Organizations is a per-realm feature flag, which we've started exposing upstream in cozystack/cozystack#3031.

One design point for nested tenancy

The single-level parent lookup works for a 2-level tree (project -> organization) but is fragile if nesting goes deeper. Could the realm/Organization owner be an explicit marker on the owning tenant, with inheritance resolving to that ancestor, rather than relying on walk-up depth?

Net: phase 1 (issuer selector + per-cluster client) fits us well. For phase 2, it's worth weighing whether Organizations cover the same goals at a fraction of the operational cost before committing to realm-per-tenant, especially if BYO-OIDC is the main driver.

[Andrei Kvapil (kvaps)]

Thanks Timofei Larkin (@lllamnyp) and mattia-eleuteri — exactly the input we needed, and the proposal has been updated to fold it in (just pushed).

What changed:

• Reframed around the agreed phasing. Phase 1 is now a per-cluster issuer selector oidc: System | CustomConfig | None + per-cluster client + structured AuthenticationConfiguration, with no auto-provisioned Keycloak groups (RBAC via a users: map → cluster-local RoleBindings). The per-tenant realm moved to a deferred Phase 2.
• Timofei Larkin (@lllamnyp)'s distinctions are first-class now: the "BYO for what" split (managed-service direct trust = CustomConfig, cozy not in path; vs federating into cozy), "couple at provisioning, decouple at ownership", and ops-vs-dev being authorization rather than directory.
• mattia-eleuteri's points are in: the identity boundary is the organization, not every tenant; Phase 2 now leads with Keycloak Organizations (one shared cozy realm + per-org brokering, cozystack/cozystack#3031) as the lighter option to evaluate before a per-tenant realm; and an explicit owner marker + inheritance instead of walk-up depth. The per-tenant realm is retained as Phase 2 Option B with its costs written down.
• Also corrected the codebase references (cozystack-basics path + emitted _namespace keys, single-level vs recursive inheritance, cozy group names, KamajiControlPlane apiVersion, --authentication-config vs --oidc-*).

Phase 1 (#3044) is the scope now. Phase 2 is explicitly not a finalized design — the doc records it as a problem space with candidate options (Organizations vs per-tenant realm), to be settled in its own proposal with a dedicated review round. Approving the Phase-1 record; shout if anything still reads wrong.

[Timofei Larkin (lllamnyp)]

mattia-eleuteri good catch on pointing out Organizations, that's exactly the missing AWS-style per-account boundary I didn't consider enough. They look a much lighter fit than realm-per-tenant for the BYO-OIDC driver. +1 on the explicit owner marker for nested trees too.

[mattia-eleuteri]

Re-reviewed. The phased reframing matches how this should be operated, so the direction LGTM from our side. Phase 1 scope (issuer selector + per-cluster client + structured authn-config) and Phase 2 being deferred with Organizations-first both fit. Two things on the RBAC layer I'd like to open up rather than block on.

1. Phase 1 RBAC: reuse the existing group concept rather than a new per-user model?

The users: map binds enumerated identities to per-cluster ClusterRoleBindings. That introduces a new, kubernetes-app-specific RBAC surface, where Cozystack already has a group concept the platform provisions and that automated provisioners drive through the Keycloak Admin API: tenant-<name>-{view,use,admin,super-admin}. An enumerated per-cluster user list is the wrong unit at platform scale; group-based binding is the natural fit and stays consistent with how the rest of Cozystack RBAC already works. There are two ways to keep the group concept for cluster kubectl, and I think it's worth discussing which:

• (a) Reuse the existing tenant-<name>-* groups. Simplest, zero new directory objects. But it couples management-cluster access to workload-cluster kubectl: anyone who can manage the tenant becomes cluster-admin on its clusters, which cuts against the ops/dev split in principle 3 (developers need kubectl but not the console).
• (b) Dedicated cluster-scoped groups, managed the same way (e.g. tenant-<name>-kubernetes-<cluster>-{view,admin}). Keeps the group mechanism + Admin API flow, and preserves the ops/dev separation. Costs a few more group objects.

Neither is obviously right, hence raising it. Our lean is (b) for the ops/dev separation, but the trade-off is real and we'd value your read.

2. Granular RBAC is its own proposal. The fine-grained, per-resource control the users:+role model gestures toward is a different-sized problem from "log a user into a cluster via OIDC", the same reasoning that moved per-tenant realms to Phase 2. Phase 1 shipping a coarse group-to-role (view/admin) split, with matrix/granular RBAC scoped as a separate design proposal, keeps this cut focused.

3. CustomConfig audience composition (open question 1) is the one piece I'd most want nailed down before it ships: a BYO issuer's own aud vs. the per-cluster audience is easy to get subtly wrong, and getting it wrong is a token-replay isolation hole rather than an ergonomics bug. Happy to test the CustomConfig path against a real external issuer once #3044 is up.

Net: 👍 on the direction and Phase 1 scope. The RBAC unit (group vs. enumerated users, and which groups) is the main thing I'd like to converge on.