Kubernetes Multi-Tenancy Patterns — Namespace Isolation vs Virtual Clusters vs Separate Clusters

Multi-tenancy in Kubernetes is not a single problem — it is a spectrum of isolation requirements that vary based on trust boundaries, compliance mandates, and operational capacity. This post examines three patterns for running multiple tenants on Kubernetes infrastructure, with specific attention to where each pattern breaks down and what compensating controls are needed.

Defining Isolation Requirements

Before choosing a multi-tenancy pattern, you need to decompose "isolation" into its constituent dimensions:

Network isolation — Can Tenant A's pods communicate with Tenant B's pods? Can they resolve each other's service DNS entries? Can they reach the Kubernetes API server endpoints of other tenants?

Compute isolation — Can Tenant A's workloads starve Tenant B of CPU or memory? Can a noisy neighbor cause eviction of another tenant's pods? Are kernel vulnerabilities a cross-tenant risk?

Storage isolation — Can Tenant A access Tenant B's persistent volumes? Are storage IOPS shared or guaranteed?

Control plane isolation — Can Tenant A list Tenant B's namespaces, secrets, or custom resources? Can a misconfigured admission webhook from one tenant block deployments for another?

Data isolation — Required by SOC 2, HIPAA, and PCI DSS in many configurations. Not just "can they access it" but "is there a credible path to access it that an auditor would flag?"

Each pattern addresses these dimensions differently, and the right choice depends on which dimensions are non-negotiable for your use case.

Pattern 1: Namespace-Per-Tenant

The most common starting point. Each tenant gets a dedicated namespace, and isolation is enforced through Kubernetes-native primitives.

RBAC Configuration

Create a namespace-scoped admin role for each tenant:

apiVersion: v1
kind: Namespace
metadata:
  name: tenant-acme
  labels:
    tenant: acme
    tier: enterprise
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: tenant-acme
  name: tenant-admin
rules:
  - apiGroups: ["", "apps", "batch"]
    resources: ["*"]
    verbs: ["*"]
  - apiGroups: ["networking.k8s.io"]
    resources: ["networkpolicies"]
    verbs: ["get", "list"]  # read-only — platform team owns policies
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  namespace: tenant-acme
  name: acme-admin-binding
subjects:
  - kind: Group
    name: tenant-acme-admins
    apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: Role
  name: tenant-admin
  apiGroup: rbac.authorization.k8s.io

Use ClusterRole aggregation to maintain a base set of permissions that all tenant roles inherit:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: tenant-base
  labels:
    rbac.authorization.k8s.io/aggregate-to-tenant-admin: "true"
rules:
  - apiGroups: [""]
    resources: ["configmaps", "secrets", "services", "pods", "pods/log"]
    verbs: ["get", "list", "watch", "create", "update", "delete"]

ResourceQuotas and LimitRanges

Without quotas, a single tenant can schedule pods that consume the entire cluster:

apiVersion: v1
kind: ResourceQuota
metadata:
  namespace: tenant-acme
  name: compute-quota
spec:
  hard:
    requests.cpu: "8"
    requests.memory: "16Gi"
    limits.cpu: "16"
    limits.memory: "32Gi"
    persistentvolumeclaims: "10"
    services.loadbalancers: "2"
    pods: "50"
---
apiVersion: v1
kind: LimitRange
metadata:
  namespace: tenant-acme
  name: default-limits
spec:
  limits:
    - default:
        cpu: "500m"
        memory: "512Mi"
      defaultRequest:
        cpu: "100m"
        memory: "128Mi"
      type: Container

NetworkPolicy Isolation

The default Kubernetes network model allows all pod-to-pod communication. You must explicitly deny cross-tenant traffic:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  namespace: tenant-acme
  name: deny-cross-tenant
spec:
  podSelector: {}
  policyTypes:
    - Ingress
    - Egress
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              tenant: acme
  egress:
    - to:
        - namespaceSelector:
            matchLabels:
              tenant: acme
    - to:  # Allow DNS resolution
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: kube-system
      ports:
        - protocol: UDP
          port: 53

Where Namespace Isolation Fails

The fundamental limitation: all tenants share the same Kubernetes API server and the same etcd instance. This means:

1. ✓CRD conflicts — If Tenant A needs CertManager v1.12 and Tenant B needs v1.14, they cannot coexist. CRDs are cluster-scoped.
2. ✓Admission webhooks — A failing webhook in one tenant's namespace can block API operations cluster-wide if the failurePolicy is set to Fail.
3. ✓Node-level attacks — Container escapes give access to other tenants' pods on the same node. Kernel CVEs (e.g., CVE-2022-0185) affect all tenants.
4. ✓API server DoS — One tenant can hammer the API server with list requests on large resource sets, degrading performance for all tenants.
5. ✓Audit trail complexity — All tenant operations appear in the same audit log. Separating them requires post-processing.

For regulated environments, auditors often flag shared control plane access as insufficient isolation. This is where virtual clusters become relevant.

Pattern 2: Virtual Clusters with vCluster

vCluster (by Loft Labs) creates lightweight virtual Kubernetes clusters inside a host cluster. Each virtual cluster has its own API server and etcd (or a backing store like SQLite or embedded etcd), but workloads are scheduled on the host cluster's nodes.

Architecture

A virtual cluster consists of:

• ✓A dedicated API server (k3s, k0s, or vanilla k8s API server)
• ✓A syncer component that maps virtual resources to host namespace resources
• ✓A backing store for the virtual cluster's etcd data

From the tenant's perspective, they have a full Kubernetes cluster. They can install CRDs, create namespaces, and run admission webhooks — all isolated within their virtual cluster.

vCluster Deployment

# vcluster.yaml
apiVersion: infrastructure.cluster.x-k8s.io/v1alpha1
kind: VCluster
metadata:
  name: tenant-acme
  namespace: vcluster-acme
spec:
  controlPlane:
    distro:
      k8s:
        enabled: true
        apiServer:
          extraArgs:
            - --audit-log-path=/var/log/audit.log
            - --audit-policy-file=/etc/kubernetes/audit-policy.yaml
    statefulSet:
      resources:
        limits:
          cpu: "1"
          memory: "2Gi"
        requests:
          cpu: "200m"
          memory: "512Mi"
  networking:
    advanced:
      fallbackHostCluster: false
    replicateServices:
      fromHost:
        - from: ingress-nginx/ingress-nginx-controller
          to: ingress/nginx
  sync:
    toHost:
      persistentVolumes:
        enabled: true
      storageClasses:
        enabled: false  # Use host storage classes
      networkPolicies:
        enabled: true
    fromHost:
      nodes:
        enabled: true
        selector:
          labels:
            tenant-pool: shared

Deploy with:

# Install vCluster CLI
curl -L -o vcluster "https://github.com/loft-sh/vcluster/releases/latest/download/vcluster-linux-amd64"
chmod +x vcluster && sudo mv vcluster /usr/local/bin/

# Create virtual cluster from manifest
vcluster create tenant-acme \
  --namespace vcluster-acme \
  --values vcluster.yaml

# Connect to the virtual cluster
vcluster connect tenant-acme --namespace vcluster-acme

# Verify — this shows the virtual cluster's resources, not the host's
kubectl get namespaces
kubectl get nodes

What vCluster Isolates

• ✓CRDs: Each virtual cluster has its own CRD registry. Tenant A's Istio installation does not conflict with Tenant B's.
• ✓Admission webhooks: Scoped to the virtual cluster. A broken webhook only affects that tenant.
• ✓RBAC: Each tenant can have cluster-admin within their virtual cluster without affecting the host.
• ✓Namespaces: Tenants can create arbitrary namespaces inside their virtual cluster.

What vCluster Does NOT Isolate

• ✓Node kernel: Workloads still share nodes unless you use dedicated node pools.
• ✓Container runtime: A container escape still gives access to the host node.
• ✓Network: Without additional CNI policies, pods from different virtual clusters can communicate at the network layer.

This is why vCluster must be paired with network-level isolation.

Pattern 3: Separate Clusters Per Tenant

Full isolation: each tenant gets a dedicated Kubernetes cluster — separate control plane, separate nodes, separate network.

When This Is the Right Choice

• ✓Regulatory mandate: Some compliance frameworks (FedRAMP High, certain healthcare regulations) require dedicated infrastructure.
• ✓Blast radius: If one tenant's cluster is compromised, others are unaffected.
• ✓Tenant autonomy: Enterprise clients who want to run their own Kubernetes version, install their own operators, and manage their own upgrades.

Operational Overhead

Managing 50 clusters means:

• ✓50 control plane upgrades per Kubernetes release cycle
• ✓50 sets of monitoring, logging, and alerting infrastructure
• ✓50 certificate rotations
• ✓Cross-cluster service mesh for any shared services

Tools that help: Cluster API for lifecycle management, Crossplane for infrastructure provisioning, Rancher or Google Anthos for fleet management.

# Cluster API manifest for provisioning a tenant cluster
apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
  name: tenant-acme
  namespace: cluster-fleet
spec:
  clusterNetwork:
    pods:
      cidrBlocks: ["10.244.0.0/16"]
    services:
      cidrBlocks: ["10.96.0.0/12"]
  controlPlaneRef:
    apiVersion: controlplane.cluster.x-k8s.io/v1beta1
    kind: KubeadmControlPlane
    name: tenant-acme-cp
  infrastructureRef:
    apiVersion: infrastructure.cluster.x-k8s.io/v1beta2
    kind: AWSCluster
    name: tenant-acme

The cost is real: at $73/month for an EKS control plane alone, 50 clusters cost $3,650/month just for control planes — before any worker nodes.

Network Isolation with Cilium

Regardless of the multi-tenancy pattern, network isolation is a hard requirement. Cilium provides eBPF-based network policies that are more expressive than standard Kubernetes NetworkPolicy.

Cilium Network Policy for Tenant Isolation

apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: tenant-isolation
  namespace: vcluster-acme
spec:
  endpointSelector: {}
  ingress:
    - fromEndpoints:
        - matchLabels:
            vcluster.loft.sh/namespace: vcluster-acme
    - fromEntities:
        - kube-system
  egress:
    - toEndpoints:
        - matchLabels:
            vcluster.loft.sh/namespace: vcluster-acme
    - toEntities:
        - kube-system
        - world  # Allow egress to external services
      toPorts:
        - ports:
            - port: "443"
              protocol: TCP
            - port: "53"
              protocol: UDP

For service mesh mTLS, Linkerd provides transparent mTLS between pods without application changes:

# Install Linkerd
linkerd install --crds | kubectl apply -f -
linkerd install | kubectl apply -f -

# Inject the proxy into tenant workloads
kubectl annotate namespace vcluster-acme linkerd.io/inject=enabled

With Linkerd, even if Cilium policies are misconfigured, cross-tenant traffic fails mutual TLS verification because certificates are scoped to the service identity.

Resource Fairness and QoS

Priority Classes

Ensure platform-critical workloads are not evicted to make room for tenant workloads:

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: platform-critical
value: 1000000
globalDefault: false
description: "Platform infrastructure (monitoring, ingress, etc.)"
---
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: tenant-standard
value: 100
globalDefault: true
description: "Default priority for tenant workloads"
---
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: tenant-batch
value: 10
preemptionPolicy: Never
description: "Low-priority batch jobs — will not preempt other pods"

Pod Disruption Budgets

Prevent tenant upgrades from taking down more than one replica at a time:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  namespace: vcluster-acme
  name: api-pdb
spec:
  maxUnavailable: 1
  selector:
    matchLabels:
      app: acme-api

Secrets Management Per Tenant

Each tenant needs secrets that are inaccessible to other tenants and to the platform team where possible.

External Secrets Operator with Per-Tenant Vault Paths

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: tenant-vault
  namespace: vcluster-acme
spec:
  provider:
    vault:
      server: "https://vault.internal:8200"
      path: "secret/tenants/acme"  # Tenant-scoped path
      auth:
        kubernetes:
          mountPath: "kubernetes"
          role: "tenant-acme"
          serviceAccountRef:
            name: "external-secrets-sa"
---
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: acme-db-credentials
  namespace: vcluster-acme
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: tenant-vault
    kind: SecretStore
  target:
    name: db-credentials
  data:
    - secretKey: username
      remoteRef:
        key: database
        property: username
    - secretKey: password
      remoteRef:
        key: database
        property: password

Vault policy for tenant isolation:

# vault-policy-tenant-acme.hcl
path "secret/data/tenants/acme/*" {
  capabilities = ["read", "list"]
}

path "secret/metadata/tenants/acme/*" {
  capabilities = ["read", "list"]
}

# Explicitly deny access to other tenants
path "secret/data/tenants/*" {
  capabilities = ["deny"]
}

Monitoring Per Tenant

Prometheus with Tenant Labels

Use Prometheus relabeling to ensure every metric carries a tenant label:

# prometheus-additional-scrape-config.yaml
- job_name: tenant-workloads
  kubernetes_sd_configs:
    - role: pod
  relabel_configs:
    - source_labels: [__meta_kubernetes_namespace]
      regex: vcluster-(.+)
      target_label: tenant
      replacement: ${1}
    - source_labels: [__meta_kubernetes_pod_label_app]
      target_label: app
  metric_relabel_configs:
    - source_labels: [tenant]
      regex: ""
      action: drop  # Drop metrics without tenant label

For multi-cluster monitoring, deploy Thanos sidecar on each Prometheus instance and aggregate at a central Thanos query endpoint.

Cost Allocation with OpenCost

# Install OpenCost
helm install opencost opencost/opencost \
  --namespace opencost \
  --set opencost.prometheus.internal.serviceName=prometheus-server \
  --set opencost.ui.enabled=true

# Query per-tenant costs
curl -s "http://opencost:9003/allocation/compute?window=7d&aggregate=namespace" \
  | jq '.data[] | to_entries[] | {tenant: .key, cost: .value.totalCost}'

OpenCost tracks CPU, memory, GPU, storage, and network costs at the pod level, aggregated by namespace (which maps to tenant in the namespace-per-tenant model) or by labels.

Decision Matrix

Case Study: SaaS Platform with 50 Enterprise Clients

A B2B SaaS platform serving 50 enterprise clients needed to guarantee data isolation for SOC 2 Type II compliance. Each client stored sensitive financial data, and the auditor required demonstrable isolation at the network and secrets layer.

Evaluation

Namespace isolation was the first approach evaluated. The platform team set up RBAC, NetworkPolicies, and ResourceQuotas per namespace. During the security review, two issues emerged: (1) a CRD conflict between two tenants requiring different versions of a custom operator, and (2) the auditor flagged shared etcd as a risk vector for data leakage. Namespace isolation was ruled out.

Separate clusters were evaluated next. At 50 tenants, this meant 50 EKS clusters. The control plane cost alone was $3,650/month, but the real cost was operational: the platform team of 4 engineers could not manage 50 cluster upgrades per Kubernetes release cycle. Cluster API helped with provisioning, but day-2 operations (certificate rotation, CNI upgrades, monitoring configuration) were still per-cluster. This approach was ruled out as operationally untenable.

Virtual clusters with vCluster provided the right balance. The Stripe Systems engineering team deployed vCluster on a shared EKS cluster with 3 node groups (m6i.xlarge instances). Each tenant received a virtual cluster with its own API server (k3s-based, consuming approximately 256MB RAM and 100m CPU).

Implementation

The vCluster configuration for each tenant:

# values-tenant-template.yaml
controlPlane:
  distro:
    k8s:
      enabled: true
      apiServer:
        extraArgs:
          - --audit-log-path=/var/log/kubernetes/audit.log
          - --audit-log-maxage=30
  statefulSet:
    resources:
      limits:
        cpu: "500m"
        memory: "1Gi"
      requests:
        cpu: "100m"
        memory: "256Mi"
    persistence:
      size: 5Gi
networking:
  advanced:
    fallbackHostCluster: false
sync:
  toHost:
    persistentVolumes:
      enabled: true
    networkPolicies:
      enabled: true
  fromHost:
    nodes:
      enabled: true
      selector:
        labels:
          node-pool: tenant-workloads

Provisioning a new tenant:

#!/bin/bash
TENANT_NAME=$1
NAMESPACE="vc-${TENANT_NAME}"

# Create host namespace
kubectl create namespace "${NAMESPACE}"
kubectl label namespace "${NAMESPACE}" tenant="${TENANT_NAME}"

# Deploy vCluster
vcluster create "${TENANT_NAME}" \
  --namespace "${NAMESPACE}" \
  --values values-tenant-template.yaml

# Apply Cilium network policy
cat <<EOF | kubectl apply -f -
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: tenant-isolation
  namespace: ${NAMESPACE}
spec:
  endpointSelector: {}
  ingress:
    - fromEndpoints:
        - matchLabels:
            io.kubernetes.pod.namespace: ${NAMESPACE}
    - fromEntities:
        - kube-system
  egress:
    - toEndpoints:
        - matchLabels:
            io.kubernetes.pod.namespace: ${NAMESPACE}
    - toEntities:
        - kube-system
    - toCIDR:
        - 0.0.0.0/0
      toPorts:
        - ports:
            - port: "443"
              protocol: TCP
            - port: "53"
              protocol: UDP
EOF

# Configure External Secrets for tenant
cat <<EOF | kubectl apply -f -
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: tenant-vault
  namespace: ${NAMESPACE}
spec:
  provider:
    vault:
      server: "https://vault.internal:8200"
      path: "secret/tenants/${TENANT_NAME}"
      auth:
        kubernetes:
          mountPath: "kubernetes"
          role: "tenant-${TENANT_NAME}"
          serviceAccountRef:
            name: "external-secrets-sa"
EOF

echo "Tenant ${TENANT_NAME} provisioned successfully"

Cost Comparison

The vCluster approach cost 27% more than namespace isolation but passed the SOC 2 audit. It cost 69% less than separate clusters while providing equivalent isolation at the control plane and network layers.

Results

After 6 months in production:

• ✓50 virtual clusters running on 12 m6i.xlarge nodes
• ✓Zero cross-tenant security incidents
• ✓SOC 2 Type II audit passed with no findings related to tenant isolation
• ✓Tenant onboarding automated to under 10 minutes (script above)
• ✓P99 API server latency per virtual cluster: 45ms (well within SLA)

The architecture satisfied auditors because each tenant's API server, secrets, and network traffic were provably isolated, while keeping operational overhead manageable for a small platform team. The combination of vCluster for control plane isolation and Cilium for network isolation addressed every dimension of the isolation requirements without the cost explosion of dedicated clusters.