Building a Production-Grade CI/CD Pipeline for a Monorepo (GitHub Actions + Docker + Kubernetes)

Monorepos consolidate multiple services, shared libraries, and frontend applications into a single repository. This brings benefits — atomic cross-service changes, shared tooling, simplified dependency management — but it makes CI/CD significantly harder. A naive approach rebuilds and redeploys everything on every commit. This post covers how to build a CI/CD pipeline that only builds what changed, caches aggressively, and deploys safely to Kubernetes.

Why Monorepo, and the CI/CD Challenges It Creates

A monorepo works well when:

• ✓Multiple services share libraries (e.g., a common auth module, shared protobuf definitions)
• ✓Teams need to make atomic changes across service boundaries (e.g., updating an API contract and its consumer in one commit)
• ✓You want unified tooling (one ESLint config, one Dockerfile template, one CI pipeline)

The CI/CD challenges:

1. ✓Build scope: Which services need rebuilding when libs/auth/index.ts changes? Every service that imports it.
2. ✓Test scope: Which integration tests should run? Only those covering affected services.
3. ✓Pipeline time: Without affected detection, a 12-service monorepo runs all 12 builds on every commit.
4. ✓Caching: Docker layer caches are invalidated differently per service. npm/pip caches are shared but can conflict.
5. ✓Deployment ordering: If Service A depends on Service B's new API, deploy B first.

Repository Structure

monorepo/
├── services/
│   ├── payment-api/
│   │   ├── src/
│   │   ├── tests/
│   │   ├── Dockerfile
│   │   ├── package.json
│   │   └── helm/
│   │       ├── Chart.yaml
│   │       ├── values.yaml
│   │       ├── values-dev.yaml
│   │       ├── values-staging.yaml
│   │       └── values-prod.yaml
│   ├── user-api/
│   ├── notification-service/
│   ├── order-api/
│   ├── inventory-api/
│   ├── search-service/
│   ├── analytics-worker/
│   └── gateway/
├── libs/
│   ├── auth/              # Shared authentication library
│   │   ├── src/
│   │   ├── package.json
│   │   └── tsconfig.json
│   ├── database/           # Shared database utilities
│   └── logging/            # Structured logging library
├── frontend/
│   ├── src/
│   ├── Dockerfile
│   └── package.json
├── .github/
│   └── workflows/
│       ├── ci.yaml
│       └── deploy.yaml
├── scripts/
│   └── affected.sh
├── package.json            # Root workspace config
└── turbo.json              # Turborepo configuration

Affected Detection

The core of monorepo CI/CD is determining which services changed. This requires understanding the dependency graph.

Dependency Map

Define which services depend on which libraries:

{
  "payment-api": ["libs/auth", "libs/database", "libs/logging"],
  "user-api": ["libs/auth", "libs/database", "libs/logging"],
  "notification-service": ["libs/logging"],
  "order-api": ["libs/auth", "libs/database", "libs/logging"],
  "inventory-api": ["libs/database", "libs/logging"],
  "search-service": ["libs/logging"],
  "analytics-worker": ["libs/database", "libs/logging"],
  "gateway": ["libs/auth", "libs/logging"],
  "frontend": []
}

Affected Detection Script

#!/bin/bash
# scripts/affected.sh
# Determines which services are affected by changes in the current commit

set -euo pipefail

BASE_REF=${1:-"origin/main"}
HEAD_REF=${2:-"HEAD"}

# Get list of changed files
CHANGED_FILES=$(git diff --name-only "$BASE_REF"..."$HEAD_REF")

# Dependency map (service -> space-separated dependency paths)
declare -A DEPS
DEPS[payment-api]="libs/auth libs/database libs/logging"
DEPS[user-api]="libs/auth libs/database libs/logging"
DEPS[notification-service]="libs/logging"
DEPS[order-api]="libs/auth libs/database libs/logging"
DEPS[inventory-api]="libs/database libs/logging"
DEPS[search-service]="libs/logging"
DEPS[analytics-worker]="libs/database libs/logging"
DEPS[gateway]="libs/auth libs/logging"
DEPS[frontend]=""

AFFECTED=()

for SERVICE in "${!DEPS[@]}"; do
  SERVICE_AFFECTED=false

  # Check if files in the service directory changed
  if echo "$CHANGED_FILES" | grep -q "^services/${SERVICE}/"; then
    SERVICE_AFFECTED=true
  fi

  # Check if the frontend directory changed
  if [[ "$SERVICE" == "frontend" ]] && echo "$CHANGED_FILES" | grep -q "^frontend/"; then
    SERVICE_AFFECTED=true
  fi

  # Check if any dependency changed
  for DEP in ${DEPS[$SERVICE]}; do
    if echo "$CHANGED_FILES" | grep -q "^${DEP}/"; then
      SERVICE_AFFECTED=true
      break
    fi
  done

  if $SERVICE_AFFECTED; then
    AFFECTED+=("$SERVICE")
  fi
done

# Check if CI config itself changed (rebuild everything)
if echo "$CHANGED_FILES" | grep -q "^\.github/workflows/\|^scripts/"; then
  echo '["payment-api","user-api","notification-service","order-api","inventory-api","search-service","analytics-worker","gateway","frontend"]'
  exit 0
fi

# Output as JSON array for GitHub Actions matrix
printf '%s\n' "${AFFECTED[@]}" | jq -R . | jq -s -c .

Usage:

chmod +x scripts/affected.sh
./scripts/affected.sh origin/main HEAD
# Output: ["payment-api","user-api","gateway"]

GitHub Actions Workflow

CI Pipeline

# .github/workflows/ci.yaml
name: CI Pipeline

on:
  pull_request:
    branches: [main]
  push:
    branches: [main]

permissions:
  contents: read
  packages: write
  pull-requests: write

env:
  REGISTRY: ghcr.io
  IMAGE_PREFIX: ghcr.io/${{ github.repository_owner }}

jobs:
  detect-changes:
    runs-on: ubuntu-latest
    outputs:
      affected: ${{ steps.affected.outputs.services }}
      has_changes: ${{ steps.affected.outputs.has_changes }}
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0  # Full history for diff

      - name: Detect affected services
        id: affected
        run: |
          if [ "${{ github.event_name }}" = "pull_request" ]; then
            AFFECTED=$(./scripts/affected.sh origin/${{ github.base_ref }} HEAD)
          else
            AFFECTED=$(./scripts/affected.sh HEAD~1 HEAD)
          fi
          echo "services=${AFFECTED}" >> $GITHUB_OUTPUT
          if [ "$AFFECTED" = "[]" ]; then
            echo "has_changes=false" >> $GITHUB_OUTPUT
          else
            echo "has_changes=true" >> $GITHUB_OUTPUT
          fi
          echo "Affected services: ${AFFECTED}"

  lint-and-typecheck:
    needs: detect-changes
    if: needs.detect-changes.outputs.has_changes == 'true'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-node@v4
        with:
          node-version: '20'
          cache: 'npm'

      - run: npm ci

      - name: Lint affected services
        run: npx turbo lint --filter='...[${{ github.event.pull_request.base.sha || 'HEAD~1' }}]'

      - name: Type check affected services
        run: npx turbo typecheck --filter='...[${{ github.event.pull_request.base.sha || 'HEAD~1' }}]'

  test:
    needs: detect-changes
    if: needs.detect-changes.outputs.has_changes == 'true'
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        service: ${{ fromJson(needs.detect-changes.outputs.affected) }}
    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-node@v4
        with:
          node-version: '20'
          cache: 'npm'

      - run: npm ci

      - name: Run unit tests for ${{ matrix.service }}
        run: |
          if [ "${{ matrix.service }}" = "frontend" ]; then
            cd frontend && npm test -- --coverage
          else
            cd services/${{ matrix.service }} && npm test -- --coverage
          fi

      - name: Upload coverage
        uses: actions/upload-artifact@v4
        with:
          name: coverage-${{ matrix.service }}
          path: |
            services/${{ matrix.service }}/coverage/
            frontend/coverage/
          retention-days: 7

  build-and-push:
    needs: [detect-changes, test, lint-and-typecheck]
    if: github.ref == 'refs/heads/main' && needs.detect-changes.outputs.has_changes == 'true'
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        service: ${{ fromJson(needs.detect-changes.outputs.affected) }}
    steps:
      - uses: actions/checkout@v4

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3

      - name: Log in to GitHub Container Registry
        uses: docker/login-action@v3
        with:
          registry: ${{ env.REGISTRY }}
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}

      - name: Generate image metadata
        id: meta
        run: |
          SHA_SHORT=$(echo "${{ github.sha }}" | cut -c1-7)
          echo "sha_tag=${SHA_SHORT}" >> $GITHUB_OUTPUT
          echo "full_image=${{ env.IMAGE_PREFIX }}/${{ matrix.service }}" >> $GITHUB_OUTPUT

      - name: Build and push
        uses: docker/build-push-action@v5
        with:
          context: .
          file: ${{ matrix.service == 'frontend' && 'frontend/Dockerfile' || format('services/{0}/Dockerfile', matrix.service) }}
          push: true
          tags: |
            ${{ steps.meta.outputs.full_image }}:${{ steps.meta.outputs.sha_tag }}
            ${{ steps.meta.outputs.full_image }}:latest
          cache-from: type=gha,scope=${{ matrix.service }}
          cache-to: type=gha,scope=${{ matrix.service }},mode=max
          build-args: |
            SERVICE_NAME=${{ matrix.service }}

  security-scan:
    needs: [detect-changes, build-and-push]
    if: github.ref == 'refs/heads/main' && needs.detect-changes.outputs.has_changes == 'true'
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false
      matrix:
        service: ${{ fromJson(needs.detect-changes.outputs.affected) }}
    steps:
      - name: Run Trivy vulnerability scan
        uses: aquasecurity/trivy-action@master
        with:
          image-ref: ${{ env.IMAGE_PREFIX }}/${{ matrix.service }}:latest
          format: 'sarif'
          output: 'trivy-results.sarif'
          severity: 'CRITICAL,HIGH'
          exit-code: '1'

      - name: Upload Trivy scan results
        uses: github/codeql-action/upload-sarif@v3
        if: always()
        with:
          sarif_file: 'trivy-results.sarif'

  generate-sbom:
    needs: [detect-changes, build-and-push]
    if: github.ref == 'refs/heads/main' && needs.detect-changes.outputs.has_changes == 'true'
    runs-on: ubuntu-latest
    strategy:
      matrix:
        service: ${{ fromJson(needs.detect-changes.outputs.affected) }}
    steps:
      - name: Generate SBOM with Syft
        uses: anchore/sbom-action@v0
        with:
          image: ${{ env.IMAGE_PREFIX }}/${{ matrix.service }}:latest
          format: spdx-json
          output-file: sbom-${{ matrix.service }}.spdx.json

      - name: Upload SBOM
        uses: actions/upload-artifact@v4
        with:
          name: sbom-${{ matrix.service }}
          path: sbom-${{ matrix.service }}.spdx.json

Key Design Decisions

fetch-depth: 0: Required for git diff between branches. Without full history, affected detection fails.

fail-fast: false: If payment-api tests fail, user-api tests should still run. Each service is independent.

GHA cache for Docker: cache-from: type=gha uses GitHub Actions cache for Docker layers. Each service gets its own cache scope, preventing cross-service cache pollution.

Trivy with exit-code 1: The pipeline fails on critical/high CVEs. This is a gate — images with known critical vulnerabilities do not reach production.

Multi-Stage Dockerfiles

A well-structured Dockerfile minimizes image size and build time:

# services/payment-api/Dockerfile

# Stage 1: Install dependencies (cached aggressively)
FROM node:20-alpine AS deps
WORKDIR /app
COPY package.json package-lock.json ./
COPY services/payment-api/package.json ./services/payment-api/
COPY libs/auth/package.json ./libs/auth/
COPY libs/database/package.json ./libs/database/
COPY libs/logging/package.json ./libs/logging/
RUN npm ci --workspace=services/payment-api --include-workspace-root

# Stage 2: Build (depends on source code changes)
FROM deps AS builder
COPY tsconfig.json ./
COPY libs/ ./libs/
COPY services/payment-api/ ./services/payment-api/
RUN npm run build --workspace=services/payment-api

# Stage 3: Production image (minimal runtime)
FROM gcr.io/distroless/nodejs20-debian12 AS runtime
WORKDIR /app
COPY --from=builder /app/services/payment-api/dist ./dist
COPY --from=builder /app/node_modules ./node_modules
COPY --from=builder /app/services/payment-api/node_modules ./services/payment-api/node_modules

EXPOSE 3000
USER nonroot:nonroot
CMD ["dist/index.js"]

Layer ordering matters: package.json files are copied before source code. When only source code changes, npm install is cached. The dependency layer (often 200MB+) is rebuilt only when package.json changes.

Distroless runtime: The final image contains only Node.js and the application. No shell, no package manager, no debugging tools. This reduces the attack surface dramatically — Trivy scans typically show 0 critical CVEs for distroless images.

Docker Image Tagging Strategy

# Tags generated per build:
# 1. Git SHA (short) — immutable, traceable to exact commit
ghcr.io/org/payment-api:abc123f

# 2. latest — mutable, points to most recent build
ghcr.io/org/payment-api:latest

# 3. Semantic version (for releases) — set via git tag
ghcr.io/org/payment-api:v2.3.1

Never deploy latest to production. Use the Git SHA tag for deterministic deployments. The latest tag is useful for development environments that should always run the newest build.

Integration Testing in a Monorepo

Unit tests run per-service, but integration tests often span service boundaries. The challenge: when user-api changes, should integration tests for order-api (which calls user-api) also run?

Reverse Dependency Graph

The affected detection script identifies forward dependencies (which services use a changed library). Integration tests require the reverse: which services depend on the changed service.

# Reverse dependency map — if service X changes, test these consumers
declare -A REVERSE_DEPS
REVERSE_DEPS[user-api]="order-api gateway"
REVERSE_DEPS[payment-api]="order-api"
REVERSE_DEPS[inventory-api]="order-api search-service"

When user-api changes, run integration tests for order-api and gateway in addition to user-api's own tests. This catches breaking changes in service contracts before they reach production.

Test Execution Strategy

  integration-test:
    needs: [detect-changes, test]
    if: github.ref == 'refs/heads/main'
    runs-on: ubuntu-latest
    services:
      postgres:
        image: postgres:16-alpine
        env:
          POSTGRES_DB: test
          POSTGRES_PASSWORD: test
        ports:
          - 5432:5432
      redis:
        image: redis:7-alpine
        ports:
          - 6379:6379
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: '20'
          cache: 'npm'
      - run: npm ci
      - name: Run integration tests
        env:
          DATABASE_URL: postgres://postgres:test@localhost:5432/test
          REDIS_URL: redis://localhost:6379
        run: |
          AFFECTED='${{ needs.detect-changes.outputs.affected }}'
          for SERVICE in $(echo "$AFFECTED" | jq -r '.[]'); do
            echo "Running integration tests for ${SERVICE}"
            npm run test:integration --workspace="services/${SERVICE}" || exit 1
          done

GitHub Actions service containers spin up Postgres and Redis for the integration test job. Each service's integration tests run against these shared dependencies, validating database queries, cache interactions, and inter-service API calls.

Helm Charts Per Service

Each service has its own Helm chart with per-environment values:

# services/payment-api/helm/values.yaml (defaults)
replicaCount: 1
image:
  repository: ghcr.io/org/payment-api
  tag: "latest"
  pullPolicy: IfNotPresent

resources:
  requests:
    cpu: 100m
    memory: 128Mi
  limits:
    cpu: 500m
    memory: 512Mi

service:
  type: ClusterIP
  port: 3000

env:
  NODE_ENV: production
  LOG_LEVEL: info

ingress:
  enabled: false

# services/payment-api/helm/values-prod.yaml
replicaCount: 3

image:
  tag: ""  # Set by CI/CD pipeline

resources:
  requests:
    cpu: 500m
    memory: 512Mi
  limits:
    cpu: "1"
    memory: "1Gi"

ingress:
  enabled: true
  className: nginx
  hosts:
    - host: payment-api.example.com
      paths:
        - path: /
          pathType: Prefix

autoscaling:
  enabled: true
  minReplicas: 3
  maxReplicas: 10
  targetCPUUtilizationPercentage: 70

Deployment with ArgoCD Image Updater

Rather than committing image tags back to Git (which creates noise), use ArgoCD Image Updater to watch the container registry and update deployments automatically:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: payment-api
  namespace: argocd
  annotations:
    argocd-image-updater.argoproj.io/image-list: >-
      payment=ghcr.io/org/payment-api
    argocd-image-updater.argoproj.io/payment.update-strategy: newest-build
    argocd-image-updater.argoproj.io/payment.allow-tags: regexp:^[a-f0-9]{7}$
    argocd-image-updater.argoproj.io/write-back-method: git
    argocd-image-updater.argoproj.io/write-back-target: kustomization
spec:
  source:
    repoURL: https://github.com/org/monorepo.git
    path: services/payment-api/helm
    helm:
      valueFiles:
        - values.yaml
        - values-prod.yaml
  destination:
    server: https://kubernetes.default.svc
    namespace: payment

The allow-tags regex ensures only Git SHA tags (7 hex characters) are considered — not latest, not arbitrary strings.

Pipeline Performance Optimization

Self-Hosted Runners with Spot Instances

GitHub-hosted runners are convenient but slow for Docker builds (no persistent cache, cold Docker daemon). Self-hosted runners on spot instances provide:

• ✓Persistent Docker layer cache on instance storage
• ✓Pre-pulled base images
• ✓Faster network to private container registries

# Runner deployment on Kubernetes using actions-runner-controller
apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerDeployment
metadata:
  name: monorepo-runners
spec:
  replicas: 4
  template:
    spec:
      repository: org/monorepo
      labels:
        - self-hosted
        - linux
        - monorepo
      dockerEnabled: true
      resources:
        limits:
          cpu: "4"
          memory: "8Gi"
      volumeMounts:
        - name: docker-cache
          mountPath: /var/lib/docker
      volumes:
        - name: docker-cache
          hostPath:
            path: /mnt/docker-cache
            type: DirectoryOrCreate

Caching Strategy Summary

Case Study: 8-Service Monorepo Transformation

A monorepo containing 8 Node.js microservices, 3 shared libraries (auth, database, logging), and a React frontend was running CI/CD on GitHub Actions with a single workflow that built everything on every commit.

Before

• ✓Pipeline structure: Single job, sequential builds of all 9 artifacts
• ✓Average pipeline time: 45 minutes
• ✓Docker builds: No layer caching, node:18 base image (950MB), npm install including devDependencies in runtime image
• ✓Testing: All tests run on every commit, including slow integration tests
• ✓Security scanning: None
• ✓Deployment: Manual kubectl apply by the lead developer

Changes Implemented by Stripe Systems

1. Affected detection: The bash script above was implemented, with the dependency map maintained alongside package.json workspace configuration. Average commits affected 1-2 services.

2. Matrix builds: Affected services build in parallel. With 4 runners, 4 services build simultaneously.

3. Docker optimization:

• ✓Multi-stage builds (deps → build → runtime)
• ✓Distroless base images (950MB → 89MB per image)
• ✓BuildKit GHA caching (rebuilds only changed layers)
• ✓Layer ordering (package.json before source code)

4. Test segmentation:

• ✓Unit tests: Run on affected services only (matrix strategy)
• ✓Integration tests: Run only when the service or its dependencies change
• ✓E2E tests: Run on main branch pushes only (not on PRs for speed)

5. Security gates: Trivy scan on every image, SBOM generation with Syft, secret scanning with gitleaks.

6. ArgoCD deployment: Image Updater watches GHCR and deploys new images automatically. No manual intervention.

Pipeline Timing Results

GitHub Actions Workflow (Simplified Final Version)

The final CI workflow:

name: CI
on:
  pull_request:
    branches: [main]
  push:
    branches: [main]

jobs:
  detect:
    runs-on: ubuntu-latest
    outputs:
      matrix: ${{ steps.affected.outputs.services }}
      any: ${{ steps.affected.outputs.has_changes }}
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
      - id: affected
        run: |
          SERVICES=$(./scripts/affected.sh origin/main HEAD)
          echo "services=${SERVICES}" >> $GITHUB_OUTPUT
          [ "$SERVICES" != "[]" ] && echo "has_changes=true" >> $GITHUB_OUTPUT || echo "has_changes=false" >> $GITHUB_OUTPUT

  ci:
    needs: detect
    if: needs.detect.outputs.any == 'true'
    runs-on: self-hosted
    strategy:
      fail-fast: false
      matrix:
        service: ${{ fromJson(needs.detect.outputs.matrix) }}
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: '20'
          cache: 'npm'
      - run: npm ci
      - run: npm test --workspace=${{ matrix.service == 'frontend' && 'frontend' || format('services/{0}', matrix.service) }}
      - uses: docker/setup-buildx-action@v3
      - uses: docker/build-push-action@v5
        if: github.ref == 'refs/heads/main'
        with:
          context: .
          file: ${{ matrix.service == 'frontend' && 'frontend/Dockerfile' || format('services/{0}/Dockerfile', matrix.service) }}
          push: true
          tags: ghcr.io/org/${{ matrix.service }}:${{ github.sha }}
          cache-from: type=gha,scope=${{ matrix.service }}
          cache-to: type=gha,scope=${{ matrix.service }},mode=max

The Helm values structure allowed environment-specific overrides without duplicating entire chart configurations. Combined with ArgoCD Image Updater, the pipeline achieved continuous deployment: a merge to main triggered the build, the image was pushed, ArgoCD detected the new tag, and the deployment rolled out — all without human intervention.

The total effort to implement these changes was approximately 3 weeks, including migration of all Dockerfiles, CI workflows, and Helm charts. The ongoing maintenance burden is low because the affected detection script and dependency map are the only monorepo-specific components — everything else uses standard tooling.