Shipping a New AI Service with GitOps: Kustomize Overlays + ExternalSecrets

When I first started building the GraphRAG infrastructure, I committed an API key to Git in an early prototype and got an automated email from GitHub secret scanning within minutes. That scare — the realization that secrets can leak in seconds and persist in Git history forever — led to adopting ExternalSecrets Operator and a strict “no secrets in Git, ever” policy that this GitOps setup enforces by design.

Adding a new microservice to a GitOps-managed Kubernetes cluster isn’t just writing a Deployment manifest. You need environment-specific configuration, secret injection, health probes, and a promotion path from staging to production. This article walks through the complete process using Kustomize overlays and the ExternalSecrets Operator.

Directory Structure

A clean base/overlay layout for a new service:

apps/
  my-ai-service/
    base/
      kustomization.yaml
      deployment.yaml
      service.yaml
      configmap.yaml
      externalsecret.yaml
    overlays/
      staging/
        kustomization.yaml
        patch-replicas.yaml
      production/
        kustomization.yaml
        patch-replicas.yaml
        patch-resources.yaml

Base Manifests

Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-ai-service
  labels:
    app: my-ai-service
spec:
  replicas: 1
  selector:
    matchLabels:
      app: my-ai-service
  template:
    metadata:
      labels:
        app: my-ai-service
    spec:
      containers:
        - name: app
          image: registry.example.com/my-ai-service:latest
          ports:
            - containerPort: 8080
          envFrom:
            - configMapRef:
                name: my-ai-service-config
            - secretRef:
                name: my-ai-service-secrets
          readinessProbe:
            httpGet:
              path: /health/ready
              port: 8080
            initialDelaySeconds: 10
            periodSeconds: 15
          livenessProbe:
            httpGet:
              path: /health/live
              port: 8080
            initialDelaySeconds: 30
            periodSeconds: 30
          resources:
            requests:
              cpu: 100m
              memory: 256Mi
            limits:
              memory: 512Mi

ExternalSecret

The ExternalSecret tells the ExternalSecrets Operator to pull secrets from your vault (Infisical, AWS Secrets Manager, etc.) and create a Kubernetes Secret:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: my-ai-service-secrets
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: infisical-store
    kind: ClusterSecretStore
  target:
    name: my-ai-service-secrets
    creationPolicy: Owner
  data:
    - secretKey: DATABASE_URL
      remoteRef:
        key: MY_AI_SERVICE_DATABASE_URL
    - secretKey: API_KEY
      remoteRef:
        key: MY_AI_SERVICE_API_KEY

Base Kustomization

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - deployment.yaml
  - service.yaml
  - configmap.yaml
  - externalsecret.yaml
commonLabels:
  app.kubernetes.io/name: my-ai-service
  app.kubernetes.io/part-of: platform

Staging Overlay

The staging overlay increases replicas and uses a staging-specific namespace:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: ai-staging
resources:
  - ../../base
patches:
  - path: patch-replicas.yaml
images:
  - name: registry.example.com/my-ai-service
    newTag: latest

Production Overlay

Production gets more replicas, higher resource limits, and an immutable image tag:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: ai-production
resources:
  - ../../base
patches:
  - path: patch-replicas.yaml
  - path: patch-resources.yaml
images:
  - name: registry.example.com/my-ai-service
    newTag: v1.2.3  # Immutable tag for production

Deployment Flow

1. Create base manifests     --> PR to infra repo
2. Flux detects change       --> deploys to staging
3. Smoke test passes         --> trigger production workflow
4. Workflow updates prod tag --> PR to infra repo
5. Flux detects change       --> deploys to production

Conclusion

Building this GitOps pipeline was an exercise in trust — trusting that the Git repository is the single source of truth, trusting that Flux will reconcile drift, and trusting that ExternalSecrets will keep credentials safe. Each of those trust layers was earned through a failure: a leaked API key, a corrupted production index from shared config, and a manual kubectl edit that caused an outage. The system we have now is self-healing, auditable, and reproducible, which means I sleep better when deployments happen at 2 AM via automated pipelines.

Key Takeaways

• Base/overlay structure keeps environment differences explicit and minimal
• ExternalSecrets syncs vault secrets into Kubernetes — no inline credentials
• Immutable tags in production, latest in staging for fast iteration
• Always include health probes — Flux won’t promote unhealthy deployments
• FQDN for service discovery: svc.namespace.svc.cluster.local

Next Steps

• Kubernetes Cluster Setup for AI Workloads — The cluster architecture this GitOps pipeline deploys into
• Integrating Docling OCR in a .NET Document Pipeline — An example AI service deployed using this exact pattern

Further Reading