Remote VS Code Development on Kubernetes with Dev Containers

When I first configured remote VS Code development against our k3s cluster, I expected it to be a quick afternoon project. Instead, I spent three days wrestling with intermittent disconnections, volume permission errors, and figuring out the right container lifecycle strategy. The biggest surprise was that running an SSH daemon inside the pod was completely unnecessary — sleep infinity combined with kubectl exec turned out to be both simpler and more secure. Once I got persistent storage working with a StatefulSet, the workflow became remarkably smooth, and I have not looked back at local-only development since.

Developing against a Kubernetes cluster from a local machine means port-forwarding, VPN tunnels, and environment variable gymnastics. An alternative: run your IDE’s backend inside the cluster as a persistent pod. VS Code connects remotely via the Kubernetes extension, and your code has native access to every cluster service.

Why Remote Development?

flowchart LR
    subgraph Laptop
        VSCODE["VS Code\n(UI only)"]
    end

    VSCODE -->|"kubectl exec"| POD

    subgraph "K8s Cluster"
        POD["Dev Container Pod\n(StatefulSet)"]
        PVC[("PVC\nWorkspace Storage")]
        ES["ExternalSecrets\n(Credentials)"]]
        POD --- PVC
        POD --- ES

        POD -.->|"native DNS"| PG["PostgreSQL"]
        POD -.->|"native DNS"| NATS["NATS"]
        POD -.->|"native DNS"| MINIO["MinIO"]
        POD -.->|"native DNS"| OLLAMA["Ollama"]
        POD -.->|"native DNS"| QDRANT["Qdrant"]
    end

    style VSCODE fill:#1a2744,stroke:#6366f1,color:#e2e8f0
    style POD fill:#1a2744,stroke:#22c55e,color:#e2e8f0

Pod Specification

A dev container pod with persistent storage and access to cluster services:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: dev-container
  namespace: dev
spec:
  replicas: 1
  selector:
    matchLabels:
      app: dev-container
  template:
    metadata:
      labels:
        app: dev-container
    spec:
      containers:
        - name: dev
          image: mcr.microsoft.com/dotnet/sdk:10.0
          command: ["sleep", "infinity"]
          volumeMounts:
            - name: workspace
              mountPath: /workspace
          envFrom:
            - secretRef:
                name: dev-secrets
          resources:
            requests:
              cpu: "2"
              memory: 4Gi
            limits:
              cpu: "4"
              memory: 8Gi
  volumeClaimTemplates:
    - metadata:
        name: workspace
      spec:
        accessModes: ["ReadWriteOnce"]
        resources:
          requests:
            storage: 20Gi

VS Code Connection

Option 1: Kubernetes Extension

1. Install the Kubernetes extension
2. Right-click the pod and select “Attach Visual Studio Code”
3. VS Code installs its server binary in the container and opens a remote window

Option 2: Remote SSH with kubectl exec

For a more stable connection, use the VS Code Remote - SSH extension with a ProxyCommand:

# ~/.ssh/config
Host dev-container
  ProxyCommand kubectl exec -i -n dev dev-container-0 -c dev -- sh -c "cat"
  User root
  StrictHostKeyChecking no

Workspace Sync

Keep your code in sync between local and remote using rsync or git:

# Push local changes to the pod
rsync -avz --exclude='bin/' --exclude='obj/' \
  ./src/ dev-container-0:/workspace/src/

# Or use git — clone inside the pod
kubectl exec -n dev dev-container-0 -c dev -- \
  git clone https://github.com/org/repo.git /workspace/repo

VS Code Tasks

Define tasks that run inside the container:

{
  "version": "2.0.0",
  "tasks": [
    {
      "label": "Build",
      "type": "shell",
      "command": "kubectl exec -n dev dev-container-0 -c dev -- dotnet build /workspace/src/MyApp.sln"
    },
    {
      "label": "Run Tests",
      "type": "shell",
      "command": "kubectl exec -n dev dev-container-0 -c dev -- dotnet test /workspace/src/MyApp.sln"
    },
    {
      "label": "Start API",
      "type": "shell",
      "command": "kubectl exec -n dev dev-container-0 -c dev -- dotnet run --project /workspace/src/MyApp.Api/MyApp.Api.csproj",
      "isBackground": true
    }
  ]
}

Service Access

Inside the dev pod, all cluster services are reachable via DNS:

# From inside the dev container:
curl http://ollama.ai.svc.cluster.local:11434/api/tags
psql -h postgres-rw.data-layer.svc.cluster.local -U myapp
nats sub -s nats://nats.messaging.svc.cluster.local:4222 ">"

No port forwarding, no VPN, no environment variable juggling.

Secrets

Mount secrets from your vault using ExternalSecrets, just like production:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: dev-secrets
  namespace: dev
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: infisical-store
    kind: ClusterSecretStore
  target:
    name: dev-secrets
  data:
    - secretKey: ConnectionStrings__PostgresDB
      remoteRef:
        key: DEV_POSTGRES_CONNECTION_STRING

Key Takeaways

• Run your dev environment inside the cluster for native service access
• Use a StatefulSet with persistent storage so your workspace survives restarts
• Connect via VS Code Kubernetes extension or Remote SSH with kubectl exec
• Mount secrets with ExternalSecrets — same mechanism as staging/production
• Define VS Code tasks that kubectl exec into the pod for build/test/run

Moving to an in-cluster development workflow was one of the most impactful changes I made to my homelab setup. The immediate payoff is eliminating the “works on my machine” problem entirely — when your IDE runs in the same network as your services, the gap between development and production shrinks dramatically. The initial setup takes effort, but the daily time savings in not managing port-forwards and VPN tunnels have been well worth it.

Next Steps

• Add GPU passthrough to the dev container for testing ML workloads locally
• Experiment with DevPod as an alternative to manual StatefulSet management
• Configure VS Code profiles to auto-select the right extensions and settings for remote sessions
• Set up pre-commit hooks that run inside the container for consistent formatting

Related Reading

• VS Code Remote Development
• Kubernetes StatefulSets
• Dev Containers Specification
• Persistent Volumes in Kubernetes