Does an OPC need Kubernetes? I switched to Nomad, and a service is 100 lines of config

Once you run more than a handful of services, docker compose stops being enough: a database, a reverse proxy, a container registry, a local LLM, hourly backups. The machines multiply too, some at home, one in the cloud. Which container runs where, who restarts it when it dies, how you move it to another box: all of that lives in your head.

The next step looks like Kubernetes. But one person on Kubernetes spends the first month installing add-ons: an ingress controller, cert-manager, a CSI driver, an operator for everything. Each one needs upgrading, and each one breaks.

In March 2024 I deleted my Kubernetes manifests and moved to Nomad. Two and a half years later, everything I run is 14 files and 1,333 lines of HCL, under 100 lines per service on average. A deploy is one command. No CI, no Helm, no operators. For one person and a few machines, Nomad costs far less than Kubernetes and gives up nothing.

Kubernetes is heavy in concepts, not in binaries

k3s is a single binary too. The weight of Kubernetes was never the download size.

To run one service with a domain, a certificate and persistent storage on Kubernetes, you write a Deployment, a Service, an Ingress, a Secret, a ConfigMap, a PVC and a StorageClass, then install an ingress controller and cert-manager. Each object has its own schema, spread over several files or folded into a Helm chart.

Nomad has three levels: job → group → task. Routing, secrets, storage and restart policy are blocks in that one file.

The migration left a number. The same three services were 9 YAML files, 226 lines on Kubernetes and 3 files, 126 lines on Nomad.

One service, one file

Here is my Postgres job with the details stripped. The full file is 111 lines, backups included. Each numbered comment is one thing that takes an object plus an add-on on Kubernetes:

variable "pg_sec" { type = string }   # no default: forget to pass it and the submit fails

job "postgres" {
  constraint {                        # only nodes that can mount Ceph volumes
    attribute = "${meta.has_rbd}"
    value     = "true"
  }

  group "db" {
    reschedule { unlimited = true }   # node gone? start on another one

    task "postgres" {
      driver = "docker"
      config {
        image = "pgvector/pgvector:pg18"
        mount {                       # 1. storage: a name and a size, the driver does the rest
          type   = "volume"
          source = "postgres-data"
          target = "/var/lib/postgresql"
          volume_options {
            driver_config {
              name    = "wetopi/rbd"
              options = { size = "<MB>" }
            }
          }
        }
      }
      template {                      # 2. secrets: environment variables by the time the process starts
        data        = var.pg_sec
        destination = "secrets/pg.env"
        env         = true
      }
      service {                       # 3. routing: Traefik sees these tags and opens a TCP route
        name = "postgres"
        port = "db"
        tags = [
          "traefik.enable=true",
          "traefik.tcp.routers.pg.entrypoints=postgres",
          "traefik.tcp.routers.pg.rule=HostSNI(`*`)",
        ]
      }
    }

    task "backup" {                   # 4. backup: not a container; pg_dumpall into restic every hour
      driver = "exec"
      lifecycle {
        hook    = "poststart"
        sidecar = true
      }
    }
  }
}

A deploy is one command, and secrets never touch disk

nomad job run -var pg_sec="$(sops -d secrets/postgres.sec.env)" cluster/postgres.hcl

Straight from the laptop to the cluster. No deploy script, no CI.

Secrets live in git encrypted with sops. They are decrypted at the moment of submit and passed in; no copy stays on any node. Forget to pass one and the submit fails.

A Kubernetes Secret is base64. To keep secrets in git you add sealed-secrets or external-secrets, then a GitOps pipeline on top. Every layer needs your private key to work. For one person with one laptop, submitting directly is safer.

Scheduling on four labels

Every node carries four labels: which cluster, which site, whether it is the public edge, whether it can mount Ceph volumes. Jobs constrain on those four and nothing else. No affinity, no spread.

Rebuild a machine, give it the same labels, and its jobs come back. No job file has ever contained a hostname.

Kubernetes does this with nodeSelector too.

Machines in several places are the normal case for Nomad

An OPC’s machines are scattered by nature: a few boxes at home with the disks and the GPU, one in the cloud with a public IP. I want one cluster and one deploy path.

A Nomad client joins as long as it can reach the servers, across NAT or across the internet, and tasks are not required to talk to each other. My nodes are on one Tailscale network and Nomad runs on top of it.

A site is a label. A service that runs in two places is one file with two groups. My Traefik is exactly that: one job, two groups, two ingress planes, one in the cloud and one at home, and they fail separately. Nomad has datacenters and regions built in; I have not needed either.

Kubernetes assumes a cluster is one datacenter. Every pod must reach every other pod on a flat network, so spanning sites means stretching an overlay across the internet; Services route to any replica with no sense of distance; persistent volumes are pinned to one site. The official answer is one cluster per site plus Karmada or ClusterMesh on top. One person running N clusters and a federation layer is a disaster.

Routes travel with the service

To expose a service, add a few tags to its service block: the hostname and which certificate to use. Traefik reads the tags and the route exists; it requests the certificate itself.

Redeploy the service and the route re-registers. Traefik’s config does not change.

The same job on Kubernetes: Ingress, an ingress controller, cert-manager, a ClusterIssuer, a Certificate.

Storage is a Docker volume plugin

The mount block in the skeleton is the whole story: a name and a size. The driver creates the volume on Ceph the first time and mounts it on whichever node the job lands on. No CSI, no StorageClass, no PVC, no PV.

Lose a node and the job restarts elsewhere with its volume. A second mount is refused, so two instances can never write the same data.

Not only containers

The Postgres backup task runs under the exec driver: a plain process instead of a container, piping pg_dumpall into restic every hour. The local LLM is llama.cpp plus a GPU, declared as one device in the job.

Kubernetes runs containers only. Nomad ships docker, exec, java and qemu drivers, and they mix in one cluster.

The costs

No Helm charts. Other people’s install docs are compose files or Helm charts, and you translate them to HCL yourself.
One more agent. I use Consul for service discovery, so every node runs two agents. Nomad now has service discovery built in, and a fresh setup can skip Consul.
A new owner. HashiCorp moved to the BSL license in 2023 and was bought by IBM in 2025. Self-hosted use is unaffected and releases keep coming, but the ecosystem will never snowball the way Kubernetes’ did.
No managed offering. Every cloud sells managed Kubernetes. Nobody sells managed Nomad.

When Kubernetes is still the answer

You have a team, or plan to hire. Kubernetes is the common language, and nobody has to learn a second one.
You use a managed cluster and never touch a node.
You depend on an operator that only exists for Kubernetes.

One person, your own machines, a few dozen services at most: pick Nomad. 1,333 lines of config fit into an AI’s context window in one go, so it sees the whole cluster while changing one service. On Kubernetes, cert-manager’s CRDs alone are longer than that.

How many machines do you run alone, and on what? If you switched in either direction, reply and say why.