Running a k0s cluster with the GCP cloud controller for internal tooling

Back in my DevOps days at Blibli, we needed a home for internal utility tooling — small services, cron jobs, glue that didn’t belong on the production GKE clusters but still deserved real orchestration. I ended up building that home as a k0s cluster with the GCP cloud controller manager wired in, and it turned into a crash course in how Kubernetes controllers actually work.

Why k0s

The alternatives were K3s (which I’d already used during my internship) and plain kubeadm. I picked k0s for this one mainly for its operational shape: a single binary, a declarative k0s.yaml for cluster config, and an opinionated-but-escapable set of defaults. For a cluster maintained by a small team alongside a dozen other responsibilities, the upgrade story mattered more than anything else — k0s lets you stage controller upgrades without babysitting etcd by hand.

The interesting part: the cloud controller

Running on GCP VMs without a cloud controller manager means your nodes are just pets with kubelet installed. Adding the GCP CCM changed that: node objects got proper provider IDs, zone labels started flowing, and Services of type LoadBalancer could provision GCP load balancers directly. That last one was the real win.

The catch was that nobody seemed to have done this outside GKE — or at least nobody had written it down. There was no proper documentation for installing the GCP CCM on a bring-your-own cluster, so most of the work was reading the controller’s source and GKE’s manifests to reverse-engineer what it expected. A few things bit me along the way:

• Provider IDs are load-bearing. If a kubelet’s --provider-id doesn’t match what the CCM expects, the node registers but nothing can manage it. I found this out when the cluster autoscaler started crashing on nodes whose IDs didn’t match its expectations. The fix I landed on was a small controller of my own that watched Node objects and reconciled their provider IDs to the correct form — my first real Kubernetes controller, written out of necessity. Reconcile loops, informers, and all the ways a Node object can fight back; nothing teaches you how controllers work like having to build one to stop a crash loop.
• Credentials scoping. The controller needs a service account with just enough to manage load balancers and node metadata. Tempting as it is to reuse a broad infra account, a narrow custom role paid off the first time someone asked “what can this cluster actually touch?”
• Internal load balancers by default. This cluster served internal tooling, so every LoadBalancer Service got the internal-facing annotation. Exposing an internal tool on a public IP by accident is the kind of mistake you only make once.

The rest of the machinery

Around the cluster itself, the setup was glue I mostly wrote once and then forgot about:

• Masters came up with Ansible. A playbook installed k0s, generated the join tokens, and brought the control plane up in the right order.
• Poor man’s node pools with Terraform. Groups of Compute Engine VMs defined in Terraform — not a GKE NodePool, but the same idea — each with a startup script that installed the k0s worker and registered the node to the cluster on first boot.
• Audit logs to Elasticsearch. I enabled the Kubernetes audit log on the API server and shipped it with Fluent Bit into Elasticsearch, so “who changed what on this cluster” had a better answer than “check bash history.”

What I’d do differently

I’d GitOps the cluster add-ons from day one instead of hand-applying the CCM manifests, and I’d replace the startup-script bootstrap with Packer-built images much earlier — they arrived late and removed a whole class of “works on the first node” drift.

For a utility cluster, k0s plus a properly configured cloud controller was the right call: boring to run, and no managed-cluster bill for workloads that didn’t justify one.