Kubernetes from the inside

Kubernetes has hundreds of object kinds and thousands of fields. Memorising them does not scale — and you don't have to. Almost everything in the system is the same small idea repeated: you write down what you want, and a loop keeps making reality match it. Once you can see those loops, a failing rollout, a Pod stuck in Pending or a Service with no traffic stops being mysterious: you ask which loop should be acting here, and what is it seeing? Every chapter below takes one mechanism apart with a diagram you can poke — and a button that breaks it on purpose.

Facts were checked against the official Kubernetes documentation for v1.37 (and the Helm, kind, Skaffold, Tilt, Argo CD, Prometheus, etcd and OpenTelemetry docs) in September 2026. Kubernetes ships new minor releases every few months, so check the release notes for anything newer.

The guide has levels. Chapters and blocks carry a chip — L1 essentials, L2 practitioner, L3 internals, L4 production. Choose your depth with the switch in the bottom-left corner; deeper material hides until you want it. The map below jumps anywhere and ticks off what you've read.

Here's the one idea the rest of this page hangs on: you never tell Kubernetes what to do. You tell it what you want. Then a crowd of small programs spends the rest of the cluster's life trying to make reality match.

The Kubernetes docs use a thermostat. You set 21 °C: that's the desired state. The room is at 17 °C: that's the current state. The thermostat keeps switching the heating on or off to close the gap. It doesn't run a script called "heat the room". It just keeps comparing.

Almost every Kubernetes object carries both halves. The spec is what you want, and you write it. The status is what's true right now, and the system writes it. When you create a Deployment with replicas: 3, you're writing a spec. Nobody has started anything yet.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
spec:              # ← you write this: desired state
  replicas: 3
  ...
status:            # ← the cluster writes this
  readyReplicas: 2 #   current state (one Pod still starting)

A controller is the thermostat. It watches one kind of object, compares spec with reality, and acts to close the gap. Then it does it again. Forever. That's the reconciliation loop: observe → diff → act.

That's the misconception to drop right now: "kubectl starts containers." It doesn't. kubectl sends your object to the API server, which stores it. That's it. The Deployment controller, the ReplicaSet controller, the scheduler and the kubelet each notice the new object on their own and do their part. kubectl has usually exited before the first container exists.

Notice what happened when you stopped the controller: nothing broke loudly. The spec still said 3, the status quietly said 2, and nobody closed the gap. When the controller came back, it didn't need to know what had happened while it was away. It compared, saw a gap, and acted. That's why controllers compare whole states instead of replaying a to-do list.

A cluster is two kinds of machines. The control plane makes global decisions: where things run, what to do when something dies. The nodes (worker machines) actually run your containers. Every cluster needs at least one worker node. In production the control plane usually spans several machines too.

Here is the cast. Learn these names once; they appear in every error message you'll ever read.

The shape to remember is a hub and spoke. All API usage from nodes and the Pods on them terminates at the API server; none of the other control plane components are designed to expose remote services. The scheduler doesn't call the kubelet. The controller-manager doesn't call the scheduler. They all watch the API server and write back to it. In a standard setup, the API server is the only component that talks to etcd.

Nobody gives orders. Everybody reads the shared whiteboard and does their bit.

With the scheduler dead, the Deployment and ReplicaSet still appeared, and three Pod objects existed in etcd. They just never got a node, so no kubelet ever claimed them. They sit in Pending. A Pod whose scheduler can't place it stays unscheduled until it can be placed; nothing times out on its own.

You never run a bare container in Kubernetes. The smallest thing you deploy is a Pod: one or more containers that are scheduled together, on the same node, sharing some things.

What they share is the key. Every container in a Pod shares the network namespace: the same IP address and the same port space. So two containers in one Pod reach each other on localhost, and they can't both listen on port 8080. They can also share volumes declared on the Pod, which is how a helper container reads the files your app writes.

Containers in different Pods have different IPs. They talk over the network, like strangers.

spec:
  initContainers:
  - name: migrate          # runs to completion first
    image: myapp-migrations:1.4
  - name: log-agent        # sidecar: keeps running
    image: fluent/fluent-bit:3.0
    restartPolicy: Always
  containers:
  - name: web              # your app
    image: myapp:1.4
    ports:
    - containerPort: 8080

Init containers run before your app, one at a time, in order. Each must finish successfully before the next starts. If one fails, the kubelet restarts it until it succeeds (unless the Pod's restartPolicy is Never, in which case the whole Pod fails). Perfect for "wait for the database" or "run migrations".

A sidecar is an init container with restartPolicy: Always. It starts in order like other init containers, but it doesn't have to exit: once it's started, the kubelet moves on to the next one, and it keeps running next to your app. At shutdown, sidecars are stopped after the main containers, in reverse order. Native sidecars have been stable since Kubernetes 1.33.

A Pod has a phase, a one-word summary. Each container inside also has a state: Waiting, Running or Terminated.

When a container exits, the Pod's restartPolicy decides what happens: Always (the default), OnFailure or Never. Restarts happen in place, on the same node, in the same Pod. The kubelet waits between restarts with an exponential backoff: 10 s, 20 s, 40 s… capped at 300 s (5 minutes). After a container has run 10 minutes without problems, the backoff resets.

CrashLoopBackOff is not an error. It's the kubelet telling you "I'm in the waiting part of that backoff". The error is whatever made your container exit. Go read its logs.

kubectl debug -it web-0 \
  --image=busybox:1.28 --target=web

You almost never create Pods yourself. You create a Deployment. The Deployment creates a ReplicaSet, and the ReplicaSet creates the Pods. Three objects, two controllers, each doing one small job.

Why the middle layer? Because a ReplicaSet is "N copies of this exact Pod template". When you change the template (a new image, a new env var), the Deployment creates a second ReplicaSet for the new template and slowly shifts replicas from old to new. That's a rolling update.

The ReplicaSet's name is [deployment-name]-[hash]. The hash comes from the Pod template and is stamped on the ReplicaSet and its Pods as the pod-template-hash label, so two generations never claim each other's Pods. Don't touch that label.

kubectl get rs
NAME             DESIRED   CURRENT   READY   AGE
web-7c9f6d5b8d   0         0         0       2d
web-5d8b9c7f4a   4         4         4       3m

old template kept at 0 replicas, so you can roll back (output shape as in the docs; names illustrative)

Only a change to .spec.template starts a rollout. Scaling from 4 to 10 replicas doesn't: same template, same ReplicaSet, just more Pods.

Two knobs control how aggressive a rollout is:

The rollout only moves forward when new Pods become ready. And "ready" is decided by a probe. The kubelet can run three kinds of probe against each container, and they do very different things:

Here's the misconception this chapter kills: "liveness means healthy enough for traffic." No. Readiness decides traffic. Liveness decides restarts. Point a liveness probe at an endpoint that checks your database, let the database hiccup, and the kubelet restarts every replica at once. The docs warn about exactly this cascade.

readinessProbe:
  httpGet: { path: /ready, port: 8080 }
  periodSeconds: 10     # default 10
  timeoutSeconds: 1     # default 1
  failureThreshold: 3   # default 3
livenessProbe:
  httpGet: { path: /livez, port: 8080 }

Look at what the broken v3 did not do: take your service down. It created as many new Pods as the surge allowed, they never became ready, so the controller could only remove as many old Pods as maxUnavailable permits. Then it stopped. Kubernetes doesn't roll back for you; after progressDeadlineSeconds it only reports ProgressDeadlineExceeded. You (or your CI) run:

kubectl rollout status deployment/web
kubectl rollout history deployment/web
kubectl rollout undo deployment/web
kubectl rollout undo deployment/web --to-revision=2

Pods come and go. Every rollout from the last chapter replaced all of them, and every new Pod got a new IP. So how does your frontend find web? Not by Pod IP. By a Service: a stable name and a stable virtual IP in front of a changing set of Pods.

apiVersion: v1
kind: Service
metadata:
  name: web
  namespace: shop
spec:
  selector:
    app: web        # which Pods are behind me
  ports:
  - port: 80        # what clients call
    targetPort: 8080  # where the Pod listens

Three moving parts make that YAML work, and none of them is the Service object itself:

1. The EndpointSlice controller (in the controller-manager) finds Pods matching the selector and writes their IPs into EndpointSlice objects, with a condition per Pod saying whether it's ready.
2. kube-proxy on every node watches Services and EndpointSlices and programs the node's packet forwarding (iptables rules, by default on Linux) so traffic to the Service's virtual IP lands on one of the backends.
3. Cluster DNS gives the Service a name: web.shop.svc.cluster.local. From inside the shop namespace, plain web works; from elsewhere, web.shop.

The two failures you just caused look identical from the client (timeouts or refused connections) but have different fixes. A readiness failure removes one Pod: the EndpointSlice controller drops its IP from every matching Service, and traffic flows to the rest. A selector typo matches nothing: the Service exists, DNS resolves, the ClusterIP is there, and there's simply nobody behind it.

nameserver 10.32.0.10
search test.svc.cluster.local svc.cluster.local cluster.local
options ndots:5

Your image should be the same in dev, staging and prod. What changes is configuration, and Kubernetes keeps it outside the image in two objects: a ConfigMap for ordinary settings and a Secret for sensitive ones. Both are key-value maps, and both are limited to 1 MiB. Pods use them from their own namespace.

apiVersion: v1
kind: ConfigMap
metadata:
  name: web-config
data:
  GREETING: hello
  app.properties: |
    cache.ttl=60

A container can consume it three ways without writing any Kubernetes code: in its command-line args, as environment variables, or as files in a volume. (The fourth way is your code reading the API directly.) The choice matters more than it looks, because they behave differently when the ConfigMap changes:

One more trap: if a Pod references a ConfigMap (or a key) that doesn't exist and the reference isn't marked optional: true, the Pod won't start.

Secrets: base64 is not encryption

A Secret looks like a ConfigMap whose values are base64-encoded. That encoding hides nothing (the docs call such values "obscured but not secret"): anyone can decode it (you can also write plain text in stringData and let the API encode it). Here's the part people miss: by default, Secrets are stored unencrypted in etcd.

What Kubernetes does do: a Secret is only sent to a node if a Pod on that node needs it, and the kubelet keeps mounted Secrets in tmpfs (memory), not on disk, deleting its copy when the Pod goes away.

The downward API: let the Pod know itself

Your app sometimes needs facts about its own Pod: its name for logs, its namespace, the node it landed on, its IP, its memory limit. The downward API exposes them without calling the API server:

env:
- name: POD_NAME
  valueFrom:
    fieldRef: { fieldPath: metadata.name }
- name: NODE_NAME
  valueFrom:
    fieldRef: { fieldPath: spec.nodeName }
- name: MEM_LIMIT
  valueFrom:
    resourceFieldRef: { resource: limits.memory }

Most fields work as env vars or as volume files. The exceptions: the whole metadata.labels and metadata.annotations maps are only available as volume files (single keys like metadata.labels['app'] work in env too), while spec.nodeName, status.podIP and status.hostIP are env-only.

Every container can declare two numbers per resource. The request is what you ask the scheduler to set aside for you. The limit is the ceiling the kernel enforces on the node. They sound like two sides of one coin. They're read by two different components at two different moments.

The scheduler only looks at requests. It adds up the requests of every Pod already placed on a node and refuses to place yours if the sum would exceed the node's capacity for CPU or memory, even if the node is idle right now. That's deliberate: the docs say it protects against a shortage later, "for example, during a daily peak in request rate."

resources:
  requests:
    cpu: 500m        # half a CPU, "500 millicpu"
    memory: 256Mi
  limits:
    cpu: "1"
    memory: 512Mi

Units first. cpu: 0.1 and cpu: 100m are the same thing, one hundred millicpu. Memory uses suffixes: Mi and Gi are powers of two, M and G powers of ten. That trips people up exactly once.

Here's the answer. CPU is compressible: if you use too much, the kernel simply makes you wait. The docs call a CPU limit "a hard limit the kernel enforces", and container runtimes "don't terminate Pods or containers for excessive CPU usage." You get throttled, and your latency climbs.

Memory is not compressible. You can't hand bytes back slowly. When a container uses more than its memory limit and the kernel detects memory pressure, the OOM killer steps in, killing processes inside the container's cgroup. On cgroup v2 the kubelet sets memory.oom.group by default, so all of the container's processes are killed together (singleProcessOOMKill: true restores the old one-victim behaviour). The container dies, and kubectl describe pod shows Reason: OOMKilled and Exit Code: 137. If the container can restart, Kubernetes restarts it.

What if you set a limit but no request? Then Kubernetes copies the limit into the request, unless an admission mechanism such as a LimitRange filled in a default first. A LimitRange is a per-namespace policy object: it can set default requests and limits, enforce min/max values and cap the ratio between limit and request.

Look at the pod cards in the figure: each one got a QoS class the moment it was created, computed from its requests and limits. You never set it yourself.

When a node runs short of memory, the kubelet starts evicting Pods. The docs spell out the ranking: first Pods whose usage exceeds their requests (ordered by Priority, then by how far over they are), and only then Pods using less than they asked for. The kubelet doesn't read the QoS class directly, but the outcome lines up with it: BestEffort Pods, whose request is zero, are always "over", so they tend to go first.

kubectl patch pod web --subresource resize \
  --patch '{"spec":{"containers":[{"name":"app",
  "resources":{"limits":{"cpu":"800m"}}}]}}'

Every Kubernetes object has the same shape: apiVersion and kind say what it is, metadata says which one (name, namespace, labels, annotations), spec is what you want, and status is what the system reports back. You write spec. Controllers write status. If you're typing status: into a manifest, something went sideways.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
  namespace: shop
  labels: {app: web, tier: frontend}
  annotations:
    team.example.com/oncall: "#shop-alerts"
spec:
  replicas: 3
  selector:
    matchLabels: {app: web}
  template: …

Labels are for selecting: Services, Deployments and kubectl get -l app=web all find things by label. Selectors come in two flavours: equality-based (app=web, tier!=db) and set-based (env in (prod,staging)). There's no OR across conditions. Annotations are for everything else: non-identifying metadata that tools read and nobody selects on.

Namespaces split one cluster into named scopes. A fresh cluster starts with four: default, kube-node-lease, kube-public and kube-system. The docs advise against using default for production. Some objects, like Nodes and PersistentVolumes, aren't namespaced at all.

There are three ways to manage objects: imperative commands (kubectl create deployment …), imperative config (kubectl create -f, replace -f) and declarative config (kubectl apply -f). The docs are blunt: manage an object with one technique only, because mixing them "results in undefined behavior." For anything that lives in Git, use apply.

Server-side apply (SSA, stable since 1.22) moves the merge logic into the API server. You run kubectl apply --server-side, and kubectl sends your manifest under a field manager name ("kubectl" by default; pick your own with --field-manager). The server records which manager owns which fields. If you apply a change to a field another manager owns, you get a conflict. You then have three options: force it and become the sole owner, drop the field from your manifest and give up the claim, or change your value to match theirs and share ownership.

The figure shows the classic fight: your CI pipeline applies replicas: 3 from Git while someone patches production by hand. Client-side apply would quietly flip the value back on the next deploy. SSA makes the disagreement visible. The docs' own example is the HorizontalPodAutoscaler: once an HPA owns replicas, the fix is to remove replicas from your manifest, not to force it back.

Now the daily toolbox. You'll use these more often than apply itself:

If every code change means build, push to a registry, deploy and wait for a pull, you'll stop testing in Kubernetes and start testing in production. The fix has two parts: a cluster on your laptop, and a tool that watches your files and does the boring steps for you.

For the laptop cluster, four tools cover almost everyone. They differ mostly in how they run the nodes and how your freshly built image gets into the cluster.

That image-loading column is where the most common local bug lives. Say you build my-app:latest, load it into kind and deploy. You get ErrImagePull. Why? If you don't set imagePullPolicy and the tag is :latest (or missing), Kubernetes sets the policy to Always. The kubelet goes to Docker Hub looking for an image that only exists on your laptop.

Now the loop itself. Skaffold's skaffold dev watches your source files. On a change it rebuilds the image (or syncs files into the running container), pushes if needed, and redeploys. It spots a local cluster from the kubectl context name (kind-…, k3d-…, minikube, docker-desktop) and loads the image instead of pushing it. Its default tagger is gitCommit (uncommitted changes get a -dirty suffix, so two dirty builds can share a tag), but Skaffold never references images by tag alone in the manifests it deploys: pushed images get tag@sha256:…, and images loaded into a local cluster get an extra immutable tag named after the image ID. That neatly avoids the stale-tag trap.

Tilt does the same job with a Tiltfile and tilt up. Its live_update copies changed files into the container (sync steps) and runs commands (run steps) instead of rebuilding. Files that match fall_back_on trigger a full rebuild. Skaffold has the equivalent in its sync rules. Either way, a template or script change lands in a second or two.

Don't forget the last mile: kubectl port-forward svc/web 8080:80 gets you to the app without an Ingress. Both Skaffold and Tilt can set up port-forwards for you.

You have one app and three environments. Copy-pasting the YAML three times works for about a week. Then two tools compete for the job, and they take opposite approaches: Kustomize patches plain YAML, while Helm renders templates from values.

Kustomize is built into kubectl (since 1.14). A directory with a kustomization.yaml is a unit you can render with kubectl kustomize dir/ or apply with kubectl apply -k dir/. A base holds the shared resources. An overlay is another kustomization that lists the base in resources and adds changes on top. The base doesn't know its overlays exist.

# overlays/prod/kustomization.yaml
resources:
- ../../base
namePrefix: prod-
images:
- name: shop/web
  newTag: "1.4.2"
patches:
- path: replicas.yaml

Kustomize's generators earn their keep. A configMapGenerator creates a ConfigMap whose name ends in a hash of its content, and Kustomize rewrites every reference to it. Change a value and you get a new name, so the Deployment's Pod template changes and a rollout happens. That matters because ConfigMaps consumed as environment variables are not updated in running Pods: they need a restart.

Helm treats your app as a chart: Chart.yaml, a values.yaml of defaults and a templates/ directory of Go templates. helm install or helm upgrade --install renders the templates with your values and creates a release. Every upgrade adds a revision. helm history lists them and helm rollback web goes back to the previous one (or to a revision number you pass).

Where does Helm keep that history? In the cluster: by default, release information is stored in Secrets in the release's namespace (HELM_DRIVER can switch that to ConfigMaps or SQL). Values merge with the right-most wins: several -f files, then --set overrides. Helm 4 defaults to server-side apply for new releases, so the field-ownership rules from the kubectl chapter apply to Helm too.

A Service gets traffic to your Pods from inside the cluster. To route HTTP from outside by host name and path, you need an L7 router. Kubernetes has two APIs for describing one. Neither API routes a single packet on its own.

That's the misconception to kill first: an Ingress object does nothing by itself. The docs: "You must have an Ingress controller to satisfy an Ingress. Only creating an Ingress resource has no effect." The controller (NGINX, Traefik, HAProxy, a cloud load balancer…) watches Ingress objects and configures a real proxy. With no controller installed, your Ingress sits in etcd and nobody reads it.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata: {name: shop}
spec:
  ingressClassName: nginx
  rules:
  - host: shop.example.com
    http:
      paths:
      - path: /api
        pathType: Prefix
        backend:
          service: {name: api, port: {number: 80}}

Each path needs a pathType: Exact, Prefix (matches whole path elements split by /, so /api matches /api/v1 but not /apix) or ImplementationSpecific. When several paths match, the longest one wins. ingressClassName picks the controller; leave it out and the default IngressClass applies, if there is one.

Anything beyond host and path (header matching, traffic splitting, rewrites) was left to controller-specific annotations, and each controller spelled them differently. And the Ingress API is now frozen: it's GA and won't be removed, but it gets no new features. The project recommends Gateway API instead.

Gateway API is an add-on: CRDs you install (often with your chosen implementation). Its kinds are modelled on roles (infrastructure provider, cluster operator, application developer). A GatewayClass names the controller that implements it. A Gateway is an instance of traffic-handling infrastructure, such as a cloud load balancer, with its listeners. Application developers typically attach HTTPRoutes (or GRPCRoutes) to it with parentRefs. Header matching and weighted traffic are part of the spec, not annotations.

kind: HTTPRoute
spec:
  parentRefs: [{name: shop-gw}]
  hostnames: ["shop.example.com"]
  rules:
  - backendRefs:
    - {name: web-v1, port: 80, weight: 90}
    - {name: web-v2, port: 80, weight: 10}

A container's filesystem is disposable: when it crashes, the kubelet restarts it with a clean state. Volumes are how data outlives that. The simplest volume, emptyDir, survives container crashes but is "deleted permanently" when the Pod leaves the node. Fine for scratch space and caches, useless for a database.

For real persistence Kubernetes splits the problem in three. A PersistentVolume (PV) is a piece of storage in the cluster; it isn't namespaced. A PersistentVolumeClaim (PVC) is your namespaced request for storage: "10Gi, ReadWriteOnce". A StorageClass describes a kind of storage and the provisioner that creates it. Create a PVC that names a class (or rely on the default class) and a PV is provisioned dynamically and bound to it.

Now put a database in a Deployment and scale it to three. All three replicas mount the same PVC, have random names and come up in any order. Databases hate that. A StatefulSet gives each Pod a sticky identity instead: an ordinal (web-0, web-1, …), a stable DNS name through a headless Service (web-0.nginx.default.svc.cluster.local), and its own PVC created from volumeClaimTemplates (www-web-0, www-web-1, …). You have to create the headless Service (clusterIP: None) yourself.

By default (podManagementPolicy: OrderedReady) Pods are created in order 0..N-1, each only after its predecessors are Running and Ready, and they're removed in reverse order. Parallel drops the ordering when you don't need it.

Notice what survived the scale-down: the PVCs. Deleting or scaling down a StatefulSet does not delete its volumes, on purpose ("data safety … is generally more valuable than an automatic purge"). If you want cleanup, persistentVolumeClaimRetentionPolicy (stable since 1.32) lets you set whenDeleted and whenScaled to Delete. Both default to Retain.

So far everything you deployed was a Deployment: a process that should run forever, and if it exits, something is wrong. But a lot of real work doesn't look like that. A database migration should run once and stop. A report should run every night at 2 a.m. A log shipper should run on every machine. Each of those shapes has its own controller, and picking the wrong one is how you get a migration that runs forever in a restart loop.

Takeaway: pick the controller by the shape of the work — forever (Deployment), until done (Job), on a clock (CronJob), once per node (DaemonSet), with a stable identity (StatefulSet). The misconception to kill: "a CronJob runs exactly once per tick". The docs say it runs approximately once, and sometimes twice or not at all — so your jobs must be idempotent.

A Job counts successes. You say how many Pods must finish successfully (completions) and how many may run at once (parallelism, default 1). When a Pod fails, the Job controller makes a new one after an exponential back-off (10 s, 20 s, 40 s … capped at six minutes), and gives up once it has used backoffLimit retries — 6 by default.

apiVersion: batch/v1
kind: Job
metadata:
  name: migrate-db
spec:
  completions: 1
  backoffLimit: 3
  activeDeadlineSeconds: 600   # hard stop
  ttlSecondsAfterFinished: 3600 # auto-delete
  template:
    spec:
      restartPolicy: Never     # Never or OnFailure
      containers:
      - name: migrate
        image: registry.example.com/app:1.42.0
        args: ["migrate", "--to", "latest"]

Two fields save you from surprises. activeDeadlineSeconds is a wall-clock limit for the whole Job and takes precedence over backoffLimit. ttlSecondsAfterFinished lets the TTL controller delete the finished Job (and its Pods) for you, so completed Jobs don't pile up.

CronJob: a Job factory with a clock

A CronJob doesn't run your code — it creates a Job on a cron schedule, and the Job creates Pods. That extra layer is where the interesting knobs live.

apiVersion: batch/v1
kind: CronJob
metadata:
  name: nightly-report
spec:
  schedule: "0 2 * * *"
  timeZone: "Europe/Warsaw"
  concurrencyPolicy: Forbid
  startingDeadlineSeconds: 300
  successfulJobsHistoryLimit: 3
  failedJobsHistoryLimit: 1
  jobTemplate:
    spec:
      backoffLimit: 2
      template:
        spec:
          restartPolicy: Never
          containers:
          - name: report
            image: registry.example.com/report:2.3.1

DaemonSet: one per node

A DaemonSet makes sure every node (or every node matching a nodeSelector / affinity) runs one copy of a Pod. Nodes join → Pods appear; nodes leave → Pods are garbage-collected. The docs' examples are exactly the things you'll meet in chapter 15: a log collector, a node monitoring agent, a storage daemon. As an app developer you rarely write one, but you'll see them in every kubectl get pods -A.

"Kubernetes autoscales" is true in the same way "a car drives itself" is true if you count cruise control. There are three separate autoscalers, and each turns a different dial: the HorizontalPodAutoscaler (HPA) changes how many Pods, the VerticalPodAutoscaler (VPA) changes how big each Pod's requests are, and a node autoscaler (Cluster Autoscaler or Karpenter) changes how many machines there are.

Takeaway: they chain. HPA adds Pods; if they don't fit, they go Pending; the node autoscaler sees Pending Pods and adds a node. And every link in that chain reasons about resource requests, not real usage. The misconception to kill: "the cluster autoscaler adds nodes when CPU is high". It doesn't look at CPU at all — it adds nodes when Pods can't be scheduled.

HPA: a control loop with a formula

The HPA is a controller inside kube-controller-manager. Every 15 seconds (by default) it reads the metric, compares it to your target, and computes:

desiredReplicas = ceil( currentReplicas × currentMetric / targetMetric )

For CPU with a utilization target, "currentMetric" is the Pods' average usage as a percentage of their CPU request. That's the first place requests matter: if a container has no CPU request, utilization is undefined and the HPA takes no action for that metric. If the ratio is within 10 % of 1.0 (the default tolerance) nothing happens, so the replica count doesn't jitter.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: api
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: api
  minReplicas: 2
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 50

The one-liner equivalent from the docs' walkthrough is kubectl autoscale deployment php-apache --cpu=50% --min=1 --max=10, and kubectl get hpa shows the live target as TARGET 0% / 50%-style columns.

Where the numbers come from

CPU and memory come from the Metrics API (metrics.k8s.io), which is usually served by the metrics-server add-on — installed separately. Anything else (requests per second, queue length) needs an adapter that serves custom.metrics.k8s.io or external.metrics.k8s.io, typically from your monitoring system. Chapter 15 shows the whole pipeline.

VPA: right-size the requests

The VerticalPodAutoscaler is not part of core Kubernetes: it's a CRD (autoscaling.k8s.io/v1) plus three components you install — a Recommender that studies historical usage, an Updater that acts on drift, and an admission controller that writes the recommended requests into new Pods.

Node autoscaling: make Pending Pods schedulable

A node autoscaler watches for Pods the scheduler can't place and provisions nodes for them; it also consolidates — removes underutilized nodes when their Pods would fit elsewhere. Both decisions use Pod requests, not real usage. Too-low requests: a new node doesn't actually help the Pod run. Too-high requests: nodes that could be removed never are.

Here's the thing nobody tells you on day one: Kubernetes produces a lot of telemetry, but it keeps almost none of it. Container logs live in files on the node until they're rotated or the Pod is evicted. Events disappear after an hour. The Metrics API only answers "how much CPU right now?". A monitoring system is something you — or your platform team — add.

Takeaway: think of it as three layers. (1) Sources that Kubernetes and your app already expose (stdout, Events, /metrics, spans). (2) Collectors that you run in the cluster, usually as DaemonSets or small Deployments (a log agent, Prometheus, kube-state-metrics, an OpenTelemetry Collector). (3) Backends that store and query (a log store, a time-series database, a trace store). The misconception to kill: "kubectl logs and kubectl top are my monitoring". They are windows onto short-lived, node-local data — the moment a Pod is evicted or a node dies, that data is gone.

The signal map

Every question you'll ask in an incident maps to one of these rows. Learn the "kept for" column: it's why the postmortem can't find anything.

Logs: stdout is the API

Your job as an app developer is simple: write logs to stdout and stderr. The container runtime captures both streams and writes them — in the standardized CRI logging format — to files under /var/log/pods on the node. The kubelet rotates those files (containerLogMaxSize, default 10Mi; containerLogMaxFiles, default 5) and serves them through the API when you run kubectl logs.

kubectl logs deploy/api            # one Pod of the Deployment
kubectl logs api-7c9f -c app -f    # one container, follow
kubectl logs api-7c9f --previous   # the crashed instance
kubectl logs -l app=api --since=10m

Two built-in limits bite everyone once. kubectl logs only reads the latest log file, so after a rotation older lines aren't visible through it (write 40 MiB with a 10 MiB rotation and you get at most 10 MiB back). And the kubelet keeps only one terminated container's logs per container — that's what --previous shows. If the Pod is evicted, its logs go with it.

Kubernetes has no built-in cluster-level log store, so the docs list the patterns you choose from:

Events: the cluster's short-term memory

Events are API objects that components write when something happens: Scheduled, Pulling, BackOff, FailedScheduling, Unhealthy (probe failed). They're the first thing to read when a Pod misbehaves — and the API server deletes them after --event-ttl, one hour by default.

kubectl events --for pod/api-7c9f --watch
kubectl describe pod api-7c9f   # Events at the bottom

Metrics: two pipelines, not one

The resource metrics pipeline is tiny on purpose. cAdvisor, built into the kubelet, measures CPU and memory per container; metrics-server (an add-on) pulls them from each kubelet's /metrics/resource endpoint and serves them as the metrics.k8s.io API. That's what kubectl top, the HPA and the VPA read. Memory is the working set; CPU is average core usage. It deliberately offers the minimum for autoscaling.

The full metrics pipeline is where monitoring actually happens. Every Kubernetes component exposes metrics in the Prometheus text format on /metrics (the kubelet adds /metrics/cadvisor, /metrics/resource and /metrics/probes). Prometheus uses a pull model: it discovers targets through the Kubernetes API (kubernetes_sd_configs with roles like node, pod, service, endpointslice) and scrapes each one every scrape_interval (global default 1m). Short batch jobs that die before a scrape can push through an intermediary gateway instead.

That's the gap kube-state-metrics fills. It doesn't measure usage at all: it watches the API server and turns object state into metrics — labels and annotations, start and termination times, status and phase (every container, for example, gets a kube_pod_container_info series). The docs' own example query finds every Pod that isn't ready:

count(kube_pod_status_ready{condition="false"})
  by (namespace, pod)

Traces: following one request

For your own services, tracing means instrumenting the app with an OpenTelemetry SDK, propagating the trace context on every outgoing call, and exporting spans over OTLP to a Collector. Kubernetes can join in: the API server and the kubelet export their own spans over OTLP/gRPC (default port 4317; both stable since 1.34) and the API server propagates W3C Trace Context, so a slow kubectl apply can be followed through the API server to its calls to webhooks and etcd.

Security in Kubernetes answers three different questions, with three different tools. Who may call the API, and do what? — ServiceAccounts and RBAC. What may my container do on the node? — securityContext, checked by Pod Security Standards. Who may open a network connection to my Pod? — NetworkPolicy.

Takeaway: every one of those starts either closed or wide open, and you need to know which. RBAC denies by default. NetworkPolicy allows by default. The misconception to kill: "I created a NetworkPolicy, so traffic is restricted". Without a network plugin that enforces it, the object has no effect at all.

ServiceAccounts: your Pod's identity

Every namespace gets a ServiceAccount named default, and a Pod that doesn't say otherwise runs as it. It has no permissions beyond API discovery. Pick your own with spec.serviceAccountName; Kubernetes then mounts a short-lived, automatically rotated token at /var/run/secrets/kubernetes.io/serviceaccount/token (with ca.crt and namespace next to it). If your app never talks to the API server, turn that off:

spec:
  serviceAccountName: api
  automountServiceAccountToken: false

RBAC: additive allow-lists

RBAC has four objects. A Role (namespaced) or ClusterRole (cluster-wide) lists verbs on resources. A RoleBinding or ClusterRoleBinding grants a role to subjects — users, groups, ServiceAccounts. Rules are purely additive: there are no deny rules, and a request that no rule allows is denied with HTTP 403.

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata: {name: pod-reader, namespace: dev}
rules:
- apiGroups: [""]
  resources: ["pods", "pods/log"]
  verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata: {name: ci-reads-pods, namespace: dev}
subjects:
- kind: ServiceAccount
  name: ci-bot
  namespace: dev
roleRef:
  kind: Role
  name: pod-reader
  apiGroup: rbac.authorization.k8s.io

Test instead of guessing — kubectl auth can-i asks the authorizer directly, and --as impersonates someone else (if you're allowed to impersonate):

kubectl auth can-i list pods -n dev \
  --as=system:serviceaccount:dev:ci-bot    # yes
kubectl auth can-i delete secrets -n dev \
  --as=system:serviceaccount:dev:ci-bot    # no

securityContext: what the process may do

Unless you say otherwise, a container runs as whatever user its image specifies, and allowPrivilegeEscalation defaults to true when unset. You can set these at Pod level and override per container:

spec:
  securityContext:
    runAsNonRoot: true
    runAsUser: 10001
    seccompProfile: {type: RuntimeDefault}
  containers:
  - name: app
    securityContext:
      allowPrivilegeEscalation: false
      readOnlyRootFilesystem: true
      capabilities: {drop: ["ALL"]}

That block passes the Restricted Pod Security Standard. The standards are three cumulative profiles, enforced per namespace by the built-in Pod Security admission controller via labels:

kubectl label ns dev \
  pod-security.kubernetes.io/enforce=baseline \
  pod-security.kubernetes.io/warn=restricted

Modes: enforce rejects violating Pods, audit annotates the audit log, warn prints a warning to the client. Subtle point: enforce checks Pods, not Deployments — so a bad Deployment is accepted and its ReplicaSet then fails to create Pods. warn and audit also check workload templates, which is why you want warn on.

NetworkPolicy: firewall rules by label

By default every Pod accepts every connection and may connect anywhere. A Pod becomes isolated for a direction as soon as any policy selecting it lists that direction in policyTypes; from then on only what some policy allows gets through, and policies combine as a union. The usual recipe: default-deny per namespace, then allow what you need.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: {name: default-deny-ingress}
spec:
  podSelector: {}
  policyTypes: ["Ingress"]
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: {name: api-from-frontend}
spec:
  podSelector: {matchLabels: {app: api}}
  ingress:
  - from:
    - podSelector: {matchLabels: {app: frontend}}
    ports: [{port: 8080}]

You've seen the core idea of Kubernetes many times now: you declare desired state, a controller reconciles. GitOps applies that same idea one level up. The desired state of the whole cluster lives in a Git repository, and a controller inside the cluster (Argo CD, Flux) keeps pulling it and applying it.

Takeaway: CI builds and tests an immutable image; a commit to a config repo declares which image runs where; a GitOps controller pulls and reconciles. Promotion is a Git change, rollback is a Git revert, and drift is something the controller detects. The misconception to kill: "CD means the CI job runs kubectl apply against prod". That works, but it gives CI cluster-admin keys and nothing notices when reality drifts from Git.

Immutable images: tags lie, digests don't

A tag is a movable pointer; a digest (@sha256:…) is the content hash and never changes. The Kubernetes docs say to avoid :latest in production — you can't tell which version is running and can't roll back properly — and to use a meaningful tag such as v1.42.0 and/or a digest. If you give both (app:1.42.0@sha256:…), only the digest is used for pulling.

So the CI rule is: build once, tag with something unique (a version or the commit SHA), push, and promote that same image from environment to environment. Never rebuild for prod.

Environments: namespaces or clusters?

A common shape (a pattern, not a rule): namespaces for dev and previews, a separate cluster for production. Per-env differences live in Kustomize overlays or Helm values (chapter 10).

Push vs pull

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata: {name: api-prod, namespace: argocd}
spec:
  project: default
  source:
    repoURL: https://git.example.com/deploy.git
    targetRevision: main
    path: envs/prod/api
  destination:
    server: https://kubernetes.default.svc
    namespace: api
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

Two flags in that automated block are off by default, and both are safety catches. Without prune, a resource you deleted from Git stays in the cluster. Without selfHeal, a manual change in the cluster makes the app OutOfSync but isn't reverted.

Promotion and progressive delivery

Promotion is just moving the same image reference from envs/dev to envs/prod — by a pull request a human approves, or by a bot after dev's checks pass. Flux ships image-automation controllers aimed at automating that tag update.

Progressive delivery limits the blast radius of that change. A Deployment's rolling update already replaces Pods gradually and gates on readiness (chapter 4). For canaries with traffic weights and automatic analysis, tools like Argo Rollouts (a controller + CRDs for blue-green, canary and canary analysis) shift traffic step by step and use canary analysis — the metrics from chapter 15 — to decide whether to go on. Traffic-weighted canaries need something that can split traffic, like the weighted routes of chapter 11.

In chapter 2 you learned that only the API server talks to etcd. Now open that box. Every write you send, whether from kubectl apply, a controller or a Helm release, walks through the same ordered pipeline. Any stage can stop it, and each stage fails with its own HTTP status code.

Takeaway: the order is authentication → flow control → authorization → mutating admission → validation → validating admission → etcd. The misconception to kill: "the API server just stores what I send". By the time your object reaches etcd, mutating admission may have added fields to it, and several policies you never wrote may have voted on it.

The stages, one by one

Admission only runs on requests that create, modify, delete or connect to an object. Plain reads (get, list, watch) stop after authorization. Admission is also stricter than authentication and authorization in one way: if any admission controller rejects, the whole request is rejected immediately.

Mutating controllers run one after another, each seeing the previous one's changes. Validating webhooks run in parallel, because they only say yes or no. That's why policy that must see the final object belongs in the validating phase.

Policies without a server: ValidatingAdmissionPolicy

Webhooks work, but each one is a TLS server you have to run and keep alive. A ValidatingAdmissionPolicy (stable since v1.30) is a CEL expression that the API server evaluates in-process. It takes two objects: the policy itself and a binding that says where it applies and what to do on failure.

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
  name: max-replicas.example.com
spec:
  failurePolicy: Fail
  matchConstraints:
    resourceRules:
    - apiGroups: ["apps"]
      apiVersions: ["v1"]
      operations: ["CREATE", "UPDATE"]
      resources: ["deployments"]
  validations:
  - expression: "object.spec.replicas <= 5"
---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicyBinding
metadata:
  name: max-replicas-binding.example.com
spec:
  policyName: max-replicas.example.com
  validationActions: [Deny]

A binding's validationActions can be Deny, Warn (the client gets a warning) or Audit (recorded in the audit event). Deny and Warn can't be combined, so a safe rollout is [Warn, Audit] first, then [Deny]. The mutating twin, MutatingAdmissionPolicy, is stable since v1.36.

Kubernetes has hundreds of clients writing to the same objects at once, and no locks. Two ideas make that safe and fast: every object carries a resourceVersion, and nobody polls, everybody watches.

Takeaway: an update carrying a stale resourceVersion is rejected with 409 Conflict, so you re-read and retry. And a watch is a replay from a version number, not a live feed that exists forever: history is kept for a limited time, and if you fall behind you get 410 Gone and must list again. The misconception to kill: "when two clients update the same object, the last write wins".

resourceVersion: a compare-and-swap token

Every object has metadata.resourceVersion, the version of that object as stored in etcd. When you update (HTTP PUT), you send back the version you read. If someone changed the object in the meantime, the API server answers 409 Conflict instead of silently overwriting their change. The right reaction is: GET again, re-apply your change, PUT again. client-go ships a helper for exactly this loop (retry.RetryOnConflict).

A patch without a resourceVersion is unconditional; add one if you need lost-update detection. Server-side apply (chapter 8) handles conflicts per field manager instead.

List, then watch

Every controller, and kubectl get -w, does the same two-step dance. The list response has a collection resourceVersion; the watch starts from it and streams every change after it:

GET /api/v1/namespaces/test/pods
→ 200 {"metadata":{"resourceVersion":"10245"},
       "items":[...]}

GET /api/v1/namespaces/test/pods?watch=1
    &resourceVersion=10245
→ {"type":"ADDED",   "object":{... "10596" ...}}
  {"type":"MODIFIED","object":{... "11020" ...}}

If the connection drops, the client resumes the watch from the last version it saw. No events are missed, as long as that version is still in history. The docs say clusters using etcd 3 keep changes for the last 5 minutes by default. Ask for anything older and you get 410 Gone: clear your cache, list again, and watch from the new version.

Bookmarks soften this. With allowWatchBookmarks=true the server may send BOOKMARK events that carry only a fresh resourceVersion, meaning "you're up to date until here", so a quiet watch still has a recent version to resume from. Don't rely on their timing: the server may send them at any interval, or not at all.

Chapter 1 drew the controller as a loop: observe, diff, act. Here is the machinery inside that loop, and the two metadata fields that decide what happens when you delete something: ownerReferences and finalizers.

Takeaway: a controller doesn't react to events, it reacts to state. An event only tells it which object to look at again. Deletion is a process, not an instant: owners, dependents and finalizers decide its order. The misconception to kill: "kubectl delete removes the object right away". With a finalizer on it, the object can sit in Terminating forever.

Informer → workqueue → reconcile

The client-go machinery that almost every controller uses has four parts:

1. A Reflector does the list-then-watch from the previous chapter for one resource type.
2. The informer keeps a local, indexed cache of every object and calls your event handlers on add/update/delete. All reads in reconcile come from this cache, not from the API server.
3. The handlers don't do work. They compute the object's key (namespace/name) and put it on a workqueue. The queue is "stingy": a key added five times before a worker picks it up is processed once, and never by two workers at the same time.
4. Workers pop a key and call reconcile: read the current object from the cache, compare desired with actual, act. On error the key goes back in with per-item exponential back-off.

That's what level-triggered means. The Kubernetes API conventions say behaviour should be level-based rather than edge-based: if replicas goes 2 → 5 → 3 while your controller is busy, it only ever needs to see 3. Missed or merged events are harmless, because reconcile always looks at the whole current state.

Owners and dependents

A ReplicaSet created by a Deployment has metadata.ownerReferences pointing at the Deployment; each Pod points at its ReplicaSet. Controllers set these automatically, together with blockOwnerDeletion: true. Owner references can't cross namespaces: a namespaced dependent may have a namespaced owner in the same namespace or a cluster-scoped owner.

When you delete an owner, the garbage collector (a controller in kube-controller-manager) follows those references:

Finalizers: "not yet, I have cleanup to do"

A finalizer is just a string in metadata.finalizers. When you delete an object that has one, the API server doesn't remove it. It sets deletionTimestamp, returns 202 Accepted, and waits. The controller that owns the finalizer sees the timestamp, does its cleanup (release a cloud load balancer, detach a volume) and removes its string. When the list is empty, the object is gone.

A real example: kubernetes.io/pvc-protection keeps a PersistentVolumeClaim in Terminating while a Pod still uses it. Once an object is being deleted, you can remove finalizers but not add new ones.

kubectl get cm app -o jsonpath='{.metadata.finalizers}'
kubectl patch cm app --type=merge \
  -p '{"metadata":{"finalizers":null}}'

kube-scheduler has exactly one job: for each Pod with no nodeName, pick a node and write that choice back to the API server as a binding. It never starts a container. The kubelet on the chosen node sees the Pod through its watch and does the rest (next chapter).

Takeaway: every attempt runs the same funnel. Filter drops the nodes that can't run the Pod (the rest are feasible), score ranks the feasible ones, and bind commits the winner; ties are broken at random. The misconception to kill: "the scheduler puts Pods where CPU is idle". The resource filter compares a node's free capacity with the Pod's requests (chapter 7). Live usage doesn't enter the decision.

The scheduling framework

Internally, each attempt is split into a scheduling cycle (choose a node) and a binding cycle (apply the choice). Scheduling cycles run one Pod at a time; binding cycles may run concurrently. Every step is a named extension point where plugins hook in, and the built-in behaviour is itself a set of plugins:

Telling the scheduler what you want

spec:
  topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: topology.kubernetes.io/zone
    whenUnsatisfiable: DoNotSchedule
    labelSelector:
      matchLabels: {app: web}
  tolerations:
  - key: gpu
    operator: Exists
    effect: NoSchedule

The "IgnoredDuringExecution" in affinity names means what it says: if node labels change after the Pod is placed, the Pod keeps running. Taints are different: a NoSchedule taint only affects new Pods, but a NoExecute taint evicts running Pods that don't tolerate it. (PreferNoSchedule is the soft version.) Even without your own constraints, the scheduler applies default spreading: maxSkew: 3 per hostname and maxSkew: 5 per zone, both ScheduleAnyway.

Priority and preemption

A PriorityClass maps a name to an integer (user classes go up to 1,000,000,000; the built-in system-cluster-critical and system-node-critical sit above that). Pods without a class get the globalDefault class's value, or 0. Higher-priority Pods are queued first.

If no node fits a Pod, PostFilter tries preemption: find a node where evicting lower-priority Pods would make room, set the Pod's status.nominatedNodeName, and delete the victims, which get their normal graceful termination. PodDisruptionBudgets are respected on a best-effort basis only. A class with preemptionPolicy: Never jumps the queue but never evicts anyone.

Once a Pod has a nodeName, the control plane is done with it. From here on it's the kubelet on that node: it turns a PodSpec into running containers, reports status back, and keeps telling the control plane "I'm alive".

Takeaway: the kubelet reconciles Pods on its node through the CRI, and proves it's alive with a Lease renewed every 10 seconds. When that stops, the control plane marks the node Unknown after 50 s, taints it, and evicts its Pods only after their 300-second toleration runs out. The misconception to kill: "when a node dies, its Pods are rescheduled right away". With defaults it's closer to six minutes.

From PodSpec to processes

The kubelet runs a sync loop: it collects work for the Pods assigned to its node (from its API watch, and from static Pod files), and per-Pod workers drive each one towards its spec. To actually run anything it's a gRPC client of the container runtime (containerd, CRI-O) through the Container Runtime Interface. Since v1.26 the runtime must speak the v1 CRI API.

1. Ask the runtime for a pod sandbox: the shared network namespace and cgroup parent. On Linux it's held by the pause container you met in chapter 3.
2. The runtime calls the CNI plugin to give the sandbox its network and IP (chapter 23).
3. Pull images as needed, run init containers in order, then start the app containers with the cgroup limits from your requests and limits (chapter 7).
4. The PLEG (Pod Lifecycle Event Generator) periodically polls the runtime for container state and wakes the sync loop, which writes Pod status to the API server. That polling is why status can lag reality slightly.

Image pulls

If you omit imagePullPolicy, the API server fills it in once, at creation:

It isn't recomputed when you later change the tag, so switching a Deployment to :latest keeps IfNotPresent. The kubelet pulls images one at a time by default (serializeImagePulls: false allows parallel pulls). A failing pull ends in ImagePullBackOff, with retries backing off up to 300 seconds.

Heartbeats and what happens when they stop

Every Node has a Lease with the same name in kube-node-lease (chapter 2). The kubelet renews it every 10 s (lease duration 40 s), which is a cheap update. The heavier Node .status is only posted when something changes or every 5 minutes. The node controller checks nodes every 5 s:

Evictions are rate-limited (by default one node per 10 seconds) and slow down or stop when a large part of a zone is unhealthy, so a network blip doesn't empty the cluster. You can tune the wait per Pod: a shorter tolerationSeconds fails over faster, a longer one keeps stateful Pods in place through short partitions.

tolerations:
- key: node.kubernetes.io/unreachable
  operator: Exists
  effect: NoExecute
  tolerationSeconds: 60

Node-pressure eviction

The kubelet also evicts Pods on its own when the node runs short. Default hard thresholds on Linux: memory.available<100Mi, nodefs.available<10%, imagefs.available<15%, nodefs.inodesFree<5%, imagefs.inodesFree<5%. It sets a pressure condition (the node gets e.g. a memory-pressure taint), then ranks Pods: first those whose usage exceeds their requests, by priority and by how far over they are. Hard evictions use a 0 s grace period, so your preStop hook doesn't get its usual time. Soft thresholds come with a grace period you configure.

Chapter 5 gave you the model: a Service is a stable virtual IP, kube-proxy programs rules, DNS gives it a name. Now follow a packet, and see which knobs change its path.

Takeaway: a Service IP isn't a host or a proxy process. It's a rule in every node's kernel that rewrites the destination (DNAT) to one Pod IP when a connection starts. Where that Pod is, whether the source IP survives, and how many DNS lookups happen first are all configurable. The misconception to kill: "kube-proxy load-balances every request". It picks a backend per connection, and a long-lived connection stays on that one Pod.

The pod network model

Kubernetes demands three things and leaves the implementation to a plugin: every Pod gets its own cluster-wide unique IP; Pods can reach all other Pods, on any node, directly, without NAT; and agents on a node (like the kubelet) can reach all Pods on that node. A CNI plugin builds this (bridges, routes, overlays or cloud-native routing). The container runtime loads the CNI plugins; since v1.24 the kubelet no longer manages them. NetworkPolicy (chapter 16) is also enforced by the plugin, or not at all.

kube-proxy modes

In the iptables example from the docs, the per-Service rules jump to per-endpoint rules, which DNAT to the chosen backend when the client connects, without rewriting the client IP for in-cluster traffic. The kernel's conntrack table remembers that translation for the rest of the connection. So a gRPC or HTTP/2 client with one long-lived connection sends every request to the same Pod, and scaling up doesn't help it until it reconnects. Balance per request in the client or in a proxy/mesh.

Traffic policies and topology

A Local traffic policy beats trafficDistribution for its traffic type. The older topology-aware routing (annotation service.kubernetes.io/topology-mode: Auto) lets the EndpointSlice controller put zone hints on endpoints, allocated in proportion to each zone's allocatable CPU; it backs off when endpoints can't be allocated evenly.

CoreDNS and ndots:5

You saw the Pod's resolv.conf in chapter 5: three search domains and options ndots:5. The resolv.conf man page defines ndots as the number of dots a name needs before it is tried as an absolute name first. api.stripe.com has two dots, fewer than five, so the resolver walks the search list first:

api.stripe.com.shop.svc.cluster.local  NXDOMAIN
api.stripe.com.svc.cluster.local       NXDOMAIN
api.stripe.com.cluster.local           NXDOMAIN
api.stripe.com                         ✓

Kubernetes picks 5 so that every name it generates (like web.shop.svc.cluster.local, four dots) still goes through the search list. The price is extra queries for every external name. For a Pod that calls external APIs a lot, lower it:

spec:
  dnsConfig:
    options:
    - name: ndots
      value: "2"

With ndots:2, web and web.shop still expand, while api.stripe.com is tried as-is first. Many resolvers also treat a name with a trailing dot (api.stripe.com.) as fully qualified and skip the search list; test that with your language's resolver before relying on it. Pods with hostNetwork: true need dnsPolicy: ClusterFirstWithHostNet to use cluster DNS; ClusterFirst is the default (the policy named Default is not).

Everything you have met so far — Pods, Deployments, Services — is a type the API server knows. But the API is not a closed list. You can add your own nouns: a Postgres, a Certificate, a CronTab. And once the API knows the noun, you can write the control loop from chapter 1 that makes it true.

Takeaway: a CustomResourceDefinition (CRD) adds a new resource type — storage, REST endpoints, kubectl get, watch, RBAC — and a custom controller gives it behavior. The pair is what people call an operator. The misconception to kill: "installing a CRD makes something happen". In the docs' words, on their own custom resources "let you store and retrieve structured data". No controller, no action.

A CRD is itself just an object. Apply it, and a few seconds later the API server serves a new endpoint such as /apis/stable.example.com/v1/namespaces/*/crontabs/…. The CRD's name must be <plural>.<group>. Here is the docs' example, trimmed:

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: crontabs.stable.example.com
spec:
  group: stable.example.com
  scope: Namespaced
  names:
    plural: crontabs
    singular: crontab
    kind: CronTab
    shortNames: [ct]
  versions:
  - name: v1
    served: true
    storage: true   # exactly one version
    schema:
      openAPIV3Schema:
        type: object
        properties:
          spec:
            type: object
            properties:
              cronSpec: {type: string}
              replicas: {type: integer}

That schema must be structural in apiextensions.k8s.io/v1: every field has a type, and the logical junctors (allOf, anyOf…) can only restrict fields that are also declared outside them. The payoff is pruning: any field the schema doesn't know is removed before the object is stored. A typo like replcas: 5 doesn't lurk in etcd — it simply disappears. If you really want free-form JSON somewhere, mark that node x-kubernetes-preserve-unknown-fields: true.

The status subresource is the contract between you and the controller. With it enabled, you own .spec and the controller owns .status: a PUT to the resource ignores status changes, and a PUT to /status ignores everything except status. Also, .metadata.generation increments on every change except to .metadata or .status — so a controller can copy the generation it last acted on into status.observedGeneration, exactly as the built-in Deployment does, and you can tell "done" from "not looked at yet".

The operator pattern, per the docs, captures "the key aim of a human operator who is managing a service": the know-how of installing, scaling, backing up and repairing, written as a controller. It usually runs in your cluster as an ordinary Deployment and marks the objects it creates with ownerReferences, so that deleting the CR garbage-collects its children. You don't need to write one from scratch: kubebuilder, the Operator Framework, the Java Operator SDK, Kopf (Python), kube-rs (Rust) and others are listed on the docs page.

A single control plane node is a single point of failure — not for your running Pods (they keep running), but for every change. No scheduling, no scaling, no rollouts, no self-healing. High availability is about keeping the brain online when a machine dies.

Takeaway: an HA control plane is three different kinds of redundancy stacked together: etcd replicates by quorum, kube-apiserver runs active-active behind a load balancer, and controller-manager and scheduler run active-standby using a Lease as a lock. The misconception to kill: "more etcd members means more fault tolerance". Four members survive exactly as many failures as three.

Quorum math. etcd needs a majority — (n/2)+1, integer division — to agree on every write. Three members need two, so you can lose one. Four need three, so you can still lose only one. Five survive two. That's why the etcd docs recommend odd sizes and "probably … no more than seven": each extra member adds replication work to every write. Lose the majority and etcd "cannot accept more writes"; per the Kubernetes docs, running Pods may keep running, but no new Pods can be scheduled.

kube-apiserver is stateless — all state is in etcd — so every instance serves traffic. Workers and kubectl reach them through a load balancer. With kubeadm, you create a TCP load balancer first and pass its address to kubeadm init --control-plane-endpoint "LB_DNS:PORT" --upload-certs; further control plane nodes join with kubeadm join … --control-plane --certificate-key …. (The uploaded certificates are deleted after two hours as a safeguard.)

Controllers must not run twice. Two schedulers both binding the same Pod would be chaos, so kube-controller-manager and kube-scheduler use leader election: each instance tries to hold a Lease object in kube-system. The holder renews it (by default every 2 s retry, giving up if it can't renew within 10 s); the standbys wait until the lease has gone unrenewed for 15 s and then take it. The same Lease API powers node heartbeats (chapter 22).

kubectl -n kube-system get lease kube-scheduler \
  -o jsonpath='{.spec.holderIdentity}'

Your rolling update (chapter 4) waits for readiness before sending traffic to new Pods. So why do you still see a handful of 502s on every deploy? Because the dangerous part is not the start of a Pod. It's the end.

Takeaway: when a Pod is deleted, two things start at the same time: the kubelet begins shutting the containers down, and the control plane starts taking the Pod out of Service endpoints. If your app exits before every load balancer and node has heard the news, requests still arrive at a closed door. The misconception to kill: "a rolling update is zero-downtime by default". It is, only if your app shuts down politely.

The termination sequence, from the docs:

1. You delete the Pod (or a rollout, drain or scale-down does). The API server records a deadline — default grace period 30 s (terminationGracePeriodSeconds) — and the Pod shows as Terminating.
2. The kubelet runs the container's preStop hook, if any. The hook must complete before the stop signal is sent — and the grace countdown already started.
3. The container runtime sends SIGTERM (or the image's STOPSIGNAL) to PID 1 in each container.
4. At the same time as step 2, the endpoint is marked terminating and ready: false in its EndpointSlices, and every kube-proxy and ingress controller reacts when its watch delivers the change.
5. When the grace period runs out, anything still alive gets SIGKILL. (If a preStop hook is still running, the kubelet grants a one-off 2 s extension first.)

The fix is a pause. Keep accepting requests for a few seconds after deletion, so routing can catch up, then drain gracefully: stop taking new connections, finish in-flight requests, exit. The simplest way is a preStop sleep — since 1.34 the built-in sleep action is stable, so you don't even need a sleep binary in your image:

spec:
  terminationGracePeriodSeconds: 45
  containers:
  - name: api
    lifecycle:
      preStop:
        sleep:
          seconds: 5   # let routing catch up

Budget the grace period as preStop + your slowest drain. The docs' own example: grace 60 s, hook 55 s, app needs 10 s to stop → killed before it finishes. And make sure SIGTERM actually reaches your process: if PID 1 is a shell script that doesn't forward signals, your app never hears it and dies by SIGKILL at the deadline.

PodDisruptionBudgets protect you from voluntary disruptions that go through the Eviction API — above all kubectl drain, which cordons a node and then evicts its Pods. A PDB says how many Pods of an app must stay up (minAvailable) or may be down (maxUnavailable), as a number or a percentage. If an eviction would break the budget, the API answers 429 Too Many Requests and drain retries until it can proceed or times out.

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata: {name: api-pdb}
spec:
  maxUnavailable: 1
  selector:
    matchLabels: {app: api}
  unhealthyPodEvictionPolicy: AlwaysAllow

unhealthyPodEvictionPolicy: AlwaysAllow (stable since 1.31) is the docs' recommendation: without it, a PDB whose Pods are already crash-looping waits for them to become healthy, and your node drain hangs on a broken app.

Kubernetes ships a new minor version every few months, and each release branch gets patch support for only about fourteen months (the patch-releases page lists roughly four active branches at a time). Put those facts together: if you never upgrade, you're out of support in little over a year. Upgrading is not a project, it's a habit.

Takeaway: upgrade one minor version at a time, control plane first (kube-apiserver, then controller-manager and scheduler), then the kubelets node by node with a drain. Components may lag behind the API server, never lead it. The misconception to kill: "I can jump 1.34 → 1.37 in one go, or start with the nodes". kube-apiserver can't skip minor versions, and a kubelet newer than its API server is unsupported.

With kubeadm, the ritual on the first control plane node is kubeadm upgrade plan then kubeadm upgrade apply v1.37.x; other control plane nodes and every worker run kubeadm upgrade node. For each kubelet: drain, upgrade the kubelet package, restart it, uncordon. The kubeadm docs are blunt about it: for a minor upgrade you must drain the node first. That drain is where chapter 26's PodDisruptionBudgets and preStop hooks pay for themselves.

kubectl drain node-a --ignore-daemonsets
# upgrade kubelet + kubectl packages, then:
sudo systemctl daemon-reload
sudo systemctl restart kubelet
kubectl uncordon node-a

Deprecated APIs are the real upgrade risk. Kubernetes removes old beta API versions on a schedule: a beta version is deprecated at most 9 months or 3 releases after it appears, and stops being served 9 months or 3 releases after that (whichever is longer). GA versions are never removed within a major version. When 1.25 stopped serving batch/v1beta1 CronJob and policy/v1beta1 PodDisruptionBudget, manifests still using those versions simply failed to apply. Your YAML, charts and tools are what break first.

Sooner or later a cluster is shared: several teams, several apps, maybe several customers. A namespace looks like the obvious fence. It's a start — but on its own it is a folder, not a wall.

Takeaway: a tenant boundary is a namespace plus the policies you attach to it: RBAC (chapter 16), a ResourceQuota, a LimitRange, Pod Security Admission labels and a default-deny NetworkPolicy. The misconception to kill: "namespaces isolate workloads". By default Pods in different namespaces can talk to each other, share the same nodes and compete for the same capacity.

ResourceQuota caps a namespace's total: sums of requests.cpu, limits.memory, storage, and object counts like count/deployments.apps or pods. A create that would exceed it is rejected with 403 Forbidden and a message naming the constraint. Two surprises: a quota on CPU or memory means every Pod must declare those values or it may be rejected; and a Deployment over quota is created fine — it's the ReplicaSet's Pod creations that fail, so the error shows up as a ReplicaFailure / FailedCreate condition, not in your terminal.

apiVersion: v1
kind: ResourceQuota
metadata: {name: team-a, namespace: team-a}
spec:
  hard:
    pods: "4"
    requests.cpu: "2"
    requests.memory: 4Gi
    limits.memory: 8Gi

LimitRange fixes the first surprise: it applies default requests and limits to containers that don't set them, and enforces per-container min/max — at admission only, never on running Pods (chapter 7 showed how defaults interact with limits). Quota plus LimitRange is the standard pair.

kubectl label ns team-a \
  pod-security.kubernetes.io/enforce=restricted \
  pod-security.kubernetes.io/warn=restricted

Namespaces or clusters? Some things can't be split by namespace at all: CRDs, webhooks, nodes and the API server itself are shared. The docs describe isolation as a spectrum from "soft" to "hard"; for tenants that don't trust each other, a dedicated cluster — or a virtual control plane per tenant — may be the easier answer.

Most Kubernetes incidents look scary and turn out to be one of seven things. The trick is to stop guessing and follow the same path every time: what state is the Pod in? → what do the events say? → what did the container say before it died?

Takeaway: the STATUS column tells you which layer is failing — scheduler (Pending), kubelet/registry (ImagePullBackOff), config (CreateContainerConfigError), your process (CrashLoopBackOff, OOMKilled) or networking (Running but unreachable). The misconception to kill: "CrashLoopBackOff is the error". It's only the kubelet saying "I'm waiting before restarting this again". The error is in the previous container's logs and exit code.

Back-off is a clock, not a verdict. After a crash, the kubelet restarts the container with an exponential back-off delay; if the container then runs fine for a while (the docs' example: 10 minutes), the back-off resets. Image pulls behave the same way, with the delay capped at 300 seconds. That's why "it fixed itself after five minutes" often just means the next retry finally happened after you fixed the registry.

kubectl debug is for the day your image has no shell (distroless, scratch). It adds an ephemeral container (stable since 1.25) to a running Pod: it can share the target container's process namespace, has no ports or probes, and is never restarted.

# shell next to a running container
kubectl debug -it api-7c9f --image=busybox:1.28 \
  --target=api
# copy of the Pod with a different command
kubectl debug api-7c9f -it --copy-to=api-dbg \
  --container=api -- sh
# a shell on the node; its root fs is at /host
kubectl debug node/node-a -it --image=ubuntu

You now know what's inside the box: etcd quorum, certificates, leader election, version skew, a steady stream of upgrades. Somebody has to own all of that. The last decision in this guide is who — and whether you need the box at all.

Takeaway: a managed control plane takes the brain off your plate; your workloads, manifests, policies and API migrations remain yours. And for a small app, Kubernetes' flexibility can cost more than it gives. The misconception to kill: "managed Kubernetes means no ops" — deprecated APIs, PDBs, quotas and graceful shutdown still land on your desk.

The docs' own list of what you can hand off: a managed control plane ("let the provider manage the scale and availability of the cluster's control plane, as well as handle patches and upgrades"), managed worker nodes, integrations with storage, registries and identity, or going fully serverless — running workloads "without managing a cluster at all", paying for CPU, memory and disk requests.

When is Kubernetes overkill? This part is opinion, so weigh it as such. The docs describe Kubernetes as "not a traditional, all-inclusive PaaS": it "provides the building blocks for building developer platforms". If you have one or two services and no platform team, a PaaS, a serverless container service, or Docker Compose on a single VM gives you most of the value — deploys, restarts, a URL — with a fraction of the concepts. Kubernetes starts paying off when you have many services, several teams, a need for portability, or workloads (batch, stateful, GPU) that simple platforms don't cover.

Multi-cluster is the other end of the scale. Reasons to run more than one cluster: hard tenant isolation (chapter 28), limiting blast radius, and staying up when a region fails — the production docs suggest spreading one cluster across zones in the same region, and a cluster like that shares its region's fate. The cost: every cluster has its own upgrades, add-ons and policies to keep in sync, which is why GitOps (chapter 17) becomes non-optional once you have several.

Cheat sheet