Concept notes · Kubernetes

Pod traffic, names and shutdown

Which connections a Pod may open, where a Service sends them, how a Pod is named, and how it is asked to stop.

Names below are fictional: namespace sample-app, Deployment greeter, Service quote-api, StatefulSet tally. Substitute your own.

Egress policy is an allow-list

By default a Pod may open connections to anything. Once at least one NetworkPolicy that includes Egress in its policyTypes selects a Pod, that Pod is isolated for egress: it may open only the connections that some policy selecting it allows. If policyTypes is left out, Egress is included whenever the policy has egress rules.

Policies only add permissions; there is no deny rule. The allowed set is the union of every policy that selects the Pod. Each egress rule allows the destinations in its to list on the listed ports. A to entry can be a podSelector, a namespaceSelector, both in one entry (both must match), or an ipBlock; separate entries are alternatives. An empty or missing to means every destination; a missing ports means every port. Replies to an allowed connection are allowed automatically. Enforcement is done by the cluster’s network plugin: a plugin without NetworkPolicy support accepts the objects and enforces nothing.

How it fails. A connection that worked before a policy appeared now times out or is refused, depending on the plugin. Every connection the selected Pod opens is judged against the list, not only the one the policy was written for.

Observe it.

kubectl get networkpolicy -n sample-app
kubectl describe networkpolicy greeter-egress -n sample-app

describe prints which Pods the policy selects, its policy types, and each allowed destination with its ports.

kubectl get pods -n sample-app --show-labels

Compare real labels with every selector: the policy’s own podSelector and each peer selector in to.

kubectl exec -n sample-app deploy/greeter -- curl -sS -m 3 http://quote-api:8080/health

Test from the selected Pod itself, using a client the image contains. A timeout from the Pod while the same request succeeds from an unselected Pod points at policy rather than at the destination.

What good looks like. Each Pod is selected by exactly the policies you intended. Every destination the workload needs connects from inside the Pod, and one destination it should not reach still fails: the negative control that proves the policy is enforced.

A Service spreads connections, not requests

A ClusterIP Service is a virtual address. kube-proxy programs every node so that the first packet of a new connection to that address is sent to one ready endpoint: chosen at random in iptables and nftables modes, round robin by default in IPVS mode. Connection tracking then keeps every later packet of that connection on the same Pod.

Requests are not balanced individually. HTTP keep-alive, client connection pools, HTTP/2 and gRPC carry many requests over one connection, so a long-lived connection talks to one backend for its whole life. sessionAffinity: ClientIP goes further and pins each client address to one endpoint, for 10,800 seconds by default.

How it fails. Load is uneven while every Pod is Ready: a few Pods are busy and others idle. Pods added by a scale-up, or created to replace others, receive little traffic until clients open new connections. When a Pod goes away, its clients’ connections break and reconnect elsewhere at once.

Observe it.

kubectl get endpointslices -n sample-app -l kubernetes.io/service-name=quote-api

The addresses that can receive new connections. Every Pod you expect to share the load should appear.

kubectl get service quote-api -n sample-app -o yaml

Read sessionAffinity and, if present, sessionAffinityConfig.

kubectl top pod -n sample-app -l app=quote-api

CPU and memory per Pod, from the metrics server. Skew between identical Pods with the same ready state is the signature of connection-level spreading. Per-Pod request counts from logs or metrics show the same thing more directly.

What good looks like. Requests per Pod are roughly even across all ready Pods, including the newest, within the time clients take to open new connections.

StatefulSet Pods have names you can call

A StatefulSet gives its Pods stable ordinal names: tally-0, tally-1, tally-2. By default they are created in that order, each after the previous one is Running and Ready. Each Pod can own a PersistentVolumeClaim from volumeClaimTemplates that stays with its name when the Pod is rescheduled.

The StatefulSet’s serviceName names a headless Service (clusterIP: None). Through it, cluster DNS publishes one record per Pod, tally-0.tally.sample-app.svc.cluster.local (the last part is the cluster domain), and the Service name itself returns the addresses of all ready Pods instead of one virtual IP. Records are published for ready Pods only, unless the Service sets publishNotReadyAddresses: true. The name is stable; the address behind it is not. A recreated Pod usually gets a new IP, and DNS follows it.

How it fails. A peer cannot be reached by name: the lookup returns NXDOMAIN, returns nothing yet, or returns an address that no longer answers because a client kept an old answer. Each Pod’s own health can look perfect while members cannot find each other.

Observe it.

kubectl get statefulset tally -n sample-app -o yaml
kubectl get service tally -n sample-app -o yaml

Read spec.serviceName and the replica count, then confirm the named Service exists, has clusterIP: None, and selects the StatefulSet’s Pods.

kubectl get pods -n sample-app -l app=tally -o wide

Names, readiness and current IPs. Only ready Pods have records.

kubectl exec -n sample-app tally-0 -- nslookup tally-1.tally.sample-app.svc.cluster.local

Resolve a peer from inside a member by its full name. Short names such as tally-1.tally rely on the client applying the search list, and some clients, such as busybox nslookup, do not for dotted names. The image needs nslookup. Compare the answer with the IP above.

What good looks like. Every member is Ready, each pod.service name resolves from a peer to that Pod’s current IP, and the headless Service name returns every ready member.

How a Pod is asked to stop

Deletion, rollouts, scale-down and drains all end a Pod the same way. The Pod is marked terminating, and in parallel the control plane marks its endpoints as terminating so Services stop sending it new connections. Clients that have not yet seen that change can still arrive for a short while.

The kubelet then runs each container’s preStop hook, if one is defined. When the hook returns, it sends SIGTERM (or the image’s configured stop signal) to the container’s main process, PID 1. One clock, terminationGracePeriodSeconds (30 seconds by default), covers the hook and the shutdown together. Any container still running when it ends receives SIGKILL. If the hook itself is still running at that point, the kubelet allows a single two-second extension, then kills.

How it fails. Work still in flight when SIGKILL arrives is cut off: clients see resets or 5xx responses, and jobs stop half done. A container killed this way reports exit code 137 (128 + 9) with reason Error, not OOMKilled. A process that never handles SIGTERM, for example a shell script as PID 1 that does not pass the signal on, always runs until the deadline and dies by SIGKILL.

Observe it.

kubectl get pod greeter-7c9d5f-x2k4p -n sample-app -o yaml

Read terminationGracePeriodSeconds and each container’s lifecycle.preStop. After a restart, lastState.terminated holds the exit code and reason.

kubectl events -n sample-app --for pod/greeter-7c9d5f-x2k4p

Shows Killing (“Stopping container”) and any hook failures with their times.

kubectl logs -n sample-app -f greeter-7c9d5f-x2k4p

Follow a Pod during a rollout. A clean shutdown logs its last messages before the stream ends; output that stops mid-work suggests the kill came first.

What good looks like. The hook plus the application’s own shutdown fit inside the grace period with margin, containers exit on SIGTERM before the deadline, and a rollout under load loses no requests.

References