Izzy

OOMKilled without a Java OutOfMemoryError

17 min read

The short version

  • The JVM’s footprint is much bigger than -Xmx. The kernel kills on total RSS, not on heap.
  • So the heap can be healthy, GC can be calm, and the container still dies. No exception, no heap dump — SIGKILL leaves nothing behind.
  • Exit code 137 = 128 + 9 (SIGKILL). It says how the process died, not why.
  • 137 has two completely different causes: your container went over its own limit, or the node ran out and your pod was picked as the victim. The fix is different for each, so find out which one before tuning anything.

Signals: why the exit code is 137

A container’s exit code is 128 + signal number. That is the whole story behind the magic numbers:

SignalExit codeSent by
SIGTERM (15)143kubelet, first, on any delete or rollout
SIGKILL (9)137the OOM killer, or kubelet at grace-period end
SIGQUIT (3)—you, for a thread dump — the JVM keeps running

kill -9 is not “a harder stop” — it is a categorically different thing. SIGKILL cannot be caught, blocked or ignored. The process never runs another instruction:

  • no shutdown hooks, no finally blocks, no @PreDestroy
  • no heap dump, no final log flush, no in-flight request drained
  • nothing written to stdout explaining what happened

So any plan that begins “when we run out of memory the app will gracefully…” is wrong for OOMKilled. There is no hook. The only defence is not reaching the limit.

137 does not always mean OOM

The kubelet also SIGKILLs a container that outlives terminationGracePeriodSeconds (default 30s) after its SIGTERM. Same exit code, completely different bug. The reason disambiguates:

kubectl get pod POD -o jsonpath='{.status.containerStatuses[*].lastState.terminated.reason}'
  • OOMKilled → memory. This note.
  • Error with exit 137 shortly after a delete/rollout → your app ignored SIGTERM and was killed at the grace period. Usually a shell wrapper: sh -c "java -jar app.jar" makes the shell PID 1, and it does not forward signals. Use exec java -jar app.jar so the JVM becomes PID 1.

PID 1 in a namespace gets kernel protection: signals it has installed no handler for are discarded, even SIGKILL, when sent from inside the namespace. The kubelet and the OOM killer act from outside it, so they are unaffected — but it is why kill -9 1 from your own exec session may appear to do nothing.

Why it happens: two views of the same memory

The confusion in almost every 137 investigation is that the kernel and the JVM are measuring different things.

The kernel’s view: touch, charge, reclaim, kill

  1. Allocating is not using. malloc/mmap hands out virtual address space. Physical pages are only assigned on first write (a page fault). A JVM that reserves a 4 Gi heap consumes almost no physical memory until it touches it.
  2. The cgroup is charged as pages are faulted in, against memory.max (v2) / memory.limit_in_bytes (v1).
  3. When a charge would exceed the limit, the kernel tries to reclaim first: drop clean page cache, write back dirty pages, swap out anonymous pages.
  4. The heap is anonymous memory, and Kubernetes nodes run without swap. Anonymous pages have nowhere to go, so there is nothing to reclaim.
  5. Reclaim fails → the cgroup OOM killer fires, picks the highest oom_score in that cgroup, and sends SIGKILL.

Two consequences worth internalising:

  • Page cache counts toward the cgroup but is reclaimable. That is why container_memory_usage_bytes overstates pressure for anything doing file I/O, and why container_memory_working_set_bytes (usage minus reclaimable file pages) is the honest number.
  • Memory is incompressible. Over-subscribed CPU gets throttled and everything runs slower. Over-subscribed memory gets somebody killed. There is no degraded mode.

The JVM’s view: reserved ≠ committed ≠ used

Three different numbers, and people routinely compare the wrong two:

  • reserved — virtual address space claimed up front (all of -Xmx at startup). Costs no physical memory, counts for nothing.
  • committed — backed by the OS and faulted in. This is the number that counts against your limit.
  • used — what live objects actually occupy; what jvm_memory_used_bytes reports on your dashboard. A subset of committed.

The kernel kills on RSS, which tracks committed-and-touched. Your dashboard shows used. So “the heap was only 40% used” is not evidence of anything.

And critically: GC frees used, not necessarily committed. A heap that spiked once during a nightly batch stays committed at that size. G1 uncommits on a full GC and with -XX:G1PeriodicGCInterval; ZGC and Shenandoah return memory more readily; -XX:MaxHeapFreeRatio / -XX:MinHeapFreeRatio tune the shrinking. This produces the classic shape: RSS ratchets upward, never comes back down, and crosses the limit days after the deploy.

Which is the argument for -XX:+AlwaysPreTouch: it commits and touches the whole heap at startup. You pay a slower start and get a truthful RSS from second one — you discover you are over-provisioned at boot rather than at 3am under load.

What the JVM uses beyond the heap

  • Metaspace (class metadata) — Spring Boot typically 60–120 Mi.
  • Code cache (JIT-compiled code) — ~50 Mi at steady state with tiered compilation.
  • Thread stacks — -Xss, default 1 Mi per thread. Tomcat’s 200 + Hikari + Kafka consumers + HTTP client pools add up fast.
  • Direct and mapped byte buffers — Netty, NIO, gRPC. Off-heap, and often the surprise.
  • GC native structures — G1 remembered sets, card tables.
  • Native libraries and the allocator itself (glibc arenas — see below).

OutOfMemoryError is not only about the heap either. It also covers Direct buffer memory, unable to create native thread, Metaspace and Requested array size exceeds VM limit. “Unable to create native thread” is a native failure that surfaces as a Java error, so the two worlds are not as separate as they look.

Triage lanes

Work these in order. Each lane has one question and one command.

Lane 0 — was it even your container, and was it near its own limit?

kubectl get pod POD -o jsonpath='{.status.containerStatuses[*].lastState.terminated}' | jq
kubectl get events --field-selector involvedObject.name=POD --sort-by=.lastTimestamp

Then look at what memory actually did in the minutes before the kill (Prometheus, not kubectl top — see below):

container_memory_working_set_bytes{pod="POD", container="app"}
kube_pod_container_resource_limits{pod="POD", resource="memory"}
  • Working set was riding your limit → your container. Continue to Lane 1.
  • Working set was nowhere near your limit → the node was exhausted and the kernel picked you. Skip to when the node runs out. Tuning MaxRAMPercentage here fixes nothing.

Also check the node’s kernel log. The cgroup OOM report lists RSS per process, which instantly tells you whether the JVM was the victim or something else in the pod:

kubectl debug node/NODE -it --image=busybox -- dmesg -T | grep -i -A5 'killed process'

The kernel kills a process, not a container. If it takes a sidecar, an exec probe, or a jcmd you ran to debug, the container survives in a confusing half-state and the restart count doesn’t move.

Lane 1 — does the JVM even see the container limit?

If it doesn’t, it sizes the heap off the node’s RAM and you get exactly this symptom, no matter what percentage you set.

kubectl exec POD -- java -XX:+PrintFlagsFinal -version | grep -E 'MaxHeapSize|UseContainerSupport'

MaxHeapSize should be a fraction of your limit. If it looks like a quarter of the node, container detection is broken. Two causes:

  • JDK older than 8u191 — no UseContainerSupport at all.
  • A cgroups v2 host with a JDK older than 8u372 / 11.0.16 / 15. Detection silently falls back to host memory.

The second one is the live trap today: RHEL 9, Ubuntu 22.04+, Debian 12 and recent managed node images all default to cgroup v2. A service pinned to Java 11.0.15 on a modern node will quietly do this. Check the node:

stat -fc %T /sys/fs/cgroup    # cgroup2fs = v2, tmpfs = v1

Fix by upgrading the JDK. Do not work around it with flags.

Lane 2 — heap or non-heap?

Compare the container’s total against what the JVM admits to (Actuator/Micrometer):

container_memory_working_set_bytes{pod="POD"}
jvm_memory_used_bytes{area="heap"}
jvm_memory_used_bytes{area="nonheap"}
  • Heap climbing and not coming down after a full GC → leak. Heap dump, Eclipse MAT.
  • Heap flat, container total climbing → non-heap. Lane 3.

For a real OutOfMemoryError, dump to a path that outlives the pod:

-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/dumps

This flag is useless for plain OOMKilled — SIGKILL throws no error, so the hook never fires. To take a dump by hand, and for the pauses and pitfalls involved, see getting data out of a live pod.

Lane 3 — non-heap: JVM-tracked or native?

# add -XX:NativeMemoryTracking=summary, restart, then:
kubectl exec POD -- jcmd 1 VM.native_memory summary

(No jcmd in the image? jattach 1 jcmd "VM.native_memory summary" — see below.)

Two caveats that cost people days:

  • NMT only accounts for the JVM’s own allocations. A native library that mallocs is invisible to it, so NMT can look perfectly clean while RSS climbs.
  • NMT committed ≠ RSS. It has ~5–10% overhead, so turn it on to investigate, not by default.

If NMT is clean but RSS grows, the usual suspects:

  • glibc malloc arenas — up to 8 × cores, fragmenting. Try MALLOC_ARENA_MAX=2, or switch to jemalloc / a musl base image. This is the classic “everything looks fine but RSS climbs” cause.
  • Thread count — jcmd 1 Thread.print | grep -c '^"'. Bound your pools.
  • Direct buffers — cap them: -XX:MaxDirectMemorySize=... (otherwise it defaults to roughly -Xmx, i.e. it can double your budget).

Getting data out of a live pod

Ordered by what is actually reachable, not by what is nicest.

1. Thread dump with no tools at all — SIGQUIT

kubectl exec POD -- kill -3 1
kubectl logs POD --tail=1000

The JVM prints every thread’s stack to stdout and keeps running, so it lands in your normal log pipeline. This works in distroless images with no JDK, no shell and no debugger. Take three dumps a few seconds apart: one tells you where threads are, three tell you what is actually stuck.

2. jcmd — if the image ships a JDK

jstack and jmap are legacy; jcmd supersedes both.

kubectl exec POD -- jcmd 1 Thread.print
kubectl exec POD -- jcmd 1 GC.heap_info
kubectl exec POD -- jcmd 1 GC.class_histogram          # cheap, often enough to spot the leak
kubectl exec POD -- jcmd 1 VM.native_memory summary    # needs NativeMemoryTracking
kubectl exec POD -- jcmd 1 GC.heap_dump /dumps/heap.hprof

3. jattach — when the image is JRE-only or distroless

Most production images ship a JRE or are distroless, so jcmd is not there. jattach is a ~24 KB static binary that speaks the JVM’s dynamic attach protocol and gives you threaddump, dumpheap, inspectheap and jcmd without a JDK. Bake it into the image, or bring it in with an ephemeral debug container:

kubectl debug -it POD --image=alpine --target=app -- sh
# --target shares the app container's process namespace, so the JVM is PID 1 here
./jattach 1 threaddump
./jattach 1 dumpheap /dumps/heap.hprof

Attach needs the same UID as the JVM and access to its /tmp socket, and it is disabled outright by -XX:+DisableAttachMechanism.

4. JFR — the one to have running before the incident

Java Flight Recorder is built into OpenJDK 11+ (backported to 8u262), costs roughly 1% overhead, and records allocation, GC and thread data continuously. Turn it on by default:

-XX:StartFlightRecording=disk=true,maxsize=256m,dumponexit=true,filename=/dumps/app.jfr

kubectl exec POD -- jcmd 1 JFR.dump name=1 filename=/dumps/now.jfr

Analyse in JDK Mission Control. For a slow leak, JFR’s allocation profile usually finds the culprit faster than a heap dump does — and unlike a dump, it survives the thing you are debugging.

5. VisualVM / JMX — honest assessment

It needs JMX exposed, RMI hostname wrangling, and a port-forward:

kubectl port-forward POD 9010:9010
-Dcom.sun.management.jmxremote.port=9010
-Dcom.sun.management.jmxremote.rmi.port=9010
-Djava.rmi.server.hostname=127.0.0.1

Fine for a staging repro where you want to poke around live. Rarely the right tool on a production pod: the JMX surface is a security decision, the client is heavy, and everything it shows you is available more cheaply from JFR plus Micrometer. Reach for it third, not first.

6. async-profiler — CPU and allocation flame graphs

Same author as jattach. Excellent, but it needs perf_event_paranoid relaxed or SYS_ADMIN, so cluster policy often blocks it. Its allocation mode (-e alloc) is the fastest route from “RSS climbs” to “this call site”.

Heap dumps: the cautions that actually bite

  • A dump is roughly the size of the live heap. Writing it inside the container can push you over the memory limit (page cache) or over the ephemeral-storage limit, which gets you evicted while investigating an OOMKill. Write to a mounted volume with room.

  • It is a full stop-the-world pause — seconds to tens of seconds on a multi-gigabyte heap. Take the pod out of service first (kubectl label pod POD app- to drop it from the Service selector) so you do not cause an outage while debugging one.

  • Getting the file out: kubectl cp requires tar inside the image, which distroless does not have. Fall back to streaming it:

    kubectl exec POD -- cat /dumps/heap.hprof > heap.hprof

  • Analyse with Eclipse MAT, and for anything large run it headless — MAT’s GUI will happily OOM on a dump bigger than your laptop:

    ./ParseHeapDump.sh heap.hprof org.eclipse.mat.api:suspects

Combining with access and application logs

Dumps tell you what is on the heap. Logs tell you when and why now — you need both.

  • kubectl logs POD --previous is the only in-cluster record of what the app was doing when it was killed, and it disappears once the pod is rescheduled. Grab it first, before anything else.

  • Because SIGKILL writes no shutdown line, the last access-log entry is the marker for the moment of death. Line it up against the memory curve.

  • The shape of the correlation names the bug. Memory tracking cumulative request count since start → leak. Memory tracking a specific endpoint, payload size or a cron schedule → spike, and you can size for it.

  • Keep GC logs on always, they are nearly free:

    -Xlog:gc*,gc+heap=debug:file=/dumps/gc.log:time,uptime:filecount=5,filesize=10m

    Heap-after-full-GC ratcheting up over days is a leak. Heap-after-full-GC flat while RSS climbs is native — go to Lane 3.

  • Tie it together with a trace id: the memory spike usually belongs to one tenant, one endpoint or one unbounded query.

Sizing the container

Non-heap overhead is roughly constant, not proportional to the heap. That single fact drives the whole strategy: percentage flags are fine on big containers and dangerous on small ones.

limit ≈ heap
      + metaspace              (60–120 Mi for Spring Boot)
      + code cache             (~50 Mi)
      + threads × Xss          (1 Mi each by default)
      + direct buffers
      + GC structures + JVM baseline (~50 Mi)
      + 10% slack, never under 100 Mi
LimitHeap strategy
≤ 512 MiExplicit -Xmx, ~50%. Consider -XX:+UseSerialGC, and -Xss512k if the stacks are shallow.
1 Gi~55–60%, still worth being explicit.
2–4 Gi-XX:MaxRAMPercentage=65–70.
≥ 8 Gi-XX:MaxRAMPercentage=75–80.

Worked example of why 75% is wrong small: a 512 Mi limit at 75% is a 384 Mi heap, leaving 128 Mi for metaspace + code cache + stacks + everything else. That config is the bug.

Defaults worth setting

resources:
  requests:
    memory: 1Gi # equal to limits -> Guaranteed QoS, see below
  limits:
    memory: 1Gi
env:
  - name: JAVA_TOOL_OPTIONS
    value: "-XX:MaxRAMPercentage=60 -XX:+ExitOnOutOfMemoryError"
  • -XX:+ExitOnOutOfMemoryError is the important one. After an OutOfMemoryError the JVM is in an undefined state — threads died mid-operation, possibly holding locks. Let it die and be restarted rather than limp along still passing its readiness probe.
  • JAVA_TOOL_OPTIONS is picked up by every JVM in the container, including jcmd, jstack and keytool, and prints Picked up JAVA_TOOL_OPTIONS to stderr. JDK_JAVA_OPTIONS (JDK 9+) only affects the java launcher.

When the node runs out: QoS, OOM scores and eviction

Everything above assumes your container exceeded its own limit. When the node is short on memory, two separate systems can kill your pod — and only one of them looks like OOMKilled.

QoS classes

The class is derived from your requests and limits; you don’t set it directly.

ClassCondition
GuaranteedEvery container has requests == limits, for both cpu and memory.
BurstableAt least one request or limit is set, but not Guaranteed.
BestEffortNo requests or limits at all.

1. Node-level OOM kill (kernel) — looks like exit 137

When memory is exhausted faster than the kubelet can react, the kernel OOM killer runs and picks a victim by oom_score_adj, which the kubelet writes per container from the QoS class:

Classoom_score_adj
Guaranteed-997
BestEffort1000
Burstablemin(max(2, 1000 − (1000 × memoryRequest) / nodeCapacity), 999)

Higher score = killed first. Read the Burstable formula: the larger your memory request relative to the node, the lower your score, the safer you are. So yes — the pod that requests less is the one that gets killed when a neighbour’s growth exhausts the node.

On a 16 Gi node, sorted by the order the kernel would pick them:

PodRequestScoreKilled
D — BestEffortnone10001st
A — Burstable256 Mi~9852nd
B — Burstable8 Gi5003rd
C — Guaranteed2 Gi−997last

The sting: pod A shows OOMKilled, exit 137, even though it never came close to its own limit. Someone reading only the pod’s own status concludes “A leaks memory” and spends a day tuning MaxRAMPercentage on an innocent service. Lane 0 above exists to catch exactly this.

Known sharp edge: the Burstable formula uses only the request/capacity ratio — it does not consider your PriorityClass. A high-priority Burstable pod with a small request is still killed early (kubernetes#131169).

2. Kubelet node-pressure eviction — does not look like 137

Before things get that bad, the kubelet watches memory.available against its eviction thresholds (Linux hard default: memory.available<100Mi) and proactively evicts pods. Ranking, in order:

  1. Whether usage exceeds requests. Pods over their request go first; pods living within their request are evicted last.
  2. Pod Priority. Lower priority first.
  3. How far usage exceeds the request. Biggest offender first.

This produces a pod with status Failed / reason Evicted — not exit 137, no OOMKilled. If you see Evicted, you’re in this lane, not the JVM’s.

kubectl get pods --field-selector status.phase=Failed -A
kubectl describe node NODE | grep -A5 Conditions   # MemoryPressure

3. Scheduler preemption — different thing again

A pending higher-priority pod that can’t be scheduled will preempt lower-priority running pods to make room. That’s the scheduler at admission time, not memory pressure, and victims get a graceful termination. Look for Preempted in events.

What to take from this

  • requests == limits (Guaranteed) is the single highest-value setting. It buys oom_score_adj = -997 and puts you last in the eviction ranking. For anything you care about, do this.
  • Under-requesting to pack more onto a node makes you the first victim, both in the kernel’s ranking and the kubelet’s. The “savings” are paid back as pages.
  • Memory is incompressible. CPU over-subscription gets throttled; memory over-subscription gets somebody killed. Swap is traditionally disabled on Kubernetes nodes (recent versions have beta node swap support, but most clusters don’t run it), so there is no graceful degradation path.
  • A node can pass every dashboard while a single Burstable neighbour with a big limit and a small request quietly sets up this failure.

Things that surprise people

  • kubectl top pod will not catch the spike that killed you — it’s a sampled gauge over ~15–60s. Use it for a sanity check, never for a post-mortem.
  • The CPU limit changes your memory footprint. Runtime.availableProcessors() respects the cpu limit (JDK 10+), which sets GC and JIT thread counts, which are native memory. Raising a CPU limit can cause a memory OOMKill.
  • Restart counts lie if the kernel killed a non-PID-1 process in the pod.
  • initialDelaySeconds interacts with this: a JVM restarting into a memory-pressured node can be killed before it ever passes readiness, producing a crash loop that looks like a startup bug.

Sources

Backlinks