JVM Memory Errors: Diagnose Heap Exhaustion, StackOverflow, and Leaks

Read the error literally, then verify the process limit

java.lang.OutOfMemoryError: Java heap space means the JVM could not allocate an object in the Java heap after collection attempts. Other messages point elsewhere: Metaspace concerns class metadata, Direct buffer memory concerns native direct buffers, and unable to create native thread may mean native memory or an operating-system thread limit. A container can also kill the process for exceeding its memory limit without the JVM producing an OutOfMemoryError at all.

Start with the exact exception, JVM flags, container memory limit, process RSS, GC logs, and thread count. Do not assume -Xmx is the whole process footprint: metaspace, code cache, thread stacks, direct buffers, native libraries, and JVM structures consume memory too.

Heap pressure versus a leak

A heap that rises during a burst and falls after garbage collection may reflect a high allocation rate, not a leak. A leak is a reachability problem: objects that are no longer useful remain reachable through a longer-lived owner. Compare post-GC occupancy over time, capture a heap dump near the failure, and inspect dominator trees and retained sizes. Follow the shortest path from a large retained object back to a static cache, listener, queue, session, or request context.

Common causes include unbounded maps, caches without eviction, listeners never removed, queues without backpressure, and request objects retained by thread-local state. Fix ownership and lifecycle first. A bounded cache, explicit cleanup, or streaming instead of materializing a whole result set often addresses the real root cause.

StackOverflowError

A StackOverflowError usually means the current thread exhausted its call stack. Look for recursion without a sound termination condition, recursive object formatting, cyclic graph traversal, or an unexpectedly deep parser or callback chain. The stack trace often repeats the same methods. Replace unbounded recursion with an iterative worklist where appropriate; raising -Xss can postpone failure while increasing per-thread native memory consumption.

A production investigation sequence

1. Preserve the full error, GC log, JVM command line, container limit, and recent traffic/deploy changes.
2. Distinguish heap, metaspace, direct memory, native thread, and external OOM-kill symptoms.
3. Capture a heap dump or native-memory summary when safe; consider dump size and sensitive data handling.
4. Compare post-GC live set and retained paths across snapshots.
5. Reproduce with representative traffic, then bound the responsible cache, queue, fan-out, or object lifetime.
6. Recheck latency, allocation rate, pause time, RSS, and headroom after the fix.

A larger heap may be part of a capacity correction. It is not evidence that the defect is fixed. Prefer a diagnosis tied to retained ownership and measured process limits.