In the programming languages like C, C++ – Developers have to both create and de-allocate the objects. If they accidentally miss to deallocate, those objects will start to build up in the memory leading to OutOfMemoryError. To address this issue, most modern programming languages like Python, Java, .NET, Golang… have automatic garbage collection capability. This feature frees up the developers from writing explicit code to de-allocate objects from memory.
This wonderful feature comes up with a major side effect – “Garbage Collection (GC) Pauses”. Yes, whenever an automatic garbage collector runs to reclaim unreferenced objects from memory, it pauses (i.e. freezes) the application. This application pause can span from a few milliseconds to several seconds (sometimes several minutes in few extreme cases).
These intermittent application freezes can silently degrade performance, disrupt user experience, and impact service availability. In this post let’s discuss what triggers the application freezes, what are their side effects, what are the tools and techniques to solve them.
What Causes Application To Freeze?
Modern applications tend to create thousands of objects even to service a simple vanilla sign-on request. Thus, at any given moment all our enterprise applications will tend to have millions/billions of objects in memory. It makes the garbage collector work more complex, because the garbage collector has to scan each and every object and their reference chains (which can span to several degrees). During this scanning process, if the application is allowed to run, they will repoint the object pointers. Thus, Garbage Collector freezes our applications, until it completes the work. This freeze can surface in two different forms:
1. Long GC Pause
2. Consecutive GC Events
Let’s understand the difference between them in this section.
- Long GC Pause: Due to lack of proper GC tuning, sometimes a single GC event can take a long time to complete. Here is a real case study of a Robotics application which was suffering from 330 seconds (i.e. 5+ minutes) of pause time for a single GC event. Below is the GC Pause time graph indicating the long pause of a single GC event
Fig: Single GC Event taking 5+ minutes to complete
- Consecutive GC Events: Due to memory problems sometimes GC events can run consecutively back-to-back. When they run consecutively, the application freezes for a longer duration. When multiple GC Events run consecutively, the GC behavior graph will look as shown in the below figure.
Fig: GC Events Running Consecutively
How to Solve Long GC Pause?
Long GC pauses surfaces because of lack of GC Tuning. Here is a blog post that provides 9 strategies to reduce long GC Pause. In nutshell they are:
- Start tuning from scratch: Clear out legacy JVM arguments and begin fresh, JVM auto-tuning can often outperform old, accumulated configs
- Resize the heap: Ensure your heap isn’t under- or over-allocated, too small leads to frequent GCs, too large delays reclamation
- Right GC algorithm: Choose the appropriate collector (Serial, Parallel, G1, ZGC, etc.) aligned with your latency and throughput needs
- Adjust internal memory region sizes: Optimize young, old, and metaspace sizes will also reduce long GC pause time.
- Tune GC algorithm settings: Use targeted JVM flags tailored to your chosen GC (e.g. G1, ZGC) rather than relying on defaults
- Address the causes of GC events: Analyze GC logs to identify triggers (allocation or promotion failures) and adjust configurations accordingly
- Disable explicit GC: Suppress System.gc() calls using flags like -XX:+DisableExplicitGC to prevent unnecessary full GCs
- Allocate sufficient system capacity: Ensure adequate CPU, I/O, and GC threads are provided in the device in which your application is running
- Reduce object creation rate: Lower the object churn through profiling and refactoring since fewer allocations mean fewer GC events
How to solve Consecutively Running GC Events?
You can avoid GC Events running consecutively (i.e. back-to-back) due to one of the following 3 reasons:
- Under Allocated Heap Space: If your application creates a lot more objects than your allocated heap size (-Xmx), Garbage Collector has to run quite frequently to reclaim the free space. Especially during your peak traffic volume period, GC events have to run back-to-back to keep up with the application’s object creation rate. To address this problem, one has to increase the heap size by passing the JVM argument ‘-Xmx’.
- Under Allocated Metaspace: Metaspace is the region where JVM stores metadata information of your application (such as class definitions, method definitions…). This region will get overloaded under following circumstances:
- a) When you add several 3rd party frameworks/libraries to your application, their class definitions will also tend to occupy this region.
- b) When your application uses java reflection to create class definitions at run time
- c) When your application uses Groovy, a javascript type of scripting language to execute certain functionality of the application.
In such circumstance you will start to see GC events running consecutively as shown in the below graph
Fig: GC Events continuously running due to Metaspace Overload
In this graph you can notice an interesting behavior – GC events are continuously running even while memory is at the bottom. Metaspace is a small region in JVM’s memory. When this region fills up, it will trigger a Full GC event. This Full GC event runs on both the heap region and Metaspace region. Since the Full GC event clears all the unreferenced objects from the heap region (which is of large size), memory drops to the bottom. However since there is a leak (or overload) in the Metaspace region, the full GC event runs continuously.
When this sort of problem happens, you need to increase the Metaspace region memory size by passing the argument ‘-XX:MaxMetaspaceSize’.
3. Memory Leak in the Code: When there is a memory leak in the code, then also you will see the GC Events to be continuously running. When you confront such problems, you can follow the steps mentioned in this post to diagnose the problem: OutOfMemoryError: Java Heap Space. In nutshell here are the two quick steps to identify the root cause of memory leak:
a. Capture Heap Dump: Heap Dump is basically a snapshot of memory, which tell what are the objects that are present in the memory. What is the data present in them, who is referencing those objects… You need this artifact to troubleshoot memory leak. This post provides 8 options to capture Heap Dump, you can use the option that fits your needs.
b. Analyze Heap Dump: Once you have captured the Heap Dump, you can use Heap Dump analysis tools such as HeapHero, Eclipse MAT, Jhat… to analyze the heap dumps. These tools analyze the heap dump and identify the root cause of memory leak.
What are the Symptoms of Application Freezes?
When Application suffers from freezes, you will start to see following symptoms:
- Degradation In Response Time: Since Garbage Collection pauses the entire application, any customer transactions that are processed by the application will also be paused when GC event runs. Thus, customer transactions will stall, resulting in poor latency.
- Failing Health Check: When Garbage Collection events can run consecutively, application will be stopped for a prolonged duration. If any health check calls come during that time, it will get rejected. If a health check fails, that particular node will be decommissioned from the load balancer pool, which will cause even more availability headaches.
- Spike in CPU consumption: Modern application tends to create thousands of objects even to service a vanilla request. So at a given moment there are millions/billions of objects in memory. Since Garbage Collector must scan each and every one of the objects and their reference chain, it takes a lot of CPU cycles. Thus, if GC events run for a longer period, CPU consumption will spike up. Here is a case study where Uber (a Car Ride sharing App) had to optimize their GC settings to reduce the CPU cycles.
How to Study the Garbage Collection Impact on your Application?
The best way to study your application’s Garbage Collection Performance is by enabling the GC logging. You can pass these JVM arguments to enable the GC log in your application.
Once you have captured the GC log, upload it to GC log analysis tools like GCeasy. Tool analyzes the Garbage Collection log file and generates GC analysis reports instantly. In the report you are going to see GC Key Performance Indicators (KPIs) such as:
a. Memory Footprint
b. GC Pause Time
c. GC Throughput
d. CPU Time
and several more metrics and interactive graphs.
Fig: GC KPIs reported by GCeasy
If your application suffers from intermittent freezes or any other memory problems, the tool will highlight it to you. It will also provide targeted recommendations and GC settings to address those problems.
Fig: GCeasy tool provides recommendations to address Application Freezes
Conclusion
Application freezes often stem from GC pauses, poor memory settings or memory leaks. Tools like GCeasy and HeapHero help you quickly analyze GC logs and heap dumps, highlight the root causes, and offer targeted recommendations. If your app is freezing intermittently, don’t guess, use these tools & techniques outlined in this post to get clarity and restore stability.
Share your Thoughts!