OutOfMemoryError Metaspace in Jenkins: Root Causes, Diagnostics & Production Fixes

Jenkins – popular CI/CD pipeline is used for several critical operations in the organization such as building applications, conducting automated tests, deployments in pre-prod and prod environments, … If Jenkins is down, engineers’ productivity will be severely hampered. Thus, extra care is given to major organizations to keep them up 24 x 7. 

While Java has 9 types of OutOfMemoryError, Jenkins is susceptible to 8 of them. In this blog series, we systematically walk through each of those 8 types, helping you identify, diagnose, and fix them. This post covers one of them.

Occasionally it can experience java.lang.OutOfMemoryError: Metaspace, which would disrupt entire Jenkins availability. In this post let’s discuss what does ‘java.lang.OutOfMemoryError: Metaspace’ mean, how to isolate the root cause of it quickly, what are it’s temporary and permanent fix, even better how to prevent them from happening.

Immediate Stabilization Steps – OutOfMemoryError Metaspace in Jenkins

When Jenkins experience ‘java.lang.OutOfMemoryError: Metaspace’ here are the options one can take to stabilize the Jenkins immediately (basically first-aid):

1. Restart the JVM: When ‘java.lang.OutOfMemoryError: Metaspace’ happens in Jenkins, it will put JVM into an instable state. It’s dangerous to run Jenkins in this setting, as it can result in erroneous behavior. Thus, it’s highly recommended to restart the JVM, so that it will come back in a clean slate.

2. Increase Metaspace size: ‘java.lang.OutOfMemoryError: Metaspace’ happens in Jenkins due to lack space in the Metaspace region of the JVM Memory. Thus increase the Metaspace memory region size. You can increase the Metaspace memory region by passing following arguments to your JVM: 

-XX:MaxMetaspaceSize=<size> Sets the upper limit for Metaspace 

-XX:MetaspaceSize=<size> Sets the initial threshold that triggers the first Garbage Collection (GC) for Metaspace.

Why OutOfMemoryError Metaspace Happens in Jenkins?

To better understand OutOfMemoryError Metaspace, we first need to understand different JVM Memory regions. Here is a video clip that gives a good introduction about different JVM memory regions. But in nutshell, JVM has following memory regions:

Fig: JVM Memory Regions

1. Young Generation: Newly created application objects are stored in this region.
2. Old Generation: Application objects that are living for longer duration are promoted from the Young Generation to the Old Generation. Basically this region holds long lived objects.
3. Metaspace: Class definitions, method definitions and other metadata that are required to execute your program are stored in the Metaspace region. This region was added in Java 8. Before that metadata definitions were stored in the PermGen. Since Java 8, PermGen was replaced by Metaspace.
4. Threads: Each application thread requires a thread stack. Space allocated for thread stacks, which contain method call information and local variables are stored in this region.
5. Code Cache: Memory areas where compiled native code (machine code) of methods is stored for efficient execution are stored in this region.
6. Direct Buffer: ByteBuffer objects are used by modern framework (i.e. Spring WebClient) for efficient I/O operations. They are stored in this region.
7. GC (Garbage Collection): Memory required for automatic garbage collection to work is stored in this region. 
8. JNI (Java Native Interface): Memory for interacting with native libraries and code written in other languages are stored in this region.
9. misc: There are areas specific to certain JVM implementations or configurations, such as the internal JVM structures or reserved memory spaces, they are classified as ‘misc’ regions.

This video covers OutOfMemoryError: Metaspace from a Java perspective, the same underlying concept that applies to Jenkins.

When Jenkins loads a large number of plugins, pipeline scripts, and their associated classes into the Metaspace region of JVM memory, exceeding the allocated limit (set via -XX:MaxMetaspaceSize), the JVM throws java.lang.OutOfMemoryError: Metaspace, causing Jenkins to become unresponsive or crash.

Fig: ‘java.lang.OutOfMemoryError: Metaspace’

Root Causes of OutOfMemoryError Metaspace in Jenkins

‘java.lang.OutOfMemoryError: Metaspace’ in Jenkins is potentially caused because of the following reasons:

1. Creating large number of dynamic classes: If your Jenkins plugins uses JavaScript /Groovy kind of scripting languages or Java Reflection to create new classes at runtime. 
2. Loading large number of classes: Either your Jenkins installation itself has a lot of classes, or it uses a lot of 3rd party plugins/libraries which have a lot of classes in it.
3. Loading large number of class loaders: Your Jenkins installation or the 3rd party plugins is loading a lot of class loaders.

How to Diagnose the OutOfMemoryError: Metaspace Problem in Jenkins (Step-by-Step)

To diagnose ‘OutOfMemoryError: Metaspace’ in Jenkins, we need to inspect the contents of the Metaspace region. Upon inspecting the contents, you can figure out the leaking area of the application code. Here is a blog post that describes a few different approaches to inspect the contents of the Metaspace region. You can choose the approach that suits your requirements. My favorite options are:

1. -verbose:class: If you are running Jenkins on Java version 8 or below, then you can use this option. When you pass the ‘-verbose:class’ option to your Jenkins application during startup, it will print all the classes that are loaded into memory. Loaded classes will be printed in the standard error stream (i.e. console, if you aren’t routing your error stream to a log file). Example:

java {app_name} -verbose:class

When we passed the ‘-verbose:class’ flag to the above program, in the console we started to see the following lines being printed:

[Loaded com.buggyapp.MetaspaceObjecta97f62c5-0f71-4702-8521-c312f3668f47 from __JVM_DefineClass__]
[Loaded com.buggyapp.MetaspaceObject70967d20-609f-42c4-a2c4-b70b50592198 from __JVM_DefineClass__]
[Loaded com.buggyapp.MetaspaceObjectf592a420-7109-42e6-b6cb-bc5635a6024e from __JVM_DefineClass__]
...

This is a clear indication that classes with the ‘com.buggyapp.MetaspaceObject’ prefix are being loaded frequently into memory. This is a great clue/hint to let you know from where the leak is happening in the Jenkins application.

2. -Xlog:class+load: If you are running Jenkins on Java version 9 or above, then you can use this option. When you pass the ‘-Xlog:class+load’ option to your Jenkins application during startup, it will print all the classes that are loaded into memory. Loaded classes will be printed in the file path you have configured. Example:

java {app_name} -Xlog:class+load=info:/opt/log/loadedClasses.txt

If you are still unable to determine the origination of the leak based on the class name, then you can do a deep dive by taking a heap dump from the Jenkins application. You can capture a heap dump using one of the 8 options discussed in this post. You might choose the option that fits your needs. Once the heap dump is captured from Jenkins, you need to use tools like HeapHero, JHat, … to analyze the dumps.

What is Heap Dump?

Heap Dump is basically a snapshot of your Jenkins application memory. It contains detailed information about the objects and data structures present in the memory. It will tell what objects are present in the memory, whom they are referencing, who are referencing, what is the actual customer data stored in them, what size they occupy, are they eligible for garbage collection… They provide valuable insights into the memory usage patterns of Jenkins, helping developers identify and resolve memory-related issues.

Solutions for OutOfMemoryError Metaspace in Jenkins

Here are the potential solutions to address java.lang.OutOfMemoryError: Metaspace in Jenkins:

1. Identify & Fix the Memory Leak in Jenkins: Using the diagnostic steps described in the above section find the leaking objects in the memory and fix it.

2. Remove the recently added Plugins: Whenever you add new plugins, it will occupy space in the Metaspace. Sometimes you might end up adding poorly implemented, memory inefficient plugins. Remove the recently added plugins and restart the JVM and see whether Jenkins stabilizes. 

3. Revert to Previous Jenkins Installation: If you have recently upgraded to latest version of Jenkins installation and Metaspace OutOfMemoryError started to surface after it, consider reverting to previous Jenkins installation. 

4. Increase Metaspace size: ‘java.lang.OutOfMemoryError: Metaspace’ happens in Jenkins due to lack space in the Metaspace region of the JVM Memory. Thus increase the Metaspace memory region size. You can increase the Metaspace memory region by passing following arguments to your JVM: 

-XX:MaxMetaspaceSize=<size> Sets the upper limit for Metaspace 

-XX:MetaspaceSize=<size> Sets the initial threshold that triggers the first Garbage Collection (GC) for Metaspace.

How to Prevent OutOfMemoryError Metaspace in Jenkins

Before you upgrade to new release of Jenkins or install a new Jenkins plugin in the production environment, you might be studying following key metrics in your performance lab:

• CPU Utilization
• Memory Utilization
• Response Time of key transactions

These are wonderful metrics that highlight the performance characteristics of the new release. However, several performance problems slowly build over the period of time, for example for most applications OutOfMemoryError happens only if it runs for more than 1 week. In performance lab, we don’t run such long endurance tests. 

Above mentioned metrics are more reactive indicators that don’t indicate the silently lurking problem in the environment. We recommend studying below mentioned Micro-metrics along with above reactive indicators in the performance lab and certify the release. These Micro-Metrics are good are predicting/forecasting performance problems even if they act at acute scale.

• GC Behavior Pattern: Detects memory leaks, poor GC configuration, or excessive object promotion causing GC pauses.
• Object Creation Rate: Identifies allocation surges that can trigger frequent GCs or memory pressure.
• GC Throughput: Highlights apps spending too much time in GC instead of work—can lead to CPU spikes or slowdowns.
• GC Pause Time: Surfaces stop-the-world GC events affecting responsiveness or causing thread backlogs.
• Thread Patterns: Flags CPU spikes, thread starvation, bursty load, and thread buildup from backend slowness.
• Thread States: Detects BLOCKED, DEADLOCKED, or WAITING threads due to DB chattiness, config limits, or locking.
• Thread Pool Behavior: Identifies thread exhaustion, request rejections, or poor pooling thresholds in backend services.
• TCP/IP Connection Count & States: Catches backend connection leaks, TIME_WAIT surges, or slow/unresponsive downstream services.
• Error Trends in Application Logs: Detects hidden runtime errors, JDBC leaks, logging misconfigurations, or disk issues.

yCrash tool facilitates you reporting these Micro-Metrics which will unearth several performance problems well in advance, before they silently surface in production. You can find the details on how to source and study these Micro-Metrics through yCrash from here. 

Business Impact & ROI

Isolating and fixing OutOfMemoryError in Jenkins will have considerable business impact to your organization:

1. Engineering Time Savings: yCrash dramatically reduces the time engineers spend analyzing Heap Dumps and pinpointing root causes in complex, multi-threaded applications.
   • Suppose your organization is analyzing 10 incidents per month & each analysis traditionally will take around 40 hours. 
   • With a Performance Engineer’s hourly rate at USD $100, yCrash can save approximately $480,000 annually (10 incidents x 40 hours/dump x $100/hour x 12 months) by automating root cause analysis and reducing troubleshooting time.

2. Rapid Deployments & Increased Productivity: yCrash minimizes prolonged downtime of Jenkins that can lead to delayed deployment, degeneration of engineers productivity and reputational damage of the organization. By quickly diagnosing issues, yCrash helps to prevent such large-scale impacts, protecting revenue and brand reputation.

3. Protection from Escalated Operational Consequences: Certain Jenkins outages can have severe repercussions, including escalated consequences like organizational changes or job losses. yCrash’s rapid problem isolation capabilities prevent such disruptions, allowing teams to resolve issues before they escalate to crisis levels. By maintaining operational continuity and team stability, yCrash supports a steady, resilient organizational environment and protects against the high stakes impacts that can result from unmanaged production outages.

Conclusion

Jenkins is the backbone of your organization’s CI/CD pipeline. Keeping it stable is not optional, it’s rather essential. Even though ‘java.lang.OutOfMemoryError: Metaspace’ is not a common error, it can hurt your entire Jenkins platform availability, when it surfaces. Hopefully this post has given you enough light on how to troubleshoot this problem effectively & efficiently.