ClassLoader Leaks in Hot-Reload Environments

Hot redeployment in Java – the ability to install new versions of software without restarting the JVM – is a huge time and money saver. It’s especially useful in managing web servers, where downtime is not acceptable. 

Like most good things, though, there is a downside. Hard-to-find Java classloader leaks are a well-known problem associated with deploying applications on the fly.

Why does this happen? What should we avoid when writing web apps to prevent it from happening? And how do we identify and fix the problem in production?

Let’s try to answer some of these questions.

The JVM Memory Model: Why ClassLoaders Get ‘Stuck’ 

Let’s first look at why class loader leaks happen.

We can visualize JVM memory like this:

Fig: JVM Memory Model

Briefly,

• Heap memory is a central storage area for objects created by all classes and all threads in the application. It’s managed by the JVM. Most garbage collectors (GC) split it into smaller areas to speed up memory cleaning:
  • New objects are created in the young generation, which is small and can be cleaned quickly. This is cleaned frequently in minor GC events.
  • Objects that survive several minor events are promoted to the old generation. This is larger, and cleaned less often in full GC events.
• Native memory is managed by the operating system, and has several memory pools, including:
  • Metaspace (or PermGen in older JVMs) holds class definitions and metadata;
  • Thread Space holds a stack for each running thread;
  • Code Cache holds pre-compiled code for ‘hot’, or frequently-used, methods;
  • Direct Buffer Area holds buffer areas for fast, native I/O;
  • GC is reserved for internal use by the garbage collector;
  • JNI is used by the Java Native Interface;
  • Misc is used internally by the JVM for things like symbol tables.

If the JVM runs short of space, the result is a Java OutOfMemoryError.

PermGen vs Metaspace: What’s the Difference?

Prior to Java 8, class metadata was stored in the Permanent Generation (PermGen). This was a fixed-size region inside the JVM heap.  In Java 8 and later releases, the PermGen has been replaced by the Metaspace. 

This uses native memory outside the heap, which is managed by the operating system. The Metaspace can grow dynamically up to a configured limit. Moving class definitions out of the heap improved metadata management, and is less prone to OutOfMemory errors.

For more information about JVM memory management, you may like to watch this video: JVM Explained in 10 Minutes.

The Garbage Collector has the task of trying to make sure there is always enough space for new memory requests. Working from objects that are known to be in use (Garbage Roots), it recursively follows all references through the heap, marking the objects they point to as being reachable. Garbage roots include the stack and static variables. Any unmarked items can then be cleaned from memory.

The GC also cleans the metaspace, if possible. Points to note are:

• The metaspace is only cleaned during full GC cycles.
• Each object in the heap contains a pointer known as the Klass pointer, pointing to the class definition stored in the metaspace.
• Each class in the metaspace points to its class loader.
• The GC doesn’t remove unused classes singly. Instead, it cleans the classloader, and all its dependent classes, at the same time.
• Classes can only be removed from the metaspace; therefore, if:
  • No live objects hold pointers to them;
  • No live classes point to the class loader.

Fig: Object Structure in Java Hotspot

It follows, therefore, if a single live reference still exists anywhere to one piece of instance data within an object,

• The object can’t be garbage collected;
• Therefore, the object’s class is still live;
• Therefore, the class’s class loader is still live;
• Therefore, NONE of the classes loaded by the same class loader can be garbage collected.

This is why class loader leaks occur, and why they are hard to troubleshoot.

What Causes Java Classloader Leaks When Webapps are Redeployed?

We’ll look at the class loader hierarchy in a web server, taking Tomcat as an example, before looking at the hot deployment procedure and what is likely to cause class loader leaks.

1. Analyzing the Tomcat ClassLoader Hierarchy 

Typically, web servers such as Tomcat have a separate class loader for each web app. This is important because it allows each web app to be isolated from others. Each can therefore have its own version of:

• Classes;
• Libraries;
• Frameworks.

Each app is independent of every other app.

Let’s look at the class loader structure in Tomcat for a server that’s running WebAppA and WebAppB, each of which use classes from the shared library LibA.

Fig: Class Loader Structure in Tomcat

The bootstrap, platform and system class loaders are standard within all JVMs. They’re responsible for loading core Java classes, standard JDK modules and application classes respectively.

Underneath this structure, Tomcat has its own layers of class loaders.

The Tomcat Common Classloader loads classes for the Tomcat APIs, and also for shared libraries such as logging functionality.

Underneath this level, each web app has its own WebApp Classloader, responsible for loading the classes that make up the application.

2. Hot Deployment in Tomcat

Let’s now briefly look at what happens on a hot redeploy.

• The Tomcat HostConfig detects a new version of an app;
• It signals the existing app to stop accepting new requests, but allows it to complete existing transactions;
• It sends a shutdown signal to the original app, which calls:
  • The contextDestroyed() method in the servlet context lister, if one is defined;
  • The destroy() method in each servlet.
• The application is expected to use these methods to clean up:
  • Resources such as file handles and database connections;
  • References to its variables held in external apps, such as caches in shared libraries;
  • Threads and thread pools;
  • Services, timers etc.
• The class loader is destroyed;
• The old app’s WAR files etc. are removed;
• The new version is deployed with its own class loader, and it begins accepting requests.

As you can imagine, if the web app doesn’t do the clean-up thoroughly, pointers will still exist, which keep the class loader alive. This means that none of the old app’s classes can be removed from the metaspace.

If several deployments take place while the JVM is still running, the problem multiplies over time.

3. 5 Common Causes of Metaspace Leaks during Redeployment 

Here are some common causes of class loader leaks.

1. Threads not stopped, or ThreadLocal variables not cleared;
2. Fields, especially static fields, referenced from external libraries;
3. JDBC drivers not deregistered;
4. Logging frameworks not shut down correctly;
5. Cleaner or reference queues not shut down and removed

How to Troubleshoot Java Classloader Leaks in Production

Let’s look at the symptoms of class loader leaks, and how we can go about finding and fixing the problem.

1. Symptoms of a ClassLoader Leak: CPU Spikes & Metaspace Growth 

If we have a class loader leak, then over time we’ll see performance degrading, getting worse with each new version deployed. The system becomes slow and unresponsive, and CPU usage becomes progressively heavier. This is because, when metaspace is nearly full, the GC keeps running to try to clear space, even though the heap is not full. Since it’s not able to clear the leak, it will reach a point where full GC cycles run back to back, and eventually the server crashes with the error java.lang.OutOfMemoryError: Metaspace. In Java versions 7 or earlier, the error would instead be java.lang.OutOfMemoryError: Permgen space. 

If we analyze the GC log using a tool such as GCeasy, we may see a pattern similar to the image below. 

Fig: GCeasy Graph of Heap Size Over Time

Full GC events are indicated by red triangles. Initially, the GC runs normally, cleaning heap space on minor GC events. When the metaspace becomes full, GC runs continually, but doesn’t clear any memory from the heap. Since the GC is now using most of the CPU time, the application stalls and no new memory is created in the heap either.

Restarting the server will temporarily fix the performance problem, but it will recur if the app is not repaired.

2. Troubleshooting Java Classloader Leaks: A Worked Example

Let’s illustrate the troubleshooting steps by working through an example. We’ll create a small web app running on Tomcat, and we’ll use the Tomcat logs and a heap dump analyzer to see what’s happening. We used HeapHero to analyze the dumps, but you may prefer another tool, such as Eclipse MAT or JMC.

We’ll begin with a bug-free servlet and see what’s in memory when it’s deployed. We’ll then add buggy code that causes a class loader leak, and deploy it a couple of times. We’ll look at the Tomcat logs, which are our best resource for finding leaks, and also analyze a new heap dump. Finally, we’ll fix the bug, restart the server to clear the buggy version from memory, and deploy the fixed version a few times to make sure it’s not leaking.

Before we start, let’s look at a heap dump of Tomcat with no webapps deployed.

Fig: Tomcat Class Loaders At Startup

We took a heap dump of Tomcat, loaded it into HeapHero and examined the resulting interactive report. In the class histogram section, we grouped by class loader rather than class, and searched for ClassLoader. We see three types of loader. We can then compare this baseline to subsequent heap dumps.

Now let’s deploy a web app that looks like this:

package eg;
import jakarta.servlet.ServletException;
import jakarta.servlet.annotation.WebServlet;
import jakarta.servlet.http.HttpServlet;
import jakarta.servlet.http.HttpServletRequest;
import jakarta.servlet.http.HttpServletResponse;
import java.io.IOException;
import java.io.PrintWriter;
/**
 * Servlet implementation class HelloServlet
 */
@WebServlet("/hello")
public class HelloServlet extends HttpServlet {
    protected void doGet(HttpServletRequest req,
                         HttpServletResponse resp)
            throws IOException {
        PrintWriter rsp_writer = resp.getWriter();
        rsp_writer.println("Hello Tomcat");
        
    }
    
}

This is a single simple servlet that displays some text.

If we now look at a new heap dump, we’ll see that we have an extra class loader for the app.

Fig: Heap Dump of Tomcat Running One App

If we deployed more apps, we’d see each one had its own ParallelWebappClassLoader.

Deploying this app several times won’t change the heap dump output, since it doesn’t have a class loader leak. However, if a full GC has not yet run, we may still see the old class loader and its classes. To avoid confusion, force GC to run before taking the dump using this command: jcmd <PID> GC.run

Let’s now introduce a bug.

package eg;
import jakarta.servlet.ServletException;
import jakarta.servlet.annotation.WebServlet;
import jakarta.servlet.http.HttpServlet;
import jakarta.servlet.http.HttpServletRequest;
import jakarta.servlet.http.HttpServletResponse;
import java.io.IOException;
import java.io.PrintWriter;
/**
 * Servlet implementation class HelloServlet
 */
@WebServlet("/hello")
public class HelloServlet extends HttpServlet {
	
    Thread backgroundThread;
	
    protected void doGet(HttpServletRequest req,
                         HttpServletResponse resp)
            throws IOException {
        PrintWriter rsp_writer = resp.getWriter();
        rsp_writer.println("Hello Tomcat");
// To introduce a classloader leak, start a thread that
// will never terminate
// ---------------------------------------------------
        backgroundThread = new Thread(() -> {
        	
            try {
            	
                while (true) {                   
                    Thread.sleep(1000);
                }
            } catch (InterruptedException e) {
                Thread.currentThread().interrupt();
            }
        });
        backgroundThread.start();
    }
}

We’ve created a thread that loops forever. As it stands, when the app is redeployed, the thread won’t be stopped, and it will continue to run in the background. The object backgroundThread can’t be garbage collected, and it holds a link to the Thread class in the metaspace. This in turn links to the app’s class loader, keeping it alive, so none of the classes that make up this app can be garbage collected either.

Tomcat is very good at helping us to troubleshoot class loader leaks. If we look at the Tomcat log after we’ve redeployed the app, we’ll see entries like this:

May 10, 2026 5:05:42 PM org.apache.catalina.loader.WebappClassLoaderBase clearReferencesThreads
WARNING: The web application [Example1] appears to have started a thread named [Thread-1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 java.base/java.lang.Thread.sleepNanos0(Native Method)
 java.base/java.lang.Thread.sleepNanos(Thread.java:551)
 java.base/java.lang.Thread.sleep(Thread.java:582)
 eg.HelloServlet.lambda$0(HelloServlet.java:29)
 java.base/java.lang.Thread.run(Thread.java:1516)

Not only has it given us a warning, but it’s given us a stack trace that directs us straight to the problem. Often, this is all we need to find class loader leaks in Tomcat apps.

Unfortunately, not all web servers are this kind to us, so let’s look a bit further and see what we’d expect to find in a heap dump.

Fig: Heap Dump of Tomcat After Redeploying App with ClassLoader leak

We now have a total of 5 class loaders, two of which are class ParallelWebappClassLoader. If we expand these two entries to see more information about them, we see they both belong to the same servlet, which immediately shows us there’s a class loader leak.

Further down in the HeapHero report, there’s a list of duplicate classes, if any are found.

Fig: Duplicate Classes Report from HeapHero

This again indicates a class loader leak.

To find out what’s holding references to the extra class loader, we can explore the dominator tree to trace what’s holding references to it that prevent it being garbage collected. 

3. Fixing the ClassLoader Leak

Once we’ve identified what’s holding the reference, we can fix the code.As stated earlier, when Tomcat shuts down a web app, it calls the destroy() method in each servlet, if one has been defined. It also checks to see if a context listener has been registered. We can either register the listener when we configure the web app, or simply register it by adding the annotation @WebListener to a listener class. The listener class may look like this:

@WebListener
public class CleanupListener implements ServletContextListener {
    @Override
    public void contextDestroyed(ServletContextEvent sce) {
        // Shutdown Actions
    }
}

Resources that are shared between servlets and other classes should be released in the context listener. Resources that belong to one servlet only can be released in its destroy() method.To fix the buggy version of our sample servlet, we can add a destroy() method that interrupts the thread:

package eg;
import jakarta.servlet.ServletException;
import jakarta.servlet.annotation.WebServlet;
import jakarta.servlet.http.HttpServlet;
import jakarta.servlet.http.HttpServletRequest;
import jakarta.servlet.http.HttpServletResponse;
import java.io.IOException;
import java.io.PrintWriter;
/**
 * Servlet implementation class HelloServlet
 */
@WebServlet("/hello")
public class HelloServlet extends HttpServlet {
	
    Thread backgroundThread;
	
    protected void doGet(HttpServletRequest req,
                         HttpServletResponse resp)
            throws IOException {
        PrintWriter rsp_writer = resp.getWriter();
        rsp_writer.println("Hello Tomcat");
        backgroundThread = new Thread(() -> {
        	
            try {
            	
                while (true) {                   
                    Thread.sleep(1000);
                }
            } catch (InterruptedException e) {
                Thread.currentThread().interrupt();
            }
        });
        backgroundThread.start();
    }
    // -------------------------------------------------
    // Destroy method added to fix the class loader leak
    // -------------------------------------------------  
    public void destroy() {
    	if(backgroundThread!=null)
    		backgroundThread.interrupt();
    }
    
    
}

After restarting the Tomcat server to get rid of the leaked classes, we can now deploy this app as many times as we like, and we’ll still see only a single class loader for it, provided the GC has run before the dump is taken.

What Should We Check Before Redeploying a Java Webapp?

To avoid introducing class loader leaks, we recommend using the following checklist to make sure nothing will hold a reference to the class loader after shutdown.

Conclusion

Hot deployment is a great time saver, but we need to take care that Java classloader leaks aren’t introduced when we redeploy a webapp

Whenever we see performance degrading after a deployment, we need to look into the possibility of leaks. The server logs are the first thing we need to check. Some servers, like Tomcat, include warnings that take us straight to the leaky code. 

Our next most useful tool is a heap dump analyzer, where we can use the class histogram to find duplicate class loaders and explore the dominator tree to find the leak.

To prevent leaks, any of our servlets that create objects or resources should have a destroy() method, where we can carry out clean-ups. For more complex apps, we can add a context listener that contains clean-up code in a contextDestroyed() method.

Lastly, we can use the checklist above to make sure nothing will retain a reference to the class loader when the app is shut down.