How To Control Memory Allocation Strategy in Third-Party Library Code

Third-party libraries are invaluable for adding extra functionality to your application for very little effort. But what do you do if third party code goes rogue, and uses too much memory?

In theory, you can’t control library memory allocation in Java. It’s like adopting a cat: once it’s in your house, it does what it wants. There are, however, a few strategies you can use to solve memory issues relating to imported libraries.

It depends, of course, on whether you’re using a Java library, or a native library/code written in some other language.

This article looks at both scenarios, and includes suggestions that may help solve the problem.

Out of Memory Errors in Java

Memory issues in third-party code can cause one of two problems. They can either seriously degrade performance, or they can result in the system crashing altogether with an Out of Memory error.

In Java, there are 9 types of system failures caused by OutOfMemoryErrors. Third party code written in Java can cause any of these, whereas native library code may affect the JNI area, threads or the device as a whole.

Controlling Library Memory Allocation in Java Applications

Let’s start with a few suggestions that apply equally to libraries written in Java or other languages.

• Check the library’s documentation or, if possible, its source code.
  • Does it have any configuration settings that relate to memory usage?
  • Does it have clean-up methods that release unused memory? These often have names such as release(), free() or close(().
  • Do any of the methods allow you to specify settings relevant to memory usage?
• Check your own Java code to make sure it’s using the library correctly. Are there bugs such as recursive method calls, or memory leaks related to the parameters you’re passing?

If the library is open source, you could create your own modified version with better memory management strategies.

Libraries Written in Java

If the library was written in Java, there are several options.

Java libraries may cause memory issues in the heap, the metaspace or direct buffers. In extreme cases, they can cause the device as a whole to run out of memory. If the application has run out of memory, first establish from the error message which area of memory is affected.

Next, assess where the problem lies by taking a heap dump and analyzing it using a tool such as HeapHero. This should give you a good idea of whether there is an actual memory leak in the code, or whether the application simply needs more memory.

If the application needs more memory, increase the size of the affected area – the heap, the metaspace or the direct buffer area – using JVM runtime arguments.

Look at the areas where you do have control, in particular, managing the variables you pass to the third-party library.  Are they defined within the correct scope, or dereferenced when no longer needed, so they will be garbage collected when the task is complete? Are you using the best class of object for the application? For example, direct buffers not only increase performance, but they also take the load off the heap.

Ensure that the library classes are dereferenced when they’re no longer needed, to save memory in the metaspace. 

Selecting the best garbage collector algorithm for the task, and tuning the garbage collector, can resolve many types of memory problems.

The HeapHero and GCeasy tools are very useful for resolving memory related problems in Java.

Libraries From Native Code or Other Languages

These are much more difficult when it comes to controlling memory allocations.

As always, the starting point is to check your own code:

• Check the objects passed as parameters to the third-party code. Are they defined within the correct scope, or dereferenced when no longer needed?
• Are there any loops that don’t have a valid termination condition?
• Are the parameters you pass defined as the right type of object?

If you need to monitor the actual internal memory of the library, the normal Java tools won’t help you since it will not be stored within the JVM. You’ll need to use tools such as dtrace, pmap or pstack. The Java Hotspot Probe can also be helpful. All of these are discussed in Oracle’s documentation. 

You can also set the JVM switch XX:NativeMemoryTracking=summary.  This allows you to obtain statistics while the program is running using the command:

jcmd <pid> VM.native_memory summary

If you’re using Java 22 or later, it’s worth considering using the new Foreign Function and Memory API (FFM). This offers much more control over memory usage, and also allows you to call foreign functions within a session, which can be closed when complete.

If all else fails, a rogue library may be invoked within a separate process using the ProcessBuilder class. This returns a Process object, which you can use to control and, if necessary, kill the library class process.

Conclusion

Although we don’t have direct control over library memory allocation in Java, we are not entirely helpless.

A well-written library should have methods for cleaning up memory, and may very likely allow us to configure memory parameters. We can also ensure our own code is free from memory leaks and other bugs.

For native libraries, the new FFM API offers much better memory management.

And, in the worst-case scenario, the native code can be called as a separate process, giving us some control over its actions.