There are different flavors of OutOfMemoryError. One of the flavours of OutOfMemoryError is ‘Kill Process or sacrifice child’. This post discusses what triggers this ‘Kill Process or sacrifice child’ OutOfMemoryError and potential solutions to diagnose this problem.
What triggers Out Of memory: Kill Process or sacrifice child?
When processes in the host tend to consume more memory than the available memory in the RAM, Operating System based on it’s internal heuristics computations will kill certain processes. In this circumstance, the processes which get killed by the Operating system will experience ‘OutOfMemoryError: Kill Process or sacrifice child’. This problem is more pervasive in the container’s world as well.
Simulating Out Of memory: Kill Process or sacrifice child
In this section, let’s see how to simulate ‘Out Of memory: Kill Process or sacrifice child.’
We launched an AWS EC2 instance r5a.2xlarge Amazon Linux instance. This type of EC2 instance has 64GB RAM.
Fig 1: No Processes running in a r5a.2xlarge host that has 64GB RAM
In this EC2 instance, we launched one Java program. We configured this Java program to consume 20GB of RAM. So this means there is only an additional ~43.5GB of RAM is left. i.e.,
= 64 GB total RAM – 20GB Java Process – 0.5GB Kernel Processes
= 43.5GB
Fig 2: Launched a Java process that will occupy 20GB RAM
Now we went ahead and launched the BuggyApp. BuggyApp is a simple open-source chaos engineering program that is capable of generating various performance problems. We configured BuggyApp to keep creating objects. We set the initial heap size (i.e., Xms) of this program to be 1GB and maximum heap size (i.e., Xmx) to be 50GB and launched the BuggyApp. It means when BuggyApp starts, it’s initial memory consumption will be 1GB, and it has the potential to consume up to 50GB. But if you note, there is only room for 43.5GB in the RAM.
Fig 3: Launching BuggyApp with initial heap size to be 1GB and Max Heap size to be 50GB
BuggyApp got launched successfully, but when its memory consumption reached 43GB, the Operating system killed the BuggyApp program with the ‘Out Of memory: Kill Process or sacrifice child’ error. Operating system took this serious action so that other processes on the system can be safeguarded.
[ 1584.087068] Out of memory: Kill process 3070 (java) score 547 or sacrifice child
[ 1584.094170] Killed process 3070 (java) total-vm:56994588kB, anon-rss:35690996kB, file-rss:0kB, shmem-rss:0kB
Above is the kernel log message printed on this device. You can note that the Operating System is killing our BuggyApp java process (whose Process Id is 3070). Note Operating System has internal algorithms and heuristics and keeps track of score for each process. Score this BuggyApp got was 547, thus it killed this process.
Fig 4: BuggyApp experiencing Out Of memory: Kill Process or sacrifice child
Solutions for Out Of memory: Kill Process or sacrifice child
Here are the potential solutions to fix ‘Out Of memory: Kill Process or sacrifice child’:
- Increase the RAM capacity of the device in which your applications are running
- Reduce the unnecessary processes that are running on the device.
- Reduce the memory consumption of the processes which are consuming a lot of memory. You can use tools like yCrash, which will help you to optimize the memory consumption.
Leave a Reply