Performance on Akshay Deshpande

Memory management : Java containers on K8s

Sat, 08 Feb 2025 00:00:00 +0000

This page documents a few aspects of memory management on Java containers on K8s clusters.

For java containers, memory management on K8s have various factors:

Xmx and Xms limits managed by java
Request/limit values for the container
HPA policies used for scaling the number of pods

Misconfigurations / misunderstanding of any of these parameters leads to OOMs of java containers on K8s clusters.

Memory management on java containers:

-XX:+UseContainerSupport is enabled by default form java 10+

[Kubernetes]: CPU and Memory Request/Limits for Pods

Sun, 14 Jan 2024 00:00:00 +0000

In this write up, we will try and explore how to make the most out of the resources in K8s cluster for the Pods on them.

Resource Types:

When it comes to resources on Kubernetes cluster, they can be fairly divided in to two categories:

compressible:
- If the usage of this resource for an application goes beyond the max, it can be throttled without directly killing the application/process.
- example : cpu - if a container consumes too much of compressible resource, they are throttled

[Memory-metrics]: Linux /proc interface

Mon, 22 Aug 2022 00:00:00 +0000

This writeup is more of a demo to showcase the power of “proc” (process information pseudo-filesystem) interface in linux to get the memory details of process, and also a quick brief on the power of “proc interface”.

In the current trend of building abstraction over abstractions in software/tooling, very few tend to care about the source of truth of a metrics. There are various APM / Monitoring tools to get the memory details of a process for a linux system, but when the need arises, I believe, one must know the ways of going closer to the source of truth on a linux system and verify things.

[Performance] : Understanding CPU Time

Sat, 24 Jul 2021 00:00:00 +0000

As a Performance Engineer, time and again you will come across a situation where you want to profile CPU of a system. The reasons might be many; like, CPU usage being high, you want to trace a method to see its CPU cost or you suspect CPU times for a slow transaction.

You might use one of the various profilers out there to do this. (I use yourkit and Jprofiler). All these profilers report the CPU costs in terms of CPU Time, when you profile the CPU. This time is not the equivalent of your watch time.

Elastic Search Best practices

Mon, 14 Jun 2021 00:00:00 +0000

These are the self-notes from managing 100+ node ES cluster, reading through various resources and a lot of production incidents due to unhealthy ES.

Memory

Always choose ES_HEAP_SIZE 50% of the total available memory. Sorting and aggregations both can be memory hungry, so enough heap space to accommodate these is required. This property is set inside the /etc/init.d/elasticsearch file.
A machine with 64 GB of RAM is ideal; however, 32 GB and 16 GB machines are also common. Less than 8 GB tends to be counterproductive (you end up needing smaller machines), and greater than 64 GB has problems in pointer compression.

CPU

Choose a modern processor with multiple cores. If you need to choose between faster CPUs or more cores, choose more cores. The extra concurrency that multiple cores offer will far outweigh a slightly faster clock speed. The number of threads is dependent on the number of cores. The more cores you have, the more threads you get for indexing, searching, merging, bulk, or other operations.

Disks

If you can afford SSDs, they are far superior to any spinning media. SSD-backed nodes see boosts in both querying and indexing performance.
Avoid network-attached storage (NAS) to store data.

Network

The faster the network you have, the more performance you will get in a distributed system. Low latency helps to ensure that nodes communicate easily, while a high bandwidth helps in shard movement and recovery.
Avoid clusters that span multiple data centers even if the data centers are collocated in close proximity. Definitely avoid clusters that span large geographic distances.

General consideration

It is better to prefer medium-to-large boxes. Avoid small machines because you don’t want to manage a cluster with a thousand nodes, and the overhead of simply running Elasticsearch is more apparent on such small boxes.
Always use a Java version greater than JDK1.7 Update 55 from Oracle and avoid using Open JDK.
A master node does not require much resources. In a cluster with 2 Terabytes of data having 100s of indexes, 2 GB of RAM, 1 Core CPU, and 10 GB of disk space is good enough for the master nodes. In the same scenario, the client nodes with 8 GB of RAM each and 2 Core CPUs is a very good configuration to handle millions of requests. The configuration of data nodes is completely dependent on the speed of indexing, the type of queries, and aggregations. However, they usually need very high configurations such as 64 GB of RAM and 8 Core CPUs.

Some other important configuration changes

[Performance] : What does CPU% usage tell us ?

Fri, 28 May 2021 00:00:00 +0000

When you come across a system which is misbehaving, majority of the times the first metrics that we look at is CPU usage. But do we really understand what CPU usage of a system tells us ? In this article let us try and understand what X % usage of a system really means.

One of the easy ways to check on CPU is “top” command.

The “%Cpu(s)” metrics seen above is a combination of different components.

[Performance] : Using iperf3 tool for Network throughput test

Sun, 31 Jan 2021 00:00:00 +0000

In this world of Microservices and the distributed systems, a single request (generally) hops through multiple servers before being served. More often than not, these hops are also across the Network cards making the Network Performance the source of slowness in the application.
These parameters makes the need to measure Network performance between servers/systems more critical for benchmarking or debugging.

Iperf3 is one of the open source tools which can be used for network throughput measurement. Below are some of its features.

[Performance] : Java Thread Dumps - Part2

Wed, 30 Dec 2020 00:00:00 +0000

In the previous article about Java Thread Dumps (link here) we looked in to a few basics on Thread dumps(When to take?, How to take?, Sneak peaks? etc.)

In this write up, I wanted to mention a few tools which can ease the process of collecting and analyzing thread dumps.

Collecting multiple thread dumps:

I prefer command-line over any APM tools for taking thread dumps. The best way for analyzing threads is to collect a few thread dumps (5 to 10) and look through the transition in the state of threads.

Thinking in-terms of Performance

Fri, 20 Nov 2020 00:00:00 +0000

A few short thoughts / ideas wrt of Performance centric product.

In this world of infinite scaling of computes, pay close attention to common choke points. Like DB, storage(s) etc, which are shared by all the computes.
Majority of the reads and writes have to happen in Bulk operations and NOT as single read/writes. Specially when there are 100’s-1000’s of reads/writes/deletes on storage(s).
Threads. Pay close attention to which part of the entire flow is multi-threaded. Sometimes, only a small part of the flow is multi-threaded, but entire application is called multi-threaded, which is wrong.

[Performance] : Java Thread Dumps - Part1

Thu, 22 Oct 2020 00:00:00 +0000

This is first of a two parts article which talks about:

What are thread dumps?
When to take thread dumps ?
How to take thread dumps ?
What is inside a thread dumps ?
What to look for in a thread dump?

Majority of the systems today are mutlicore and hyper-threaded. Threading at the software level allows us to take advantage of a system’s mutlicores to achieve the desired pace and efficiency of the application operations. Along with pace and efficiency, multi-threading brings its own set of problems w.r.t thread contentions, thread racing, high CPU usage etc. In this write up we will see how to debug these problems by taking thread dumps on java applications.

[Performance] : Flame Graphs

Sun, 26 Apr 2020 00:00:00 +0000

In the previous article we explored the basic capabilities of linux Perf_tool.
In this write-up I am trying to extend these capabilities and show how to generate and read Flame Graphs for analyzing the profiles generated with Perf_tool.

How to generate Flame Graphs ?

To start with, we will need perf_tools linux profiler to capture the profile first. Follow the steps under “How to setup perf tool?” in the previous article.
Now if you collect a profile of the CPU using perf_tool setup in the above step, there is a possibility that you might see a lot of symbol link values in the place of Function names.

[Performance] : Profiling with linux Perf command-line tool

Sun, 15 Mar 2020 00:00:00 +0000

Most of the Performance Engineers use some sort of profiling tools like Yourkit, Jprofiler or some APM tools like Newrelic, Datadog, Appdynmics etc. Although these tools are easy to use out of the box and help with Observability, they don’t give a complete picture of a Performance problem at occasions.
This is where perf Linux profiler comes in handy.

This write up is an attempt to explain :
- What is perf Linux profiler ?
- How to set it up ?
- What are its capabilities ?

[Performance] : Java's built in diagnostic tool - Jstat

Wed, 22 Jan 2020 00:00:00 +0000

When it comes to Performance Monitoring and analysis, we tend to think of full fledged license tools like Dynatrace, Newrelic, Appdynamics, Yourkit etc. However, if it is a java application which is under diagnosis, java’s built in tools are a good place to start.

Java comes with a set of built-in diagnostic tools like - Jconsole, jcmd, jstat, jmap, jstack, jvisualvm, jfr and many more. Each of them help in tackling a kind of problem. For the scope of this article, lets look in to how jstat is useful as diagnostic tools.

[Monitoring] - Paging and Swapping in Memory

Mon, 09 Dec 2019 00:00:00 +0000

As a continuation after understanding Virtual Memory in the previous article, this article tries to explain theways to monitor the same.

Memory can be looked at with two perspectives: Utilization and Saturation.
Utilization tells the memory usage. Checking free/used memory reflects the Utilization.
Saturation tells if the memory is used at its full capacity and how the system is using Virtual memory to deal with memory crunch.
In other words, if demands for memory exceed the amount of main memory, main memory becomes saturated. The operating system may then free memory by employing paging, swapping, and, on Linux, the OOM killer. Any of these activities is an indicator of main memory saturation.
Also, it is important to understand that Paging and Swapping are two different things. More details about the same in the previous article.

[Understanding] : Virtual Memory

Mon, 04 Nov 2019 00:00:00 +0000

As a Performance Engineer you will come across Virtual memory very often specially when monitoring or debugging Memory issues. Virtual memory along with below mentioned terminologies are used very loosely across the industry.
- Page, Page frame, Page fault, Minor/Major fault, Paging, Swapping etc.

This article is an attempt to understand Virtual memory in detail theoretically, in the context of Computer Architecture.

What is virtual memory?

Virtual memory is not the real memory. It is an abstraction layer provided to each process. It is meant to simplify the software development, leaving the physical memory placement to operating system.
To put in very simple terms, the purpose of Virtual memory is to use the hard disk as an extension of RAM, thus increasing the available address space a process can use. Using Virtual memory, system can address more memory than it actually has, and it uses the hard drive to hold the excess. This area on the hard drive is called a page file, because it holds chunks of main memory on the hard drive.

[Understanding] : How to read a G1GC log file.

Fri, 11 Oct 2019 00:00:00 +0000

As a Performance Engineer, time and again you will need to look in to GC logs to see how jvm is handling garbage collection.
With G1GC being the default gc for java versions 9 & above, one needs to know what the G1GC log actually reads like.

To get an understanding of G1GC, here is an in-depth material on it from Oracle tutorials. - link

To begin with, there are 50+ jvm parameters for selecting different GC algorithms and customizing them as per requirement.
Below link has cheat sheet for all the jvm parameters that you can select from. - PDF

[Performance Debugging] : Root causing "Too many open files" issue

Fri, 28 Jun 2019 00:00:00 +0000

Operating system : Linux

This is a very straight forward write-up on how to root cause “Too many open files” error seen during high load Performance Testing.

This article talks about:

The ulimit parameter “open files”,
Soft and Hard ulimits
What happens when the process overflows the upper limit
How to root cause the source of file reference leak.

Scenario :

During a load test, as the load increased, I was seeing failures in transaction with error “Too many open files”.