Observability

This writeup talks about different metric type that an application can emit as telemetry. It intends to cover the case of when to use which metric types and the usecase for them. Metrics is one of the oldest forms of telemetry. There are many APM solutions, whose billing model runs on the number of metrics you send to them. But it is often not clear to app devs on what kind of metrics to emit from applications. Infact, the types of metrics can be confusing sometimes. ...

This note is a mental model for how Prometheus discovers and scrapes metrics in Kubernetes. The lens I want to keep throughout is: Where will the scrape config file sit? (Prometheus repo vs application repo) In which namespace will the serviceMonitor sit? (and how Prometheus finds it) At a high level there are two ways to tell Prometheus about a /metrics endpoint: Static via in the Prometheus config file. Dynamic via (CRD from Prometheus Operator) with label‑based discovery. ...

I attended Kubecon India 2025 held in Hyderabad this year. I mainly focused and attended talks related to Observability and Scalable designs. Now that the sessions are uploaded to Youtube, linking the ones which I really enjoyed. Observability at Scale With Monitoring as Code: Grafana, Prometheus, & Tempo - Vipin GopalaKrishnapillai & Saiabhinay Bommakanti, Amway Global - link Predictable auto scaling with keda - link Observability - tenant centric metrics - link Building observability platform for Edge compute nodes - link ...

I enjoy philosophy. Stoic philosophy in particular. Philosophy, I think, helps us revalidate our purpose. It acts as a yard stick and makes sure that we are not moving away from our First-Principles. Applying the same to Software Engineering, in my opinion, every team should have a “Design Philosophy”. What is that one yard stick which teams can use for making better decisions. Infact, it is done in some forms in a few cases. Some call it Guiding-Principles. Some call it MVPs. I call it “Design Philosophy”. ...

The intention of this page is to put together the Observability Signal Guidelines which will provide the required visibility into the systems without hurting the cost aspect of the solution. Three basic observability signals that any application emits are: Metrics, Traces and Logs The general question is - When to emit what signal? The answer lies in the intent behind the signal being emitted. What do you intend to measure with the Observability signal that you are emitting? ...

Observability is crucial for modern distributed systems, enabling engineers to monitor, debug, and optimize their applications effectively. OpenTelemetry (Otel) has emerged as a comprehensive, vendor-neutral observability framework for collecting, processing, and exporting telemetry data such as traces, metrics, and logs. This blog post will explore how custom processors in OpenTelemetry can significantly enhance your observability strategy, making it highly customizable and powerful. The repo link where I have implemented a very simple Otel-Custom-Processor. https://github.com/AkshayD110/otel-custom-processor/tree/master ...

This page documents a few aspects of memory management on Java containers on K8s clusters. For java containers, memory management on K8s have various factors: Xmx and Xms limits managed by java Request/limit values for the container HPA policies used for scaling the number of pods Misconfigurations / misunderstanding of any of these parameters leads to OOMs of java containers on K8s clusters. Memory management on java containers: -XX:+UseContainerSupport is enabled by default form java 10+ ...

There isn’t a single book or article on observability (O11y) where there isn’t a mention of MELT. (Metrics, Events, Traces, Logs) While these four are the building blocks of telemetry data in Observability, all the four components haven’t evolved at the same rate. In this write-up, I delve into metrics in observability (mainly custom metrics) and argue how the overuse of metrics is turning into madness and directly impacting the cost of observability. ...