Cloud

This writeup talks about different metric type that an application can emit as telemetry. It intends to cover the case of when to use which metric types and the usecase for them. Metrics is one of the oldest forms of telemetry. There are many APM solutions, whose billing model runs on the number of metrics you send to them. But it is often not clear to app devs on what kind of metrics to emit from applications. Infact, the types of metrics can be confusing sometimes. ...

This note is a mental model for how Prometheus discovers and scrapes metrics in Kubernetes. The lens I want to keep throughout is: Where will the scrape config file sit? (Prometheus repo vs application repo) In which namespace will the serviceMonitor sit? (and how Prometheus finds it) At a high level there are two ways to tell Prometheus about a /metrics endpoint: Static via in the Prometheus config file. Dynamic via (CRD from Prometheus Operator) with label‑based discovery. ...

In most of distributed datastore systems, there are a lot of techical terms to describe the behavior of the system. While these terms, like, “Leader”, “Follower”, “Replication”, “Consistency”, etc., are widely used and helpful, what I feel missing are the details about internal relationship between these terms. Analogically, while the map of the field is great, it is also important to understand how the soil, water, and sunlight interact to help the plants grow. ...

When studying Chapter 4 of Designing Data-Intensive Applications (Encoding and Evolution), I quickly encounters a level of granularity that seems mechanical: binary formats, schema evolution, and serialization techniques. Yet behind this technical scaffolding lies something conceptually deeper. Encoding is not merely a process of serialization; it is the very grammar through which distributed systems express and interpret meaning. It is the act that allows a system’s internal thoughts — the data in memory — to be externalized into a communicable form. Without it, a database, an API, or a Kafka stream would be nothing but incomprehensible noise. ...

The intention of this page is to put together the Observability Signal Guidelines which will provide the required visibility into the systems without hurting the cost aspect of the solution. Three basic observability signals that any application emits are: Metrics, Traces and Logs The general question is - When to emit what signal? The answer lies in the intent behind the signal being emitted. What do you intend to measure with the Observability signal that you are emitting? ...

I recently did a brush up course on Networking/Protocols 101’s. Making my notes public. Networking Basics / Protocols: Mental model to think about all protocols: A few notes about different Protocols: Network layer protocol: Transport layer protocol: TCP: UDP: Other Protocols: HTTP: SMTP: XMPP: MQTT: AWS Networking: Local Zone: Edge location: Understanding IPv4, IPv6 IPv4 IPv6 Classes IPv4 How to read CIDR noation: AWS VPC: Interacting with VPC: Internet Gateway (IGW) Subnet: Route table NACL: Networking Basics / Protocols: There are 7 layers of communications as defined in OSI (Open systems Interconnection) model ...

Observability is crucial for modern distributed systems, enabling engineers to monitor, debug, and optimize their applications effectively. OpenTelemetry (Otel) has emerged as a comprehensive, vendor-neutral observability framework for collecting, processing, and exporting telemetry data such as traces, metrics, and logs. This blog post will explore how custom processors in OpenTelemetry can significantly enhance your observability strategy, making it highly customizable and powerful. The repo link where I have implemented a very simple Otel-Custom-Processor. https://github.com/AkshayD110/otel-custom-processor/tree/master ...

This page documents a few aspects of memory management on Java containers on K8s clusters. For java containers, memory management on K8s have various factors: Xmx and Xms limits managed by java Request/limit values for the container HPA policies used for scaling the number of pods Misconfigurations / misunderstanding of any of these parameters leads to OOMs of java containers on K8s clusters. Memory management on java containers: -XX:+UseContainerSupport is enabled by default form java 10+ ...

In distributed systems, the principle of message passing between nodes is a core concept. But this leads to an inevitable question: How can we ensure that a message was successfully delivered to its destination? To address this, there are three types of delivery semantics commonly employed: • At Most Once • At Least Once • Exactly Once Each of these offers different guarantees and trade-offs when it comes to message delivery. Let’s break down each one: ...