Skip to main content


Showing posts from 2017

Paper on Gray Failures

A great paper annotated by The Morning Paper, this time on the subject of gray failures. There are a handful of interesting takeaways from this one: It is important that monitoring on a system aligns with the clients of the system's definition of failure. The cycle of failure is inevitable unless proper root causes are identified. Some personal observations: A potential observability gap is the difference between proximate and root cause. For example a service may fail because a proxy server returns an error. The error may be due to timing out a request to an upstream server. The cause of that could be the fact that latency has increased on the upstream server. The root cause of that could be that the working set of the server no longer fully fits in memory and the server is swapping leading to latency. The server itself is not failing any requests, but the proxy is throwing the res

Operational Metrics and Alerts for Distributed Software Systems

This post will be about operational metrics and alerts for distributed software systems. What do I mean by that? I mean the metrics and alerts that allow operations personel to detect failure of of a distributed software system and helps them to quickly diagnose what is wrong. Metrics The metrics are measurements of characteristics of the system collected at regular(ish) intervals and stored somewhere for processing - rendering into graphs, triggering alert notifications, etc. Metrics can be divided into 3 categories: input metrics, output metrics, and process metrics. Input metrics are measures of the inputs to the system, for example, the number of user requests, counts of particular characteristics of the requests - where they are from, how large the request data is, counts of particular features in the request (for example, which resources/items/products are being asked for). Output metrics are measures of the output of the system. Examples of these would include orders s