Observability

Observability is the ability to understand the internal state of a system by examining its external outputs. It is a broader capability than logging, and the gap between the two becomes obvious the first time a system reaches a state nobody designed for. This is the first post in a series on observability. It covers what the term means, why logs alone leave you debugging in production, the commercial case for investing in observability, and the bar an application has to clear before you can say it has it. ...

AWS Lambda forwards everything you write to stdout into CloudWatch for free, and it keeps doing so even after your handler has returned. The moment you want richer signals than text logs, or you want to send them anywhere other than CloudWatch, that free path stops being enough: you have to push the data over the network yourself. On a function that freezes between invocations, every way of doing that trades off against one of two things. You either pay Lambda’s per-millisecond billing for the time spent waiting on a remote endpoint, or you risk dropping the telemetry entirely when the environment is suspended out from under you. This post walks the approaches from simplest to most reliable and names where each one bills you or loses signals. The problem is not specific to CloudWatch; it applies to any serverless environment that scales to zero. ...