Error monitoring for a server that reports hundreds of thousands of errors

A live production server produces errors continuously. Most of them repeat, some are not yours at all, and in a list they all look the same. The question is not «how many errors were there» but «what do I fix today».

Collected without touching the customer's code

Errors are pulled from the system's own feed, every five minutes, with a cursor to continue from. There is no agent to install and no library to add to the code: what the system already reports is what gets collected.

One signature, one group

The same error repeats thousands of times and is not thousands of problems. Every event gets a signature made of three parts (module, function and a normalised message), and events sharing a signature fall into one group with a counter.

Not every error is your bug

Every group is sorted into three kinds: a real error, an external service (SMS, mail) and noise: bots, scanners, probes for addresses that do not exist. The sorting is explicit rules rather than a guess, and the default is «real»: what we failed to recognise is not hidden.

A rating that says what comes first

Each group gets a severity from four things: frequency, kind, a return after being closed, and when it was last seen. Noise stays low always; an external service is capped at medium; a group that came back moves up a step; one not seen for three days moves down.

A «growth trend» term is deliberately absent: the accumulated history is not deep enough, and a measure that cannot be computed honestly would look like a measurement.

One letter for a new group, a separate letter for a returning one

A new group produces one letter. If a closed group comes back, a different letter goes out, one that says so in words rather than with a badge: «appeared» and «came back» call for different things, and an identical subject line would teach you to read neither.

The error text passes through identifier masking both in the letter and on screen. An error message carries whatever the user typed: a national id in a query parameter, a phone from a form, and a letter that has gone out cannot be recalled.

Where in the code it came from

The class where the error fell is extracted from the call stack. This does not always work: a stack does not arrive with every error, and it does not always contain a frame of our own.

Areas, so you can ask «what is broken in payments»

Groups can be given an area label (payments, registration, the admin panel), and then the question stops being «which error is the most frequent» and becomes «what is wrong in this part of the system». Eight such labels exist at present.

The first run does not bury you in everything that ever happened

When a feed that has been running for years is connected, everything in it is «new» as far as we are concerned, and without a rule of its own the first run would send hundreds of letters at once, all of which would be deleted unread.

So groups that existed before the connection are marked as known silently, and a narrow window is left for what genuinely appeared just now. On the live project that avoided 849 unnecessary letters, and the one group that really was born in that hour got written about.

A task opens from here

A task is opened from a group in one action, and the task remembers which error it came from. There is also a solution field and a mechanism that looks for similar groups already solved.

And plainly: on the live project exactly one such task has been opened, and a solution is recorded for zero groups out of 902. The mechanism exists; the material in it does not yet.

What the customer sees

Nothing. The error screens belong to the provider. What reaches the customer is whatever became a task or a report, not the list itself.