Module: AceMQ::AMQP::Telemetry

Defined in:: lib/acemq/amqp/telemetry.rb,
lib/acemq/amqp/telemetry/open_telemetry.rb

Overview

What the library counts, and where it sends the numbers.

The names are shared with the Java, Go, .NET and Python libraries, so a dashboard or an alert written against one service reads the same against the next. Java publishes them through Micrometer and .NET through System.Diagnostics.Metrics; Ruby has no standard metrics interface at all, so this counts them and hands them to whatever you already run.

The vocabulary is Java's MetricNames, which is the family's. It says what a message did in the tag rather than in the metric name — one acemq.consume.total tagged outcome rather than five counters — so a dashboard sums the whole and breaks it down without knowing the list in advance, and a new outcome does not need a new panel.

The interface

An observer is anything answering three methods:

def count(metric, delta = 1, **labels)   # a counter goes up
def observe(metric, value, **labels)     # a distribution takes a sample
def gauge(metric, value, **labels)       # a current value is set

observe is a distribution rather than a timer. Durations go through it in seconds, and CONSUME_ATTEMPTS puts an attempt number through the same method, because the summary worth having is the same one either way.

The tags

exchange, routing.key, message.type, queue, outcome, target, pipeline and step are what this library attaches. The wider family vocabulary also has transport. routing.key and message.type are not legal Prometheus label names: a dot is not allowed in one, and a scrape containing routing.key="order.placed" is rejected whole rather than in part. Registry#to_prometheus renders them as routing_key and message_type; an observer of your own that talks to Prometheus has to do the same.

Three methods and no dependency, deliberately. Depending on a metrics gem would put every service using this library on the same one, and the choice between Prometheus, statsd and a log line belongs to the application. Registry is here for when the numbers themselves are all that is wanted, and OpenTelemetry for spans, which are a different shape again and live in an interceptor rather than in an observer.

Every method is called on the path a message takes, from whatever thread is publishing or handling, so an observer has to be safe to call from several at once and must not block. Anything an observer raises is swallowed: a metrics backend that is down is not a reason to stop delivering messages, and a library that let it become one would have made observability a source of outages.

Defined Under Namespace

Modules: Outcome Classes: None, OpenTelemetry, Registry, Reporter

Constant Summary collapse

PUBLISH_TOTAL = Publishes, tagged with exchange, routing.key, message.type and outcome — the tags Java's MetricNames puts on a publish. One counter rather than two: outcome=confirmed is a message the broker took, outcome=failed one it would not, and the sum of the two is how many times this process tried — a number that used to need adding up.

"acemq.publish.total"

PUBLISH_DURATION = How long a publish took, in seconds, from the call to the broker's answer, tagged the same way as PUBLISH_TOTAL. Recorded for every outcome, so a slow failed is told apart from a slow confirmed.

"acemq.publish.duration"

CONSUME_TOTAL = Deliveries handled, tagged with queue and outcome. What the consumer decided — not what the handler asked for. Exactly one acemq.consume.total series goes up per delivery, and the outcome tag is the word naming what really happened: acked, retried, rejected, dead_lettered or parked. A handler asking for a retry on its last attempt is tagged dead_lettered and not retried, because that is what the consumer will really do with the message, and it is the word its span carries too. rejected is kept apart from dead_lettered even though both end in the dead-letter queue: a message the handler refused on purpose is the system working and one that ran out of attempts is not. parked is a third thing again — nothing could read it, which no number of further attempts will change.

"acemq.consume.total"

CONSUME_DURATION = How long handlers take, in seconds, tagged with queue and outcome. Tagged with the outcome because the interesting question is almost always about one of them: a p99 that includes the failures is a different number from the p99 of the work that succeeded, and a queue whose retries are slow and whose acks are fast looks healthy without this tag.

"acemq.consume.duration"

CONSUME_ATTEMPTS = Which attempt each delivery was, tagged with queue. A distribution rather than a counter, and it is two numbers in one. Its sample count is how many deliveries this consumer was given, counted on the way in — the difference between a queue nothing is reading and a queue one thing is stuck on. Its maximum and mean say how many goes those deliveries are taking, so a rising distribution is a dependency starting to struggle before any of it has reached the dead-letter queue.

"acemq.consume.attempts"

CONSUME_IN_FLIGHT = How many messages are being handled right now, tagged with queue.

"acemq.consume.in.flight"

RETRIED_TOTAL = Messages sent to a retry queue, tagged with queue. The same deliveries as acemq.consume.total tagged outcome=retried, counted again under a name of their own. Kept because a retry rate is the number most often wanted on its own and this is the way to have it without a tag filter — and because Java, Go, .NET and Python all keep it, so an alert written once reads the same against all five.

"acemq.messages.retried.total"

DEAD_LETTERED_TOTAL = Messages sent to a dead-letter queue, tagged with queue. The counterpart of RETRIED_TOTAL, and the same deliveries as acemq.consume.total tagged outcome=dead_lettered. A message the handler rejected on purpose is not counted here: both end in the same queue, and only the word keeps them apart.

"acemq.messages.dead.lettered.total"

REQUEST_TOTAL = Request-reply round trips, tagged with routing.key, message.type and outcome — Outcome::ANSWERED, Outcome::TIMED_OUT or Outcome::FAILED. The caller's number rather than the responder's. It counts what the thread that called Patterns::Requester#call experienced, which is the publish, the responder's work and the reply's trip home added together — and the three cannot be told apart from in here, which is exactly why the caller's own view is worth a metric of its own. timed_out is not a failure of this library and is deliberately kept apart from failed: it says a reply did not arrive in time, not that anything went wrong. The request may still be queued, still being handled, or long since done with the reply lost on the way back.

"acemq.request.total"

REQUEST_DURATION = How long a round trip took, in seconds, tagged the same way as REQUEST_TOTAL. Recorded for the calls that timed out as well, where it is the deadline rather than the work — which is the point: a timed_out distribution sitting exactly on the timeout is what a deadline set too short looks like from outside.

"acemq.request.duration"

TAG_ROUTING_KEY = The label a request metric names its destination with, and the one it names the envelope's type with. Spelled with the dots Java and .NET use rather than with underscores, because these are the same two series read on the same dashboard and a tag that differs by a punctuation mark is a panel that silently covers four languages out of five. Registry turns them into routing_key and message_type on the way to Prometheus, which is the only place the dots are illegal.

:"routing.key"

TAG_MESSAGE_TYPE =

:"message.type"

PIPELINE_RUN_TOTAL = Pipeline runs that finished, tagged with pipeline, step and outcome — completed when the route ran out, ended_early when a step decided the message goes no further.

"acemq.pipeline.run.total"

PIPELINE_RUN_DURATION = How long a message had existed when it left a pipeline, in seconds, tagged with pipeline. The age of the envelope rather than the time in the last step, so this is the whole run: the envelope was created when the message entered and carried through every hop.

"acemq.pipeline.run.duration"

SET_ASIDE_FAILED = Messages that could not be moved to their dead-letter or parking queue, because the republish itself failed. Worth an alert, and usually a queue that was never declared. Nothing is lost: the delivery is never settled, so the broker redelivers it. What it looks like from outside is a handler failing over and over on the same message, which is a different problem with a different fix — and this counter is the only thing that tells the two apart. The Go and Python libraries raise the same counter where they reject the message to the broker instead, so an alert written once reads the same against all three.

"acemq.messages.set.aside.failed"

RUNG_MISSING = Long retries that had to wait in the consumer because the rung queue they were meant to wait on is not on the broker. Worth an alert. Nothing breaks and no message is lost: the retry still happens and the wait still happens. What is lost is the reason the rung exists — a consumer restarted mid-wait now turns a five-minute backoff into no wait at all — and without this there would be no sign of it, because a topology that was never applied looks exactly like one that was until something has to wait.

"acemq.retry.rung.missing"