acemq_amqp.topology
The exchanges, queues and bindings a service expects to find.
Declaring them one call at a time works, and stops working the moment somebody needs to know what a service will do to a broker before it does it. A topology can be printed, checked and applied, which is the difference between a deployment somebody approves and one they find out about::
topology = (
Topology()
.exchange("orders-events", "topic")
.queue("shipping-orders", dead_letter=True)
.binding("shipping-orders", "orders-events", "order.placed")
)
print(topology)
await topology.apply(transport)
Mistakes are raised where they are made rather than collected for later. Go accumulates them on the builder because a Go builder has nowhere else to put them; Python has an exception and a traceback that points at the line that is wrong, which is more useful than a message that says a topology is bad.
Two exchanges are this library's own rather than the caller's:
RETRY_EXCHANGE returns an expired rung message to the queue it came
from, and DEAD_LETTER_EXCHANGE reaches the dead-letter and parking
queues. Both are declared, and bound, by Topology.queue() when it is asked
for retries or for dead-lettering, so that the whole arrangement appears in
Topology.plan() and nothing depends on a caller remembering a binding.
A consumer declares that same half again when it starts, through
declare_where_failures_go(), without waiting to be asked. Declaring it
twice costs two round trips and nothing else, because both declarations carry
the same arguments; not declaring it at all costs a message, because a service
deployed without its topology republishes a dead letter to a queue the broker
cannot route to, and the broker discards what it cannot route in silence.
A durable queue asked for here is a quorum queue. That is the default in
every AceMQ library, and it is a contract rather than a preference for the same
reason the rung arguments are: a Java service and a Python service consuming
orders both declare orders, and a queue type is compared as strictly as
any other argument, so the second one to arrive is answered
PRECONDITION_FAILED and cannot consume at all. See Topology.queue()
for what stays classic and why.
1# Copyright 2026 AceMQ. 2# 3# Licensed under the Apache License, Version 2.0 (the "License"); 4# you may not use this file except in compliance with the License. 5# You may obtain a copy of the License at 6# 7# https://www.apache.org/licenses/LICENSE-2.0 8# 9# Unless required by applicable law or agreed to in writing, software 10# distributed under the License is distributed on an "AS IS" BASIS, 11# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12# See the License for the specific language governing permissions and 13# limitations under the License. 14 15"""The exchanges, queues and bindings a service expects to find. 16 17Declaring them one call at a time works, and stops working the moment somebody 18needs to know what a service will do to a broker *before* it does it. A topology 19can be printed, checked and applied, which is the difference between a 20deployment somebody approves and one they find out about:: 21 22 topology = ( 23 Topology() 24 .exchange("orders-events", "topic") 25 .queue("shipping-orders", dead_letter=True) 26 .binding("shipping-orders", "orders-events", "order.placed") 27 ) 28 print(topology) 29 await topology.apply(transport) 30 31Mistakes are raised where they are made rather than collected for later. Go 32accumulates them on the builder because a Go builder has nowhere else to put 33them; Python has an exception and a traceback that points at the line that is 34wrong, which is more useful than a message that says a topology is bad. 35 36Two exchanges are this library's own rather than the caller's: 37:data:`RETRY_EXCHANGE` returns an expired rung message to the queue it came 38from, and :data:`DEAD_LETTER_EXCHANGE` reaches the dead-letter and parking 39queues. Both are declared, and bound, by :meth:`Topology.queue` when it is asked 40for retries or for dead-lettering, so that the whole arrangement appears in 41:meth:`Topology.plan` and nothing depends on a caller remembering a binding. 42 43A consumer declares that same half again when it starts, through 44:func:`declare_where_failures_go`, without waiting to be asked. Declaring it 45twice costs two round trips and nothing else, because both declarations carry 46the same arguments; not declaring it at all costs a message, because a service 47deployed without its topology republishes a dead letter to a queue the broker 48cannot route to, and the broker discards what it cannot route in silence. 49 50A durable queue asked for here is a **quorum** queue. That is the default in 51every AceMQ library, and it is a contract rather than a preference for the same 52reason the rung arguments are: a Java service and a Python service consuming 53``orders`` both declare ``orders``, and a queue type is compared as strictly as 54any other argument, so the second one to arrive is answered 55``PRECONDITION_FAILED`` and cannot consume at all. See :meth:`Topology.queue` 56for what stays classic and why. 57""" 58 59from __future__ import annotations 60 61from collections.abc import Mapping 62from dataclasses import dataclass, field 63from datetime import timedelta 64from typing import Any 65 66from . import naming 67from .retry import RetryPolicy 68from .transport import ExchangeSpec, QueueSpec, Transport 69 70#: Where the broker sends a message this queue rejects or expires. 71DEAD_LETTER_EXCHANGE_ARG = "x-dead-letter-exchange" 72 73#: What routing key it is sent under, which a direct exchange matches against 74#: its bindings. 75DEAD_LETTER_ROUTING_KEY_ARG = "x-dead-letter-routing-key" 76 77#: How long a message may sit on a queue before the broker expires it, in 78#: milliseconds. On the queue, never on the message: see :meth:`Topology.queue`. 79MESSAGE_TTL_ARG = "x-message-ttl" 80 81#: The exchange a rung queue dead-letters through on its way back to the queue 82#: the message came from, bound on the source queue's own name. 83#: 84#: This constant and :data:`DEAD_LETTER_EXCHANGE` are the only two places that 85#: name an exchange this library manages, which is deliberate. A rung is a 86#: contract rather than a preference: two services consuming the same queue 87#: declare the same ``{queue}.retry.{delay}`` by name, and a rung declared with 88#: different arguments answers the second service ``PRECONDITION_FAILED`` and 89#: leaves it unable to consume at all. Python used to dead-letter a rung through 90#: the default exchange, which needs no exchange and no binding and works 91#: perfectly well on its own — but Java has always used a named exchange, most 92#: of the released code follows Java, and two libraries cannot both be right 93#: about one queue. This is the settled answer. 94RETRY_EXCHANGE = "acemq.retry" 95 96#: The exchange the dead-letter and parking queues are reached through, each 97#: bound on its own name. 98DEAD_LETTER_EXCHANGE = "acemq.dlx" 99 100#: What kind of queue to declare. Fixed at declaration: a queue that exists 101#: cannot be turned into another kind, only deleted and declared again. 102QUEUE_TYPE_ARG = "x-queue-type" 103 104#: The kind a durable queue asked for through :meth:`Topology.queue` is declared 105#: as, which is what Java, Go, .NET and Ruby declare too. 106#: 107#: There is no matching constant for a classic queue, and that is deliberate: a 108#: classic queue is declared by leaving :data:`QUEUE_TYPE_ARG` off altogether, 109#: which is what the other libraries send and therefore the only spelling a 110#: broker will find equivalent to theirs. 111QUORUM_QUEUE_TYPE = "quorum" 112 113#: How both managed exchanges are declared: direct, because every binding on 114#: them matches a queue name exactly, and durable, because a topology that 115#: vanished with the broker would leave rungs dead-lettering into nothing. 116_MANAGED_EXCHANGE = ExchangeSpec(kind="direct", durable=True) 117 118 119def rung_args(source: str, delay: timedelta) -> dict[str, Any]: 120 """The arguments a ``{source}.retry.{delay}`` queue must be declared with. 121 122 Exactly three keys, and the same three in every AceMQ library: the delay as 123 ``x-message-ttl``, :data:`RETRY_EXCHANGE` as the dead-letter target, and the 124 source queue as the routing key it is sent under. Nothing consumes a rung; 125 the time-to-live is the only thing that ever takes a message out of one, and 126 the binding made by :meth:`Topology.queue` is what brings it home. 127 128 The TTL is on the queue rather than on each message, which is the whole 129 reason a policy needs several queues instead of one. RabbitMQ expires 130 messages only from the head of a queue, so a single queue holding 131 per-message TTLs releases nothing while a message with a long one sits at 132 the front: a thirty-second wait queued behind a ten-minute wait becomes a 133 ten-minute wait. The delays that come out would bear no relation to the ones 134 that went in, and the bug would only appear under the load that puts two 135 different waits on the queue at once. 136 137 :param source: the queue being consumed, which is where an expired message 138 goes back to 139 :param delay: how long the rung holds a message 140 :returns: the argument table, which is a contract and is pinned by a test 141 """ 142 return { 143 MESSAGE_TTL_ARG: int(delay.total_seconds() * 1000), 144 DEAD_LETTER_EXCHANGE_ARG: RETRY_EXCHANGE, 145 DEAD_LETTER_ROUTING_KEY_ARG: source, 146 } 147 148 149@dataclass(frozen=True, slots=True) 150class Binding: 151 """Messages matching a routing key going from an exchange to a queue.""" 152 153 queue: str 154 exchange: str 155 routing_key: str = "" 156 157 def __str__(self) -> str: 158 return f"{self.exchange} -> {self.queue} ({self.routing_key})" 159 160 161@dataclass(frozen=True, slots=True) 162class PlanAction: 163 """One thing applying a topology would do.""" 164 165 kind: str 166 name: str 167 detail: str = "" 168 169 def __str__(self) -> str: 170 if not self.detail: 171 return f"declare {self.kind} {self.name}" 172 return f"declare {self.kind} {self.name} ({self.detail})" 173 174 175@dataclass(frozen=True, slots=True) 176class _NamedQueue: 177 name: str 178 spec: QueueSpec 179 180 181@dataclass(frozen=True, slots=True) 182class _NamedExchange: 183 name: str 184 spec: ExchangeSpec 185 186 187@dataclass 188class Topology: 189 """A description of what a service needs a broker to have. 190 191 Every method that adds something returns the topology, so a description 192 reads as one expression. The order things are added in does not matter: 193 :meth:`apply` declares exchanges first, then queues, then the bindings 194 between them, because that is the order a broker will accept them in. 195 """ 196 197 _exchanges: list[_NamedExchange] = field(default_factory=list) 198 _queues: list[_NamedQueue] = field(default_factory=list) 199 _bindings: list[Binding] = field(default_factory=list) 200 201 def exchange( 202 self, 203 name: str, 204 kind: str = "topic", 205 *, 206 durable: bool = True, 207 auto_delete: bool = False, 208 args: Mapping[str, Any] | None = None, 209 ) -> Topology: 210 """Adds an exchange. 211 212 :param name: what to call it 213 :param kind: ``direct``, ``topic``, ``fanout`` or ``headers`` 214 :param durable: survives a broker restart 215 :param auto_delete: goes away when its last binding does 216 :param args: broker-specific arguments 217 :returns: this topology 218 """ 219 if not name: 220 raise ValueError("acemq: an exchange needs a name") 221 if not kind: 222 raise ValueError(f"acemq: exchange {name!r} needs a kind (direct, topic, fanout)") 223 self._exchanges.append( 224 _NamedExchange( 225 name, 226 ExchangeSpec( 227 kind=kind, 228 durable=durable, 229 auto_delete=auto_delete, 230 args=dict(args or {}), 231 ), 232 ) 233 ) 234 return self 235 236 def queue( 237 self, 238 name: str, 239 *, 240 durable: bool = True, 241 auto_delete: bool = False, 242 exclusive: bool = False, 243 quorum: bool | None = None, 244 dead_letter: bool = False, 245 retry: RetryPolicy | None = None, 246 args: Mapping[str, Any] | None = None, 247 ) -> Topology: 248 """Adds a queue, and the queues that catch what it cannot handle. 249 250 A durable queue is a **quorum** queue unless it is asked to be something 251 else. That is what Java has always declared and what the other libraries 252 declare now, and it has to be the same everywhere: the queue type is one 253 more argument the broker compares, so a Java service and a Python 254 service that disagree about ``orders`` cannot both consume it. Pass 255 ``quorum=False`` for a classic queue, which is declared by leaving 256 ``x-queue-type`` off entirely — the spelling every AceMQ library uses, 257 and therefore the only one a broker finds equivalent to theirs. 258 259 ``exclusive``, ``auto_delete`` and ``durable=False`` all mean a queue 260 that belongs to one connection, and RabbitMQ refuses a quorum queue that 261 is any of those. Such a queue is declared classic without being asked, 262 because the alternative is a declaration the broker rejects with a 263 message that never mentions the word quorum. Asking for both — a queue 264 that is ``quorum=True`` and exclusive — is refused here instead, where 265 the contradiction is written down. 266 267 A queue whose ``args`` already name a kind keeps it: that is how 268 :func:`acemq_amqp.patterns.stream` declares a stream, and a caller who 269 wrote ``x-queue-type`` down meant it. Writing it down *and* passing 270 ``quorum`` is two answers to one question and is refused. 271 272 ``dead_letter`` declares ``{name}.dlq`` alongside it and points the 273 broker at it, so a message this queue rejects or lets expire lands 274 somewhere an operator can find by name rather than disappearing. It 275 declares ``{name}.parked`` too, because a consumer on this queue sends 276 anything it cannot decode there — and a library that parks messages into 277 a queue nobody declared has only moved the disappearance somewhere else. 278 279 Neither dead-letters itself: a dead-letter queue that dead-letters is a 280 loop, and a loop is how a poison message becomes an outage. Both are 281 reached through :data:`DEAD_LETTER_EXCHANGE`, bound on their own names, 282 which is what Java has always done and is therefore what a broker shared 283 with a Java service already has. A service that wants its own 284 dead-letter exchange instead should leave ``dead_letter`` alone and pass 285 the arguments in ``args``. 286 287 ``retry`` declares the rung queues that policy's long waits need: one 288 ``{name}.retry.{delay}`` per distinct delay at or above the policy's 289 threshold, each declared with :func:`rung_args` — the delay as 290 ``x-message-ttl``, :data:`RETRY_EXCHANGE` as the dead-letter target, and 291 this queue as the routing key. The binding that brings an expired 292 message home, ``{name}`` to :data:`RETRY_EXCHANGE` on ``{name}``, is 293 made here and not left to a caller to remember: without it every rung 294 expires into an exchange that routes nowhere, and a retry that is 295 dropped by the broker looks exactly like one that is still waiting. 296 297 It takes the policy rather than a list of delays on purpose. The rungs a 298 consumer will publish to are derived from the policy it is running, so 299 anything else here would be a second copy of the same list, free to drift 300 from the first — and the way that drift shows up is a retry published to 301 a queue nobody declared, at the moment the service is already failing. 302 One value produces both, or the topology is not a description of what the 303 service needs. 304 305 :param name: what to call it 306 :param durable: survives a broker restart 307 :param auto_delete: goes away when its last consumer does 308 :param exclusive: usable only by the connection that declared it 309 :param quorum: replicated rather than classic. ``None``, the default, 310 means quorum for a durable queue and classic for one belonging to a 311 single connection 312 :param dead_letter: also declare and wire ``{name}.dlq``, and declare 313 ``{name}.parked`` 314 :param retry: also declare the rung queues this policy's long waits use 315 :param args: broker-specific arguments 316 :returns: this topology 317 :raises ValueError: when the queue cannot be what it was asked to be 318 """ 319 if not name: 320 raise ValueError("acemq: a queue needs a name") 321 322 arguments: dict[str, Any] = dict(args or {}) 323 kind = _queue_type( 324 name, 325 arguments, 326 durable=durable, 327 auto_delete=auto_delete, 328 exclusive=exclusive, 329 quorum=quorum, 330 ) 331 if kind is not None: 332 arguments[QUEUE_TYPE_ARG] = kind 333 if dead_letter: 334 # Refused rather than resolved, because either answer would be a 335 # guess about which of two conflicting instructions was meant. 336 conflicting = sorted( 337 key 338 for key in arguments 339 if key in (DEAD_LETTER_EXCHANGE_ARG, DEAD_LETTER_ROUTING_KEY_ARG) 340 ) 341 if conflicting: 342 raise ValueError( 343 f"acemq: queue {name!r} asks for dead_letter=True and also sets " 344 + ", ".join(conflicting) 345 + "; pick one" 346 ) 347 arguments[DEAD_LETTER_EXCHANGE_ARG] = DEAD_LETTER_EXCHANGE 348 arguments[DEAD_LETTER_ROUTING_KEY_ARG] = naming.dead_letter_queue(name) 349 350 self._queues.append( 351 _NamedQueue( 352 name, 353 QueueSpec( 354 durable=durable, 355 auto_delete=auto_delete, 356 exclusive=exclusive, 357 args=arguments, 358 ), 359 ) 360 ) 361 if dead_letter: 362 self._managed_exchange(DEAD_LETTER_EXCHANGE) 363 # Classic, deliberately, whatever the queue in front of them is. 364 # Java declares both QueueType.CLASSIC and these are shared names: 365 # a dead-letter queue nobody consumes gains nothing from being 366 # replicated, and gaining it here would cost a Java service its 367 # declaration of the same queue. 368 for target in (naming.dead_letter_queue(name), naming.parked_queue(name)): 369 self._queues.append(_NamedQueue(target, QueueSpec(durable=durable))) 370 self._bindings.append( 371 Binding(queue=target, exchange=DEAD_LETTER_EXCHANGE, routing_key=target) 372 ) 373 if retry is not None: 374 rungs = retry.broker_rungs() 375 if rungs: 376 self._managed_exchange(RETRY_EXCHANGE) 377 # Classic too, and for the same reason as ``.dlq``: a rung is 378 # declared by every service consuming the source queue, Java 379 # declares it QueueType.CLASSIC, and :func:`rung_args` is the 380 # whole argument table — three keys, no ``x-queue-type``. 381 for rung in rungs: 382 self._queues.append( 383 _NamedQueue( 384 naming.retry_queue(name, rung), 385 QueueSpec(durable=durable, args=rung_args(name, rung)), 386 ) 387 ) 388 # One binding brings every expired message back to the queue it 389 # came from. Made here rather than offered as something to add, 390 # because a missing binding loses every retry silently: the 391 # broker drops what a direct exchange cannot route and says 392 # nothing, so the queue looks quiet rather than broken. 393 self._bindings.append( 394 Binding(queue=name, exchange=RETRY_EXCHANGE, routing_key=name) 395 ) 396 return self 397 398 def _managed_exchange(self, name: str) -> None: 399 """Declares one of this library's own exchanges, once. 400 401 Several queues in one topology each need :data:`RETRY_EXCHANGE` and 402 :data:`DEAD_LETTER_EXCHANGE`, and declaring either twice is what 403 :meth:`validate` refuses. A caller who has already named the same 404 exchange keeps their own spec: theirs is the deliberate one. 405 """ 406 if any(entry.name == name for entry in self._exchanges): 407 return 408 self._exchanges.append(_NamedExchange(name, _MANAGED_EXCHANGE)) 409 410 def binding(self, queue: str, exchange: str, routing_key: str = "") -> Topology: 411 """Routes messages matching a key from an exchange to a queue. 412 413 :param queue: where messages end up, which this topology must declare 414 :param exchange: where they come from, which this topology must declare 415 :param routing_key: what to match, empty for a fanout 416 :returns: this topology 417 """ 418 self._bindings.append(Binding(queue=queue, exchange=exchange, routing_key=routing_key)) 419 return self 420 421 @property 422 def queues(self) -> list[str]: 423 """The queue names this topology declares, in declaration order.""" 424 return [entry.name for entry in self._queues] 425 426 @property 427 def exchanges(self) -> list[str]: 428 """The exchange names this topology declares, in declaration order.""" 429 return [entry.name for entry in self._exchanges] 430 431 @property 432 def bindings(self) -> list[Binding]: 433 """The bindings this topology declares.""" 434 return list(self._bindings) 435 436 def validate(self) -> None: 437 """Reports what is wrong with the description itself. 438 439 A binding naming a queue the topology does not declare is the mistake 440 worth catching here. The broker would accept it if the queue happened to 441 exist already, and the service would then depend on something nothing 442 declares — which works until the day it is deployed somewhere new. 443 444 :raises ValueError: when the topology cannot be right 445 """ 446 seen_queues: set[str] = set() 447 for queue in self._queues: 448 if queue.name in seen_queues: 449 raise ValueError(f"acemq: the topology declares queue {queue.name!r} twice") 450 seen_queues.add(queue.name) 451 452 seen_exchanges: set[str] = set() 453 for exchange in self._exchanges: 454 if exchange.name in seen_exchanges: 455 raise ValueError( 456 f"acemq: the topology declares exchange {exchange.name!r} twice" 457 ) 458 seen_exchanges.add(exchange.name) 459 460 for binding in self._bindings: 461 if binding.queue not in seen_queues: 462 raise ValueError( 463 f"acemq: binding {binding} names queue {binding.queue!r}, " 464 "which this topology does not declare" 465 ) 466 if not binding.exchange: 467 raise ValueError( 468 f"acemq: binding {binding} names the default exchange, " 469 "which cannot be bound to" 470 ) 471 if binding.exchange not in seen_exchanges: 472 raise ValueError( 473 f"acemq: binding {binding} names exchange {binding.exchange!r}, " 474 "which this topology does not declare" 475 ) 476 477 def plan(self) -> list[PlanAction]: 478 """What :meth:`apply` would do, without doing it. 479 480 Deliberately not a difference against the live broker: AMQP offers no 481 way to enumerate what is there without the management API, and a plan 482 that quietly guessed would be worse than one that is honest about being 483 a statement of intent. 484 485 :returns: one action per thing that would be declared 486 :raises ValueError: when the topology cannot be right 487 """ 488 self.validate() 489 actions = [ 490 PlanAction("exchange", entry.name, _describe(entry.spec)) 491 for entry in self._exchanges 492 ] 493 actions += [ 494 PlanAction("queue", entry.name, _describe(entry.spec)) for entry in self._queues 495 ] 496 actions += [ 497 PlanAction( 498 "binding", binding.queue, f"from {binding.exchange} on {binding.routing_key}" 499 ) 500 for binding in self._bindings 501 ] 502 return actions 503 504 async def apply(self, transport: Transport) -> None: 505 """Declares everything, in the order a broker needs. 506 507 It stops at the first failure. A queue that already exists with 508 different settings is refused by the broker with ``PRECONDITION_FAILED``, 509 and that refusal is passed on rather than swallowed: it means this 510 service and the broker disagree about what the queue is, and carrying on 511 would leave the service using a queue that is not the one it asked for. 512 513 :param transport: where to declare it 514 :raises ValueError: when the topology cannot be right 515 """ 516 self.validate() 517 for exchange in self._exchanges: 518 await transport.declare_exchange(exchange.name, exchange.spec) 519 for queue in self._queues: 520 await transport.declare_queue(queue.name, queue.spec) 521 for binding in self._bindings: 522 await transport.bind(binding.queue, binding.exchange, binding.routing_key) 523 524 def __str__(self) -> str: 525 try: 526 actions = self.plan() 527 except ValueError as failure: 528 return f"Topology(invalid: {failure})" 529 header = ( 530 f"Topology: {len(self._exchanges)} exchanges, " 531 f"{len(self._queues)} queues, {len(self._bindings)} bindings" 532 ) 533 return "\n".join([header, *(f" {action}" for action in actions)]) 534 535 536def _queue_type( 537 name: str, 538 args: Mapping[str, Any], 539 *, 540 durable: bool, 541 auto_delete: bool, 542 exclusive: bool, 543 quorum: bool | None, 544) -> str | None: 545 """What to declare a queue as, or ``None`` to leave the argument off. 546 547 ``None`` is classic and is a real answer rather than a missing one: every 548 AceMQ library declares a classic queue by sending no ``x-queue-type`` at 549 all, so this matches Java's ``QueueType.CLASSIC`` argument for argument. 550 551 RabbitMQ does in fact normalise the two — a queue declared with no 552 ``x-queue-type`` and one declared with ``classic`` are the same queue to 553 the broker, checked in both orders against a real one, while quorum against 554 either is refused. So sending ``classic`` would also work. Sending nothing 555 is still what this does, because "declare exactly what the other libraries 556 declare" is a rule that holds without anyone having to remember which 557 arguments the broker happens to normalise and which it compares literally. 558 559 :param name: the queue, for the message when the answer is a refusal 560 :param args: what the caller wrote down, which may already name a kind 561 :param durable: survives a broker restart 562 :param auto_delete: goes away when its last consumer does 563 :param exclusive: usable only by the connection that declared it 564 :param quorum: what the caller asked for, or ``None`` for the default 565 :returns: the value for :data:`QUEUE_TYPE_ARG`, or ``None`` for classic 566 :raises ValueError: when what was asked for cannot exist 567 """ 568 written = args.get(QUEUE_TYPE_ARG) 569 if written is not None: 570 if quorum is not None: 571 raise ValueError( 572 f"acemq: queue {name!r} sets {QUEUE_TYPE_ARG}={written!r} and also passes " 573 f"quorum={quorum!r}; pick one" 574 ) 575 return str(written) 576 577 # A queue tied to one connection cannot be a quorum queue. RabbitMQ refuses 578 # exclusive, auto-delete and transient outright, and the refusal it sends 579 # back does not say which of the three it objected to. 580 per_connection = exclusive or auto_delete or not durable 581 if quorum is None: 582 return None if per_connection else QUORUM_QUEUE_TYPE 583 if not quorum: 584 return None 585 if per_connection: 586 refused = ", ".join( 587 label 588 for label, set_ in ( 589 ("exclusive", exclusive), 590 ("auto_delete", auto_delete), 591 ("durable=False", not durable), 592 ) 593 if set_ 594 ) 595 raise ValueError( 596 f"acemq: queue {name!r} asks for quorum=True and also {refused}; " 597 "a quorum queue belongs to the broker rather than to a connection, " 598 "so it can be none of those" 599 ) 600 return QUORUM_QUEUE_TYPE 601 602 603async def declare_where_failures_go( 604 transport: Transport, queue: str, retry: RetryPolicy | None = None 605) -> None: 606 """Declares the queues a consumer of ``queue`` will need when it fails. 607 608 ``acemq.dlx``, ``{queue}.dlq``, ``{queue}.parked`` and the two bindings that 609 reach them; and, when ``retry`` has waits long enough to be held by the 610 broker, ``acemq.retry``, one rung queue per distinct wait, and the binding 611 that brings an expired message back to ``queue``. Exactly what 612 :meth:`Topology.queue` writes for the same arguments, and exactly what Java's 613 consumer declares when it starts. 614 615 Called by :meth:`acemq_amqp.Connection.consume` before it subscribes, which 616 is the point of it: everything here is somewhere a message goes when the 617 handler has given up, and a message republished to a queue that is not there 618 is discarded by the broker without a word. Applying a topology first is still 619 the right way to run a service — it is reviewable, and it declares the source 620 queue with the ``x-dead-letter-exchange`` this cannot — but a service that was 621 deployed without one should lose a message to a bug rather than to a missing 622 queue. 623 624 The source queue is deliberately not declared. It belongs to whoever set the 625 service up, who chose its type and its arguments, and guessing wrong is a 626 ``PRECONDITION_FAILED`` that stops the consumer starting at all. 627 628 Safe to call repeatedly, and safe to call after :meth:`Topology.apply` or 629 before it: every declaration here carries the same arguments the topology 630 would send, so the second one to arrive is a no-op rather than a refusal. 631 632 :param transport: where to declare it 633 :param queue: the queue being consumed 634 :param retry: the policy the consumer is running, whose long waits decide 635 which rungs exist. ``None``, or a policy that holds every wait in the 636 consumer, declares only the dead-letter half 637 """ 638 await transport.declare_exchange(DEAD_LETTER_EXCHANGE, _MANAGED_EXCHANGE) 639 for target in (naming.dead_letter_queue(queue), naming.parked_queue(queue)): 640 # Classic, whatever the queue in front of them is, for the reason 641 # :meth:`Topology.queue` gives: these are shared names, and a Java 642 # service declaring the same queue declares it classic. 643 await transport.declare_queue(target, QueueSpec(durable=True)) 644 await transport.bind(target, DEAD_LETTER_EXCHANGE, target) 645 646 # Keyed by name rather than by delay. Two delays can render to the same name, 647 # and a second queue by that name with a different time-to-live is not a 648 # second rung — it is the PRECONDITION_FAILED that stops this consumer 649 # starting. The first delay to reach a name owns it, which is what Java does. 650 rungs: dict[str, timedelta] = {} 651 for delay in retry.broker_rungs() if retry is not None else []: 652 rungs.setdefault(naming.retry_queue(queue, delay), delay) 653 if not rungs: 654 return 655 656 await transport.declare_exchange(RETRY_EXCHANGE, _MANAGED_EXCHANGE) 657 for name, delay in rungs.items(): 658 await transport.declare_queue( 659 name, QueueSpec(durable=True, args=rung_args(queue, delay)) 660 ) 661 # One binding brings every expired message home. Without it a rung expires 662 # into an exchange that routes nowhere, and the broker drops what it cannot 663 # route in silence: the retry looks like one still waiting rather than one 664 # that is gone. 665 await transport.bind(queue, RETRY_EXCHANGE, queue) 666 667 668def _describe(spec: QueueSpec | ExchangeSpec) -> str: 669 """A spec as the line an operator would want in a deployment log.""" 670 parts: list[str] = [] 671 if isinstance(spec, ExchangeSpec): 672 parts.append(spec.kind) 673 parts.append("durable" if spec.durable else "transient") 674 if spec.auto_delete: 675 parts.append("auto-delete") 676 if isinstance(spec, QueueSpec) and spec.exclusive: 677 parts.append("exclusive") 678 parts += [f"{key}={spec.args[key]!r}" for key in sorted(spec.args)] 679 return ", ".join(parts)
120def rung_args(source: str, delay: timedelta) -> dict[str, Any]: 121 """The arguments a ``{source}.retry.{delay}`` queue must be declared with. 122 123 Exactly three keys, and the same three in every AceMQ library: the delay as 124 ``x-message-ttl``, :data:`RETRY_EXCHANGE` as the dead-letter target, and the 125 source queue as the routing key it is sent under. Nothing consumes a rung; 126 the time-to-live is the only thing that ever takes a message out of one, and 127 the binding made by :meth:`Topology.queue` is what brings it home. 128 129 The TTL is on the queue rather than on each message, which is the whole 130 reason a policy needs several queues instead of one. RabbitMQ expires 131 messages only from the head of a queue, so a single queue holding 132 per-message TTLs releases nothing while a message with a long one sits at 133 the front: a thirty-second wait queued behind a ten-minute wait becomes a 134 ten-minute wait. The delays that come out would bear no relation to the ones 135 that went in, and the bug would only appear under the load that puts two 136 different waits on the queue at once. 137 138 :param source: the queue being consumed, which is where an expired message 139 goes back to 140 :param delay: how long the rung holds a message 141 :returns: the argument table, which is a contract and is pinned by a test 142 """ 143 return { 144 MESSAGE_TTL_ARG: int(delay.total_seconds() * 1000), 145 DEAD_LETTER_EXCHANGE_ARG: RETRY_EXCHANGE, 146 DEAD_LETTER_ROUTING_KEY_ARG: source, 147 }
The arguments a {source}acemq_amqp.retry.{delay} queue must be declared with.
Exactly three keys, and the same three in every AceMQ library: the delay as
x-message-ttl, RETRY_EXCHANGE as the dead-letter target, and the
source queue as the routing key it is sent under. Nothing consumes a rung;
the time-to-live is the only thing that ever takes a message out of one, and
the binding made by Topology.queue() is what brings it home.
The TTL is on the queue rather than on each message, which is the whole reason a policy needs several queues instead of one. RabbitMQ expires messages only from the head of a queue, so a single queue holding per-message TTLs releases nothing while a message with a long one sits at the front: a thirty-second wait queued behind a ten-minute wait becomes a ten-minute wait. The delays that come out would bear no relation to the ones that went in, and the bug would only appear under the load that puts two different waits on the queue at once.
Parameters
- source: the queue being consumed, which is where an expired message goes back to
- delay: how long the rung holds a message :returns: the argument table, which is a contract and is pinned by a test
150@dataclass(frozen=True, slots=True) 151class Binding: 152 """Messages matching a routing key going from an exchange to a queue.""" 153 154 queue: str 155 exchange: str 156 routing_key: str = "" 157 158 def __str__(self) -> str: 159 return f"{self.exchange} -> {self.queue} ({self.routing_key})"
Messages matching a routing key going from an exchange to a queue.
162@dataclass(frozen=True, slots=True) 163class PlanAction: 164 """One thing applying a topology would do.""" 165 166 kind: str 167 name: str 168 detail: str = "" 169 170 def __str__(self) -> str: 171 if not self.detail: 172 return f"declare {self.kind} {self.name}" 173 return f"declare {self.kind} {self.name} ({self.detail})"
One thing applying a topology would do.
188@dataclass 189class Topology: 190 """A description of what a service needs a broker to have. 191 192 Every method that adds something returns the topology, so a description 193 reads as one expression. The order things are added in does not matter: 194 :meth:`apply` declares exchanges first, then queues, then the bindings 195 between them, because that is the order a broker will accept them in. 196 """ 197 198 _exchanges: list[_NamedExchange] = field(default_factory=list) 199 _queues: list[_NamedQueue] = field(default_factory=list) 200 _bindings: list[Binding] = field(default_factory=list) 201 202 def exchange( 203 self, 204 name: str, 205 kind: str = "topic", 206 *, 207 durable: bool = True, 208 auto_delete: bool = False, 209 args: Mapping[str, Any] | None = None, 210 ) -> Topology: 211 """Adds an exchange. 212 213 :param name: what to call it 214 :param kind: ``direct``, ``topic``, ``fanout`` or ``headers`` 215 :param durable: survives a broker restart 216 :param auto_delete: goes away when its last binding does 217 :param args: broker-specific arguments 218 :returns: this topology 219 """ 220 if not name: 221 raise ValueError("acemq: an exchange needs a name") 222 if not kind: 223 raise ValueError(f"acemq: exchange {name!r} needs a kind (direct, topic, fanout)") 224 self._exchanges.append( 225 _NamedExchange( 226 name, 227 ExchangeSpec( 228 kind=kind, 229 durable=durable, 230 auto_delete=auto_delete, 231 args=dict(args or {}), 232 ), 233 ) 234 ) 235 return self 236 237 def queue( 238 self, 239 name: str, 240 *, 241 durable: bool = True, 242 auto_delete: bool = False, 243 exclusive: bool = False, 244 quorum: bool | None = None, 245 dead_letter: bool = False, 246 retry: RetryPolicy | None = None, 247 args: Mapping[str, Any] | None = None, 248 ) -> Topology: 249 """Adds a queue, and the queues that catch what it cannot handle. 250 251 A durable queue is a **quorum** queue unless it is asked to be something 252 else. That is what Java has always declared and what the other libraries 253 declare now, and it has to be the same everywhere: the queue type is one 254 more argument the broker compares, so a Java service and a Python 255 service that disagree about ``orders`` cannot both consume it. Pass 256 ``quorum=False`` for a classic queue, which is declared by leaving 257 ``x-queue-type`` off entirely — the spelling every AceMQ library uses, 258 and therefore the only one a broker finds equivalent to theirs. 259 260 ``exclusive``, ``auto_delete`` and ``durable=False`` all mean a queue 261 that belongs to one connection, and RabbitMQ refuses a quorum queue that 262 is any of those. Such a queue is declared classic without being asked, 263 because the alternative is a declaration the broker rejects with a 264 message that never mentions the word quorum. Asking for both — a queue 265 that is ``quorum=True`` and exclusive — is refused here instead, where 266 the contradiction is written down. 267 268 A queue whose ``args`` already name a kind keeps it: that is how 269 :func:`acemq_amqp.patterns.stream` declares a stream, and a caller who 270 wrote ``x-queue-type`` down meant it. Writing it down *and* passing 271 ``quorum`` is two answers to one question and is refused. 272 273 ``dead_letter`` declares ``{name}.dlq`` alongside it and points the 274 broker at it, so a message this queue rejects or lets expire lands 275 somewhere an operator can find by name rather than disappearing. It 276 declares ``{name}.parked`` too, because a consumer on this queue sends 277 anything it cannot decode there — and a library that parks messages into 278 a queue nobody declared has only moved the disappearance somewhere else. 279 280 Neither dead-letters itself: a dead-letter queue that dead-letters is a 281 loop, and a loop is how a poison message becomes an outage. Both are 282 reached through :data:`DEAD_LETTER_EXCHANGE`, bound on their own names, 283 which is what Java has always done and is therefore what a broker shared 284 with a Java service already has. A service that wants its own 285 dead-letter exchange instead should leave ``dead_letter`` alone and pass 286 the arguments in ``args``. 287 288 ``retry`` declares the rung queues that policy's long waits need: one 289 ``{name}.retry.{delay}`` per distinct delay at or above the policy's 290 threshold, each declared with :func:`rung_args` — the delay as 291 ``x-message-ttl``, :data:`RETRY_EXCHANGE` as the dead-letter target, and 292 this queue as the routing key. The binding that brings an expired 293 message home, ``{name}`` to :data:`RETRY_EXCHANGE` on ``{name}``, is 294 made here and not left to a caller to remember: without it every rung 295 expires into an exchange that routes nowhere, and a retry that is 296 dropped by the broker looks exactly like one that is still waiting. 297 298 It takes the policy rather than a list of delays on purpose. The rungs a 299 consumer will publish to are derived from the policy it is running, so 300 anything else here would be a second copy of the same list, free to drift 301 from the first — and the way that drift shows up is a retry published to 302 a queue nobody declared, at the moment the service is already failing. 303 One value produces both, or the topology is not a description of what the 304 service needs. 305 306 :param name: what to call it 307 :param durable: survives a broker restart 308 :param auto_delete: goes away when its last consumer does 309 :param exclusive: usable only by the connection that declared it 310 :param quorum: replicated rather than classic. ``None``, the default, 311 means quorum for a durable queue and classic for one belonging to a 312 single connection 313 :param dead_letter: also declare and wire ``{name}.dlq``, and declare 314 ``{name}.parked`` 315 :param retry: also declare the rung queues this policy's long waits use 316 :param args: broker-specific arguments 317 :returns: this topology 318 :raises ValueError: when the queue cannot be what it was asked to be 319 """ 320 if not name: 321 raise ValueError("acemq: a queue needs a name") 322 323 arguments: dict[str, Any] = dict(args or {}) 324 kind = _queue_type( 325 name, 326 arguments, 327 durable=durable, 328 auto_delete=auto_delete, 329 exclusive=exclusive, 330 quorum=quorum, 331 ) 332 if kind is not None: 333 arguments[QUEUE_TYPE_ARG] = kind 334 if dead_letter: 335 # Refused rather than resolved, because either answer would be a 336 # guess about which of two conflicting instructions was meant. 337 conflicting = sorted( 338 key 339 for key in arguments 340 if key in (DEAD_LETTER_EXCHANGE_ARG, DEAD_LETTER_ROUTING_KEY_ARG) 341 ) 342 if conflicting: 343 raise ValueError( 344 f"acemq: queue {name!r} asks for dead_letter=True and also sets " 345 + ", ".join(conflicting) 346 + "; pick one" 347 ) 348 arguments[DEAD_LETTER_EXCHANGE_ARG] = DEAD_LETTER_EXCHANGE 349 arguments[DEAD_LETTER_ROUTING_KEY_ARG] = naming.dead_letter_queue(name) 350 351 self._queues.append( 352 _NamedQueue( 353 name, 354 QueueSpec( 355 durable=durable, 356 auto_delete=auto_delete, 357 exclusive=exclusive, 358 args=arguments, 359 ), 360 ) 361 ) 362 if dead_letter: 363 self._managed_exchange(DEAD_LETTER_EXCHANGE) 364 # Classic, deliberately, whatever the queue in front of them is. 365 # Java declares both QueueType.CLASSIC and these are shared names: 366 # a dead-letter queue nobody consumes gains nothing from being 367 # replicated, and gaining it here would cost a Java service its 368 # declaration of the same queue. 369 for target in (naming.dead_letter_queue(name), naming.parked_queue(name)): 370 self._queues.append(_NamedQueue(target, QueueSpec(durable=durable))) 371 self._bindings.append( 372 Binding(queue=target, exchange=DEAD_LETTER_EXCHANGE, routing_key=target) 373 ) 374 if retry is not None: 375 rungs = retry.broker_rungs() 376 if rungs: 377 self._managed_exchange(RETRY_EXCHANGE) 378 # Classic too, and for the same reason as ``.dlq``: a rung is 379 # declared by every service consuming the source queue, Java 380 # declares it QueueType.CLASSIC, and :func:`rung_args` is the 381 # whole argument table — three keys, no ``x-queue-type``. 382 for rung in rungs: 383 self._queues.append( 384 _NamedQueue( 385 naming.retry_queue(name, rung), 386 QueueSpec(durable=durable, args=rung_args(name, rung)), 387 ) 388 ) 389 # One binding brings every expired message back to the queue it 390 # came from. Made here rather than offered as something to add, 391 # because a missing binding loses every retry silently: the 392 # broker drops what a direct exchange cannot route and says 393 # nothing, so the queue looks quiet rather than broken. 394 self._bindings.append( 395 Binding(queue=name, exchange=RETRY_EXCHANGE, routing_key=name) 396 ) 397 return self 398 399 def _managed_exchange(self, name: str) -> None: 400 """Declares one of this library's own exchanges, once. 401 402 Several queues in one topology each need :data:`RETRY_EXCHANGE` and 403 :data:`DEAD_LETTER_EXCHANGE`, and declaring either twice is what 404 :meth:`validate` refuses. A caller who has already named the same 405 exchange keeps their own spec: theirs is the deliberate one. 406 """ 407 if any(entry.name == name for entry in self._exchanges): 408 return 409 self._exchanges.append(_NamedExchange(name, _MANAGED_EXCHANGE)) 410 411 def binding(self, queue: str, exchange: str, routing_key: str = "") -> Topology: 412 """Routes messages matching a key from an exchange to a queue. 413 414 :param queue: where messages end up, which this topology must declare 415 :param exchange: where they come from, which this topology must declare 416 :param routing_key: what to match, empty for a fanout 417 :returns: this topology 418 """ 419 self._bindings.append(Binding(queue=queue, exchange=exchange, routing_key=routing_key)) 420 return self 421 422 @property 423 def queues(self) -> list[str]: 424 """The queue names this topology declares, in declaration order.""" 425 return [entry.name for entry in self._queues] 426 427 @property 428 def exchanges(self) -> list[str]: 429 """The exchange names this topology declares, in declaration order.""" 430 return [entry.name for entry in self._exchanges] 431 432 @property 433 def bindings(self) -> list[Binding]: 434 """The bindings this topology declares.""" 435 return list(self._bindings) 436 437 def validate(self) -> None: 438 """Reports what is wrong with the description itself. 439 440 A binding naming a queue the topology does not declare is the mistake 441 worth catching here. The broker would accept it if the queue happened to 442 exist already, and the service would then depend on something nothing 443 declares — which works until the day it is deployed somewhere new. 444 445 :raises ValueError: when the topology cannot be right 446 """ 447 seen_queues: set[str] = set() 448 for queue in self._queues: 449 if queue.name in seen_queues: 450 raise ValueError(f"acemq: the topology declares queue {queue.name!r} twice") 451 seen_queues.add(queue.name) 452 453 seen_exchanges: set[str] = set() 454 for exchange in self._exchanges: 455 if exchange.name in seen_exchanges: 456 raise ValueError( 457 f"acemq: the topology declares exchange {exchange.name!r} twice" 458 ) 459 seen_exchanges.add(exchange.name) 460 461 for binding in self._bindings: 462 if binding.queue not in seen_queues: 463 raise ValueError( 464 f"acemq: binding {binding} names queue {binding.queue!r}, " 465 "which this topology does not declare" 466 ) 467 if not binding.exchange: 468 raise ValueError( 469 f"acemq: binding {binding} names the default exchange, " 470 "which cannot be bound to" 471 ) 472 if binding.exchange not in seen_exchanges: 473 raise ValueError( 474 f"acemq: binding {binding} names exchange {binding.exchange!r}, " 475 "which this topology does not declare" 476 ) 477 478 def plan(self) -> list[PlanAction]: 479 """What :meth:`apply` would do, without doing it. 480 481 Deliberately not a difference against the live broker: AMQP offers no 482 way to enumerate what is there without the management API, and a plan 483 that quietly guessed would be worse than one that is honest about being 484 a statement of intent. 485 486 :returns: one action per thing that would be declared 487 :raises ValueError: when the topology cannot be right 488 """ 489 self.validate() 490 actions = [ 491 PlanAction("exchange", entry.name, _describe(entry.spec)) 492 for entry in self._exchanges 493 ] 494 actions += [ 495 PlanAction("queue", entry.name, _describe(entry.spec)) for entry in self._queues 496 ] 497 actions += [ 498 PlanAction( 499 "binding", binding.queue, f"from {binding.exchange} on {binding.routing_key}" 500 ) 501 for binding in self._bindings 502 ] 503 return actions 504 505 async def apply(self, transport: Transport) -> None: 506 """Declares everything, in the order a broker needs. 507 508 It stops at the first failure. A queue that already exists with 509 different settings is refused by the broker with ``PRECONDITION_FAILED``, 510 and that refusal is passed on rather than swallowed: it means this 511 service and the broker disagree about what the queue is, and carrying on 512 would leave the service using a queue that is not the one it asked for. 513 514 :param transport: where to declare it 515 :raises ValueError: when the topology cannot be right 516 """ 517 self.validate() 518 for exchange in self._exchanges: 519 await transport.declare_exchange(exchange.name, exchange.spec) 520 for queue in self._queues: 521 await transport.declare_queue(queue.name, queue.spec) 522 for binding in self._bindings: 523 await transport.bind(binding.queue, binding.exchange, binding.routing_key) 524 525 def __str__(self) -> str: 526 try: 527 actions = self.plan() 528 except ValueError as failure: 529 return f"Topology(invalid: {failure})" 530 header = ( 531 f"Topology: {len(self._exchanges)} exchanges, " 532 f"{len(self._queues)} queues, {len(self._bindings)} bindings" 533 ) 534 return "\n".join([header, *(f" {action}" for action in actions)])
A description of what a service needs a broker to have.
Every method that adds something returns the topology, so a description
reads as one expression. The order things are added in does not matter:
apply() declares exchanges first, then queues, then the bindings
between them, because that is the order a broker will accept them in.
202 def exchange( 203 self, 204 name: str, 205 kind: str = "topic", 206 *, 207 durable: bool = True, 208 auto_delete: bool = False, 209 args: Mapping[str, Any] | None = None, 210 ) -> Topology: 211 """Adds an exchange. 212 213 :param name: what to call it 214 :param kind: ``direct``, ``topic``, ``fanout`` or ``headers`` 215 :param durable: survives a broker restart 216 :param auto_delete: goes away when its last binding does 217 :param args: broker-specific arguments 218 :returns: this topology 219 """ 220 if not name: 221 raise ValueError("acemq: an exchange needs a name") 222 if not kind: 223 raise ValueError(f"acemq: exchange {name!r} needs a kind (direct, topic, fanout)") 224 self._exchanges.append( 225 _NamedExchange( 226 name, 227 ExchangeSpec( 228 kind=kind, 229 durable=durable, 230 auto_delete=auto_delete, 231 args=dict(args or {}), 232 ), 233 ) 234 ) 235 return self
Adds an exchange.
Parameters
- name: what to call it
- kind:
direct,topic,fanoutorheaders - durable: survives a broker restart
- auto_delete: goes away when its last binding does
- args: broker-specific arguments :returns: this topology
237 def queue( 238 self, 239 name: str, 240 *, 241 durable: bool = True, 242 auto_delete: bool = False, 243 exclusive: bool = False, 244 quorum: bool | None = None, 245 dead_letter: bool = False, 246 retry: RetryPolicy | None = None, 247 args: Mapping[str, Any] | None = None, 248 ) -> Topology: 249 """Adds a queue, and the queues that catch what it cannot handle. 250 251 A durable queue is a **quorum** queue unless it is asked to be something 252 else. That is what Java has always declared and what the other libraries 253 declare now, and it has to be the same everywhere: the queue type is one 254 more argument the broker compares, so a Java service and a Python 255 service that disagree about ``orders`` cannot both consume it. Pass 256 ``quorum=False`` for a classic queue, which is declared by leaving 257 ``x-queue-type`` off entirely — the spelling every AceMQ library uses, 258 and therefore the only one a broker finds equivalent to theirs. 259 260 ``exclusive``, ``auto_delete`` and ``durable=False`` all mean a queue 261 that belongs to one connection, and RabbitMQ refuses a quorum queue that 262 is any of those. Such a queue is declared classic without being asked, 263 because the alternative is a declaration the broker rejects with a 264 message that never mentions the word quorum. Asking for both — a queue 265 that is ``quorum=True`` and exclusive — is refused here instead, where 266 the contradiction is written down. 267 268 A queue whose ``args`` already name a kind keeps it: that is how 269 :func:`acemq_amqp.patterns.stream` declares a stream, and a caller who 270 wrote ``x-queue-type`` down meant it. Writing it down *and* passing 271 ``quorum`` is two answers to one question and is refused. 272 273 ``dead_letter`` declares ``{name}.dlq`` alongside it and points the 274 broker at it, so a message this queue rejects or lets expire lands 275 somewhere an operator can find by name rather than disappearing. It 276 declares ``{name}.parked`` too, because a consumer on this queue sends 277 anything it cannot decode there — and a library that parks messages into 278 a queue nobody declared has only moved the disappearance somewhere else. 279 280 Neither dead-letters itself: a dead-letter queue that dead-letters is a 281 loop, and a loop is how a poison message becomes an outage. Both are 282 reached through :data:`DEAD_LETTER_EXCHANGE`, bound on their own names, 283 which is what Java has always done and is therefore what a broker shared 284 with a Java service already has. A service that wants its own 285 dead-letter exchange instead should leave ``dead_letter`` alone and pass 286 the arguments in ``args``. 287 288 ``retry`` declares the rung queues that policy's long waits need: one 289 ``{name}.retry.{delay}`` per distinct delay at or above the policy's 290 threshold, each declared with :func:`rung_args` — the delay as 291 ``x-message-ttl``, :data:`RETRY_EXCHANGE` as the dead-letter target, and 292 this queue as the routing key. The binding that brings an expired 293 message home, ``{name}`` to :data:`RETRY_EXCHANGE` on ``{name}``, is 294 made here and not left to a caller to remember: without it every rung 295 expires into an exchange that routes nowhere, and a retry that is 296 dropped by the broker looks exactly like one that is still waiting. 297 298 It takes the policy rather than a list of delays on purpose. The rungs a 299 consumer will publish to are derived from the policy it is running, so 300 anything else here would be a second copy of the same list, free to drift 301 from the first — and the way that drift shows up is a retry published to 302 a queue nobody declared, at the moment the service is already failing. 303 One value produces both, or the topology is not a description of what the 304 service needs. 305 306 :param name: what to call it 307 :param durable: survives a broker restart 308 :param auto_delete: goes away when its last consumer does 309 :param exclusive: usable only by the connection that declared it 310 :param quorum: replicated rather than classic. ``None``, the default, 311 means quorum for a durable queue and classic for one belonging to a 312 single connection 313 :param dead_letter: also declare and wire ``{name}.dlq``, and declare 314 ``{name}.parked`` 315 :param retry: also declare the rung queues this policy's long waits use 316 :param args: broker-specific arguments 317 :returns: this topology 318 :raises ValueError: when the queue cannot be what it was asked to be 319 """ 320 if not name: 321 raise ValueError("acemq: a queue needs a name") 322 323 arguments: dict[str, Any] = dict(args or {}) 324 kind = _queue_type( 325 name, 326 arguments, 327 durable=durable, 328 auto_delete=auto_delete, 329 exclusive=exclusive, 330 quorum=quorum, 331 ) 332 if kind is not None: 333 arguments[QUEUE_TYPE_ARG] = kind 334 if dead_letter: 335 # Refused rather than resolved, because either answer would be a 336 # guess about which of two conflicting instructions was meant. 337 conflicting = sorted( 338 key 339 for key in arguments 340 if key in (DEAD_LETTER_EXCHANGE_ARG, DEAD_LETTER_ROUTING_KEY_ARG) 341 ) 342 if conflicting: 343 raise ValueError( 344 f"acemq: queue {name!r} asks for dead_letter=True and also sets " 345 + ", ".join(conflicting) 346 + "; pick one" 347 ) 348 arguments[DEAD_LETTER_EXCHANGE_ARG] = DEAD_LETTER_EXCHANGE 349 arguments[DEAD_LETTER_ROUTING_KEY_ARG] = naming.dead_letter_queue(name) 350 351 self._queues.append( 352 _NamedQueue( 353 name, 354 QueueSpec( 355 durable=durable, 356 auto_delete=auto_delete, 357 exclusive=exclusive, 358 args=arguments, 359 ), 360 ) 361 ) 362 if dead_letter: 363 self._managed_exchange(DEAD_LETTER_EXCHANGE) 364 # Classic, deliberately, whatever the queue in front of them is. 365 # Java declares both QueueType.CLASSIC and these are shared names: 366 # a dead-letter queue nobody consumes gains nothing from being 367 # replicated, and gaining it here would cost a Java service its 368 # declaration of the same queue. 369 for target in (naming.dead_letter_queue(name), naming.parked_queue(name)): 370 self._queues.append(_NamedQueue(target, QueueSpec(durable=durable))) 371 self._bindings.append( 372 Binding(queue=target, exchange=DEAD_LETTER_EXCHANGE, routing_key=target) 373 ) 374 if retry is not None: 375 rungs = retry.broker_rungs() 376 if rungs: 377 self._managed_exchange(RETRY_EXCHANGE) 378 # Classic too, and for the same reason as ``.dlq``: a rung is 379 # declared by every service consuming the source queue, Java 380 # declares it QueueType.CLASSIC, and :func:`rung_args` is the 381 # whole argument table — three keys, no ``x-queue-type``. 382 for rung in rungs: 383 self._queues.append( 384 _NamedQueue( 385 naming.retry_queue(name, rung), 386 QueueSpec(durable=durable, args=rung_args(name, rung)), 387 ) 388 ) 389 # One binding brings every expired message back to the queue it 390 # came from. Made here rather than offered as something to add, 391 # because a missing binding loses every retry silently: the 392 # broker drops what a direct exchange cannot route and says 393 # nothing, so the queue looks quiet rather than broken. 394 self._bindings.append( 395 Binding(queue=name, exchange=RETRY_EXCHANGE, routing_key=name) 396 ) 397 return self
Adds a queue, and the queues that catch what it cannot handle.
A durable queue is a quorum queue unless it is asked to be something
else. That is what Java has always declared and what the other libraries
declare now, and it has to be the same everywhere: the queue type is one
more argument the broker compares, so a Java service and a Python
service that disagree about orders cannot both consume it. Pass
quorum=False for a classic queue, which is declared by leaving
x-queue-type off entirely — the spelling every AceMQ library uses,
and therefore the only one a broker finds equivalent to theirs.
exclusive, auto_delete and durable=False all mean a queue
that belongs to one connection, and RabbitMQ refuses a quorum queue that
is any of those. Such a queue is declared classic without being asked,
because the alternative is a declaration the broker rejects with a
message that never mentions the word quorum. Asking for both — a queue
that is quorum=True and exclusive — is refused here instead, where
the contradiction is written down.
A queue whose args already name a kind keeps it: that is how
acemq_amqp.patterns.stream() declares a stream, and a caller who
wrote x-queue-type down meant it. Writing it down and passing
quorum is two answers to one question and is refused.
dead_letter declares {name}.dlq alongside it and points the
broker at it, so a message this queue rejects or lets expire lands
somewhere an operator can find by name rather than disappearing. It
declares {name}.parked too, because a consumer on this queue sends
anything it cannot decode there — and a library that parks messages into
a queue nobody declared has only moved the disappearance somewhere else.
Neither dead-letters itself: a dead-letter queue that dead-letters is a
loop, and a loop is how a poison message becomes an outage. Both are
reached through DEAD_LETTER_EXCHANGE, bound on their own names,
which is what Java has always done and is therefore what a broker shared
with a Java service already has. A service that wants its own
dead-letter exchange instead should leave dead_letter alone and pass
the arguments in args.
retry declares the rung queues that policy's long waits need: one
{name}acemq_amqp.retry.{delay} per distinct delay at or above the policy's
threshold, each declared with rung_args() — the delay as
x-message-ttl, RETRY_EXCHANGE as the dead-letter target, and
this queue as the routing key. The binding that brings an expired
message home, {name} to RETRY_EXCHANGE on {name}, is
made here and not left to a caller to remember: without it every rung
expires into an exchange that routes nowhere, and a retry that is
dropped by the broker looks exactly like one that is still waiting.
It takes the policy rather than a list of delays on purpose. The rungs a consumer will publish to are derived from the policy it is running, so anything else here would be a second copy of the same list, free to drift from the first — and the way that drift shows up is a retry published to a queue nobody declared, at the moment the service is already failing. One value produces both, or the topology is not a description of what the service needs.
Parameters
- name: what to call it
- durable: survives a broker restart
- auto_delete: goes away when its last consumer does
- exclusive: usable only by the connection that declared it
- quorum: replicated rather than classic.
None, the default, means quorum for a durable queue and classic for one belonging to a single connection - dead_letter: also declare and wire
{name}.dlq, and declare{name}.parked - retry: also declare the rung queues this policy's long waits use
- args: broker-specific arguments :returns: this topology
Raises
- ValueError: when the queue cannot be what it was asked to be
411 def binding(self, queue: str, exchange: str, routing_key: str = "") -> Topology: 412 """Routes messages matching a key from an exchange to a queue. 413 414 :param queue: where messages end up, which this topology must declare 415 :param exchange: where they come from, which this topology must declare 416 :param routing_key: what to match, empty for a fanout 417 :returns: this topology 418 """ 419 self._bindings.append(Binding(queue=queue, exchange=exchange, routing_key=routing_key)) 420 return self
Routes messages matching a key from an exchange to a queue.
Parameters
- queue: where messages end up, which this topology must declare
- exchange: where they come from, which this topology must declare
- routing_key: what to match, empty for a fanout :returns: this topology
422 @property 423 def queues(self) -> list[str]: 424 """The queue names this topology declares, in declaration order.""" 425 return [entry.name for entry in self._queues]
The queue names this topology declares, in declaration order.
427 @property 428 def exchanges(self) -> list[str]: 429 """The exchange names this topology declares, in declaration order.""" 430 return [entry.name for entry in self._exchanges]
The exchange names this topology declares, in declaration order.
432 @property 433 def bindings(self) -> list[Binding]: 434 """The bindings this topology declares.""" 435 return list(self._bindings)
The bindings this topology declares.
437 def validate(self) -> None: 438 """Reports what is wrong with the description itself. 439 440 A binding naming a queue the topology does not declare is the mistake 441 worth catching here. The broker would accept it if the queue happened to 442 exist already, and the service would then depend on something nothing 443 declares — which works until the day it is deployed somewhere new. 444 445 :raises ValueError: when the topology cannot be right 446 """ 447 seen_queues: set[str] = set() 448 for queue in self._queues: 449 if queue.name in seen_queues: 450 raise ValueError(f"acemq: the topology declares queue {queue.name!r} twice") 451 seen_queues.add(queue.name) 452 453 seen_exchanges: set[str] = set() 454 for exchange in self._exchanges: 455 if exchange.name in seen_exchanges: 456 raise ValueError( 457 f"acemq: the topology declares exchange {exchange.name!r} twice" 458 ) 459 seen_exchanges.add(exchange.name) 460 461 for binding in self._bindings: 462 if binding.queue not in seen_queues: 463 raise ValueError( 464 f"acemq: binding {binding} names queue {binding.queue!r}, " 465 "which this topology does not declare" 466 ) 467 if not binding.exchange: 468 raise ValueError( 469 f"acemq: binding {binding} names the default exchange, " 470 "which cannot be bound to" 471 ) 472 if binding.exchange not in seen_exchanges: 473 raise ValueError( 474 f"acemq: binding {binding} names exchange {binding.exchange!r}, " 475 "which this topology does not declare" 476 )
Reports what is wrong with the description itself.
A binding naming a queue the topology does not declare is the mistake worth catching here. The broker would accept it if the queue happened to exist already, and the service would then depend on something nothing declares — which works until the day it is deployed somewhere new.
Raises
- ValueError: when the topology cannot be right
478 def plan(self) -> list[PlanAction]: 479 """What :meth:`apply` would do, without doing it. 480 481 Deliberately not a difference against the live broker: AMQP offers no 482 way to enumerate what is there without the management API, and a plan 483 that quietly guessed would be worse than one that is honest about being 484 a statement of intent. 485 486 :returns: one action per thing that would be declared 487 :raises ValueError: when the topology cannot be right 488 """ 489 self.validate() 490 actions = [ 491 PlanAction("exchange", entry.name, _describe(entry.spec)) 492 for entry in self._exchanges 493 ] 494 actions += [ 495 PlanAction("queue", entry.name, _describe(entry.spec)) for entry in self._queues 496 ] 497 actions += [ 498 PlanAction( 499 "binding", binding.queue, f"from {binding.exchange} on {binding.routing_key}" 500 ) 501 for binding in self._bindings 502 ] 503 return actions
What apply() would do, without doing it.
Deliberately not a difference against the live broker: AMQP offers no way to enumerate what is there without the management API, and a plan that quietly guessed would be worse than one that is honest about being a statement of intent.
:returns: one action per thing that would be declared
Raises
- ValueError: when the topology cannot be right
505 async def apply(self, transport: Transport) -> None: 506 """Declares everything, in the order a broker needs. 507 508 It stops at the first failure. A queue that already exists with 509 different settings is refused by the broker with ``PRECONDITION_FAILED``, 510 and that refusal is passed on rather than swallowed: it means this 511 service and the broker disagree about what the queue is, and carrying on 512 would leave the service using a queue that is not the one it asked for. 513 514 :param transport: where to declare it 515 :raises ValueError: when the topology cannot be right 516 """ 517 self.validate() 518 for exchange in self._exchanges: 519 await transport.declare_exchange(exchange.name, exchange.spec) 520 for queue in self._queues: 521 await transport.declare_queue(queue.name, queue.spec) 522 for binding in self._bindings: 523 await transport.bind(binding.queue, binding.exchange, binding.routing_key)
Declares everything, in the order a broker needs.
It stops at the first failure. A queue that already exists with
different settings is refused by the broker with PRECONDITION_FAILED,
and that refusal is passed on rather than swallowed: it means this
service and the broker disagree about what the queue is, and carrying on
would leave the service using a queue that is not the one it asked for.
Parameters
- transport: where to declare it
Raises
- ValueError: when the topology cannot be right
604async def declare_where_failures_go( 605 transport: Transport, queue: str, retry: RetryPolicy | None = None 606) -> None: 607 """Declares the queues a consumer of ``queue`` will need when it fails. 608 609 ``acemq.dlx``, ``{queue}.dlq``, ``{queue}.parked`` and the two bindings that 610 reach them; and, when ``retry`` has waits long enough to be held by the 611 broker, ``acemq.retry``, one rung queue per distinct wait, and the binding 612 that brings an expired message back to ``queue``. Exactly what 613 :meth:`Topology.queue` writes for the same arguments, and exactly what Java's 614 consumer declares when it starts. 615 616 Called by :meth:`acemq_amqp.Connection.consume` before it subscribes, which 617 is the point of it: everything here is somewhere a message goes when the 618 handler has given up, and a message republished to a queue that is not there 619 is discarded by the broker without a word. Applying a topology first is still 620 the right way to run a service — it is reviewable, and it declares the source 621 queue with the ``x-dead-letter-exchange`` this cannot — but a service that was 622 deployed without one should lose a message to a bug rather than to a missing 623 queue. 624 625 The source queue is deliberately not declared. It belongs to whoever set the 626 service up, who chose its type and its arguments, and guessing wrong is a 627 ``PRECONDITION_FAILED`` that stops the consumer starting at all. 628 629 Safe to call repeatedly, and safe to call after :meth:`Topology.apply` or 630 before it: every declaration here carries the same arguments the topology 631 would send, so the second one to arrive is a no-op rather than a refusal. 632 633 :param transport: where to declare it 634 :param queue: the queue being consumed 635 :param retry: the policy the consumer is running, whose long waits decide 636 which rungs exist. ``None``, or a policy that holds every wait in the 637 consumer, declares only the dead-letter half 638 """ 639 await transport.declare_exchange(DEAD_LETTER_EXCHANGE, _MANAGED_EXCHANGE) 640 for target in (naming.dead_letter_queue(queue), naming.parked_queue(queue)): 641 # Classic, whatever the queue in front of them is, for the reason 642 # :meth:`Topology.queue` gives: these are shared names, and a Java 643 # service declaring the same queue declares it classic. 644 await transport.declare_queue(target, QueueSpec(durable=True)) 645 await transport.bind(target, DEAD_LETTER_EXCHANGE, target) 646 647 # Keyed by name rather than by delay. Two delays can render to the same name, 648 # and a second queue by that name with a different time-to-live is not a 649 # second rung — it is the PRECONDITION_FAILED that stops this consumer 650 # starting. The first delay to reach a name owns it, which is what Java does. 651 rungs: dict[str, timedelta] = {} 652 for delay in retry.broker_rungs() if retry is not None else []: 653 rungs.setdefault(naming.retry_queue(queue, delay), delay) 654 if not rungs: 655 return 656 657 await transport.declare_exchange(RETRY_EXCHANGE, _MANAGED_EXCHANGE) 658 for name, delay in rungs.items(): 659 await transport.declare_queue( 660 name, QueueSpec(durable=True, args=rung_args(queue, delay)) 661 ) 662 # One binding brings every expired message home. Without it a rung expires 663 # into an exchange that routes nowhere, and the broker drops what it cannot 664 # route in silence: the retry looks like one still waiting rather than one 665 # that is gone. 666 await transport.bind(queue, RETRY_EXCHANGE, queue)
Declares the queues a consumer of queue will need when it fails.
acemq.dlx, {queue}.dlq, {queue}.parked and the two bindings that
reach them; and, when retry has waits long enough to be held by the
broker, acemq.retry, one rung queue per distinct wait, and the binding
that brings an expired message back to queue. Exactly what
Topology.queue() writes for the same arguments, and exactly what Java's
consumer declares when it starts.
Called by acemq_amqp.Connection.consume() before it subscribes, which
is the point of it: everything here is somewhere a message goes when the
handler has given up, and a message republished to a queue that is not there
is discarded by the broker without a word. Applying a topology first is still
the right way to run a service — it is reviewable, and it declares the source
queue with the x-dead-letter-exchange this cannot — but a service that was
deployed without one should lose a message to a bug rather than to a missing
queue.
The source queue is deliberately not declared. It belongs to whoever set the
service up, who chose its type and its arguments, and guessing wrong is a
PRECONDITION_FAILED that stops the consumer starting at all.
Safe to call repeatedly, and safe to call after Topology.apply() or
before it: every declaration here carries the same arguments the topology
would send, so the second one to arrive is a no-op rather than a refusal.
Parameters
- transport: where to declare it
- queue: the queue being consumed
- retry: the policy the consumer is running, whose long waits decide
which rungs exist.
None, or a policy that holds every wait in the consumer, declares only the dead-letter half