acemq_amqp
AceMQ for Python.
A client for AceMQ messaging over AMQP, speaking the same wire contract as the Java , Go and .NET libraries: the same reserved headers, the same defaults, the same retry semantics. A Python consumer reads what a Java producer writes, and the fixtures generated from the Java implementation pin that rather than leaving it to be discovered in production.
The API shape is Python's, deliberately. The contract is portable; the ergonomics are native.
Importing this package needs nothing installed. connect() reaches for
aio-pika at the moment it is called rather than at import, so a program that
only reads an envelope off a message somebody else delivered never has to have
an AMQP client at all. The blocking API lives in acemq_amqp.sync for
programs that are not running an event loop.
Reaching a real broker means reaching it over TLS: an amqps:// URL is
verified against the machine's trust store with no further configuration, and
Security in acemq_amqp.security covers a private certificate
authority, a client certificate, and a login supplied separately from the URL so
that a password never has to be written into a connection string.
1# Copyright 2026 AceMQ. 2# 3# Licensed under the Apache License, Version 2.0 (the "License"); 4# you may not use this file except in compliance with the License. 5# You may obtain a copy of the License at 6# 7# https://www.apache.org/licenses/LICENSE-2.0 8# 9# Unless required by applicable law or agreed to in writing, software 10# distributed under the License is distributed on an "AS IS" BASIS, 11# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12# See the License for the specific language governing permissions and 13# limitations under the License. 14 15"""AceMQ for Python. 16 17A client for AceMQ messaging over AMQP, speaking the same wire contract as the 18`Java <https://github.com/AceMQ-Company/acemq-java-amqp>`_, `Go 19<https://github.com/AceMQ-Company/acemq-go-amqp>`_ and `.NET 20<https://github.com/AceMQ-Company/acemq-dotnet-amqp>`_ libraries: the same 21reserved headers, the same defaults, the same retry semantics. A Python 22consumer reads what a Java producer writes, and the fixtures generated from the 23Java implementation pin that rather than leaving it to be discovered in 24production. 25 26The API shape is Python's, deliberately. The contract is portable; the 27ergonomics are native. 28 29Importing this package needs nothing installed. :func:`connect` reaches for 30aio-pika at the moment it is called rather than at import, so a program that 31only reads an envelope off a message somebody else delivered never has to have 32an AMQP client at all. The blocking API lives in :mod:`acemq_amqp.sync` for 33programs that are not running an event loop. 34 35Reaching a real broker means reaching it over TLS: an ``amqps://`` URL is 36verified against the machine's trust store with no further configuration, and 37:class:`Security` in :mod:`acemq_amqp.security` covers a private certificate 38authority, a client certificate, and a login supplied separately from the URL so 39that a password never has to be written into a connection string. 40""" 41 42from __future__ import annotations 43 44from . import headers 45from .ack import Ack, Action, FatalError, Settlement, accept, in_progress, park, reject 46from .ack import retry as _retry 47from .codec import ( 48 BYTES_CONTENT_TYPE, 49 JSON_CONTENT_TYPE, 50 TEXT_CONTENT_TYPE, 51 BytesCodec, 52 Codec, 53 CompositeCodec, 54 JsonCodec, 55 TextCodec, 56 codec_by_name, 57 codec_names, 58 register_codec, 59) 60from .connection import ( 61 BLOCKED_DETAIL, 62 DEFAULT_DRAIN_TIMEOUT, 63 DEFAULT_HEALTH_TIMEOUT, 64 AsyncHandler, 65 BrokerHealth, 66 Connection, 67 Consumer, 68 Handler, 69 Message, 70 Publisher, 71 connect, 72) 73from .envelope import Envelope 74from .errors import AceMQError, PublishError, PublishingPausedError, SecurityError 75from .interceptors import ( 76 ConsumeContext, 77 ConsumeInterceptor, 78 ConsumeNext, 79 PublishContext, 80 PublishInterceptor, 81 PublishNext, 82 SettlementListener, 83 consume_chain, 84 publish_chain, 85) 86from .naming import dead_letter_queue, parked_queue, retry_queue 87from .retry import ( 88 DEFAULT_BROKER_WAIT_THRESHOLD, 89 RetryPolicy, 90 Wait, 91 exponential_retry, 92 fixed_retry, 93 no_retry, 94) 95from .security import ( 96 DEVELOPMENT_MARKER, 97 Credentials, 98 CredentialsSource, 99 Security, 100 Verification, 101 credentials_from_environment, 102 credentials_from_file, 103 is_development_certificate, 104 without_verifying_the_broker, 105) 106from .telemetry import ( 107 METRIC_CONSUME_ATTEMPTS, 108 METRIC_CONSUME_DURATION, 109 METRIC_CONSUME_IN_FLIGHT, 110 METRIC_CONSUME_TOTAL, 111 METRIC_DEAD_LETTERED_TOTAL, 112 METRIC_OUTBOX_LAG, 113 METRIC_OUTBOX_TOTAL, 114 METRIC_PIPELINE_RUN_DURATION, 115 METRIC_PIPELINE_RUN_TOTAL, 116 METRIC_PUBLISH_DURATION, 117 METRIC_PUBLISH_TOTAL, 118 METRIC_REQUEST_DURATION, 119 METRIC_REQUEST_TOTAL, 120 METRIC_RETRIED_TOTAL, 121 METRIC_RUNG_MISSING, 122 METRIC_SET_ASIDE_FAILED, 123 DurationSummary, 124 HealthCheck, 125 HealthReport, 126 HealthStatus, 127 Metrics, 128 NullObserver, 129 Observer, 130 aggregate_health, 131 prometheus_text, 132) 133from .topology import ( 134 DEAD_LETTER_EXCHANGE, 135 QUEUE_TYPE_ARG, 136 QUORUM_QUEUE_TYPE, 137 RETRY_EXCHANGE, 138 Topology, 139 declare_where_failures_go, 140 rung_args, 141) 142from .transport import ( 143 BlockedState, 144 ConsumeSpec, 145 Delivery, 146 ExchangeSpec, 147 Outbound, 148 PublishResult, 149 QueueSpec, 150 Transport, 151) 152 153#: Returns a message to be tried again. See :func:`acemq_amqp.ack.retry`. 154#: 155#: Bound here rather than imported by name, because importing the retry module a 156#: few lines above binds ``acemq_amqp.retry`` to the module and the last write 157#: wins. A module is not what ``from acemq_amqp import retry`` should hand 158#: somebody when the other four libraries hand them the acknowledgement, and 159#: ``from acemq_amqp.retry import RetryPolicy`` still reaches the module. 160retry = _retry 161 162__version__ = "0.7.12" 163 164__all__ = [ 165 "BLOCKED_DETAIL", 166 "BYTES_CONTENT_TYPE", 167 "DEAD_LETTER_EXCHANGE", 168 "DEFAULT_BROKER_WAIT_THRESHOLD", 169 "DEFAULT_DRAIN_TIMEOUT", 170 "DEFAULT_HEALTH_TIMEOUT", 171 "DEVELOPMENT_MARKER", 172 "JSON_CONTENT_TYPE", 173 "METRIC_CONSUME_ATTEMPTS", 174 "METRIC_CONSUME_DURATION", 175 "METRIC_CONSUME_IN_FLIGHT", 176 "METRIC_CONSUME_TOTAL", 177 "METRIC_DEAD_LETTERED_TOTAL", 178 "METRIC_OUTBOX_LAG", 179 "METRIC_OUTBOX_TOTAL", 180 "METRIC_PIPELINE_RUN_DURATION", 181 "METRIC_PIPELINE_RUN_TOTAL", 182 "METRIC_PUBLISH_DURATION", 183 "METRIC_PUBLISH_TOTAL", 184 "METRIC_REQUEST_DURATION", 185 "METRIC_REQUEST_TOTAL", 186 "METRIC_RETRIED_TOTAL", 187 "METRIC_RUNG_MISSING", 188 "METRIC_SET_ASIDE_FAILED", 189 "QUEUE_TYPE_ARG", 190 "QUORUM_QUEUE_TYPE", 191 "RETRY_EXCHANGE", 192 "TEXT_CONTENT_TYPE", 193 "AceMQError", 194 "Ack", 195 "Action", 196 "AsyncHandler", 197 "BlockedState", 198 "BrokerHealth", 199 "BytesCodec", 200 "Codec", 201 "CompositeCodec", 202 "Connection", 203 "ConsumeContext", 204 "ConsumeInterceptor", 205 "ConsumeNext", 206 "ConsumeSpec", 207 "Consumer", 208 "Credentials", 209 "CredentialsSource", 210 "Delivery", 211 "DurationSummary", 212 "Envelope", 213 "ExchangeSpec", 214 "FatalError", 215 "Handler", 216 "HealthCheck", 217 "HealthReport", 218 "HealthStatus", 219 "JsonCodec", 220 "Message", 221 "Metrics", 222 "NullObserver", 223 "Observer", 224 "Outbound", 225 "PublishContext", 226 "PublishError", 227 "PublishInterceptor", 228 "PublishNext", 229 "PublishResult", 230 "Publisher", 231 "PublishingPausedError", 232 "QueueSpec", 233 "RetryPolicy", 234 "Security", 235 "SecurityError", 236 "Settlement", 237 "SettlementListener", 238 "TextCodec", 239 "Topology", 240 "Transport", 241 "Verification", 242 "Wait", 243 "__version__", 244 "accept", 245 "aggregate_health", 246 "codec_by_name", 247 "codec_names", 248 "connect", 249 "consume_chain", 250 "credentials_from_environment", 251 "credentials_from_file", 252 "dead_letter_queue", 253 "declare_where_failures_go", 254 "exponential_retry", 255 "fixed_retry", 256 "headers", 257 "in_progress", 258 "is_development_certificate", 259 "no_retry", 260 "park", 261 "parked_queue", 262 "prometheus_text", 263 "publish_chain", 264 "register_codec", 265 "reject", 266 "retry", 267 "retry_queue", 268 "rung_args", 269 "without_verifying_the_broker", 270]
Something this library refused to do.
70@dataclass(frozen=True, slots=True) 71class Ack: 72 """A handler's decision: it worked, try it again, never again, or nobody 73 can read it. 74 75 Returned rather than performed, so a handler that forgets to decide is a 76 handler that returns ``None`` and is caught immediately, rather than one 77 whose message sits unacknowledged until the connection drops and then comes 78 back — usually to the same handler, with the same outcome. 79 """ 80 81 action: Action 82 error: BaseException | None = None 83 #: How long to wait before the message is looked at again. Read for 84 #: :attr:`Action.IN_PROGRESS` only; a retry's wait is the policy's. 85 delay: timedelta | None = None 86 87 def __str__(self) -> str: 88 return self.action.value
A handler's decision: it worked, try it again, never again, or nobody can read it.
Returned rather than performed, so a handler that forgets to decide is a
handler that returns None and is caught immediately, rather than one
whose message sits unacknowledged until the connection drops and then comes
back — usually to the same handler, with the same outcome.
60class Action(Enum): 61 """The things that can be done with a delivered message.""" 62 63 ACCEPT = "accept" 64 RETRY = "retry" 65 REJECT = "reject" 66 PARK = "park" 67 IN_PROGRESS = "in_progress"
The things that can be done with a delivered message.
230@runtime_checkable 231class BlockedState(Protocol): 232 """A transport that can say whether the broker has blocked its connection. 233 234 RabbitMQ sends ``connection.blocked`` when it is low on memory or disk and 235 ``connection.unblocked`` when the alarm clears. In between it stops reading 236 from the socket, so everything sent on that connection — a publish, a 237 declaration, a health probe — waits rather than failing. From outside that 238 is indistinguishable from a broker that has silently gone away, and the two 239 want opposite responses: a blocked broker is one to wait for, a dead one is 240 one to fail over from. 241 242 Separate from :class:`Transport` because not every transport has an answer. 243 A fake in a test is never blocked and can say so; a client library that 244 keeps the state to itself cannot be asked at all, which is what ``None`` 245 below is for. It is the same shape as Java's ``isBlocked()`` and 246 ``blockedReason()``, .NET's ``IsBlocked``/``BlockedReason`` and Go's 247 ``BlockedReason()``. 248 """ 249 250 @property 251 def blocked(self) -> bool | None: 252 """Whether the broker has blocked this connection. 253 254 ``None`` rather than ``False`` when the question could not be asked at 255 all — a client that does not expose the state, a connection that is 256 between reconnections. The two are different facts and a report that 257 folds them together says a broker is fine when nobody looked. 258 """ 259 260 @property 261 def blocked_reason(self) -> str | None: 262 """What the broker said when it blocked this connection. 263 264 ``None`` when it is not blocked, and also ``None`` when the client read 265 the reason and did not keep it — which is what aio-pika does, and why 266 this is separate from :attr:`blocked` rather than a string that is empty 267 when there is nothing to say. A reason invented here would read exactly 268 like one RabbitMQ sent. 269 """
A transport that can say whether the broker has blocked its connection.
RabbitMQ sends connection.blocked when it is low on memory or disk and
connection.unblocked when the alarm clears. In between it stops reading
from the socket, so everything sent on that connection — a publish, a
declaration, a health probe — waits rather than failing. From outside that
is indistinguishable from a broker that has silently gone away, and the two
want opposite responses: a blocked broker is one to wait for, a dead one is
one to fail over from.
Separate from Transport because not every transport has an answer.
A fake in a test is never blocked and can say so; a client library that
keeps the state to itself cannot be asked at all, which is what None
below is for. It is the same shape as Java's isBlocked() and
blockedReason(), .NET's IsBlocked/BlockedReason and Go's
BlockedReason().
1771def _no_init_or_replace_init(self, *args, **kwargs): 1772 cls = type(self) 1773 1774 if cls._is_protocol: 1775 raise TypeError('Protocols cannot be instantiated') 1776 1777 # Already using a custom `__init__`. No need to calculate correct 1778 # `__init__` to call. This can lead to RecursionError. See bpo-45121. 1779 if cls.__init__ is not _no_init_or_replace_init: 1780 return 1781 1782 # Initially, `__init__` of a protocol subclass is set to `_no_init_or_replace_init`. 1783 # The first instantiation of the subclass will call `_no_init_or_replace_init` which 1784 # searches for a proper new `__init__` in the MRO. The new `__init__` 1785 # replaces the subclass' old `__init__` (ie `_no_init_or_replace_init`). Subsequent 1786 # instantiation of the protocol subclass will thus use the new 1787 # `__init__` and no longer call `_no_init_or_replace_init`. 1788 for base in cls.__mro__: 1789 init = base.__dict__.get('__init__', _no_init_or_replace_init) 1790 if init is not _no_init_or_replace_init: 1791 cls.__init__ = init 1792 break 1793 else: 1794 # should not happen 1795 cls.__init__ = object.__init__ 1796 1797 cls.__init__(self, *args, **kwargs)
250 @property 251 def blocked(self) -> bool | None: 252 """Whether the broker has blocked this connection. 253 254 ``None`` rather than ``False`` when the question could not be asked at 255 all — a client that does not expose the state, a connection that is 256 between reconnections. The two are different facts and a report that 257 folds them together says a broker is fine when nobody looked. 258 """
Whether the broker has blocked this connection.
None rather than False when the question could not be asked at
all — a client that does not expose the state, a connection that is
between reconnections. The two are different facts and a report that
folds them together says a broker is fine when nobody looked.
260 @property 261 def blocked_reason(self) -> str | None: 262 """What the broker said when it blocked this connection. 263 264 ``None`` when it is not blocked, and also ``None`` when the client read 265 the reason and did not keep it — which is what aio-pika does, and why 266 this is separate from :attr:`blocked` rather than a string that is empty 267 when there is nothing to say. A reason invented here would read exactly 268 like one RabbitMQ sent. 269 """
What the broker said when it blocked this connection.
None when it is not blocked, and also None when the client read
the reason and did not keep it — which is what aio-pika does, and why
this is separate from blocked rather than a string that is empty
when there is nothing to say. A reason invented here would read exactly
like one RabbitMQ sent.
1977@dataclass(frozen=True, slots=True) 1978class BrokerHealth: 1979 """A connection as a :class:`~acemq_amqp.telemetry.HealthCheck`. 1980 1981 So that a broker check and an application's own — a database, a downstream 1982 service — can be handed to 1983 :func:`~acemq_amqp.telemetry.aggregate_health` together, which is what a 1984 readiness endpoint actually needs. 1985 1986 A blocked broker comes back through here as **up, with the reason**, and 1987 nothing in between folds it into anything else: the aggregate takes the 1988 worst status it is given, so a check that reported back pressure as degraded 1989 would quietly degrade a report that had decided otherwise. The deadline is 1990 the same argument — :attr:`timeout` is shorter than the aggregate's, so that 1991 a broker which has stopped answering is described by this check rather than 1992 by the aggregate giving up on it. 1993 1994 :param connection: what to ask 1995 :param label: what to call it in a combined report 1996 :param timeout: how long to give the broker's answer. See 1997 :meth:`Connection.health` 1998 """ 1999 2000 connection: Connection 2001 label: str = "broker" 2002 timeout: float = DEFAULT_HEALTH_TIMEOUT 2003 2004 @property 2005 def name(self) -> str: 2006 return self.label 2007 2008 async def check(self) -> HealthReport: 2009 return await self.connection.health(self.timeout)
A connection as a ~acemq_amqp.telemetry.HealthCheck.
So that a broker check and an application's own — a database, a downstream
service — can be handed to
~acemq_amqp.telemetry.aggregate_health() together, which is what a
readiness endpoint actually needs.
A blocked broker comes back through here as up, with the reason, and
nothing in between folds it into anything else: the aggregate takes the
worst status it is given, so a check that reported back pressure as degraded
would quietly degrade a report that had decided otherwise. The deadline is
the same argument — timeout is shorter than the aggregate's, so that
a broker which has stopped answering is described by this check rather than
by the aggregate giving up on it.
Parameters
- connection: what to ask
- label: what to call it in a combined report
- timeout: how long to give the broker's answer. See
Connection.health()
145class BytesCodec: 146 """Passes bodies through without touching them. 147 148 For a payload that is already encoded — an image, something another system's 149 serializer produced — and for reading a message this process has no type 150 for. Replaying a dead-lettered message uses it: the bytes that were 151 committed are the bytes that should go back, and re-encoding through a class 152 that has since gained a field would produce something else. 153 154 It answers for every content type, which is why it has to be asked for 155 rather than found: put it last in a :class:`CompositeCodec`, or nothing 156 after it is ever reached. 157 """ 158 159 @property 160 def content_type(self) -> str: 161 return BYTES_CONTENT_TYPE 162 163 def encode(self, payload: Any) -> bytes: 164 if payload is None: 165 return b"" 166 if isinstance(payload, bytes): 167 return payload 168 if isinstance(payload, bytearray | memoryview): 169 return bytes(payload) 170 if isinstance(payload, str): 171 return payload.encode("utf-8") 172 raise TypeError( 173 f"acemq: BytesCodec cannot encode a {type(payload).__name__}; " 174 "it takes bytes or a string" 175 ) 176 177 def decode(self, body: bytes, content_type: str | None = None) -> Any: 178 return bytes(body) 179 180 def can_decode(self, content_type: str | None) -> bool: 181 return True
Passes bodies through without touching them.
For a payload that is already encoded — an image, something another system's serializer produced — and for reading a message this process has no type for. Replaying a dead-lettered message uses it: the bytes that were committed are the bytes that should go back, and re-encoding through a class that has since gained a field would produce something else.
It answers for every content type, which is why it has to be asked for
rather than found: put it last in a CompositeCodec, or nothing
after it is ever reached.
163 def encode(self, payload: Any) -> bytes: 164 if payload is None: 165 return b"" 166 if isinstance(payload, bytes): 167 return payload 168 if isinstance(payload, bytearray | memoryview): 169 return bytes(payload) 170 if isinstance(payload, str): 171 return payload.encode("utf-8") 172 raise TypeError( 173 f"acemq: BytesCodec cannot encode a {type(payload).__name__}; " 174 "it takes bytes or a string" 175 )
53@runtime_checkable 54class Codec(Protocol): 55 """Reads and writes message bodies in one format.""" 56 57 @property 58 def content_type(self) -> str: 59 """What this codec writes onto a message it encodes.""" 60 61 def encode(self, payload: Any) -> bytes: 62 """Turns a payload into bytes. 63 64 :param payload: what the application wants to send 65 :returns: the body 66 :raises TypeError: when the payload is not something this codec writes 67 """ 68 69 def decode(self, body: bytes, content_type: str | None = None) -> Any: 70 """Turns bytes back into a payload. 71 72 :param body: the message body 73 :param content_type: what the sender said it was, or ``None`` when the 74 sender said nothing 75 :returns: the payload 76 :raises FatalError: when the body cannot be read, which is fatal because 77 the same bytes will not read any better on a second attempt 78 """ 79 80 def can_decode(self, content_type: str | None) -> bool: 81 """Whether this codec should be offered a message. 82 83 :param content_type: the sender's content type, or ``None`` when there 84 was none 85 :returns: whether this codec claims it 86 """
Reads and writes message bodies in one format.
1771def _no_init_or_replace_init(self, *args, **kwargs): 1772 cls = type(self) 1773 1774 if cls._is_protocol: 1775 raise TypeError('Protocols cannot be instantiated') 1776 1777 # Already using a custom `__init__`. No need to calculate correct 1778 # `__init__` to call. This can lead to RecursionError. See bpo-45121. 1779 if cls.__init__ is not _no_init_or_replace_init: 1780 return 1781 1782 # Initially, `__init__` of a protocol subclass is set to `_no_init_or_replace_init`. 1783 # The first instantiation of the subclass will call `_no_init_or_replace_init` which 1784 # searches for a proper new `__init__` in the MRO. The new `__init__` 1785 # replaces the subclass' old `__init__` (ie `_no_init_or_replace_init`). Subsequent 1786 # instantiation of the protocol subclass will thus use the new 1787 # `__init__` and no longer call `_no_init_or_replace_init`. 1788 for base in cls.__mro__: 1789 init = base.__dict__.get('__init__', _no_init_or_replace_init) 1790 if init is not _no_init_or_replace_init: 1791 cls.__init__ = init 1792 break 1793 else: 1794 # should not happen 1795 cls.__init__ = object.__init__ 1796 1797 cls.__init__(self, *args, **kwargs)
57 @property 58 def content_type(self) -> str: 59 """What this codec writes onto a message it encodes."""
What this codec writes onto a message it encodes.
61 def encode(self, payload: Any) -> bytes: 62 """Turns a payload into bytes. 63 64 :param payload: what the application wants to send 65 :returns: the body 66 :raises TypeError: when the payload is not something this codec writes 67 """
Turns a payload into bytes.
Parameters
- payload: what the application wants to send :returns: the body
Raises
- TypeError: when the payload is not something this codec writes
69 def decode(self, body: bytes, content_type: str | None = None) -> Any: 70 """Turns bytes back into a payload. 71 72 :param body: the message body 73 :param content_type: what the sender said it was, or ``None`` when the 74 sender said nothing 75 :returns: the payload 76 :raises FatalError: when the body cannot be read, which is fatal because 77 the same bytes will not read any better on a second attempt 78 """
Turns bytes back into a payload.
Parameters
- body: the message body
- content_type: what the sender said it was, or
Nonewhen the sender said nothing :returns: the payload
Raises
- FatalError: when the body cannot be read, which is fatal because the same bytes will not read any better on a second attempt
80 def can_decode(self, content_type: str | None) -> bool: 81 """Whether this codec should be offered a message. 82 83 :param content_type: the sender's content type, or ``None`` when there 84 was none 85 :returns: whether this codec claims it 86 """
Whether this codec should be offered a message.
Parameters
- content_type: the sender's content type, or
Nonewhen there was none :returns: whether this codec claims it
224class CompositeCodec: 225 """Picks a codec by the message's content type. 226 227 For a queue carrying more than one format — during a migration, or where 228 several producers were written at different times:: 229 230 codec = CompositeCodec(JsonCodec(), TextCodec()) 231 232 The first codec is what it writes. All of them are offered a message to 233 read, in the order they were given, and the first that claims the content 234 type *and* succeeds wins. Order matters when two overlap: :class:`BytesCodec` 235 answers for everything, so anything after it would never be reached. 236 """ 237 238 def __init__(self, *codecs: Codec) -> None: 239 if not codecs: 240 raise ValueError("acemq: a CompositeCodec needs at least one codec in it") 241 self._codecs: tuple[Codec, ...] = codecs 242 243 @property 244 def codecs(self) -> tuple[Codec, ...]: 245 """The codecs, in the order they are tried.""" 246 return self._codecs 247 248 @property 249 def content_type(self) -> str: 250 """The first codec's, since that is the one that encodes.""" 251 return self._codecs[0].content_type 252 253 def encode(self, payload: Any) -> bytes: 254 return self._codecs[0].encode(payload) 255 256 def decode(self, body: bytes, content_type: str | None = None) -> Any: 257 """Reads a body with whichever codec recognises the content type. 258 259 With no content type nothing can be ruled out, so every codec is a 260 candidate rather than the first being assumed right. Candidates are 261 tried in turn and the first that reads the body wins — so a message that 262 claims one format and is written in another is still read — and only 263 when every one has refused is anything raised, carrying all the reasons 264 rather than just the last. 265 """ 266 if content_type is None: 267 candidates = list(self._codecs) 268 else: 269 candidates = [codec for codec in self._codecs if codec.can_decode(content_type)] 270 271 if not candidates: 272 raise FatalError( 273 f"acemq: no codec here reads {content_type!r}; " 274 f"this one holds {self._describe()}" 275 ) 276 277 refusals: list[str] = [] 278 for codec in candidates: 279 try: 280 return codec.decode(body, content_type) 281 except Exception as failure: 282 # Collected rather than raised: a codec refusing says nothing 283 # about the next one, and the message that matters at the end is 284 # every reason together rather than whichever came last. 285 refusals.append(f"{type(codec).__name__}: {failure}") 286 287 raise FatalError( 288 "acemq: no codec could read this message. Tried " + "; ".join(refusals) 289 ) 290 291 def can_decode(self, content_type: str | None) -> bool: 292 return any(codec.can_decode(content_type) for codec in self._codecs) 293 294 def _describe(self) -> str: 295 return ", ".join(codec.content_type for codec in self._codecs) 296 297 def __repr__(self) -> str: 298 return f"CompositeCodec({self._describe()})"
Picks a codec by the message's content type.
For a queue carrying more than one format — during a migration, or where several producers were written at different times::
codec = CompositeCodec(JsonCodec(), TextCodec())
The first codec is what it writes. All of them are offered a message to
read, in the order they were given, and the first that claims the content
type and succeeds wins. Order matters when two overlap: BytesCodec
answers for everything, so anything after it would never be reached.
243 @property 244 def codecs(self) -> tuple[Codec, ...]: 245 """The codecs, in the order they are tried.""" 246 return self._codecs
The codecs, in the order they are tried.
248 @property 249 def content_type(self) -> str: 250 """The first codec's, since that is the one that encodes.""" 251 return self._codecs[0].content_type
The first codec's, since that is the one that encodes.
256 def decode(self, body: bytes, content_type: str | None = None) -> Any: 257 """Reads a body with whichever codec recognises the content type. 258 259 With no content type nothing can be ruled out, so every codec is a 260 candidate rather than the first being assumed right. Candidates are 261 tried in turn and the first that reads the body wins — so a message that 262 claims one format and is written in another is still read — and only 263 when every one has refused is anything raised, carrying all the reasons 264 rather than just the last. 265 """ 266 if content_type is None: 267 candidates = list(self._codecs) 268 else: 269 candidates = [codec for codec in self._codecs if codec.can_decode(content_type)] 270 271 if not candidates: 272 raise FatalError( 273 f"acemq: no codec here reads {content_type!r}; " 274 f"this one holds {self._describe()}" 275 ) 276 277 refusals: list[str] = [] 278 for codec in candidates: 279 try: 280 return codec.decode(body, content_type) 281 except Exception as failure: 282 # Collected rather than raised: a codec refusing says nothing 283 # about the next one, and the message that matters at the end is 284 # every reason together rather than whichever came last. 285 refusals.append(f"{type(codec).__name__}: {failure}") 286 287 raise FatalError( 288 "acemq: no codec could read this message. Tried " + "; ".join(refusals) 289 )
Reads a body with whichever codec recognises the content type.
With no content type nothing can be ruled out, so every codec is a candidate rather than the first being assumed right. Candidates are tried in turn and the first that reads the body wins — so a message that claims one format and is written in another is still read — and only when every one has refused is anything raised, carrying all the reasons rather than just the last.
1221class Connection: 1222 """A connection to a broker, and the defaults everything on it inherits. 1223 1224 Built by :func:`connect` in the ordinary case. Taking a transport directly 1225 is for a fake in a test, or for a connection made with settings a URL cannot 1226 carry. 1227 1228 :param transport: how to reach the broker 1229 :param codec: what publishers and consumers use unless they say otherwise 1230 :param origin: what to stamp on published messages, conventionally 1231 ``service@host`` 1232 :param retry: what consumers use unless they say otherwise. The default is 1233 one delivery and no second chance, so a handler that calls 1234 ``retry()`` on a connection with no policy dead-letters the message and 1235 says so in the reason. That is deliberate: the alternative default is an 1236 immediate requeue, which is a hot loop against a broker that nobody 1237 asked for 1238 :param prefetch: how many unacknowledged messages a consumer holds 1239 :param max_outstanding_publishes: how many publishes may be waiting for the 1240 broker at once. The ceiling that turns a memory leak into backpressure: 1241 without one, a caller publishing faster than the broker answers 1242 accumulates unconfirmed messages until the process dies, which looks 1243 like throughput right up to the moment it does not. A thousand by 1244 default, which is what Java's ``maxOutstandingPublishes`` and .NET's 1245 ``MaxOutstandingPublishes`` default to 1246 :param confirm_timeout: how long a publish waits for one of those permits, 1247 and then for the broker's confirm, before giving up. Ten seconds by 1248 default, the same as Java's ``confirmTimeout``. Reaching it raises 1249 rather than waiting on, because a publisher stalled for ever behind a 1250 broker that has stopped answering is the failure this bound exists to make visible 1251 :param on_publish: cross-cutting behaviour to wrap every publish on this 1252 connection in, outermost first. See :mod:`acemq_amqp.interceptors` 1253 :param on_consume: the same around every handler. 1254 :meth:`intercept_publish` and :meth:`intercept_consume` add more later 1255 :param observer: where the numbers go. Nothing by default, so a service 1256 that wants none pays for none. See :mod:`acemq_amqp.telemetry` 1257 """ 1258 1259 def __init__( 1260 self, 1261 transport: Transport, 1262 *, 1263 codec: Codec | None = None, 1264 origin: str | None = None, 1265 retry: RetryPolicy | None = None, 1266 prefetch: int = DEFAULT_PREFETCH, 1267 max_outstanding_publishes: int = DEFAULT_MAX_OUTSTANDING_PUBLISHES, 1268 confirm_timeout: timedelta = DEFAULT_CONFIRM_TIMEOUT, 1269 on_publish: Sequence[PublishInterceptor] = (), 1270 on_consume: Sequence[ConsumeInterceptor] = (), 1271 observer: Observer | None = None, 1272 ) -> None: 1273 if prefetch < 0: 1274 raise ValueError(f"acemq: prefetch must not be negative, got {prefetch}") 1275 if max_outstanding_publishes < 1: 1276 raise ValueError( 1277 "acemq: max_outstanding_publishes must be at least 1, got " 1278 f"{max_outstanding_publishes}" 1279 ) 1280 if confirm_timeout <= timedelta(0): 1281 raise ValueError( 1282 f"acemq: confirm_timeout must be positive, got {confirm_timeout}" 1283 ) 1284 self._transport = transport 1285 self._observer: Observer = observer or NullObserver() 1286 self._codec = codec or JsonCodec() 1287 self._origin = origin or default_origin() 1288 self._retry = retry or no_retry() 1289 self._prefetch = prefetch 1290 self._max_outstanding = max_outstanding_publishes 1291 self._confirm_timeout = confirm_timeout 1292 # Built here rather than on first use. An asyncio.Semaphore has bound no 1293 # loop at construction since 3.10, so it is safe to make one outside a 1294 # running loop, and making it here means every publish on this connection 1295 # shares the one counter however many tasks are publishing. 1296 self._outstanding = asyncio.Semaphore(max_outstanding_publishes) 1297 self._in_flight_publishes = 0 1298 self._consumers: list[Consumer] = [] 1299 self._publishing: list[PublishInterceptor] = list(on_publish) 1300 self._consuming: list[ConsumeInterceptor] = list(on_consume) 1301 self._closed = False 1302 1303 @property 1304 def transport(self) -> Transport: 1305 """The transport underneath, for anything this library does not wrap.""" 1306 return self._transport 1307 1308 @property 1309 def codec(self) -> Codec: 1310 """The codec publishers and consumers use unless told otherwise.""" 1311 return self._codec 1312 1313 @property 1314 def origin(self) -> str: 1315 """What published messages are stamped with.""" 1316 return self._origin 1317 1318 @property 1319 def retry(self) -> RetryPolicy: 1320 """The retry policy consumers use unless told otherwise.""" 1321 return self._retry 1322 1323 @property 1324 def observer(self) -> Observer: 1325 """Where this connection's numbers go.""" 1326 return self._observer 1327 1328 @property 1329 def max_outstanding_publishes(self) -> int: 1330 """How many publishes may be waiting for the broker at once.""" 1331 return self._max_outstanding 1332 1333 @property 1334 def confirm_timeout(self) -> timedelta: 1335 """How long a publish waits for room before giving up.""" 1336 return self._confirm_timeout 1337 1338 @property 1339 def outstanding_publishes(self) -> int: 1340 """How many publishes are waiting for the broker right now. 1341 1342 What a stalled publisher looks like from outside: a number pinned at 1343 :attr:`max_outstanding_publishes` is a broker that has stopped answering, 1344 and it says so before the first :class:`~acemq_amqp.PublishError` does. 1345 """ 1346 # Counted here rather than read off the semaphore, which keeps its count 1347 # private. Raised after the permit is taken and lowered before it is 1348 # given back, both inside publish_raw, so it never reads high. 1349 return self._in_flight_publishes 1350 1351 @property 1352 def consumers(self) -> tuple[Consumer, ...]: 1353 """The consumers running on this connection. 1354 1355 A snapshot, and public because a health check that has to say whether 1356 the consumers are alive should not need private access to find them — 1357 the same rule an interceptor follows. 1358 """ 1359 return tuple(self._consumers) 1360 1361 @property 1362 def closed(self) -> bool: 1363 """Whether this connection has been closed.""" 1364 return self._closed 1365 1366 @property 1367 def blocked(self) -> bool | None: 1368 """Whether the broker has blocked this connection, or ``None`` when the 1369 transport cannot be asked. 1370 1371 RabbitMQ blocks a connection when it is low on memory or disk, and while 1372 it lasts it stops reading that connection's socket: everything sent — 1373 a publish, a declaration, :meth:`health`'s own probe — waits rather than 1374 failing. It is the broker protecting itself and it is not an error, so 1375 it is worth telling apart from the broker having gone away, which looks 1376 identical from outside and wants the opposite response. 1377 1378 ``None`` means the question could not be asked at all, which is a 1379 different fact from ``False`` and is reported as one: a transport that 1380 does not implement 1381 :class:`~acemq_amqp.transport.BlockedState`, or one whose client has 1382 moved the state out of reach. See :attr:`blocked_reason` for why there 1383 may be a boolean and no reason to go with it. 1384 """ 1385 transport = self._transport 1386 return transport.blocked if isinstance(transport, BlockedState) else None 1387 1388 @property 1389 def blocked_reason(self) -> str | None: 1390 """What the broker said when it blocked this connection, if anything. 1391 1392 ``None`` when the connection is not blocked, and also ``None`` when the 1393 transport is blocked but cannot say why — which is what the RabbitMQ 1394 transport answers, because aio-pika reads RabbitMQ's reason and does not 1395 keep it. Reporting the block without the reason is the honest half; 1396 inventing a reason would read exactly like one the broker sent. 1397 """ 1398 transport = self._transport 1399 return transport.blocked_reason if isinstance(transport, BlockedState) else None 1400 1401 async def health(self, timeout: float = DEFAULT_HEALTH_TIMEOUT) -> HealthReport: 1402 """Whether this connection can reach its broker, and its consumers run. 1403 1404 The broker half is a declaration of a temporary queue, which is the 1405 cheapest thing AMQP offers that actually proves the connection works. A 1406 TCP connection that is open but wedged — the broker paused, the network 1407 black-holing — answers a socket-level check exactly as a healthy one 1408 does, right up until something is asked of it. 1409 1410 The consumer half is the one a probe usually wants and nothing outside 1411 can see. A consumer whose workers have died without it being closed is 1412 one the broker is still sending messages to and nothing is reading, and 1413 from outside that is indistinguishable from a quiet queue. 1414 1415 **A blocked connection is up, with the reason.** The broker blocks a 1416 connection when it is low on disk or memory, and an application that 1417 fails its own readiness check for it is one an orchestrator restarts 1418 into the same blocked broker, having thrown away whatever it was 1419 holding. ``parts["blocked"]`` carries the state either way, and is 1420 ``None`` when the transport could not be asked — see :attr:`blocked`. 1421 1422 **The probe is bounded.** A blocked broker stops reading the socket, so 1423 the round trip this makes is precisely the thing that does not come back 1424 when something is wrong; without a deadline of its own a health check 1425 hangs, and a readiness endpoint that hangs takes the instance out of 1426 rotation with no report at all. A probe that runs out of time is 1427 abandoned rather than waited for, because a request that cannot be 1428 answered cannot reliably be cancelled either. 1429 1430 It costs a round trip, so it is not something to call per request. Wire 1431 it to a readiness probe and let the probe's interval decide how often. 1432 1433 :param timeout: how long to give the broker's answer, three seconds by 1434 default. Shorter than the deadline 1435 :func:`~acemq_amqp.telemetry.aggregate_health` puts around a whole 1436 set of checks, so that this report rather than that one is what an 1437 operator reads when a broker stops answering 1438 :returns: the report. Degraded means working but worth an alert 1439 """ 1440 checked = datetime.now(timezone.utc) 1441 consumers = self.consumers 1442 stalled = [consumer.queue for consumer in consumers if not consumer.running] 1443 blocked = self.blocked 1444 parts: dict[str, Any] = { 1445 "consumers": len(consumers), 1446 "in-flight": sum(consumer.in_flight for consumer in consumers), 1447 "blocked": blocked, 1448 } 1449 reason = self.blocked_reason 1450 if reason: 1451 parts["blocked-reason"] = reason 1452 1453 if self._closed: 1454 return HealthReport( 1455 HealthStatus.DOWN, "the connection has been closed", checked, parts 1456 ) 1457 1458 if blocked: 1459 # Asked before the probe rather than after it, because the probe is 1460 # what a blocked broker will not answer: a round trip started here 1461 # would spend the whole deadline arriving at the answer this already 1462 # has. 1463 return self._blocked_report(checked, parts, stalled, reason) 1464 1465 # Asking whether a queue nothing has ever declared exists, rather than 1466 # declaring a temporary one. It is the same round trip and it proves the 1467 # same thing, and it creates nothing at all: an exclusive queue is only 1468 # released when the channel that declared it closes, so a probe that 1469 # declared one would leave a queue on the broker for the life of the 1470 # connection and a new one behind every restart. 1471 probe = f"acemq-health-{uuid.uuid4().hex}" 1472 started = time.monotonic() 1473 try: 1474 await self._probe_within(probe, timeout) 1475 except asyncio.TimeoutError: 1476 # A round trip that does not come back is the ordinary way a block 1477 # shows itself, and the notification may have arrived while this was 1478 # waiting. Asked again rather than assumed, because "the broker is 1479 # protecting itself" and "the broker has gone" are the same silence. 1480 blocked = self.blocked 1481 parts["blocked"] = blocked 1482 reason = self.blocked_reason 1483 if reason: 1484 parts["blocked-reason"] = reason 1485 if blocked: 1486 return self._blocked_report(checked, parts, stalled, reason) 1487 return HealthReport( 1488 HealthStatus.DOWN, 1489 f"the broker did not answer within {timeout:g}s", 1490 checked, 1491 parts, 1492 ) 1493 except Exception as failure: 1494 parts["error"] = _describe(failure) 1495 return HealthReport( 1496 HealthStatus.DOWN, f"the broker did not answer: {failure}", checked, parts 1497 ) 1498 parts["round-trip"] = round(time.monotonic() - started, 4) 1499 1500 if stalled: 1501 # Degraded rather than down: the connection works, and a replacement 1502 # instance would almost certainly stall the same way. Worth waking 1503 # somebody; not worth taking this one out of rotation. 1504 return HealthReport( 1505 HealthStatus.DEGRADED, 1506 _stalled_detail(stalled), 1507 checked, 1508 {**parts, "stalled": sorted(stalled)}, 1509 ) 1510 return HealthReport(HealthStatus.UP, "", checked, parts) 1511 1512 def _blocked_report( 1513 self, 1514 checked: datetime, 1515 parts: dict[str, Any], 1516 stalled: list[str], 1517 reason: str | None, 1518 ) -> HealthReport: 1519 """What a blocked connection reports: up, with the reason. 1520 1521 Up for the block itself, and still degraded when a consumer of this 1522 process has stopped reading — that is this instance's own failure rather 1523 than the broker's, it outlives the alarm, and a block is not a reason to 1524 stop saying so. 1525 """ 1526 detail = self._blocked_detail(reason) 1527 if stalled: 1528 return HealthReport( 1529 HealthStatus.DEGRADED, 1530 f"{_stalled_detail(stalled)}; and {detail}", 1531 checked, 1532 {**parts, "stalled": sorted(stalled)}, 1533 ) 1534 return HealthReport(HealthStatus.UP, detail, checked, parts) 1535 1536 @staticmethod 1537 def _blocked_detail(reason: str | None) -> str: 1538 """The blocked sentence, with the broker's reason when there is one. 1539 1540 The reason follows a colon, which is how Go and Ruby write it too — 1541 ``blockedPrefix + ": " + reason`` and ``"#{BLOCKED}: #{reason}"``. One 1542 shape across the family means one alert rule reads a blocked broker 1543 whatever language the service happens to be written in. 1544 """ 1545 return BLOCKED_DETAIL if not reason else f"{BLOCKED_DETAIL}: {reason}" 1546 1547 async def _probe_within(self, name: str, timeout: float) -> None: 1548 """One round trip to the broker, given ``timeout`` to come back. 1549 1550 The probe runs as a task that is abandoned rather than awaited when the 1551 deadline passes, which is the difference between a bounded check and one 1552 that merely looks bounded: cancelling a request to a broker that has 1553 stopped reading its socket means waiting for a cancellation that travels 1554 the same way the request did. 1555 1556 :raises asyncio.TimeoutError: when the broker did not answer in time 1557 """ 1558 probe = asyncio.ensure_future(self._probe(name)) 1559 try: 1560 done, _ = await asyncio.wait({probe}, timeout=timeout) 1561 except asyncio.CancelledError: 1562 # The caller gave up on the whole check. Waiting on a task is not 1563 # cancelling it, so it is cancelled here rather than left running 1564 # against a broker nobody is listening to any more. 1565 probe.cancel() 1566 probe.add_done_callback(_retrieved) 1567 raise 1568 if not done: 1569 probe.cancel() 1570 # Whatever it ends as is nobody's news now, and an abandoned task 1571 # whose exception is never retrieved is a warning at collection time 1572 # about a failure that has already been reported as a timeout. 1573 probe.add_done_callback(_retrieved) 1574 raise asyncio.TimeoutError 1575 # Raises what the probe raised, which is what the caller reports. 1576 probe.result() 1577 1578 async def _probe(self, name: str) -> None: 1579 """One round trip to the broker that leaves nothing behind. 1580 1581 A transport that cannot be asked whether a queue exists gets the next 1582 cheapest thing: a temporary queue, removed again straight away. Both 1583 prove the same thing, which is that the connection answers rather than 1584 merely being open. 1585 1586 The temporary queue is classic, and has to be. It is exclusive and 1587 auto-deleting so that it goes when the channel does, and RabbitMQ 1588 refuses a quorum queue that is either — a health check that could not be 1589 declared would report a healthy broker as down. 1590 """ 1591 if isinstance(self._transport, QueueAdmin): 1592 await self._transport.queue_exists(name) 1593 return 1594 await self._transport.declare_queue( 1595 name, QueueSpec(durable=False, auto_delete=True, exclusive=True) 1596 ) 1597 1598 @property 1599 def publish_interceptors(self) -> tuple[PublishInterceptor, ...]: 1600 """What wraps every publish, outermost first. 1601 1602 A snapshot rather than the list itself, so a publish in progress cannot 1603 be handed a chain that is being edited underneath it. 1604 """ 1605 return tuple(self._publishing) 1606 1607 @property 1608 def consume_interceptors(self) -> tuple[ConsumeInterceptor, ...]: 1609 """What wraps every handler, outermost first.""" 1610 return tuple(self._consuming) 1611 1612 def intercept_publish(self, interceptor: PublishInterceptor) -> Connection: 1613 """Wraps every publish on this connection in ``interceptor``. 1614 1615 Registered ones run in the order they were added, so this one is 1616 innermost until the next is added. Read at publish time rather than 1617 copied into each publisher, so an interceptor registered during start-up 1618 applies to publishers that already exist — which is what makes it usable 1619 from a framework's wiring rather than only from the line that connects. 1620 1621 Adding at start-up is still what to do. A message already in flight will 1622 not see it, and an interceptor whose whole job is to stamp every message 1623 is not doing it if the first few went without. 1624 1625 :param interceptor: what to wrap the publish in. See 1626 :mod:`acemq_amqp.interceptors` 1627 :returns: this connection, so registrations read as one expression 1628 """ 1629 self._publishing.append(interceptor) 1630 return self 1631 1632 def intercept_consume(self, interceptor: ConsumeInterceptor) -> Connection: 1633 """Wraps every handler on this connection in ``interceptor``. 1634 1635 :param interceptor: what to wrap the handler in 1636 :returns: this connection 1637 """ 1638 self._consuming.append(interceptor) 1639 return self 1640 1641 async def declare(self, topology: Topology) -> None: 1642 """Applies a topology to this connection's broker. 1643 1644 :param topology: what the broker should have 1645 """ 1646 await topology.apply(self._transport) 1647 1648 def publisher( 1649 self, 1650 exchange: str = "", 1651 routing_key: str = "", 1652 *, 1653 codec: Codec | None = None, 1654 persistent: bool = True, 1655 mandatory: bool = False, 1656 ) -> Publisher: 1657 """Builds a publisher for an exchange and routing key. 1658 1659 Publish straight to a queue by leaving the exchange empty: the default 1660 exchange routes to the queue whose name matches the routing key. 1661 1662 :param exchange: where to publish, empty for the default exchange 1663 :param routing_key: what to publish under, or a queue name 1664 :param codec: a codec other than this connection's 1665 :param persistent: ask the broker to write these messages to disk 1666 :param mandatory: fail when a message reaches no queue at all, rather 1667 than letting the broker drop it silently 1668 :returns: the publisher 1669 """ 1670 return Publisher( 1671 self, 1672 exchange, 1673 routing_key, 1674 codec=codec, 1675 persistent=persistent, 1676 mandatory=mandatory, 1677 ) 1678 1679 async def consume( 1680 self, 1681 queue: str, 1682 handler: Handler, 1683 *, 1684 codec: Codec | None = None, 1685 retry: RetryPolicy | None = None, 1686 prefetch: int | None = None, 1687 concurrency: int = 1, 1688 tag: str = "", 1689 args: Mapping[str, Any] | None = None, 1690 declare: bool = True, 1691 ) -> Consumer: 1692 """Reads messages from a queue until the returned consumer is closed. 1693 1694 Before it subscribes it declares the queues it will need when a message 1695 fails: ``{queue}.dlq``, ``{queue}.parked``, ``acemq.dlx`` and, when the 1696 retry policy has waits the broker holds, ``acemq.retry`` and the rungs. 1697 See :func:`acemq_amqp.topology.declare_where_failures_go` for exactly 1698 what and why. The source queue is not among them, and neither is 1699 anything a producer needs, so this is not a replacement for declaring a 1700 topology — it is the floor underneath a service that was deployed 1701 without one. 1702 1703 :param queue: what to read 1704 :param handler: what to do with a message; returning an 1705 :class:`~acemq_amqp.Ack` is the decision 1706 :param codec: a codec other than this connection's 1707 :param retry: a policy other than this connection's 1708 :param prefetch: how many unacknowledged messages to hold 1709 :param concurrency: how many messages to work on at once. One by 1710 default, which keeps a queue's messages in order; raising it trades 1711 that order for throughput, which is the right trade for handlers 1712 that spend their time waiting on something else 1713 :param tag: what to call this consumer to the broker 1714 :param args: broker-specific consumer arguments 1715 :param declare: declare the queues above before subscribing. On by 1716 default, because the alternative default loses messages silently. 1717 Turn it off for a login with no ``configure`` permission on the 1718 vhost, which is refused by the broker rather than ignored, and for a 1719 tool draining a queue it does not own — a one-off reader of 1720 ``orders.dlq`` has no business creating ``orders.dlq.dlq``. A 1721 consumer started this way still publishes to those queues, so 1722 somebody else has to have declared them 1723 :returns: the running consumer 1724 """ 1725 if self._closed: 1726 raise AceMQError("acemq: this connection is closed") 1727 1728 consumer = Consumer( 1729 self, 1730 queue, 1731 handler, 1732 codec=codec or self._codec, 1733 retry=retry or self._retry, 1734 concurrency=concurrency, 1735 declare=declare, 1736 ) 1737 self._consumers.append(consumer) 1738 try: 1739 await consumer._start( 1740 self._prefetch if prefetch is None else prefetch, 1741 tag, 1742 args or {}, 1743 ) 1744 except BaseException: 1745 self._untrack(consumer) 1746 raise 1747 return consumer 1748 1749 async def publish_raw( 1750 self, exchange: str, routing_key: str, message: Outbound 1751 ) -> PublishResult: 1752 """Sends bytes that are already encoded, with headers already rendered. 1753 1754 For the retry and dead-letter hops, and for anything else replaying a 1755 message recorded earlier: the payload's class may not exist any more, 1756 and re-encoding it would produce different bytes from the ones that were 1757 committed. Ordinary publishing goes through :class:`Publisher`. 1758 1759 **This is the one place the outstanding-publish bound is taken.** Every 1760 publish in the library comes through here — a :class:`Publisher`, a 1761 retry rung, a dead letter, a replay — so one permit counts them all, and 1762 :meth:`Publisher.send_all` cannot put more messages on the wire at once 1763 than :attr:`max_outstanding_publishes` allows however long the list it 1764 was given is. 1765 1766 :param exchange: where to publish, empty for the default exchange 1767 :param routing_key: what to publish under 1768 :param message: the message, encoded 1769 :returns: what the broker said 1770 :raises PublishingPausedError: when the broker has blocked this 1771 connection. Nothing was sent; retry once it unblocks 1772 :raises PublishError: when every permit is taken and none came free 1773 within :attr:`confirm_timeout`, or when the broker did not confirm 1774 this publish within it. The second may or may not have arrived 1775 """ 1776 # Refused rather than queued, before a permit or a byte is spent. A 1777 # blocked broker stops reading the socket, so a publish sent now sits in 1778 # the client's write buffer with no confirm coming until the alarm 1779 # clears, then times out as a failure that may or may not have arrived. 1780 # Go and .NET refuse at the same point for the same reason. ``None`` 1781 # means the state could not be read and is not treated as blocked. 1782 if self.blocked is True: 1783 reason = self.blocked_reason 1784 raise PublishingPausedError( 1785 message.message_id, 1786 exchange, 1787 routing_key, 1788 ( 1789 f"{self._blocked_detail(reason)}. Nothing was sent; back off and " 1790 "retry once the broker unblocks" 1791 ), 1792 ) 1793 1794 # Taken before the message is written and not after, which is the whole 1795 # point: a bound applied afterwards has already let the message into 1796 # memory. The permit is given back in the ``finally`` below on every 1797 # path there is — the publish returning, the publish raising, and this 1798 # task being cancelled while the broker is still thinking — because a 1799 # permit lost on any one of those is a connection that publishes a 1800 # thousand more messages and then stops for ever. 1801 try: 1802 await asyncio.wait_for( 1803 self._outstanding.acquire(), self._confirm_timeout.total_seconds() 1804 ) 1805 except asyncio.TimeoutError as stalled: 1806 # Said rather than waited out. A publisher parked behind a broker 1807 # that has stopped answering is indistinguishable from a quiet 1808 # service, and this sentence is the one Java raises word for word so 1809 # that an operator reading a log has one thing to recognise. 1810 raise PublishError( 1811 message.message_id, 1812 exchange, 1813 routing_key, 1814 ( 1815 f"{self._max_outstanding} publishes are already waiting for a " 1816 f"confirm and none completed within {self._confirm_timeout}. " 1817 "The broker is not keeping up; publish more slowly rather than " 1818 "buffering more." 1819 ), 1820 ) from stalled 1821 1822 self._in_flight_publishes += 1 1823 try: 1824 # Bounded as well, by the same timeout. RabbitMQ says a connection is 1825 # blocked only once it publishes, so the first publish after an alarm 1826 # is written and then held with no confirm until the alarm clears -- 1827 # minutes, or never. Said here as a failure, not a refusal: it was 1828 # written, so it may have arrived. The publish after it sees the 1829 # block and is refused above. .NET bounds the same wait the same way. 1830 result = await asyncio.wait_for( 1831 self._transport.publish(exchange, routing_key, message), 1832 self._confirm_timeout.total_seconds(), 1833 ) 1834 except asyncio.TimeoutError as unconfirmed: 1835 raise PublishError( 1836 message.message_id, 1837 exchange, 1838 routing_key, 1839 ( 1840 f"the broker did not confirm this publish within " 1841 f"{self._confirm_timeout}. It may or may not have been stored, " 1842 "so treat it as neither sent nor lost" 1843 ), 1844 ) from unconfirmed 1845 finally: 1846 self._in_flight_publishes -= 1 1847 self._outstanding.release() 1848 1849 # A nack is the broker declining to keep a message it was asked to take, and 1850 # it is raised here rather than returned. 1851 # 1852 # It used to be returned, and `confirmed` was read nowhere in this library 1853 # except a telemetry label -- so a caller who did not inspect the result 1854 # carried on believing the message had gone somewhere, and Java, .NET, Go and 1855 # Ruby all raised on the same event. Worse than the wrong belief: the outbox 1856 # relay removes a record because the publish did not raise, so a nack deleted 1857 # a record for a message the broker had refused. At-most-once, in the pattern 1858 # whose whole purpose is not losing anything. 1859 # 1860 # Checked here because this is the one place every publish in the library 1861 # comes through -- a Publisher, a retry rung, a dead letter, a replay, the 1862 # outbox -- so one check covers all of them, and anything that settles or 1863 # deletes on the strength of a publish returning now hears the refusal. 1864 # 1865 # Distinct from unroutable, which the Publisher path reports separately and 1866 # which sets `unroutable`: the broker could route this one and would not keep 1867 # it, and the two want different responses. 1868 # 1869 # So a returned message is passed back rather than raised here. The 1870 # RabbitMQ transport reports a Basic.Return as not confirmed as well as 1871 # not routed, and raising it as a nack hid ``routed`` from every caller 1872 # that reads it: a mandatory Publisher said "refused" rather than 1873 # "reached no queue", and the retry and dead-letter hops never reached 1874 # the fallbacks written for a queue that is not there. 1875 if not result.confirmed and result.routed: 1876 raise PublishError( 1877 message.message_id, 1878 exchange, 1879 routing_key, 1880 ( 1881 "the broker refused to confirm it. The message reached the broker " 1882 "and the broker has not taken responsibility for it, so it must be " 1883 "treated as not sent" 1884 ), 1885 ) 1886 1887 return result 1888 1889 async def pull(self, queue: str) -> Delivery | None: 1890 """Takes one message off a queue, or ``None`` when there is none waiting. 1891 1892 For tools rather than for services: it is a round trip per message where 1893 :meth:`consume` gets a stream. What it can do that a consumer cannot is 1894 say that a queue is empty, which is what anything working through a 1895 backlog and then stopping — a replay, a drain — has to know. 1896 1897 The delivery comes back unsettled, so the caller decides whether it is 1898 gone or goes back. 1899 1900 :param queue: what to read from 1901 :returns: the delivery, or ``None`` 1902 """ 1903 return await self._source().pull(queue) 1904 1905 async def queue_exists(self, queue: str) -> bool: 1906 """Whether a queue is on the broker, creating nothing.""" 1907 return await self._admin().queue_exists(queue) 1908 1909 async def message_count(self, queue: str) -> int: 1910 """How many messages are waiting on a queue.""" 1911 return await self._admin().message_count(queue) 1912 1913 async def delete_queue(self, queue: str) -> None: 1914 """Removes a queue and every message still on it.""" 1915 await self._admin().delete_queue(queue) 1916 1917 def _admin(self) -> QueueAdmin: 1918 if not isinstance(self._transport, QueueAdmin): 1919 raise AceMQError( 1920 f"acemq: the {type(self._transport).__name__} transport cannot manage queues" 1921 ) 1922 return self._transport 1923 1924 def _source(self) -> MessageSource: 1925 if not isinstance(self._transport, MessageSource): 1926 raise AceMQError( 1927 f"acemq: the {type(self._transport).__name__} transport cannot be " 1928 "asked for one message at a time" 1929 ) 1930 return self._transport 1931 1932 def _untrack(self, consumer: Consumer) -> None: 1933 if consumer in self._consumers: 1934 self._consumers.remove(consumer) 1935 1936 async def close(self, *, timeout: float | None = DEFAULT_DRAIN_TIMEOUT) -> bool: 1937 """Stops every consumer on this connection and releases it. 1938 1939 Consumers first, and their handlers given up to ``timeout`` to finish, 1940 so a message being worked on when this is called is settled rather than 1941 returned to the queue for somebody else to redo. The bound is one 1942 deadline for every handler of every consumer, not one each: they are 1943 all drained at once. A handler still running at the deadline is 1944 cancelled and its message left unsettled, for the broker to redeliver — 1945 see :meth:`Consumer.close`. 1946 1947 The connection is released whatever happens, including when the caller 1948 cancels closing; that cancellation is raised again afterwards. 1949 1950 :param timeout: seconds to wait for the running handlers; ``None`` 1951 waits for as long as they take 1952 :returns: ``True`` if every handler finished, ``False`` if the time ran 1953 out first 1954 """ 1955 if self._closed: 1956 return True 1957 self._closed = True 1958 try: 1959 drained = await asyncio.gather( 1960 *(consumer.close(timeout=timeout) for consumer in list(self._consumers)), 1961 return_exceptions=True, 1962 ) 1963 finally: 1964 await self._transport.close() 1965 for outcome in drained: 1966 if isinstance(outcome, BaseException): 1967 raise outcome 1968 return all(drained) 1969 1970 async def __aenter__(self) -> Connection: 1971 return self 1972 1973 async def __aexit__(self, *_: object) -> None: 1974 await self.close()
A connection to a broker, and the defaults everything on it inherits.
Built by connect() in the ordinary case. Taking a transport directly
is for a fake in a test, or for a connection made with settings a URL cannot
carry.
Parameters
- transport: how to reach the broker
- codec: what publishers and consumers use unless they say otherwise
- origin: what to stamp on published messages, conventionally
service@host - retry: what consumers use unless they say otherwise. The default is
one delivery and no second chance, so a handler that calls
retry()on a connection with no policy dead-letters the message and says so in the reason. That is deliberate: the alternative default is an immediate requeue, which is a hot loop against a broker that nobody asked for - prefetch: how many unacknowledged messages a consumer holds
- max_outstanding_publishes: how many publishes may be waiting for the
broker at once. The ceiling that turns a memory leak into backpressure:
without one, a caller publishing faster than the broker answers
accumulates unconfirmed messages until the process dies, which looks
like throughput right up to the moment it does not. A thousand by
default, which is what Java's
maxOutstandingPublishesand .NET'sMaxOutstandingPublishesdefault to - confirm_timeout: how long a publish waits for one of those permits,
and then for the broker's confirm, before giving up. Ten seconds by
default, the same as Java's
confirmTimeout. Reaching it raises rather than waiting on, because a publisher stalled for ever behind a broker that has stopped answering is the failure this bound exists to make visible - on_publish: cross-cutting behaviour to wrap every publish on this
connection in, outermost first. See
acemq_amqp.interceptors - on_consume: the same around every handler.
intercept_publish()andintercept_consume()add more later - observer: where the numbers go. Nothing by default, so a service
that wants none pays for none. See
acemq_amqp.telemetry
1259 def __init__( 1260 self, 1261 transport: Transport, 1262 *, 1263 codec: Codec | None = None, 1264 origin: str | None = None, 1265 retry: RetryPolicy | None = None, 1266 prefetch: int = DEFAULT_PREFETCH, 1267 max_outstanding_publishes: int = DEFAULT_MAX_OUTSTANDING_PUBLISHES, 1268 confirm_timeout: timedelta = DEFAULT_CONFIRM_TIMEOUT, 1269 on_publish: Sequence[PublishInterceptor] = (), 1270 on_consume: Sequence[ConsumeInterceptor] = (), 1271 observer: Observer | None = None, 1272 ) -> None: 1273 if prefetch < 0: 1274 raise ValueError(f"acemq: prefetch must not be negative, got {prefetch}") 1275 if max_outstanding_publishes < 1: 1276 raise ValueError( 1277 "acemq: max_outstanding_publishes must be at least 1, got " 1278 f"{max_outstanding_publishes}" 1279 ) 1280 if confirm_timeout <= timedelta(0): 1281 raise ValueError( 1282 f"acemq: confirm_timeout must be positive, got {confirm_timeout}" 1283 ) 1284 self._transport = transport 1285 self._observer: Observer = observer or NullObserver() 1286 self._codec = codec or JsonCodec() 1287 self._origin = origin or default_origin() 1288 self._retry = retry or no_retry() 1289 self._prefetch = prefetch 1290 self._max_outstanding = max_outstanding_publishes 1291 self._confirm_timeout = confirm_timeout 1292 # Built here rather than on first use. An asyncio.Semaphore has bound no 1293 # loop at construction since 3.10, so it is safe to make one outside a 1294 # running loop, and making it here means every publish on this connection 1295 # shares the one counter however many tasks are publishing. 1296 self._outstanding = asyncio.Semaphore(max_outstanding_publishes) 1297 self._in_flight_publishes = 0 1298 self._consumers: list[Consumer] = [] 1299 self._publishing: list[PublishInterceptor] = list(on_publish) 1300 self._consuming: list[ConsumeInterceptor] = list(on_consume) 1301 self._closed = False
1303 @property 1304 def transport(self) -> Transport: 1305 """The transport underneath, for anything this library does not wrap.""" 1306 return self._transport
The transport underneath, for anything this library does not wrap.
1308 @property 1309 def codec(self) -> Codec: 1310 """The codec publishers and consumers use unless told otherwise.""" 1311 return self._codec
The codec publishers and consumers use unless told otherwise.
1313 @property 1314 def origin(self) -> str: 1315 """What published messages are stamped with.""" 1316 return self._origin
What published messages are stamped with.
1318 @property 1319 def retry(self) -> RetryPolicy: 1320 """The retry policy consumers use unless told otherwise.""" 1321 return self._retry
The retry policy consumers use unless told otherwise.
1323 @property 1324 def observer(self) -> Observer: 1325 """Where this connection's numbers go.""" 1326 return self._observer
Where this connection's numbers go.
1328 @property 1329 def max_outstanding_publishes(self) -> int: 1330 """How many publishes may be waiting for the broker at once.""" 1331 return self._max_outstanding
How many publishes may be waiting for the broker at once.
1333 @property 1334 def confirm_timeout(self) -> timedelta: 1335 """How long a publish waits for room before giving up.""" 1336 return self._confirm_timeout
How long a publish waits for room before giving up.
1338 @property 1339 def outstanding_publishes(self) -> int: 1340 """How many publishes are waiting for the broker right now. 1341 1342 What a stalled publisher looks like from outside: a number pinned at 1343 :attr:`max_outstanding_publishes` is a broker that has stopped answering, 1344 and it says so before the first :class:`~acemq_amqp.PublishError` does. 1345 """ 1346 # Counted here rather than read off the semaphore, which keeps its count 1347 # private. Raised after the permit is taken and lowered before it is 1348 # given back, both inside publish_raw, so it never reads high. 1349 return self._in_flight_publishes
How many publishes are waiting for the broker right now.
What a stalled publisher looks like from outside: a number pinned at
max_outstanding_publishes is a broker that has stopped answering,
and it says so before the first ~acemq_amqp.PublishError does.
1351 @property 1352 def consumers(self) -> tuple[Consumer, ...]: 1353 """The consumers running on this connection. 1354 1355 A snapshot, and public because a health check that has to say whether 1356 the consumers are alive should not need private access to find them — 1357 the same rule an interceptor follows. 1358 """ 1359 return tuple(self._consumers)
The consumers running on this connection.
A snapshot, and public because a health check that has to say whether the consumers are alive should not need private access to find them — the same rule an interceptor follows.
1361 @property 1362 def closed(self) -> bool: 1363 """Whether this connection has been closed.""" 1364 return self._closed
Whether this connection has been closed.
1366 @property 1367 def blocked(self) -> bool | None: 1368 """Whether the broker has blocked this connection, or ``None`` when the 1369 transport cannot be asked. 1370 1371 RabbitMQ blocks a connection when it is low on memory or disk, and while 1372 it lasts it stops reading that connection's socket: everything sent — 1373 a publish, a declaration, :meth:`health`'s own probe — waits rather than 1374 failing. It is the broker protecting itself and it is not an error, so 1375 it is worth telling apart from the broker having gone away, which looks 1376 identical from outside and wants the opposite response. 1377 1378 ``None`` means the question could not be asked at all, which is a 1379 different fact from ``False`` and is reported as one: a transport that 1380 does not implement 1381 :class:`~acemq_amqp.transport.BlockedState`, or one whose client has 1382 moved the state out of reach. See :attr:`blocked_reason` for why there 1383 may be a boolean and no reason to go with it. 1384 """ 1385 transport = self._transport 1386 return transport.blocked if isinstance(transport, BlockedState) else None
Whether the broker has blocked this connection, or None when the
transport cannot be asked.
RabbitMQ blocks a connection when it is low on memory or disk, and while
it lasts it stops reading that connection's socket: everything sent —
a publish, a declaration, health()'s own probe — waits rather than
failing. It is the broker protecting itself and it is not an error, so
it is worth telling apart from the broker having gone away, which looks
identical from outside and wants the opposite response.
None means the question could not be asked at all, which is a
different fact from False and is reported as one: a transport that
does not implement
~acemq_amqp.transport.BlockedState, or one whose client has
moved the state out of reach. See blocked_reason for why there
may be a boolean and no reason to go with it.
1388 @property 1389 def blocked_reason(self) -> str | None: 1390 """What the broker said when it blocked this connection, if anything. 1391 1392 ``None`` when the connection is not blocked, and also ``None`` when the 1393 transport is blocked but cannot say why — which is what the RabbitMQ 1394 transport answers, because aio-pika reads RabbitMQ's reason and does not 1395 keep it. Reporting the block without the reason is the honest half; 1396 inventing a reason would read exactly like one the broker sent. 1397 """ 1398 transport = self._transport 1399 return transport.blocked_reason if isinstance(transport, BlockedState) else None
What the broker said when it blocked this connection, if anything.
None when the connection is not blocked, and also None when the
transport is blocked but cannot say why — which is what the RabbitMQ
transport answers, because aio-pika reads RabbitMQ's reason and does not
keep it. Reporting the block without the reason is the honest half;
inventing a reason would read exactly like one the broker sent.
1401 async def health(self, timeout: float = DEFAULT_HEALTH_TIMEOUT) -> HealthReport: 1402 """Whether this connection can reach its broker, and its consumers run. 1403 1404 The broker half is a declaration of a temporary queue, which is the 1405 cheapest thing AMQP offers that actually proves the connection works. A 1406 TCP connection that is open but wedged — the broker paused, the network 1407 black-holing — answers a socket-level check exactly as a healthy one 1408 does, right up until something is asked of it. 1409 1410 The consumer half is the one a probe usually wants and nothing outside 1411 can see. A consumer whose workers have died without it being closed is 1412 one the broker is still sending messages to and nothing is reading, and 1413 from outside that is indistinguishable from a quiet queue. 1414 1415 **A blocked connection is up, with the reason.** The broker blocks a 1416 connection when it is low on disk or memory, and an application that 1417 fails its own readiness check for it is one an orchestrator restarts 1418 into the same blocked broker, having thrown away whatever it was 1419 holding. ``parts["blocked"]`` carries the state either way, and is 1420 ``None`` when the transport could not be asked — see :attr:`blocked`. 1421 1422 **The probe is bounded.** A blocked broker stops reading the socket, so 1423 the round trip this makes is precisely the thing that does not come back 1424 when something is wrong; without a deadline of its own a health check 1425 hangs, and a readiness endpoint that hangs takes the instance out of 1426 rotation with no report at all. A probe that runs out of time is 1427 abandoned rather than waited for, because a request that cannot be 1428 answered cannot reliably be cancelled either. 1429 1430 It costs a round trip, so it is not something to call per request. Wire 1431 it to a readiness probe and let the probe's interval decide how often. 1432 1433 :param timeout: how long to give the broker's answer, three seconds by 1434 default. Shorter than the deadline 1435 :func:`~acemq_amqp.telemetry.aggregate_health` puts around a whole 1436 set of checks, so that this report rather than that one is what an 1437 operator reads when a broker stops answering 1438 :returns: the report. Degraded means working but worth an alert 1439 """ 1440 checked = datetime.now(timezone.utc) 1441 consumers = self.consumers 1442 stalled = [consumer.queue for consumer in consumers if not consumer.running] 1443 blocked = self.blocked 1444 parts: dict[str, Any] = { 1445 "consumers": len(consumers), 1446 "in-flight": sum(consumer.in_flight for consumer in consumers), 1447 "blocked": blocked, 1448 } 1449 reason = self.blocked_reason 1450 if reason: 1451 parts["blocked-reason"] = reason 1452 1453 if self._closed: 1454 return HealthReport( 1455 HealthStatus.DOWN, "the connection has been closed", checked, parts 1456 ) 1457 1458 if blocked: 1459 # Asked before the probe rather than after it, because the probe is 1460 # what a blocked broker will not answer: a round trip started here 1461 # would spend the whole deadline arriving at the answer this already 1462 # has. 1463 return self._blocked_report(checked, parts, stalled, reason) 1464 1465 # Asking whether a queue nothing has ever declared exists, rather than 1466 # declaring a temporary one. It is the same round trip and it proves the 1467 # same thing, and it creates nothing at all: an exclusive queue is only 1468 # released when the channel that declared it closes, so a probe that 1469 # declared one would leave a queue on the broker for the life of the 1470 # connection and a new one behind every restart. 1471 probe = f"acemq-health-{uuid.uuid4().hex}" 1472 started = time.monotonic() 1473 try: 1474 await self._probe_within(probe, timeout) 1475 except asyncio.TimeoutError: 1476 # A round trip that does not come back is the ordinary way a block 1477 # shows itself, and the notification may have arrived while this was 1478 # waiting. Asked again rather than assumed, because "the broker is 1479 # protecting itself" and "the broker has gone" are the same silence. 1480 blocked = self.blocked 1481 parts["blocked"] = blocked 1482 reason = self.blocked_reason 1483 if reason: 1484 parts["blocked-reason"] = reason 1485 if blocked: 1486 return self._blocked_report(checked, parts, stalled, reason) 1487 return HealthReport( 1488 HealthStatus.DOWN, 1489 f"the broker did not answer within {timeout:g}s", 1490 checked, 1491 parts, 1492 ) 1493 except Exception as failure: 1494 parts["error"] = _describe(failure) 1495 return HealthReport( 1496 HealthStatus.DOWN, f"the broker did not answer: {failure}", checked, parts 1497 ) 1498 parts["round-trip"] = round(time.monotonic() - started, 4) 1499 1500 if stalled: 1501 # Degraded rather than down: the connection works, and a replacement 1502 # instance would almost certainly stall the same way. Worth waking 1503 # somebody; not worth taking this one out of rotation. 1504 return HealthReport( 1505 HealthStatus.DEGRADED, 1506 _stalled_detail(stalled), 1507 checked, 1508 {**parts, "stalled": sorted(stalled)}, 1509 ) 1510 return HealthReport(HealthStatus.UP, "", checked, parts)
Whether this connection can reach its broker, and its consumers run.
The broker half is a declaration of a temporary queue, which is the cheapest thing AMQP offers that actually proves the connection works. A TCP connection that is open but wedged — the broker paused, the network black-holing — answers a socket-level check exactly as a healthy one does, right up until something is asked of it.
The consumer half is the one a probe usually wants and nothing outside can see. A consumer whose workers have died without it being closed is one the broker is still sending messages to and nothing is reading, and from outside that is indistinguishable from a quiet queue.
A blocked connection is up, with the reason. The broker blocks a
connection when it is low on disk or memory, and an application that
fails its own readiness check for it is one an orchestrator restarts
into the same blocked broker, having thrown away whatever it was
holding. parts["blocked"] carries the state either way, and is
None when the transport could not be asked — see blocked.
The probe is bounded. A blocked broker stops reading the socket, so the round trip this makes is precisely the thing that does not come back when something is wrong; without a deadline of its own a health check hangs, and a readiness endpoint that hangs takes the instance out of rotation with no report at all. A probe that runs out of time is abandoned rather than waited for, because a request that cannot be answered cannot reliably be cancelled either.
It costs a round trip, so it is not something to call per request. Wire it to a readiness probe and let the probe's interval decide how often.
Parameters
- timeout: how long to give the broker's answer, three seconds by
default. Shorter than the deadline
~acemq_amqp.telemetry.aggregate_health()puts around a whole set of checks, so that this report rather than that one is what an operator reads when a broker stops answering :returns: the report. Degraded means working but worth an alert
1598 @property 1599 def publish_interceptors(self) -> tuple[PublishInterceptor, ...]: 1600 """What wraps every publish, outermost first. 1601 1602 A snapshot rather than the list itself, so a publish in progress cannot 1603 be handed a chain that is being edited underneath it. 1604 """ 1605 return tuple(self._publishing)
What wraps every publish, outermost first.
A snapshot rather than the list itself, so a publish in progress cannot be handed a chain that is being edited underneath it.
1607 @property 1608 def consume_interceptors(self) -> tuple[ConsumeInterceptor, ...]: 1609 """What wraps every handler, outermost first.""" 1610 return tuple(self._consuming)
What wraps every handler, outermost first.
1612 def intercept_publish(self, interceptor: PublishInterceptor) -> Connection: 1613 """Wraps every publish on this connection in ``interceptor``. 1614 1615 Registered ones run in the order they were added, so this one is 1616 innermost until the next is added. Read at publish time rather than 1617 copied into each publisher, so an interceptor registered during start-up 1618 applies to publishers that already exist — which is what makes it usable 1619 from a framework's wiring rather than only from the line that connects. 1620 1621 Adding at start-up is still what to do. A message already in flight will 1622 not see it, and an interceptor whose whole job is to stamp every message 1623 is not doing it if the first few went without. 1624 1625 :param interceptor: what to wrap the publish in. See 1626 :mod:`acemq_amqp.interceptors` 1627 :returns: this connection, so registrations read as one expression 1628 """ 1629 self._publishing.append(interceptor) 1630 return self
Wraps every publish on this connection in interceptor.
Registered ones run in the order they were added, so this one is innermost until the next is added. Read at publish time rather than copied into each publisher, so an interceptor registered during start-up applies to publishers that already exist — which is what makes it usable from a framework's wiring rather than only from the line that connects.
Adding at start-up is still what to do. A message already in flight will not see it, and an interceptor whose whole job is to stamp every message is not doing it if the first few went without.
Parameters
- interceptor: what to wrap the publish in. See
acemq_amqp.interceptors:returns: this connection, so registrations read as one expression
1632 def intercept_consume(self, interceptor: ConsumeInterceptor) -> Connection: 1633 """Wraps every handler on this connection in ``interceptor``. 1634 1635 :param interceptor: what to wrap the handler in 1636 :returns: this connection 1637 """ 1638 self._consuming.append(interceptor) 1639 return self
Wraps every handler on this connection in interceptor.
Parameters
- interceptor: what to wrap the handler in :returns: this connection
1641 async def declare(self, topology: Topology) -> None: 1642 """Applies a topology to this connection's broker. 1643 1644 :param topology: what the broker should have 1645 """ 1646 await topology.apply(self._transport)
Applies a topology to this connection's broker.
Parameters
- topology: what the broker should have
1648 def publisher( 1649 self, 1650 exchange: str = "", 1651 routing_key: str = "", 1652 *, 1653 codec: Codec | None = None, 1654 persistent: bool = True, 1655 mandatory: bool = False, 1656 ) -> Publisher: 1657 """Builds a publisher for an exchange and routing key. 1658 1659 Publish straight to a queue by leaving the exchange empty: the default 1660 exchange routes to the queue whose name matches the routing key. 1661 1662 :param exchange: where to publish, empty for the default exchange 1663 :param routing_key: what to publish under, or a queue name 1664 :param codec: a codec other than this connection's 1665 :param persistent: ask the broker to write these messages to disk 1666 :param mandatory: fail when a message reaches no queue at all, rather 1667 than letting the broker drop it silently 1668 :returns: the publisher 1669 """ 1670 return Publisher( 1671 self, 1672 exchange, 1673 routing_key, 1674 codec=codec, 1675 persistent=persistent, 1676 mandatory=mandatory, 1677 )
Builds a publisher for an exchange and routing key.
Publish straight to a queue by leaving the exchange empty: the default exchange routes to the queue whose name matches the routing key.
Parameters
- exchange: where to publish, empty for the default exchange
- routing_key: what to publish under, or a queue name
- codec: a codec other than this connection's
- persistent: ask the broker to write these messages to disk
- mandatory: fail when a message reaches no queue at all, rather than letting the broker drop it silently :returns: the publisher
1679 async def consume( 1680 self, 1681 queue: str, 1682 handler: Handler, 1683 *, 1684 codec: Codec | None = None, 1685 retry: RetryPolicy | None = None, 1686 prefetch: int | None = None, 1687 concurrency: int = 1, 1688 tag: str = "", 1689 args: Mapping[str, Any] | None = None, 1690 declare: bool = True, 1691 ) -> Consumer: 1692 """Reads messages from a queue until the returned consumer is closed. 1693 1694 Before it subscribes it declares the queues it will need when a message 1695 fails: ``{queue}.dlq``, ``{queue}.parked``, ``acemq.dlx`` and, when the 1696 retry policy has waits the broker holds, ``acemq.retry`` and the rungs. 1697 See :func:`acemq_amqp.topology.declare_where_failures_go` for exactly 1698 what and why. The source queue is not among them, and neither is 1699 anything a producer needs, so this is not a replacement for declaring a 1700 topology — it is the floor underneath a service that was deployed 1701 without one. 1702 1703 :param queue: what to read 1704 :param handler: what to do with a message; returning an 1705 :class:`~acemq_amqp.Ack` is the decision 1706 :param codec: a codec other than this connection's 1707 :param retry: a policy other than this connection's 1708 :param prefetch: how many unacknowledged messages to hold 1709 :param concurrency: how many messages to work on at once. One by 1710 default, which keeps a queue's messages in order; raising it trades 1711 that order for throughput, which is the right trade for handlers 1712 that spend their time waiting on something else 1713 :param tag: what to call this consumer to the broker 1714 :param args: broker-specific consumer arguments 1715 :param declare: declare the queues above before subscribing. On by 1716 default, because the alternative default loses messages silently. 1717 Turn it off for a login with no ``configure`` permission on the 1718 vhost, which is refused by the broker rather than ignored, and for a 1719 tool draining a queue it does not own — a one-off reader of 1720 ``orders.dlq`` has no business creating ``orders.dlq.dlq``. A 1721 consumer started this way still publishes to those queues, so 1722 somebody else has to have declared them 1723 :returns: the running consumer 1724 """ 1725 if self._closed: 1726 raise AceMQError("acemq: this connection is closed") 1727 1728 consumer = Consumer( 1729 self, 1730 queue, 1731 handler, 1732 codec=codec or self._codec, 1733 retry=retry or self._retry, 1734 concurrency=concurrency, 1735 declare=declare, 1736 ) 1737 self._consumers.append(consumer) 1738 try: 1739 await consumer._start( 1740 self._prefetch if prefetch is None else prefetch, 1741 tag, 1742 args or {}, 1743 ) 1744 except BaseException: 1745 self._untrack(consumer) 1746 raise 1747 return consumer
Reads messages from a queue until the returned consumer is closed.
Before it subscribes it declares the queues it will need when a message
fails: {queue}.dlq, {queue}.parked, acemq.dlx and, when the
retry policy has waits the broker holds, acemq.retry and the rungs.
See acemq_amqp.topology.declare_where_failures_go() for exactly
what and why. The source queue is not among them, and neither is
anything a producer needs, so this is not a replacement for declaring a
topology — it is the floor underneath a service that was deployed
without one.
Parameters
- queue: what to read
- handler: what to do with a message; returning an
~acemq_amqp.Ackis the decision - codec: a codec other than this connection's
- retry: a policy other than this connection's
- prefetch: how many unacknowledged messages to hold
- concurrency: how many messages to work on at once. One by default, which keeps a queue's messages in order; raising it trades that order for throughput, which is the right trade for handlers that spend their time waiting on something else
- tag: what to call this consumer to the broker
- args: broker-specific consumer arguments
- declare: declare the queues above before subscribing. On by
default, because the alternative default loses messages silently.
Turn it off for a login with no
configurepermission on the vhost, which is refused by the broker rather than ignored, and for a tool draining a queue it does not own — a one-off reader oforders.dlqhas no business creatingorders.dlq.dlq. A consumer started this way still publishes to those queues, so somebody else has to have declared them :returns: the running consumer
1749 async def publish_raw( 1750 self, exchange: str, routing_key: str, message: Outbound 1751 ) -> PublishResult: 1752 """Sends bytes that are already encoded, with headers already rendered. 1753 1754 For the retry and dead-letter hops, and for anything else replaying a 1755 message recorded earlier: the payload's class may not exist any more, 1756 and re-encoding it would produce different bytes from the ones that were 1757 committed. Ordinary publishing goes through :class:`Publisher`. 1758 1759 **This is the one place the outstanding-publish bound is taken.** Every 1760 publish in the library comes through here — a :class:`Publisher`, a 1761 retry rung, a dead letter, a replay — so one permit counts them all, and 1762 :meth:`Publisher.send_all` cannot put more messages on the wire at once 1763 than :attr:`max_outstanding_publishes` allows however long the list it 1764 was given is. 1765 1766 :param exchange: where to publish, empty for the default exchange 1767 :param routing_key: what to publish under 1768 :param message: the message, encoded 1769 :returns: what the broker said 1770 :raises PublishingPausedError: when the broker has blocked this 1771 connection. Nothing was sent; retry once it unblocks 1772 :raises PublishError: when every permit is taken and none came free 1773 within :attr:`confirm_timeout`, or when the broker did not confirm 1774 this publish within it. The second may or may not have arrived 1775 """ 1776 # Refused rather than queued, before a permit or a byte is spent. A 1777 # blocked broker stops reading the socket, so a publish sent now sits in 1778 # the client's write buffer with no confirm coming until the alarm 1779 # clears, then times out as a failure that may or may not have arrived. 1780 # Go and .NET refuse at the same point for the same reason. ``None`` 1781 # means the state could not be read and is not treated as blocked. 1782 if self.blocked is True: 1783 reason = self.blocked_reason 1784 raise PublishingPausedError( 1785 message.message_id, 1786 exchange, 1787 routing_key, 1788 ( 1789 f"{self._blocked_detail(reason)}. Nothing was sent; back off and " 1790 "retry once the broker unblocks" 1791 ), 1792 ) 1793 1794 # Taken before the message is written and not after, which is the whole 1795 # point: a bound applied afterwards has already let the message into 1796 # memory. The permit is given back in the ``finally`` below on every 1797 # path there is — the publish returning, the publish raising, and this 1798 # task being cancelled while the broker is still thinking — because a 1799 # permit lost on any one of those is a connection that publishes a 1800 # thousand more messages and then stops for ever. 1801 try: 1802 await asyncio.wait_for( 1803 self._outstanding.acquire(), self._confirm_timeout.total_seconds() 1804 ) 1805 except asyncio.TimeoutError as stalled: 1806 # Said rather than waited out. A publisher parked behind a broker 1807 # that has stopped answering is indistinguishable from a quiet 1808 # service, and this sentence is the one Java raises word for word so 1809 # that an operator reading a log has one thing to recognise. 1810 raise PublishError( 1811 message.message_id, 1812 exchange, 1813 routing_key, 1814 ( 1815 f"{self._max_outstanding} publishes are already waiting for a " 1816 f"confirm and none completed within {self._confirm_timeout}. " 1817 "The broker is not keeping up; publish more slowly rather than " 1818 "buffering more." 1819 ), 1820 ) from stalled 1821 1822 self._in_flight_publishes += 1 1823 try: 1824 # Bounded as well, by the same timeout. RabbitMQ says a connection is 1825 # blocked only once it publishes, so the first publish after an alarm 1826 # is written and then held with no confirm until the alarm clears -- 1827 # minutes, or never. Said here as a failure, not a refusal: it was 1828 # written, so it may have arrived. The publish after it sees the 1829 # block and is refused above. .NET bounds the same wait the same way. 1830 result = await asyncio.wait_for( 1831 self._transport.publish(exchange, routing_key, message), 1832 self._confirm_timeout.total_seconds(), 1833 ) 1834 except asyncio.TimeoutError as unconfirmed: 1835 raise PublishError( 1836 message.message_id, 1837 exchange, 1838 routing_key, 1839 ( 1840 f"the broker did not confirm this publish within " 1841 f"{self._confirm_timeout}. It may or may not have been stored, " 1842 "so treat it as neither sent nor lost" 1843 ), 1844 ) from unconfirmed 1845 finally: 1846 self._in_flight_publishes -= 1 1847 self._outstanding.release() 1848 1849 # A nack is the broker declining to keep a message it was asked to take, and 1850 # it is raised here rather than returned. 1851 # 1852 # It used to be returned, and `confirmed` was read nowhere in this library 1853 # except a telemetry label -- so a caller who did not inspect the result 1854 # carried on believing the message had gone somewhere, and Java, .NET, Go and 1855 # Ruby all raised on the same event. Worse than the wrong belief: the outbox 1856 # relay removes a record because the publish did not raise, so a nack deleted 1857 # a record for a message the broker had refused. At-most-once, in the pattern 1858 # whose whole purpose is not losing anything. 1859 # 1860 # Checked here because this is the one place every publish in the library 1861 # comes through -- a Publisher, a retry rung, a dead letter, a replay, the 1862 # outbox -- so one check covers all of them, and anything that settles or 1863 # deletes on the strength of a publish returning now hears the refusal. 1864 # 1865 # Distinct from unroutable, which the Publisher path reports separately and 1866 # which sets `unroutable`: the broker could route this one and would not keep 1867 # it, and the two want different responses. 1868 # 1869 # So a returned message is passed back rather than raised here. The 1870 # RabbitMQ transport reports a Basic.Return as not confirmed as well as 1871 # not routed, and raising it as a nack hid ``routed`` from every caller 1872 # that reads it: a mandatory Publisher said "refused" rather than 1873 # "reached no queue", and the retry and dead-letter hops never reached 1874 # the fallbacks written for a queue that is not there. 1875 if not result.confirmed and result.routed: 1876 raise PublishError( 1877 message.message_id, 1878 exchange, 1879 routing_key, 1880 ( 1881 "the broker refused to confirm it. The message reached the broker " 1882 "and the broker has not taken responsibility for it, so it must be " 1883 "treated as not sent" 1884 ), 1885 ) 1886 1887 return result
Sends bytes that are already encoded, with headers already rendered.
For the retry and dead-letter hops, and for anything else replaying a
message recorded earlier: the payload's class may not exist any more,
and re-encoding it would produce different bytes from the ones that were
committed. Ordinary publishing goes through Publisher.
This is the one place the outstanding-publish bound is taken. Every
publish in the library comes through here — a Publisher, a
retry rung, a dead letter, a replay — so one permit counts them all, and
Publisher.send_all() cannot put more messages on the wire at once
than max_outstanding_publishes allows however long the list it
was given is.
Parameters
- exchange: where to publish, empty for the default exchange
- routing_key: what to publish under
- message: the message, encoded :returns: what the broker said
Raises
- PublishingPausedError: when the broker has blocked this connection. Nothing was sent; retry once it unblocks
- PublishError: when every permit is taken and none came free
within
confirm_timeout, or when the broker did not confirm this publish within it. The second may or may not have arrived
1889 async def pull(self, queue: str) -> Delivery | None: 1890 """Takes one message off a queue, or ``None`` when there is none waiting. 1891 1892 For tools rather than for services: it is a round trip per message where 1893 :meth:`consume` gets a stream. What it can do that a consumer cannot is 1894 say that a queue is empty, which is what anything working through a 1895 backlog and then stopping — a replay, a drain — has to know. 1896 1897 The delivery comes back unsettled, so the caller decides whether it is 1898 gone or goes back. 1899 1900 :param queue: what to read from 1901 :returns: the delivery, or ``None`` 1902 """ 1903 return await self._source().pull(queue)
Takes one message off a queue, or None when there is none waiting.
For tools rather than for services: it is a round trip per message where
consume() gets a stream. What it can do that a consumer cannot is
say that a queue is empty, which is what anything working through a
backlog and then stopping — a replay, a drain — has to know.
The delivery comes back unsettled, so the caller decides whether it is gone or goes back.
Parameters
- queue: what to read from
:returns: the delivery, or
None
1905 async def queue_exists(self, queue: str) -> bool: 1906 """Whether a queue is on the broker, creating nothing.""" 1907 return await self._admin().queue_exists(queue)
Whether a queue is on the broker, creating nothing.
1909 async def message_count(self, queue: str) -> int: 1910 """How many messages are waiting on a queue.""" 1911 return await self._admin().message_count(queue)
How many messages are waiting on a queue.
1913 async def delete_queue(self, queue: str) -> None: 1914 """Removes a queue and every message still on it.""" 1915 await self._admin().delete_queue(queue)
Removes a queue and every message still on it.
1936 async def close(self, *, timeout: float | None = DEFAULT_DRAIN_TIMEOUT) -> bool: 1937 """Stops every consumer on this connection and releases it. 1938 1939 Consumers first, and their handlers given up to ``timeout`` to finish, 1940 so a message being worked on when this is called is settled rather than 1941 returned to the queue for somebody else to redo. The bound is one 1942 deadline for every handler of every consumer, not one each: they are 1943 all drained at once. A handler still running at the deadline is 1944 cancelled and its message left unsettled, for the broker to redeliver — 1945 see :meth:`Consumer.close`. 1946 1947 The connection is released whatever happens, including when the caller 1948 cancels closing; that cancellation is raised again afterwards. 1949 1950 :param timeout: seconds to wait for the running handlers; ``None`` 1951 waits for as long as they take 1952 :returns: ``True`` if every handler finished, ``False`` if the time ran 1953 out first 1954 """ 1955 if self._closed: 1956 return True 1957 self._closed = True 1958 try: 1959 drained = await asyncio.gather( 1960 *(consumer.close(timeout=timeout) for consumer in list(self._consumers)), 1961 return_exceptions=True, 1962 ) 1963 finally: 1964 await self._transport.close() 1965 for outcome in drained: 1966 if isinstance(outcome, BaseException): 1967 raise outcome 1968 return all(drained)
Stops every consumer on this connection and releases it.
Consumers first, and their handlers given up to timeout to finish,
so a message being worked on when this is called is settled rather than
returned to the queue for somebody else to redo. The bound is one
deadline for every handler of every consumer, not one each: they are
all drained at once. A handler still running at the deadline is
cancelled and its message left unsettled, for the broker to redeliver —
see Consumer.close().
The connection is released whatever happens, including when the caller cancels closing; that cancellation is raised again afterwards.
Parameters
- timeout: seconds to wait for the running handlers;
Nonewaits for as long as they take :returns:Trueif every handler finished,Falseif the time ran out first
126@dataclass(slots=True) 127class ConsumeContext: 128 """A message that has arrived and been decoded, before the handler sees it. 129 130 :param queue: what it was read from 131 :param envelope: the metadata, as it will reach the handler. Changing it 132 changes what the handler is given 133 :param payload: the decoded body, likewise 134 :param body: the undecoded bytes, for an interceptor that wants to see what 135 actually arrived 136 :param content_type: what the sender said the body was, or ``None`` 137 :param routing_key: the key it arrived under 138 :param redelivered: the broker saying it has handed this one over before 139 :param state: somewhere for an interceptor to leave something for a later 140 one, or for its own way out. Empty, and this library never reads it 141 :param reports_settlement: whether whatever is driving this delivery will 142 say how it was settled. True when a consumer is, and false when an 143 interceptor chain is being run by something else — a test, or a caller 144 composing the chain by hand 145 :param reply_to: AMQP's own ``reply-to`` property as it arrived, empty when 146 the sender set none 147 """ 148 149 queue: str 150 envelope: Envelope 151 payload: Any 152 body: bytes 153 content_type: str | None 154 routing_key: str 155 redelivered: bool 156 state: dict[str, Any] = field(default_factory=dict) 157 reports_settlement: bool = False 158 reply_to: str = "" 159 _settlement_listeners: list[SettlementListener] = field( 160 default_factory=list, repr=False 161 ) 162 163 def when_settled(self, listener: SettlementListener) -> bool: 164 """Asks to be told how this delivery was settled. 165 166 What the handler answered is not what happened to the message: a 167 handler asking for a retry with no attempts left is dead-lettered, and 168 an interceptor that read only the :class:`~acemq_amqp.ack.Ack` would 169 report a retry that never happened. That gap is exactly what somebody 170 querying a trace backend for dead letters falls into, so the consumer 171 reports its decision here and the tracing adapter waits for it. 172 173 The listener is called once, on the consumer's task, after the decision 174 and before it is carried out. It should not block and must not raise; 175 one that does is logged and otherwise ignored, because a delivery still 176 has to be settled when whoever was watching it breaks. 177 178 :param listener: called with the :class:`~acemq_amqp.ack.Settlement` 179 :returns: whether anything will ever call it. ``False`` when this 180 delivery is not being driven by a consumer, in which case the 181 caller should not wait for an answer that is not coming 182 """ 183 if not self.reports_settlement: 184 return False 185 self._settlement_listeners.append(listener) 186 return True 187 188 def settled(self, settlement: Settlement) -> None: 189 """Tells the listeners what happened. Called by the consumer. 190 191 :param settlement: what the consumer decided to do with the delivery 192 """ 193 for listener in self._settlement_listeners: 194 try: 195 listener(settlement) 196 except Exception: 197 # The delivery still has to be settled. A listener is an 198 # observer, and an observer that breaks must not take the 199 # message with it. 200 log.exception( 201 "acemq: a settlement listener failed for a delivery from %s", self.queue 202 )
A message that has arrived and been decoded, before the handler sees it.
Parameters
- queue: what it was read from
- envelope: the metadata, as it will reach the handler. Changing it changes what the handler is given
- payload: the decoded body, likewise
- body: the undecoded bytes, for an interceptor that wants to see what actually arrived
- content_type: what the sender said the body was, or
None - routing_key: the key it arrived under
- redelivered: the broker saying it has handed this one over before
- state: somewhere for an interceptor to leave something for a later one, or for its own way out. Empty, and this library never reads it
- reports_settlement: whether whatever is driving this delivery will say how it was settled. True when a consumer is, and false when an interceptor chain is being run by something else — a test, or a caller composing the chain by hand
- reply_to: AMQP's own
reply-toproperty as it arrived, empty when the sender set none
163 def when_settled(self, listener: SettlementListener) -> bool: 164 """Asks to be told how this delivery was settled. 165 166 What the handler answered is not what happened to the message: a 167 handler asking for a retry with no attempts left is dead-lettered, and 168 an interceptor that read only the :class:`~acemq_amqp.ack.Ack` would 169 report a retry that never happened. That gap is exactly what somebody 170 querying a trace backend for dead letters falls into, so the consumer 171 reports its decision here and the tracing adapter waits for it. 172 173 The listener is called once, on the consumer's task, after the decision 174 and before it is carried out. It should not block and must not raise; 175 one that does is logged and otherwise ignored, because a delivery still 176 has to be settled when whoever was watching it breaks. 177 178 :param listener: called with the :class:`~acemq_amqp.ack.Settlement` 179 :returns: whether anything will ever call it. ``False`` when this 180 delivery is not being driven by a consumer, in which case the 181 caller should not wait for an answer that is not coming 182 """ 183 if not self.reports_settlement: 184 return False 185 self._settlement_listeners.append(listener) 186 return True
Asks to be told how this delivery was settled.
What the handler answered is not what happened to the message: a
handler asking for a retry with no attempts left is dead-lettered, and
an interceptor that read only the ~acemq_amqp.ack.Ack would
report a retry that never happened. That gap is exactly what somebody
querying a trace backend for dead letters falls into, so the consumer
reports its decision here and the tracing adapter waits for it.
The listener is called once, on the consumer's task, after the decision and before it is carried out. It should not block and must not raise; one that does is logged and otherwise ignored, because a delivery still has to be settled when whoever was watching it breaks.
Parameters
- listener: called with the
~acemq_amqp.ack.Settlement:returns: whether anything will ever call it.Falsewhen this delivery is not being driven by a consumer, in which case the caller should not wait for an answer that is not coming
188 def settled(self, settlement: Settlement) -> None: 189 """Tells the listeners what happened. Called by the consumer. 190 191 :param settlement: what the consumer decided to do with the delivery 192 """ 193 for listener in self._settlement_listeners: 194 try: 195 listener(settlement) 196 except Exception: 197 # The delivery still has to be settled. A listener is an 198 # observer, and an observer that breaks must not take the 199 # message with it. 200 log.exception( 201 "acemq: a settlement listener failed for a delivery from %s", self.queue 202 )
Tells the listeners what happened. Called by the consumer.
Parameters
- settlement: what the consumer decided to do with the delivery
68@dataclass(frozen=True, slots=True) 69class ConsumeSpec: 70 """How a consumer should be set up. 71 72 :param prefetch: how many unacknowledged messages the broker will send at 73 once, or zero for the transport's default 74 :param tag: what to call this consumer to the broker, which is what an 75 operator sees when working out who is holding a message 76 :param args: broker-specific consumer arguments, such as the 77 ``x-stream-offset`` a stream consumer needs to say where it starts 78 """ 79 80 prefetch: int = 0 81 tag: str = "" 82 args: Mapping[str, Any] = field(default_factory=dict)
How a consumer should be set up.
Parameters
- prefetch: how many unacknowledged messages the broker will send at once, or zero for the transport's default
- tag: what to call this consumer to the broker, which is what an operator sees when working out who is holding a message
- args: broker-specific consumer arguments, such as the
x-stream-offseta stream consumer needs to say where it starts
451class Consumer: 452 """A running subscription. Close it to stop. 453 454 One is built by :meth:`Connection.consume` rather than directly, because it 455 has to be started before it is any use and a half-built consumer that looks 456 finished is a thing somebody will hold on to. 457 458 Starting one declares the queues it will need when a message fails — its 459 dead-letter queue, its parking lot and its rungs — unless it was told not 460 to. See :func:`acemq_amqp.topology.declare_where_failures_go`. 461 """ 462 463 def __init__( 464 self, 465 connection: Connection, 466 queue: str, 467 handler: Handler, 468 *, 469 codec: Codec, 470 retry: RetryPolicy, 471 concurrency: int, 472 declare: bool = True, 473 ) -> None: 474 if concurrency < 1: 475 raise ValueError(f"acemq: concurrency must be at least 1, got {concurrency}") 476 self._connection = connection 477 self._queue = queue 478 self._handler = handler 479 self._codec = codec 480 self._retry = retry 481 self._concurrency = concurrency 482 self._declare = declare 483 self._work: asyncio.Queue[Delivery | None] = asyncio.Queue() 484 self._workers: list[asyncio.Task[None]] = [] 485 self._subscription: Subscription | None = None 486 self._in_flight = 0 487 self._closed = False 488 489 @property 490 def queue(self) -> str: 491 """The queue this consumer reads.""" 492 return self._queue 493 494 @property 495 def closed(self) -> bool: 496 """Whether this consumer has been stopped.""" 497 return self._closed 498 499 @property 500 def running(self) -> bool: 501 """Whether this consumer is subscribed and still has workers to hand a 502 delivery to. 503 504 What a health check reads. A consumer whose workers have all finished 505 without it being closed is a consumer the broker is still sending 506 messages to and nothing is reading — which looks identical to a quiet 507 queue from outside, and is the failure worth telling a probe about. 508 """ 509 return ( 510 not self._closed 511 and self._subscription is not None 512 and any(not worker.done() for worker in self._workers) 513 ) 514 515 @property 516 def in_flight(self) -> int: 517 """How many messages this consumer is working on right now.""" 518 return self._in_flight 519 520 @property 521 def _labels(self) -> dict[str, str]: 522 return {TAG_QUEUE: self._queue} 523 524 @property 525 def _observer(self) -> Observer: 526 return self._connection.observer 527 528 async def _start(self, prefetch: int, tag: str, args: Mapping[str, Any]) -> None: 529 if self._declare: 530 # Before the subscription rather than after it, so that the first 531 # message this consumer gives up on already has somewhere to go. A 532 # broker that refuses one of these declarations is a consumer that 533 # does not start, which is the honest failure: it would otherwise 534 # start and lose its first dead letter. 535 await declare_where_failures_go( 536 self._connection.transport, self._queue, self._retry 537 ) 538 539 self._workers = [ 540 asyncio.create_task(self._work_loop(), name=f"acemq-consumer-{self._queue}-{n}") 541 for n in range(self._concurrency) 542 ] 543 try: 544 self._subscription = await self._connection.transport.consume( 545 self._queue, 546 ConsumeSpec(prefetch=prefetch, tag=tag, args=dict(args)), 547 self._deliver, 548 ) 549 except BaseException: 550 # Nothing was subscribed, so nothing will arrive; the workers would 551 # otherwise sit on an empty queue for the life of the process. 552 for _ in self._workers: 553 self._work.put_nowait(None) 554 await asyncio.gather(*self._workers) 555 raise 556 557 async def _deliver(self, delivery: Delivery) -> None: 558 """Hands a delivery to the workers. 559 560 Queued rather than handled here, so that ``concurrency`` means what it 561 says: aio-pika starts a task per delivery and does not wait for the last 562 one, so without a queue of our own a prefetch of twenty would run twenty 563 handlers at once whatever the caller asked for. The queue needs no bound 564 because prefetch is one: the broker will not send a twenty-first message 565 until one of the twenty has been settled. 566 """ 567 self._work.put_nowait(delivery) 568 569 async def _work_loop(self) -> None: 570 while True: 571 delivery = await self._work.get() 572 if delivery is None: 573 return 574 try: 575 await self._handle(delivery) 576 except Exception: 577 # A failure here is this library's, not the handler's, and the 578 # message is left unsettled on purpose: the broker will hand it 579 # back rather than it disappearing because our own code broke. 580 log.exception("acemq: could not settle a delivery from %s", self._queue) 581 582 async def _handle(self, delivery: Delivery) -> None: 583 envelope = Envelope.from_headers(delivery.headers, delivery.routing_key) 584 # No counter for "a message arrived": every delivery is counted once 585 # when it is settled, on ``acemq.consume.total`` with the outcome, and 586 # the sum across the outcomes is how many arrived. One counter that 587 # leads the others by however many messages are in flight is a counter 588 # that makes an operator wonder which one is lying. 589 # 590 # The attempt is a distribution rather than a counter, and it is recorded 591 # here on arrival rather than at the settlement: it is a fact about the 592 # delivery that is true before the handler runs, and recording it after 593 # would drop every message this consumer is still working on when the 594 # process stops. A rising distribution is a dependency in trouble, and it 595 # says so before anything else does — the dead letters only move once the 596 # attempts run out. 597 self._observer.observe( 598 METRIC_CONSUME_ATTEMPTS, float(envelope.attempt), self._labels 599 ) 600 self._in_flight += 1 601 self._observer.gauge(METRIC_CONSUME_IN_FLIGHT, self._in_flight, self._labels) 602 try: 603 await self._handle_one(delivery, envelope) 604 finally: 605 self._in_flight -= 1 606 self._observer.gauge(METRIC_CONSUME_IN_FLIGHT, self._in_flight, self._labels) 607 608 async def _handle_one(self, delivery: Delivery, envelope: Envelope) -> None: 609 try: 610 payload = self._codec.decode(delivery.body, delivery.content_type) 611 except Exception as failure: 612 # A body that will not decode decodes no better next time, so it 613 # does not go round the retry schedule until it ages out. 614 # 615 # Parked rather than dead-lettered: a message that failed five times 616 # and a message nothing could read are different problems with 617 # different answers — one is usually the world, the other is usually 618 # a producer — and whoever drains the dead letters should not have 619 # to sort them by hand. 620 # 621 # Counted as a settled delivery here rather than in ``_carry_out``, 622 # which this path never reaches: no handler ran, so there is no 623 # duration to record, but the message did arrive and was dealt with 624 # and a total that missed it would not add up. 625 self._observer.count( 626 METRIC_CONSUME_TOTAL, 1, {**self._labels, TAG_OUTCOME: OUTCOME_PARKED} 627 ) 628 await self._park( 629 delivery, envelope, f"could not be decoded: {_describe(failure)}" 630 ) 631 return 632 633 context = ConsumeContext( 634 queue=self._queue, 635 envelope=envelope, 636 payload=payload, 637 body=delivery.body, 638 content_type=delivery.content_type, 639 routing_key=delivery.routing_key, 640 redelivered=delivery.redelivered, 641 # This delivery is being driven by a consumer, so how it ends will 642 # be reported. An interceptor that wants to know can wait for it. 643 reports_settlement=True, 644 reply_to=delivery.reply_to, 645 ) 646 647 started = time.monotonic() 648 try: 649 decision = await consume_chain( 650 self._connection.consume_interceptors, self._run_handler 651 )(context) 652 except Exception as failure: 653 # An exception is how Python says a thing failed, so a handler that 654 # raises is asking for the retry policy rather than confessing a 655 # bug — which is what Java does with the same situation, and what a 656 # handler written without reading the documentation will expect. 657 # Raising FatalError is how it says the opposite, and the retry path 658 # below reads that mark before it reads the request. 659 # 660 # An interceptor that raises lands here too, and deliberately: a 661 # refused message is retried and then dead-lettered rather than 662 # acknowledged as though something had processed it. 663 decision = Ack(Action.RETRY, failure) 664 665 # Timed around the interceptors as well as the handler, because what an 666 # operator wants to know is how long a message takes to deal with, and 667 # an interceptor that opens a transaction is part of dealing with it. 668 # Stopped here and recorded a few lines below rather than in a 669 # ``finally``, because the duration is tagged with the outcome and the 670 # outcome is not known until ``_decide`` has run: how long a message 671 # took and what happened to it are one question, and a p99 that mixes 672 # the messages that worked with the ones that timed out answers neither 673 # half of it. 674 elapsed = time.monotonic() - started 675 676 # Whatever the interceptors left, rather than what arrived: one that 677 # rewrote the envelope on the way in meant that rewrite for the retry 678 # and the dead letter as much as for the handler. 679 envelope = context.envelope 680 681 if isinstance(decision, Ack): 682 settlement, wait = self._decide(envelope, decision) 683 else: 684 settlement, wait = ( 685 Settlement( 686 OUTCOME_DEAD_LETTERED, 687 reason=( 688 f"the handler returned {type(decision).__name__} instead of an Ack" 689 ), 690 ), 691 None, 692 ) 693 694 self._observer.observe( 695 METRIC_CONSUME_DURATION, 696 elapsed, 697 {**self._labels, TAG_OUTCOME: settlement.outcome}, 698 ) 699 700 # Announced before it is carried out, and so before a consumer-side 701 # retry sleeps out its backoff. A listener told afterwards would be 702 # describing the delivery minutes after the decision, and the span it is 703 # writing on would have stayed open for a wait that is not work. 704 context.settled(settlement) 705 await self._carry_out(delivery, envelope, settlement, wait) 706 707 async def _run_handler(self, context: ConsumeContext) -> Ack: 708 """The innermost work: build the message the interceptors left, and run 709 the handler on it. 710 711 The message is built here rather than before the chain, so that an 712 interceptor which changed the envelope or the payload changed what the 713 handler is given rather than a copy nobody reads. 714 """ 715 message = Message( 716 payload=context.payload, 717 envelope=context.envelope, 718 routing_key=context.routing_key, 719 content_type=context.content_type, 720 redelivered=context.redelivered, 721 body=context.body, 722 reply_to=context.reply_to, 723 ) 724 returned = self._handler(message) 725 # Not an Ack is a mistake worth reporting, but it is not this method's 726 # to report: it is returned as it stands and _handle dead-letters it 727 # with a reason, which is where every other unhandleable delivery goes. 728 # An interceptor between here and there sees the same thing the handler 729 # returned, which is the honest thing to show it. 730 decision: Ack = await returned if inspect.isawaitable(returned) else returned 731 return decision 732 733 def _decide(self, envelope: Envelope, decision: Ack) -> tuple[Settlement, Wait | None]: 734 """What is going to happen to this delivery, before anything happens to 735 it. 736 737 Worked out on its own, and without touching the broker, so that it can 738 be announced first. The gap this closes is a small one to describe and a 739 wide one to fall into: a handler asking for a retry it has no attempts 740 left for is dead-lettered, and anything reading the handler's answer 741 instead of this one records a retry that never happened. 742 743 :param envelope: the message's metadata, for the attempt and the age 744 :param decision: what the handler answered 745 :returns: the settlement, and the wait when there is another attempt — 746 which the settlement does not carry, because whether a wait is spent 747 here or on a rung queue is this consumer's business and not a 748 listener's 749 """ 750 if decision.action is Action.ACCEPT: 751 return Settlement(OUTCOME_ACKED), None 752 753 if decision.action is Action.REJECT: 754 return ( 755 Settlement( 756 OUTCOME_REJECTED, 757 reason=f"the handler rejected it: {_describe(decision.error)}", 758 ), 759 None, 760 ) 761 762 if decision.action is Action.PARK: 763 # The handler read the message and found nothing it could read. The 764 # engine reaches the same conclusion for a body its codec refuses, 765 # and both end up in the same queue for the same reason: a message 766 # nobody can read is a producer problem, and it is only findable if 767 # it is not filed with the messages that merely ran out of luck. 768 return ( 769 Settlement( 770 OUTCOME_PARKED, 771 reason=f"the handler parked it: {_describe(decision.error)}", 772 ), 773 None, 774 ) 775 776 if decision.action is Action.IN_PROGRESS: 777 # Somebody else holds the message. Not a failure of this one, so the 778 # retry policy is not asked: it can neither spend an attempt nor run 779 # out and dead-letter a message nobody has failed to handle. 780 delay = decision.delay if decision.delay is not None else ZERO 781 return Settlement(OUTCOME_IN_PROGRESS, delay=delay), Wait(delay, in_broker=False) 782 783 if isinstance(decision.error, FatalError): 784 # The handler asked for a retry but marked the reason as one that 785 # will not change. Honouring the mark rather than the request is the 786 # point of having it. 787 return ( 788 Settlement( 789 OUTCOME_DEAD_LETTERED, 790 reason=( 791 "the handler reported an unprocessable message: " 792 f"{_describe(decision.error)}" 793 ), 794 ), 795 None, 796 ) 797 798 wait = self._retry.next_wait(envelope.attempt, envelope.age) 799 if wait is None: 800 return ( 801 Settlement( 802 OUTCOME_DEAD_LETTERED, 803 reason=f"{self._exhausted(envelope)}: {_describe(decision.error)}", 804 ), 805 None, 806 ) 807 808 return Settlement(OUTCOME_RETRIED, delay=wait.delay), wait 809 810 async def _carry_out( 811 self, 812 delivery: Delivery, 813 envelope: Envelope, 814 settlement: Settlement, 815 wait: Wait | None, 816 ) -> None: 817 """Does what :meth:`_decide` decided, and counts it as decided. 818 819 Every counter here is chosen from the :class:`~acemq_amqp.ack.Settlement` 820 and never from the handler's :class:`~acemq_amqp.ack.Ack`, which is the 821 same rule the delivery's span follows. A handler asking for a retry it 822 has no attempts left for is dead-lettered, and a counter that read the 823 request rather than the answer would report a retry for a message 824 nothing will ever try again — so the dead letters would be undercounted 825 by exactly the messages an operator most wants to find. 826 827 One ``acemq.consume.total`` per delivery, carrying the outcome, and then 828 whichever of the standalone counters applies. ``retried`` and 829 ``dead_lettered`` are both an outcome here and a counter of their own, 830 which is what Java does and is not a redundancy: the outcome says what 831 was decided about a delivery, and the counter says how many messages are 832 going round again or have been set aside, which is the number an alert 833 is written against. 834 """ 835 self._observer.count( 836 METRIC_CONSUME_TOTAL, 1, {**self._labels, TAG_OUTCOME: settlement.outcome} 837 ) 838 839 if settlement.outcome == OUTCOME_ACKED: 840 await delivery.ack() 841 return 842 843 if settlement.outcome == OUTCOME_PARKED: 844 # ``_park`` counts it, exactly as it does for a body that would not 845 # decode. There is one parked counter and one parked queue whether 846 # the codec or the handler was the one that could not read it. 847 await self._park(delivery, envelope, settlement.reason or "") 848 return 849 850 if wait is None: 851 await self._dead_letter(delivery, envelope, settlement.reason or "") 852 return 853 854 if settlement.outcome == OUTCOME_IN_PROGRESS: 855 await self._put_back(delivery, envelope, wait.delay) 856 return 857 858 if wait.in_broker and await self._retry_in_broker(delivery, envelope, wait.delay): 859 return 860 861 if wait.delay > ZERO: 862 # Waiting here holds the delivery, and so holds one of this 863 # consumer's prefetch slots. For a wait of a few seconds that is the 864 # cheaper of the two costs; for a longer one it is not, which is why 865 # the policy has a threshold and the long waits went to a rung queue 866 # a few lines above. 867 await asyncio.sleep(wait.delay.total_seconds()) 868 869 await self._retry_again(delivery, envelope, wait.delay) 870 871 def _exhausted(self, envelope: Envelope) -> str: 872 """Why there is no next attempt, in words an operator can act on.""" 873 if envelope.attempt >= self._retry.max_attempts: 874 attempts = self._retry.max_attempts 875 return f"exhausted {attempts} attempt{'' if attempts == 1 else 's'}" 876 return f"exceeded the maximum message age of {self._retry.max_message_age}" 877 878 async def _retry_in_broker( 879 self, delivery: Delivery, envelope: Envelope, delay: timedelta 880 ) -> bool: 881 """Parks the message on ``{queue}.retry.{delay}`` and lets the broker 882 return it. 883 884 The rung's ``x-message-ttl`` is the delay and its dead-letter target is 885 this queue, so the wait costs nothing here: no delivery held, no prefetch 886 slot spent, and — the reason it exists — nothing lost when this process 887 restarts halfway through. A consumer sleeping on a five-minute backoff 888 that dies at minute one does not resume at minute one; the broker 889 redelivers the unacknowledged message immediately, and the policy that 890 said five minutes delivers in none. 891 892 The obvious simplification is one rung queue and a per-message 893 ``expiration`` instead of several with fixed TTLs. It does not work. 894 RabbitMQ only ever expires the message at the *head* of a queue, so a 895 message with a ten-minute TTL at the front holds back every thirty-second 896 one behind it, and the delays that come out bear no resemblance to the 897 ones that went in. Never set a per-message TTL for this. 898 899 The attempt advances here exactly as it does on an immediate retry: the 900 counter belongs to the message, and a message that has been round the 901 broker is no less on its second attempt than one that waited here. 902 903 :returns: whether the rung took it. ``False`` means no such queue, and 904 the caller should fall back to waiting here 905 """ 906 rung = naming.retry_queue(self._queue, delay) 907 next_attempt = envelope.with_(attempt=envelope.attempt + 1) 908 try: 909 routed = await self._republish(delivery, rung, next_attempt) 910 except _HopRefusedError: 911 # The rung is there and would not take it. Requeued at once, with 912 # nothing else on this consumer settled, rather than waited out 913 # here: the broker is the one saying no, and holding the delivery 914 # for the whole delay would stall this consumer for nothing. 915 self._observer.count(METRIC_RETRIED_TOTAL, 1, {**self._labels, "where": "requeued"}) 916 await delivery.nack(True) 917 return True 918 if not routed: 919 # Kept even though a consumer declares its own rungs when it starts, 920 # because the rung can still be missing: a consumer started with 921 # ``declare=False`` declares nothing, and a rung deleted under a 922 # running consumer is gone whoever declared it. What has changed is 923 # what it means — it used to be an ordinary topology mistake and is 924 # now either a deliberate opt-out or somebody removing queues. 925 # 926 # Degraded rather than fatal: the message is still deliverable, and 927 # waiting for it here is what this library did before there were 928 # rungs. Counted as well as logged, because this is the one failure 929 # here that nothing else shows: the message is still retried and the 930 # wait still happens, so a dashboard reads as normal while the reason 931 # the rung exists is quietly gone. 932 self._observer.count(METRIC_RUNG_MISSING, 1, {**self._labels, "rung": rung}) 933 log.error( 934 "acemq: %s is not on the broker, so %s will wait %s in this consumer " 935 "instead; declare it with Topology().queue(%r, retry=policy), or let " 936 "the consumer declare it by leaving declare=True", 937 rung, 938 envelope.id, 939 delay, 940 self._queue, 941 ) 942 return False 943 944 self._observer.count(METRIC_RETRIED_TOTAL, 1, {**self._labels, "where": "broker"}) 945 log.info( 946 "acemq: retrying %s from %s, attempt %d of %d, after %s in %s", 947 envelope.id, 948 self._queue, 949 next_attempt.attempt, 950 self._retry.max_attempts, 951 delay, 952 rung, 953 ) 954 await delivery.ack() 955 return True 956 957 async def _retry_again( 958 self, delivery: Delivery, envelope: Envelope, delay: timedelta 959 ) -> None: 960 """Puts the message back on its own queue, one attempt further on. 961 962 Republished rather than requeued, because a requeue returns the bytes 963 the broker was given: the attempt header would still read what the 964 publisher wrote however many times the message had come round, and the 965 count would live only in this process's memory, which is the one place 966 it is lost when the process that has been failing restarts. 967 968 The cost is that the message goes to the back of the queue rather than 969 the front, so a retry is no longer in order with its neighbours. For a 970 message that has already failed once, that is the better trade. 971 """ 972 next_attempt = envelope.with_(attempt=envelope.attempt + 1) 973 try: 974 delivered = await self._republish(delivery, self._queue, next_attempt) 975 except _HopRefusedError: 976 delivered = False 977 if not delivered: 978 # The queue we are consuming has gone. Requeued rather than 979 # acknowledged, because dropping it here would lose a message over a 980 # broker change nobody told this consumer about. 981 # 982 # Counted as a retry all the same, and it is one: the settlement 983 # said ``retried``, the span says ``retried``, and the broker is 984 # about to hand the message back. Leaving it uncounted was the one 985 # place where the counters and the span disagreed about the same 986 # delivery. The label says which path it took, because a requeue 987 # keeps the attempt header it arrived with and the other two do not. 988 self._observer.count(METRIC_RETRIED_TOTAL, 1, {**self._labels, "where": "requeued"}) 989 log.error( 990 "acemq: cannot republish %s onto %s for attempt %d; returning it to the broker", 991 envelope.id, 992 self._queue, 993 next_attempt.attempt, 994 ) 995 await delivery.nack(True) 996 return 997 998 self._observer.count(METRIC_RETRIED_TOTAL, 1, {**self._labels, "where": "consumer"}) 999 log.info( 1000 "acemq: retrying %s from %s, attempt %d of %d, after %s", 1001 envelope.id, 1002 self._queue, 1003 next_attempt.attempt, 1004 self._retry.max_attempts, 1005 delay, 1006 ) 1007 await delivery.ack() 1008 1009 async def _put_back( 1010 self, delivery: Delivery, envelope: Envelope, delay: timedelta 1011 ) -> None: 1012 """Returns a message somebody else is working on, attempt unchanged. 1013 1014 :meth:`_retry_again` without the increment: the attempt counts this 1015 message's failures, and a claim held elsewhere is not one of them. 1016 """ 1017 if delay > ZERO: 1018 await asyncio.sleep(delay.total_seconds()) 1019 try: 1020 delivered = await self._republish(delivery, self._queue, envelope) 1021 except _HopRefusedError: 1022 delivered = False 1023 if delivered: 1024 await delivery.ack() 1025 return 1026 log.error( 1027 "acemq: cannot put %s back onto %s; returning it to the broker", 1028 envelope.id, 1029 self._queue, 1030 ) 1031 await delivery.nack(True) 1032 1033 async def _park(self, delivery: Delivery, envelope: Envelope, reason: str) -> None: 1034 """Sends the message to ``{queue}.parked`` with the reason attached. 1035 1036 Where a message goes when it never reached the handler at all. Somebody 1037 has to look at it, and what they need to know first is that it was 1038 unreadable rather than unlucky. 1039 """ 1040 # The same counter a dead-lettering moves, separated by the outcome 1041 # rather than by a metric of its own — which is what Java settled on, and 1042 # what makes "how much is this queue giving up on" one number that can 1043 # then be split into the two problems it is made of. 1044 self._observer.count( 1045 METRIC_DEAD_LETTERED_TOTAL, 1, {**self._labels, TAG_OUTCOME: OUTCOME_PARKED} 1046 ) 1047 await self._set_aside(delivery, envelope, naming.parked_queue(self._queue), reason) 1048 1049 async def _dead_letter(self, delivery: Delivery, envelope: Envelope, reason: str) -> None: 1050 """Sends the message to ``{queue}.dlq`` with the reason attached, and 1051 acknowledges the original. 1052 1053 Acknowledging a message that failed looks wrong and is what makes this 1054 reliable: the message has already been safely republished somewhere 1055 else, so acknowledging the original is removing the copy that has been 1056 dealt with. Rejecting it instead would either requeue it into a hot loop 1057 or, with a dead-letter exchange configured on the queue, send it 1058 somewhere this consumer did not choose and without the reason. 1059 1060 The reason travels as an envelope field, so a consumer of the 1061 dead-letter queue reads it back through the API rather than having to 1062 know the wire header name. 1063 """ 1064 self._observer.count( 1065 METRIC_DEAD_LETTERED_TOTAL, 1066 1, 1067 {**self._labels, TAG_OUTCOME: OUTCOME_DEAD_LETTERED}, 1068 ) 1069 await self._set_aside( 1070 delivery, envelope, naming.dead_letter_queue(self._queue), reason 1071 ) 1072 1073 async def _set_aside( 1074 self, delivery: Delivery, envelope: Envelope, target: str, reason: str 1075 ) -> None: 1076 """Republishes to ``target`` with the reason, then acknowledges.""" 1077 failed = envelope.with_(error=reason) 1078 try: 1079 delivered = await self._republish(delivery, target, failed) 1080 except _HopRefusedError: 1081 delivered = False 1082 if not delivered: 1083 # Rejected rather than acknowledged: without a dead-letter queue to 1084 # put it in, the broker's own dead-lettering is the last thing left 1085 # between this message and nothing. 1086 self._observer.count( 1087 METRIC_SET_ASIDE_FAILED, 1, {**self._labels, TAG_TARGET: target} 1088 ) 1089 log.error( 1090 "acemq: cannot move %s to %s (%s); rejecting it to the broker instead", 1091 envelope.id, 1092 target, 1093 reason, 1094 ) 1095 await delivery.nack(False) 1096 return 1097 1098 log.warning( 1099 "acemq: set aside %s from %s after %d attempts, into %s: %s", 1100 envelope.id, 1101 self._queue, 1102 envelope.attempt, 1103 target, 1104 reason, 1105 ) 1106 await delivery.ack() 1107 1108 async def _republish(self, delivery: Delivery, queue: str, envelope: Envelope) -> bool: 1109 """Sends the original bytes to a queue by name, and says whether they 1110 arrived. 1111 1112 Through the default exchange, which routes to the queue whose name 1113 matches the routing key, and mandatory so that a queue that is not there 1114 is an answer rather than a silence. The body goes back exactly as it 1115 came: re-encoding through a class that has since changed would replace 1116 what was committed with something else. 1117 """ 1118 try: 1119 result = await self._connection.publish_raw( 1120 "", 1121 queue, 1122 Outbound( 1123 body=delivery.body, 1124 content_type=delivery.content_type or "", 1125 message_id=envelope.id, 1126 headers=envelope.to_headers(routing_key=delivery.routing_key), 1127 persistent=True, 1128 mandatory=True, 1129 ), 1130 ) 1131 except PublishError as refused: 1132 # The broker nacked it, or would not take it while blocked. Raised 1133 # as the caller's to settle, because what to do differs: a refused 1134 # rung means requeue now, where a missing rung means wait here. 1135 # Escaping further left the delivery unsettled for the life of the 1136 # channel, which is what Ruby did before 0.7.12. 1137 log.error("acemq: the broker refused %s onto %s: %s", envelope.id, queue, refused) 1138 raise _HopRefusedError(str(refused)) from refused 1139 return result.routed 1140 1141 async def close(self, *, timeout: float | None = DEFAULT_DRAIN_TIMEOUT) -> bool: 1142 """Stops the consumer and waits, for up to ``timeout``, for the handlers 1143 already running. 1144 1145 A message being worked on when this is called is finished and settled, 1146 rather than abandoned for the broker to hand to somebody else. A message 1147 that had been delivered but not started is given back instead: it is the 1148 broker's to hand to another consumer, and holding it here only to work 1149 through a retry delay would make closing take as long as the schedule. 1150 1151 A handler still running when the time is up is cancelled, and its message 1152 is left unsettled — neither acknowledged nor rejected — so the broker 1153 hands it out again once the channel goes. Shutdown is never the reason a 1154 message is lost or dead-lettered. The same happens when the caller 1155 cancels closing, ``asyncio.wait_for`` included, and that cancellation is 1156 raised again once the consumer has been cleaned up. 1157 1158 The subscription is released last, after everything has been settled, 1159 because a settlement travels on the channel its delivery arrived on. 1160 1161 :param timeout: seconds to wait for every running handler together; 1162 ``None`` waits for as long as they take 1163 :returns: ``True`` if every handler finished, ``False`` if the time ran 1164 out first and some were cut off — the consumer is stopped either way 1165 """ 1166 if self._closed: 1167 return True 1168 self._closed = True 1169 1170 running: set[asyncio.Task[None]] = set() 1171 try: 1172 if self._subscription is not None: 1173 await self._subscription.stop() 1174 1175 while True: 1176 try: 1177 pending = self._work.get_nowait() 1178 except asyncio.QueueEmpty: 1179 break 1180 if pending is not None: 1181 await pending.nack(True) 1182 1183 for _ in self._workers: 1184 self._work.put_nowait(None) 1185 if self._workers: 1186 _, running = await asyncio.wait(self._workers, timeout=timeout) 1187 finally: 1188 # Reached with workers still running for two reasons: the deadline 1189 # passed, or the caller cancelled us. Either way they are cancelled 1190 # before the channel goes, so nothing settles a message on a channel 1191 # that is closing — and a cancellation that came from the caller is 1192 # not caught here, so it carries on out once this has run. 1193 stragglers = [worker for worker in self._workers if not worker.done()] 1194 for worker in stragglers: 1195 worker.cancel() 1196 if stragglers: 1197 await asyncio.wait(stragglers) 1198 if self._subscription is not None: 1199 await self._subscription.close() 1200 self._connection._untrack(self) 1201 1202 for worker in self._workers: 1203 # A worker that was already cancelled — a task group unwinding, a 1204 # shutdown that cancelled the loop's tasks before closing the 1205 # connection — is not news to report to a caller who is merely 1206 # closing. One that died of its own accord is. 1207 failure = None if worker.cancelled() else worker.exception() 1208 if failure is not None: 1209 raise failure 1210 return not running 1211 1212 async def __aenter__(self) -> Consumer: 1213 return self 1214 1215 async def __aexit__(self, *_: object) -> None: 1216 # Not returned: a true value from here tells the interpreter that the 1217 # exception leaving the block was handled. 1218 await self.close()
A running subscription. Close it to stop.
One is built by Connection.consume() rather than directly, because it
has to be started before it is any use and a half-built consumer that looks
finished is a thing somebody will hold on to.
Starting one declares the queues it will need when a message fails — its
dead-letter queue, its parking lot and its rungs — unless it was told not
to. See acemq_amqp.topology.declare_where_failures_go().
463 def __init__( 464 self, 465 connection: Connection, 466 queue: str, 467 handler: Handler, 468 *, 469 codec: Codec, 470 retry: RetryPolicy, 471 concurrency: int, 472 declare: bool = True, 473 ) -> None: 474 if concurrency < 1: 475 raise ValueError(f"acemq: concurrency must be at least 1, got {concurrency}") 476 self._connection = connection 477 self._queue = queue 478 self._handler = handler 479 self._codec = codec 480 self._retry = retry 481 self._concurrency = concurrency 482 self._declare = declare 483 self._work: asyncio.Queue[Delivery | None] = asyncio.Queue() 484 self._workers: list[asyncio.Task[None]] = [] 485 self._subscription: Subscription | None = None 486 self._in_flight = 0 487 self._closed = False
489 @property 490 def queue(self) -> str: 491 """The queue this consumer reads.""" 492 return self._queue
The queue this consumer reads.
494 @property 495 def closed(self) -> bool: 496 """Whether this consumer has been stopped.""" 497 return self._closed
Whether this consumer has been stopped.
499 @property 500 def running(self) -> bool: 501 """Whether this consumer is subscribed and still has workers to hand a 502 delivery to. 503 504 What a health check reads. A consumer whose workers have all finished 505 without it being closed is a consumer the broker is still sending 506 messages to and nothing is reading — which looks identical to a quiet 507 queue from outside, and is the failure worth telling a probe about. 508 """ 509 return ( 510 not self._closed 511 and self._subscription is not None 512 and any(not worker.done() for worker in self._workers) 513 )
Whether this consumer is subscribed and still has workers to hand a delivery to.
What a health check reads. A consumer whose workers have all finished without it being closed is a consumer the broker is still sending messages to and nothing is reading — which looks identical to a quiet queue from outside, and is the failure worth telling a probe about.
515 @property 516 def in_flight(self) -> int: 517 """How many messages this consumer is working on right now.""" 518 return self._in_flight
How many messages this consumer is working on right now.
1141 async def close(self, *, timeout: float | None = DEFAULT_DRAIN_TIMEOUT) -> bool: 1142 """Stops the consumer and waits, for up to ``timeout``, for the handlers 1143 already running. 1144 1145 A message being worked on when this is called is finished and settled, 1146 rather than abandoned for the broker to hand to somebody else. A message 1147 that had been delivered but not started is given back instead: it is the 1148 broker's to hand to another consumer, and holding it here only to work 1149 through a retry delay would make closing take as long as the schedule. 1150 1151 A handler still running when the time is up is cancelled, and its message 1152 is left unsettled — neither acknowledged nor rejected — so the broker 1153 hands it out again once the channel goes. Shutdown is never the reason a 1154 message is lost or dead-lettered. The same happens when the caller 1155 cancels closing, ``asyncio.wait_for`` included, and that cancellation is 1156 raised again once the consumer has been cleaned up. 1157 1158 The subscription is released last, after everything has been settled, 1159 because a settlement travels on the channel its delivery arrived on. 1160 1161 :param timeout: seconds to wait for every running handler together; 1162 ``None`` waits for as long as they take 1163 :returns: ``True`` if every handler finished, ``False`` if the time ran 1164 out first and some were cut off — the consumer is stopped either way 1165 """ 1166 if self._closed: 1167 return True 1168 self._closed = True 1169 1170 running: set[asyncio.Task[None]] = set() 1171 try: 1172 if self._subscription is not None: 1173 await self._subscription.stop() 1174 1175 while True: 1176 try: 1177 pending = self._work.get_nowait() 1178 except asyncio.QueueEmpty: 1179 break 1180 if pending is not None: 1181 await pending.nack(True) 1182 1183 for _ in self._workers: 1184 self._work.put_nowait(None) 1185 if self._workers: 1186 _, running = await asyncio.wait(self._workers, timeout=timeout) 1187 finally: 1188 # Reached with workers still running for two reasons: the deadline 1189 # passed, or the caller cancelled us. Either way they are cancelled 1190 # before the channel goes, so nothing settles a message on a channel 1191 # that is closing — and a cancellation that came from the caller is 1192 # not caught here, so it carries on out once this has run. 1193 stragglers = [worker for worker in self._workers if not worker.done()] 1194 for worker in stragglers: 1195 worker.cancel() 1196 if stragglers: 1197 await asyncio.wait(stragglers) 1198 if self._subscription is not None: 1199 await self._subscription.close() 1200 self._connection._untrack(self) 1201 1202 for worker in self._workers: 1203 # A worker that was already cancelled — a task group unwinding, a 1204 # shutdown that cancelled the loop's tasks before closing the 1205 # connection — is not news to report to a caller who is merely 1206 # closing. One that died of its own accord is. 1207 failure = None if worker.cancelled() else worker.exception() 1208 if failure is not None: 1209 raise failure 1210 return not running
Stops the consumer and waits, for up to timeout, for the handlers
already running.
A message being worked on when this is called is finished and settled, rather than abandoned for the broker to hand to somebody else. A message that had been delivered but not started is given back instead: it is the broker's to hand to another consumer, and holding it here only to work through a retry delay would make closing take as long as the schedule.
A handler still running when the time is up is cancelled, and its message
is left unsettled — neither acknowledged nor rejected — so the broker
hands it out again once the channel goes. Shutdown is never the reason a
message is lost or dead-lettered. The same happens when the caller
cancels closing, asyncio.wait_for included, and that cancellation is
raised again once the consumer has been cleaned up.
The subscription is released last, after everything has been settled, because a settlement travels on the channel its delivery arrived on.
Parameters
- timeout: seconds to wait for every running handler together;
Nonewaits for as long as they take :returns:Trueif every handler finished,Falseif the time ran out first and some were cut off — the consumer is stopped either way
151@dataclass(frozen=True, slots=True, repr=False) 152class Credentials: 153 """A username and a secret for the broker. 154 155 The secret never appears in :func:`repr` or :func:`str`, so a credential 156 logged by accident — in a dataclass dump, in an exception, in a debugger's 157 variables pane — prints the username and nothing else. That is the whole 158 reason this is a type rather than two strings, and a test pins it. 159 160 :param username: the account name 161 :param secret: the password, or the token for a broker that takes one 162 """ 163 164 username: str = "" 165 secret: str = "" 166 167 @property 168 def empty(self) -> bool: 169 """Whether these credentials carry nothing at all.""" 170 return not self.username and not self.secret 171 172 def __repr__(self) -> str: 173 if not self.username: 174 return "Credentials(<secret>)" 175 return f"Credentials(username={self.username!r}, secret=<secret>)" 176 177 __str__ = __repr__
A username and a secret for the broker.
The secret never appears in repr() or str(), so a credential
logged by accident — in a dataclass dump, in an exception, in a debugger's
variables pane — prints the username and nothing else. That is the whole
reason this is a type rather than two strings, and a test pins it.
Parameters
- username: the account name
- secret: the password, or the token for a broker that takes one
134@dataclass(frozen=True, slots=True) 135class Delivery: 136 """A message as it arrived, before any codec has looked at it. 137 138 :param body: the bytes, undecoded 139 :param content_type: what the sender said the body is, or ``None`` when the 140 sender said nothing — which is a different thing from an empty string 141 and is why the codecs distinguish them 142 :param routing_key: the key it arrived under 143 :param message_id: the AMQP message id 144 :param headers: everything the sender wrote, reserved names included 145 :param redelivered: the broker saying it has handed this message over 146 before. It is the broker's own view and says nothing about which attempt 147 this is: a requeue returns the bytes the broker was given, so the 148 attempt header still reads whatever the publisher wrote 149 :param ack: confirms the message and removes it from the queue 150 :param nack: returns the message, requeued or not 151 :param reply_to: AMQP's own ``reply-to`` property, empty when the sender did 152 not set one. A transport that cannot report it leaves it empty and the 153 responder falls back to the header, which is the only reason this has a 154 default at all 155 """ 156 157 body: bytes 158 content_type: str | None 159 routing_key: str 160 message_id: str 161 headers: Mapping[str, Any] 162 redelivered: bool 163 ack: Callable[[], Awaitable[None]] 164 nack: Callable[[bool], Awaitable[None]] 165 reply_to: str = ""
A message as it arrived, before any codec has looked at it.
Parameters
- body: the bytes, undecoded
- content_type: what the sender said the body is, or
Nonewhen the sender said nothing — which is a different thing from an empty string and is why the codecs distinguish them - routing_key: the key it arrived under
- message_id: the AMQP message id
- headers: everything the sender wrote, reserved names included
- redelivered: the broker saying it has handed this message over before. It is the broker's own view and says nothing about which attempt this is: a requeue returns the bytes the broker was given, so the attempt header still reads whatever the publisher wrote
- ack: confirms the message and removes it from the queue
- nack: returns the message, requeued or not
- reply_to: AMQP's own
reply-toproperty, empty when the sender did not set one. A transport that cannot report it leaves it empty and the responder falls back to the header, which is the only reason this has a default at all
350@dataclass(frozen=True, slots=True) 351class DurationSummary: 352 """What a :class:`Metrics` knows about a timing. 353 354 Deliberately not percentiles. Computing those needs either every sample kept 355 or a sketch, and a library that quietly did either would be making a 356 decision about memory that belongs to the application. 357 """ 358 359 count: int = 0 360 total: float = 0.0 361 fastest: float = 0.0 362 slowest: float = 0.0 363 364 @property 365 def mean(self) -> float: 366 """The average, or zero when nothing has been recorded.""" 367 return self.total / self.count if self.count else 0.0
What a Metrics knows about a timing.
Deliberately not percentiles. Computing those needs either every sample kept or a sketch, and a library that quietly did either would be making a decision about memory that belongs to the application.
45@dataclass(frozen=True, slots=True) 46class Envelope: 47 """The metadata AceMQ carries with every message. 48 49 Frozen, because an envelope describes a message that has already been 50 published or received: changing one in place would change what a log line 51 said after it was written. :meth:`with_` returns a modified copy for the 52 cases where a new message derives from an old one. 53 54 Defaults are applied when the envelope is built rather than when it is 55 read, so two libraries reading the same message agree without having to 56 agree on a second set of rules. ``correlation_id`` defaults to ``id``, an 57 attempt starts at 1, a version starts at 1, and ``type`` falls back to the 58 routing key the message was published under. 59 """ 60 61 id: str = field(default_factory=lambda: str(uuid.uuid4())) 62 type: str = "" 63 version: int = 1 64 correlation_id: str = "" 65 causation_id: str = "" 66 attempt: int = 1 67 first_seen: datetime = field(default_factory=_now) 68 origin: str = "" 69 error: str = "" 70 claim: str = "" 71 72 #: The steps of a declared route, by name, comma-separated and in order — 73 #: the form Java's ``Pipeline`` writes. Empty for a message that is not 74 #: travelling one. 75 #: 76 #: A field rather than an application header for the same reason ``claim`` 77 #: is one: the name is reserved, so the application map refuses it, and a 78 #: pattern that needs it off a delivery has nowhere else to read it from. 79 #: Carried by :meth:`with_`, so a retry, a dead-letter and a replay all keep 80 #: it, which is what makes replaying a message *resume* a route rather than 81 #: restart it. :mod:`acemq_amqp.patterns.routingslip` is what reads it. 82 route: str = "" 83 84 #: Which step of :attr:`route` this message is for, counting from zero. 85 route_position: int = 0 86 87 #: One run through a route, carried by every hop of it. 88 route_id: str = "" 89 90 #: The application's own headers. Never contains a reserved name. 91 headers: Mapping[str, Any] = field(default_factory=dict) 92 93 def __post_init__(self) -> None: 94 if not self.correlation_id: 95 object.__setattr__(self, "correlation_id", self.id) 96 if self.attempt < 1: 97 object.__setattr__(self, "attempt", 1) 98 if self.version < 1: 99 object.__setattr__(self, "version", 1) 100 if self.route_position < 0: 101 # A position before the start would send the message to an 102 # arbitrary step. Starting over is the only defensible reading, and 103 # it is the one Java takes for a position it cannot parse. 104 object.__setattr__(self, "route_position", 0) 105 # Reserved names in the application's map would be written twice and 106 # read back inconsistently, so they are refused rather than dropped: 107 # silently discarding a header somebody set is worse than saying no. 108 offending = sorted(name for name in self.headers if headers.is_reserved(name)) 109 if offending: 110 raise ValueError( 111 "these header names belong to AceMQ and cannot be set by hand: " 112 + ", ".join(offending) 113 ) 114 115 @property 116 def age(self) -> timedelta: 117 """How long since the message was first published. 118 119 The basis for giving up on age rather than on attempts, which is the 120 honest limit when a queue has been paused: a message can be on attempt 121 two and four days old. 122 """ 123 return _now() - self.first_seen 124 125 def with_(self, **changes: Any) -> Envelope: 126 """A copy with fields changed. 127 128 :param changes: the fields to replace 129 :returns: a new envelope 130 """ 131 return replace(self, **changes) 132 133 def to_headers(self, routing_key: str = "") -> dict[str, Any]: 134 """The AMQP headers for this envelope. 135 136 :param routing_key: what the message is published under, used when no 137 type was given; when that is empty too, the type is the literal 138 ``message`` (AceMQ TCK requirement AMQP-ENV-006), rather than an 139 empty string no consumer can dispatch on 140 :returns: the reserved headers, plus the application's own 141 """ 142 written: dict[str, Any] = { 143 headers.ID: self.id, 144 headers.TYPE: self.type or routing_key or DEFAULT_TYPE, 145 headers.VERSION: self.version, 146 headers.CORRELATION: self.correlation_id or self.id, 147 headers.ATTEMPT: self.attempt, 148 headers.FIRST_SEEN: _millis(self.first_seen), 149 } 150 # Absent rather than empty. A header carrying "" is a header somebody 151 # has to write a special case for at the other end. 152 for name, value in ( 153 (headers.CAUSATION, self.causation_id), 154 (headers.ORIGIN, self.origin), 155 (headers.ERROR, self.error), 156 (headers.CLAIM, self.claim), 157 ): 158 if value: 159 written[name] = value 160 161 # The position is written whenever there is a route, including the zero 162 # a first hop carries: a route with no position reads as position zero 163 # everywhere, and leaving it off would make the first hop the only one 164 # whose slip is incomplete. 165 if self.route: 166 written[headers.ROUTE] = self.route 167 written[headers.ROUTE_POSITION] = self.route_position 168 if self.route_id: 169 written[headers.ROUTE_ID] = self.route_id 170 171 written.update(self.headers) 172 return written 173 174 @classmethod 175 def from_headers(cls, raw: Mapping[str, Any] | None, routing_key: str = "") -> Envelope: 176 """Reads an envelope off a delivery. 177 178 Anything missing takes its default, and anything unreadable takes its 179 default too: a message from a producer that wrote ``x-acemq-attempt`` 180 as a string is still a message, and refusing to deliver it would hand 181 the application an outage rather than a message. 182 183 :param raw: the delivery's headers 184 :param routing_key: what it arrived on, used when no type was set 185 :returns: the envelope 186 """ 187 raw = raw or {} 188 application = { 189 name: value for name, value in raw.items() if not headers.is_reserved(name) 190 } 191 192 identifier = _text(raw.get(headers.ID)) or str(uuid.uuid4()) 193 first_seen = raw.get(headers.FIRST_SEEN) 194 195 return cls( 196 id=identifier, 197 type=_text(raw.get(headers.TYPE)) or routing_key or DEFAULT_TYPE, 198 version=_number(raw.get(headers.VERSION), 1), 199 correlation_id=_text(raw.get(headers.CORRELATION)) or identifier, 200 causation_id=_text(raw.get(headers.CAUSATION)), 201 attempt=_number(raw.get(headers.ATTEMPT), 1), 202 first_seen=_from_millis(_number(first_seen, _millis(_now()))), 203 origin=_text(raw.get(headers.ORIGIN)), 204 error=_text(raw.get(headers.ERROR)), 205 claim=_text(raw.get(headers.CLAIM)), 206 route=_text(raw.get(headers.ROUTE)), 207 # A position that will not read is taken as the start rather than 208 # dropped: a route whose position is unreadable would otherwise send 209 # the message to whichever step happened to be first in the list. 210 route_position=_number(raw.get(headers.ROUTE_POSITION), 0), 211 route_id=_text(raw.get(headers.ROUTE_ID)), 212 headers=application, 213 )
The metadata AceMQ carries with every message.
Frozen, because an envelope describes a message that has already been
published or received: changing one in place would change what a log line
said after it was written. with_() returns a modified copy for the
cases where a new message derives from an old one.
Defaults are applied when the envelope is built rather than when it is
read, so two libraries reading the same message agree without having to
agree on a second set of rules. correlation_id defaults to id, an
attempt starts at 1, a version starts at 1, and type falls back to the
routing key the message was published under.
115 @property 116 def age(self) -> timedelta: 117 """How long since the message was first published. 118 119 The basis for giving up on age rather than on attempts, which is the 120 honest limit when a queue has been paused: a message can be on attempt 121 two and four days old. 122 """ 123 return _now() - self.first_seen
How long since the message was first published.
The basis for giving up on age rather than on attempts, which is the honest limit when a queue has been paused: a message can be on attempt two and four days old.
125 def with_(self, **changes: Any) -> Envelope: 126 """A copy with fields changed. 127 128 :param changes: the fields to replace 129 :returns: a new envelope 130 """ 131 return replace(self, **changes)
A copy with fields changed.
Parameters
- changes: the fields to replace :returns: a new envelope
133 def to_headers(self, routing_key: str = "") -> dict[str, Any]: 134 """The AMQP headers for this envelope. 135 136 :param routing_key: what the message is published under, used when no 137 type was given; when that is empty too, the type is the literal 138 ``message`` (AceMQ TCK requirement AMQP-ENV-006), rather than an 139 empty string no consumer can dispatch on 140 :returns: the reserved headers, plus the application's own 141 """ 142 written: dict[str, Any] = { 143 headers.ID: self.id, 144 headers.TYPE: self.type or routing_key or DEFAULT_TYPE, 145 headers.VERSION: self.version, 146 headers.CORRELATION: self.correlation_id or self.id, 147 headers.ATTEMPT: self.attempt, 148 headers.FIRST_SEEN: _millis(self.first_seen), 149 } 150 # Absent rather than empty. A header carrying "" is a header somebody 151 # has to write a special case for at the other end. 152 for name, value in ( 153 (headers.CAUSATION, self.causation_id), 154 (headers.ORIGIN, self.origin), 155 (headers.ERROR, self.error), 156 (headers.CLAIM, self.claim), 157 ): 158 if value: 159 written[name] = value 160 161 # The position is written whenever there is a route, including the zero 162 # a first hop carries: a route with no position reads as position zero 163 # everywhere, and leaving it off would make the first hop the only one 164 # whose slip is incomplete. 165 if self.route: 166 written[headers.ROUTE] = self.route 167 written[headers.ROUTE_POSITION] = self.route_position 168 if self.route_id: 169 written[headers.ROUTE_ID] = self.route_id 170 171 written.update(self.headers) 172 return written
The AMQP headers for this envelope.
Parameters
- routing_key: what the message is published under, used when no
type was given; when that is empty too, the type is the literal
message(AceMQ TCK requirement AMQP-ENV-006), rather than an empty string no consumer can dispatch on :returns: the reserved headers, plus the application's own
174 @classmethod 175 def from_headers(cls, raw: Mapping[str, Any] | None, routing_key: str = "") -> Envelope: 176 """Reads an envelope off a delivery. 177 178 Anything missing takes its default, and anything unreadable takes its 179 default too: a message from a producer that wrote ``x-acemq-attempt`` 180 as a string is still a message, and refusing to deliver it would hand 181 the application an outage rather than a message. 182 183 :param raw: the delivery's headers 184 :param routing_key: what it arrived on, used when no type was set 185 :returns: the envelope 186 """ 187 raw = raw or {} 188 application = { 189 name: value for name, value in raw.items() if not headers.is_reserved(name) 190 } 191 192 identifier = _text(raw.get(headers.ID)) or str(uuid.uuid4()) 193 first_seen = raw.get(headers.FIRST_SEEN) 194 195 return cls( 196 id=identifier, 197 type=_text(raw.get(headers.TYPE)) or routing_key or DEFAULT_TYPE, 198 version=_number(raw.get(headers.VERSION), 1), 199 correlation_id=_text(raw.get(headers.CORRELATION)) or identifier, 200 causation_id=_text(raw.get(headers.CAUSATION)), 201 attempt=_number(raw.get(headers.ATTEMPT), 1), 202 first_seen=_from_millis(_number(first_seen, _millis(_now()))), 203 origin=_text(raw.get(headers.ORIGIN)), 204 error=_text(raw.get(headers.ERROR)), 205 claim=_text(raw.get(headers.CLAIM)), 206 route=_text(raw.get(headers.ROUTE)), 207 # A position that will not read is taken as the start rather than 208 # dropped: a route whose position is unreadable would otherwise send 209 # the message to whichever step happened to be first in the list. 210 route_position=_number(raw.get(headers.ROUTE_POSITION), 0), 211 route_id=_text(raw.get(headers.ROUTE_ID)), 212 headers=application, 213 )
Reads an envelope off a delivery.
Anything missing takes its default, and anything unreadable takes its
default too: a message from a producer that wrote x-acemq-attempt
as a string is still a message, and refusing to deliver it would hand
the application an outage rather than a message.
Parameters
- raw: the delivery's headers
- routing_key: what it arrived on, used when no type was set :returns: the envelope
52@dataclass(frozen=True, slots=True) 53class ExchangeSpec: 54 """How an exchange should be declared. 55 56 :param kind: ``direct``, ``topic``, ``fanout`` or ``headers`` 57 :param durable: survives a broker restart 58 :param auto_delete: goes away when its last binding does 59 :param args: broker-specific arguments 60 """ 61 62 kind: str = "topic" 63 durable: bool = True 64 auto_delete: bool = False 65 args: Mapping[str, Any] = field(default_factory=dict)
How an exchange should be declared.
Parameters
- kind:
direct,topic,fanoutorheaders - durable: survives a broker restart
- auto_delete: goes away when its last binding does
- args: broker-specific arguments
199class FatalError(Exception): 200 """An error no number of retries will fix. 201 202 Wrapping a cause in this says "stop now" without the handler having to know 203 how many attempts remain, which is knowledge handlers should not need. 204 205 The other languages call this ``Fatal``; the name is Python's because a 206 class name is ergonomics rather than wire contract, and Python expects an 207 exception to say that it is one. 208 """
An error no number of retries will fix.
Wrapping a cause in this says "stop now" without the handler having to know how many attempts remain, which is knowledge handlers should not need.
The other languages call this Fatal; the name is Python's because a
class name is ergonomics rather than wire contract, and Python expects an
exception to say that it is one.
583@runtime_checkable 584class HealthCheck(Protocol): 585 """Anything that can report on itself. 586 587 The library provides one for the connection; an application adds its own — a 588 database, a downstream service — and :func:`aggregate_health` combines them. 589 """ 590 591 @property 592 def name(self) -> str: 593 """What this check is called in a combined report.""" 594 595 async def check(self) -> HealthReport: 596 """The current state. 597 598 It must return quickly and must not raise: a check that hangs makes a 599 readiness probe hang with it, and a probe that hangs is a pod that never 600 comes back. 601 """
Anything that can report on itself.
The library provides one for the connection; an application adds its own — a
database, a downstream service — and aggregate_health() combines them.
1771def _no_init_or_replace_init(self, *args, **kwargs): 1772 cls = type(self) 1773 1774 if cls._is_protocol: 1775 raise TypeError('Protocols cannot be instantiated') 1776 1777 # Already using a custom `__init__`. No need to calculate correct 1778 # `__init__` to call. This can lead to RecursionError. See bpo-45121. 1779 if cls.__init__ is not _no_init_or_replace_init: 1780 return 1781 1782 # Initially, `__init__` of a protocol subclass is set to `_no_init_or_replace_init`. 1783 # The first instantiation of the subclass will call `_no_init_or_replace_init` which 1784 # searches for a proper new `__init__` in the MRO. The new `__init__` 1785 # replaces the subclass' old `__init__` (ie `_no_init_or_replace_init`). Subsequent 1786 # instantiation of the protocol subclass will thus use the new 1787 # `__init__` and no longer call `_no_init_or_replace_init`. 1788 for base in cls.__mro__: 1789 init = base.__dict__.get('__init__', _no_init_or_replace_init) 1790 if init is not _no_init_or_replace_init: 1791 cls.__init__ = init 1792 break 1793 else: 1794 # should not happen 1795 cls.__init__ = object.__init__ 1796 1797 cls.__init__(self, *args, **kwargs)
595 async def check(self) -> HealthReport: 596 """The current state. 597 598 It must return quickly and must not raise: a check that hangs makes a 599 readiness probe hang with it, and a probe that hangs is a pod that never 600 comes back. 601 """
The current state.
It must return quickly and must not raise: a check that hangs makes a readiness probe hang with it, and a probe that hangs is a pod that never comes back.
553@dataclass(frozen=True, slots=True) 554class HealthReport: 555 """What a check found. 556 557 :param status: up, down or degraded 558 :param detail: why, when it is not simply up 559 :param checked: when this was worked out, in UTC 560 :param parts: whatever the check wants to show — a consumer count, a 561 round-trip time, the reports of the checks that were combined 562 """ 563 564 status: HealthStatus 565 detail: str = "" 566 checked: datetime = field(default_factory=lambda: datetime.now(timezone.utc)) 567 parts: Mapping[str, Any] = field(default_factory=dict) 568 569 @property 570 def healthy(self) -> bool: 571 """Whether a readiness probe should pass. 572 573 Degraded passes. An instance that is slow is still an instance that 574 works, and taking it out of rotation moves the traffic to one that will 575 be just as slow with more of it. 576 """ 577 return self.status is not HealthStatus.DOWN 578 579 def __str__(self) -> str: 580 return self.status.value if not self.detail else f"{self.status.value}: {self.detail}"
What a check found.
Parameters
- status: up, down or degraded
- detail: why, when it is not simply up
- checked: when this was worked out, in UTC
- parts: whatever the check wants to show — a consumer count, a round-trip time, the reports of the checks that were combined
569 @property 570 def healthy(self) -> bool: 571 """Whether a readiness probe should pass. 572 573 Degraded passes. An instance that is slow is still an instance that 574 works, and taking it out of rotation moves the traffic to one that will 575 be just as slow with more of it. 576 """ 577 return self.status is not HealthStatus.DOWN
Whether a readiness probe should pass.
Degraded passes. An instance that is slow is still an instance that works, and taking it out of rotation moves the traffic to one that will be just as slow with more of it.
538class HealthStatus(str, Enum): 539 """How a check turned out.""" 540 541 #: Working. 542 UP = "up" 543 544 #: Not working. A readiness probe should fail on this. 545 DOWN = "down" 546 547 #: Working, but not as well as it should. Worth an alert; not worth taking 548 #: the instance out of rotation, because the replacement will almost 549 #: certainly be degraded too. 550 DEGRADED = "degraded"
How a check turned out.
89class JsonCodec: 90 """Reads and writes JSON. The default, and the format every AceMQ library 91 has without an extra dependency. 92 93 Dataclasses are encoded through :func:`dataclasses.asdict`, so a payload 94 written as a dataclass goes on the wire as an object with its field names. 95 Everything else is handed to :mod:`json` unchanged, which covers dicts, 96 lists and the scalars. 97 98 Decoding produces dicts and lists rather than a class, because the wire says 99 nothing about which class was meant. An application that wants its own type 100 back constructs it from what it gets, where it can decide what a missing 101 field means. 102 """ 103 104 @property 105 def content_type(self) -> str: 106 return JSON_CONTENT_TYPE 107 108 def encode(self, payload: Any) -> bytes: 109 if dataclasses.is_dataclass(payload) and not isinstance(payload, type): 110 payload = dataclasses.asdict(payload) 111 try: 112 return json.dumps(payload).encode("utf-8") 113 except TypeError as failure: 114 raise TypeError( 115 f"acemq: a {type(payload).__name__} is not something JSON can carry: {failure}" 116 ) from failure 117 118 def decode(self, body: bytes, content_type: str | None = None) -> Any: 119 try: 120 return json.loads(body) 121 except (ValueError, UnicodeDecodeError) as failure: 122 # Fatal rather than retryable: a body that is not JSON is not JSON 123 # the next time either, and retrying it only holds a queue open 124 # until the message ages out. 125 raise FatalError(f"acemq: this message is not JSON: {failure}") from failure 126 127 def can_decode(self, content_type: str | None) -> bool: 128 """Accepts JSON, any ``+json`` media type, and a message with none. 129 130 Answering for a message whose sender set no content type is deliberate 131 and is what the other libraries do: JSON is the default format, so an 132 untyped message is far more likely to be JSON than anything else, and 133 something has to read it. 134 """ 135 if content_type is None or content_type == "": 136 return True 137 lowered = content_type.lower() 138 return ( 139 lowered.startswith(JSON_CONTENT_TYPE) 140 or lowered.startswith("text/json") 141 or "+json" in lowered 142 )
Reads and writes JSON. The default, and the format every AceMQ library has without an extra dependency.
Dataclasses are encoded through dataclasses.asdict(), so a payload
written as a dataclass goes on the wire as an object with its field names.
Everything else is handed to json unchanged, which covers dicts,
lists and the scalars.
Decoding produces dicts and lists rather than a class, because the wire says nothing about which class was meant. An application that wants its own type back constructs it from what it gets, where it can decide what a missing field means.
108 def encode(self, payload: Any) -> bytes: 109 if dataclasses.is_dataclass(payload) and not isinstance(payload, type): 110 payload = dataclasses.asdict(payload) 111 try: 112 return json.dumps(payload).encode("utf-8") 113 except TypeError as failure: 114 raise TypeError( 115 f"acemq: a {type(payload).__name__} is not something JSON can carry: {failure}" 116 ) from failure
118 def decode(self, body: bytes, content_type: str | None = None) -> Any: 119 try: 120 return json.loads(body) 121 except (ValueError, UnicodeDecodeError) as failure: 122 # Fatal rather than retryable: a body that is not JSON is not JSON 123 # the next time either, and retrying it only holds a queue open 124 # until the message ages out. 125 raise FatalError(f"acemq: this message is not JSON: {failure}") from failure
127 def can_decode(self, content_type: str | None) -> bool: 128 """Accepts JSON, any ``+json`` media type, and a message with none. 129 130 Answering for a message whose sender set no content type is deliberate 131 and is what the other libraries do: JSON is the default format, so an 132 untyped message is far more likely to be JSON than anything else, and 133 something has to read it. 134 """ 135 if content_type is None or content_type == "": 136 return True 137 lowered = content_type.lower() 138 return ( 139 lowered.startswith(JSON_CONTENT_TYPE) 140 or lowered.startswith("text/json") 141 or "+json" in lowered 142 )
Accepts JSON, any +json media type, and a message with none.
Answering for a message whose sender set no content type is deliberate and is what the other libraries do: JSON is the default format, so an untyped message is far more likely to be JSON than anything else, and something has to read it.
166@dataclass(frozen=True, slots=True) 167class Message: 168 """A delivery that has been decoded. 169 170 :param payload: the body, read through the codec 171 :param envelope: the metadata that travelled with it 172 :param routing_key: the key it arrived under 173 :param content_type: what the sender said the body was, or ``None`` 174 :param redelivered: the broker saying it has handed this one over before 175 :param body: the undecoded bytes, for a handler that wants to see them 176 :param reply_to: AMQP's own ``reply-to`` property, empty when the sender set 177 none. A responder reads it as the fallback behind the 178 ``acemq-reply-to`` header, which is what lets a Java or .NET caller — 179 neither of which writes the header — be answered by a Python responder 180 """ 181 182 payload: Any 183 envelope: Envelope 184 routing_key: str 185 content_type: str | None 186 redelivered: bool 187 body: bytes 188 reply_to: str = ""
A delivery that has been decoded.
Parameters
- payload: the body, read through the codec
- envelope: the metadata that travelled with it
- routing_key: the key it arrived under
- content_type: what the sender said the body was, or
None - redelivered: the broker saying it has handed this one over before
- body: the undecoded bytes, for a handler that wants to see them
- reply_to: AMQP's own
reply-toproperty, empty when the sender set none. A responder reads it as the fallback behind theacemq-reply-toheader, which is what lets a Java or .NET caller — neither of which writes the header — be answered by a Python responder
370class Metrics: 371 """An :class:`Observer` that keeps the numbers in memory. 372 373 Enough to serve from a health endpoint, assert on in a test, or print on a 374 signal. It is not a substitute for a real metrics system: no histograms, no 375 percentiles, and nothing is exported anywhere. :func:`prometheus_text` 376 renders what it has for a scraper, which needs no dependency at all. 377 378 Locked rather than relying on the event loop, because the blocking API in 379 :mod:`acemq_amqp.sync` runs handlers on a thread pool and a counter read 380 from two threads at once is a counter that loses increments. 381 """ 382 383 def __init__(self) -> None: 384 self._lock = threading.Lock() 385 self._counts: dict[str, int] = {} 386 self._gauges: dict[str, int] = {} 387 self._durations: dict[str, DurationSummary] = {} 388 389 def count(self, metric: str, delta: int, labels: Mapping[str, str]) -> None: 390 key = metric_key(metric, labels) 391 with self._lock: 392 self._counts[key] = self._counts.get(key, 0) + delta 393 394 def gauge(self, metric: str, value: int, labels: Mapping[str, str]) -> None: 395 key = metric_key(metric, labels) 396 with self._lock: 397 self._gauges[key] = value 398 399 def observe(self, metric: str, seconds: float, labels: Mapping[str, str]) -> None: 400 key = metric_key(metric, labels) 401 with self._lock: 402 existing = self._durations.get(key) 403 if existing is None: 404 self._durations[key] = DurationSummary(1, seconds, seconds, seconds) 405 return 406 self._durations[key] = DurationSummary( 407 count=existing.count + 1, 408 total=existing.total + seconds, 409 fastest=min(existing.fastest, seconds), 410 slowest=max(existing.slowest, seconds), 411 ) 412 413 @property 414 def counts(self) -> dict[str, int]: 415 """Every counter, keyed by metric and labels.""" 416 with self._lock: 417 return dict(self._counts) 418 419 @property 420 def gauges(self) -> dict[str, int]: 421 """Every gauge.""" 422 with self._lock: 423 return dict(self._gauges) 424 425 @property 426 def durations(self) -> dict[str, DurationSummary]: 427 """A summary per timing: how many, the total, the fastest, the slowest.""" 428 with self._lock: 429 return dict(self._durations) 430 431 def __str__(self) -> str: 432 lines = [f"{name} {value}" for name, value in sorted(self.counts.items())] 433 lines += [f"{name} {value}" for name, value in sorted(self.gauges.items())] 434 lines += [ 435 f"{name} count={summary.count} mean={summary.mean:.4f}s" 436 for name, summary in sorted(self.durations.items()) 437 ] 438 return "\n".join(lines) or "nothing recorded yet"
An Observer that keeps the numbers in memory.
Enough to serve from a health endpoint, assert on in a test, or print on a
signal. It is not a substitute for a real metrics system: no histograms, no
percentiles, and nothing is exported anywhere. prometheus_text()
renders what it has for a scraper, which needs no dependency at all.
Locked rather than relying on the event loop, because the blocking API in
acemq_amqp.sync runs handlers on a thread pool and a counter read
from two threads at once is a counter that loses increments.
399 def observe(self, metric: str, seconds: float, labels: Mapping[str, str]) -> None: 400 key = metric_key(metric, labels) 401 with self._lock: 402 existing = self._durations.get(key) 403 if existing is None: 404 self._durations[key] = DurationSummary(1, seconds, seconds, seconds) 405 return 406 self._durations[key] = DurationSummary( 407 count=existing.count + 1, 408 total=existing.total + seconds, 409 fastest=min(existing.fastest, seconds), 410 slowest=max(existing.slowest, seconds), 411 )
413 @property 414 def counts(self) -> dict[str, int]: 415 """Every counter, keyed by metric and labels.""" 416 with self._lock: 417 return dict(self._counts)
Every counter, keyed by metric and labels.
419 @property 420 def gauges(self) -> dict[str, int]: 421 """Every gauge.""" 422 with self._lock: 423 return dict(self._gauges)
Every gauge.
425 @property 426 def durations(self) -> dict[str, DurationSummary]: 427 """A summary per timing: how many, the total, the fastest, the slowest.""" 428 with self._lock: 429 return dict(self._durations)
A summary per timing: how many, the total, the fastest, the slowest.
332class NullObserver: 333 """An observer that does nothing, which is what a connection has by default. 334 335 A real object rather than ``None`` checked for at each call site, because 336 the alternative is a branch on every publish and every delivery, and one of 337 them is eventually written the wrong way round. 338 """ 339 340 def count(self, metric: str, delta: int, labels: Mapping[str, str]) -> None: 341 return None 342 343 def observe(self, metric: str, seconds: float, labels: Mapping[str, str]) -> None: 344 return None 345 346 def gauge(self, metric: str, value: int, labels: Mapping[str, str]) -> None: 347 return None
An observer that does nothing, which is what a connection has by default.
A real object rather than None checked for at each call site, because
the alternative is a branch on every publish and every delivery, and one of
them is eventually written the wrong way round.
292@runtime_checkable 293class Observer(Protocol): 294 """Told what the library is doing. 295 296 Every method is called on the path a message takes, so none of them may 297 block and all of them have to be safe to call from several tasks at once. 298 An observer that talks to the network on each call is an observer that makes 299 every publish as slow as the network it talks to; buffer, and flush 300 somewhere else. 301 302 Nothing here is awaitable, deliberately. A metrics call that could suspend 303 would reorder the code around it and turn "count this" into a scheduling 304 decision, and no metrics library needs it. 305 """ 306 307 def count(self, metric: str, delta: int, labels: Mapping[str, str]) -> None: 308 """Adds to a counter.""" 309 310 def observe(self, metric: str, seconds: float, labels: Mapping[str, str]) -> None: 311 """Records one sample of a distribution. 312 313 A duration in seconds for every metric but one. The exception is 314 :data:`METRIC_CONSUME_ATTEMPTS`, which is a count of attempts and is 315 recorded here because this is the only distribution the interface has — 316 Java uses a Micrometer ``DistributionSummary`` for it and a ``Timer`` for 317 the rest, and this interface has one method where Java has two. 318 319 The argument keeps its name because renaming it would break every 320 observer that passes it by keyword, and because it is a duration in every 321 case but that one. An implementation that puts durations into histogram 322 buckets should give that metric its own: seconds-shaped buckets over a 323 value of 1, 2 or 3 report an attempt count as a handful of very fast 324 handlers. :class:`~acemq_amqp.prometheus.PrometheusObserver` does exactly 325 that, with :data:`~acemq_amqp.prometheus.DEFAULT_ATTEMPT_BUCKETS`. 326 """ 327 328 def gauge(self, metric: str, value: int, labels: Mapping[str, str]) -> None: 329 """Sets a current value."""
Told what the library is doing.
Every method is called on the path a message takes, so none of them may block and all of them have to be safe to call from several tasks at once. An observer that talks to the network on each call is an observer that makes every publish as slow as the network it talks to; buffer, and flush somewhere else.
Nothing here is awaitable, deliberately. A metrics call that could suspend would reorder the code around it and turn "count this" into a scheduling decision, and no metrics library needs it.
1771def _no_init_or_replace_init(self, *args, **kwargs): 1772 cls = type(self) 1773 1774 if cls._is_protocol: 1775 raise TypeError('Protocols cannot be instantiated') 1776 1777 # Already using a custom `__init__`. No need to calculate correct 1778 # `__init__` to call. This can lead to RecursionError. See bpo-45121. 1779 if cls.__init__ is not _no_init_or_replace_init: 1780 return 1781 1782 # Initially, `__init__` of a protocol subclass is set to `_no_init_or_replace_init`. 1783 # The first instantiation of the subclass will call `_no_init_or_replace_init` which 1784 # searches for a proper new `__init__` in the MRO. The new `__init__` 1785 # replaces the subclass' old `__init__` (ie `_no_init_or_replace_init`). Subsequent 1786 # instantiation of the protocol subclass will thus use the new 1787 # `__init__` and no longer call `_no_init_or_replace_init`. 1788 for base in cls.__mro__: 1789 init = base.__dict__.get('__init__', _no_init_or_replace_init) 1790 if init is not _no_init_or_replace_init: 1791 cls.__init__ = init 1792 break 1793 else: 1794 # should not happen 1795 cls.__init__ = object.__init__ 1796 1797 cls.__init__(self, *args, **kwargs)
307 def count(self, metric: str, delta: int, labels: Mapping[str, str]) -> None: 308 """Adds to a counter."""
Adds to a counter.
310 def observe(self, metric: str, seconds: float, labels: Mapping[str, str]) -> None: 311 """Records one sample of a distribution. 312 313 A duration in seconds for every metric but one. The exception is 314 :data:`METRIC_CONSUME_ATTEMPTS`, which is a count of attempts and is 315 recorded here because this is the only distribution the interface has — 316 Java uses a Micrometer ``DistributionSummary`` for it and a ``Timer`` for 317 the rest, and this interface has one method where Java has two. 318 319 The argument keeps its name because renaming it would break every 320 observer that passes it by keyword, and because it is a duration in every 321 case but that one. An implementation that puts durations into histogram 322 buckets should give that metric its own: seconds-shaped buckets over a 323 value of 1, 2 or 3 report an attempt count as a handful of very fast 324 handlers. :class:`~acemq_amqp.prometheus.PrometheusObserver` does exactly 325 that, with :data:`~acemq_amqp.prometheus.DEFAULT_ATTEMPT_BUCKETS`. 326 """
Records one sample of a distribution.
A duration in seconds for every metric but one. The exception is
METRIC_CONSUME_ATTEMPTS, which is a count of attempts and is
recorded here because this is the only distribution the interface has —
Java uses a Micrometer DistributionSummary for it and a Timer for
the rest, and this interface has one method where Java has two.
The argument keeps its name because renaming it would break every
observer that passes it by keyword, and because it is a duration in every
case but that one. An implementation that puts durations into histogram
buckets should give that metric its own: seconds-shaped buckets over a
value of 1, 2 or 3 report an attempt count as a handful of very fast
handlers. ~acemq_amqp.prometheus.PrometheusObserver does exactly
that, with ~acemq_amqp.prometheus.DEFAULT_ATTEMPT_BUCKETS.
85@dataclass(frozen=True, slots=True) 86class Outbound: 87 """A message on its way to the broker, already encoded. 88 89 :param body: the encoded payload 90 :param content_type: what the body is, for the consumer's codec 91 :param message_id: the AMQP message id, conventionally the envelope's 92 :param reply_to: AMQP's own ``reply-to`` property, naming where an answer 93 should be sent. Empty for the overwhelming majority of messages, which 94 are not questions. It is here as well as in the headers because the four 95 other libraries read the native property, and a Python responder that 96 only ever wrote a header could not be answered by a Java or .NET caller 97 :param headers: the envelope's headers plus the application's 98 :param persistent: asks the broker to write the message to disk, which is 99 not a guarantee on its own — a persistent message on a queue that is not 100 durable still dies with the broker 101 :param mandatory: asks the broker to return the message rather than drop it 102 when no queue is bound to receive it 103 """ 104 105 body: bytes 106 content_type: str = "" 107 message_id: str = "" 108 reply_to: str = "" 109 headers: Mapping[str, Any] = field(default_factory=dict) 110 persistent: bool = True 111 mandatory: bool = False
A message on its way to the broker, already encoded.
Parameters
- body: the encoded payload
- content_type: what the body is, for the consumer's codec
- message_id: the AMQP message id, conventionally the envelope's
- reply_to: AMQP's own
reply-toproperty, naming where an answer should be sent. Empty for the overwhelming majority of messages, which are not questions. It is here as well as in the headers because the four other libraries read the native property, and a Python responder that only ever wrote a header could not be answered by a Java or .NET caller - headers: the envelope's headers plus the application's
- persistent: asks the broker to write the message to disk, which is not a guarantee on its own — a persistent message on a queue that is not durable still dies with the broker
- mandatory: asks the broker to return the message rather than drop it when no queue is bound to receive it
86@dataclass(slots=True) 87class PublishContext: 88 """A message on its way out, before it is encoded. 89 90 Mutable on purpose, and mutable rather than rebuilt-and-returned: an 91 interceptor that only wants to add one header should not have to reconstruct 92 the rest, and one that reads what an earlier interceptor wrote needs to be 93 looking at the same object rather than at a copy of an earlier version. 94 95 :param exchange: where it is going. Changing it redirects the message 96 :param routing_key: what it is published under 97 :param envelope: the metadata. Reserved header names are still refused when 98 the envelope is rendered, whoever wrote them 99 :param payload: what is about to be encoded, before the codec sees it, so an 100 interceptor can change the message and not only its metadata 101 :param persistent: whether the broker is asked to write it to disk 102 :param mandatory: whether reaching no queue is an error rather than a 103 silence 104 :param reply_to: where an answer should be sent, written to AMQP's own 105 ``reply-to`` property. Empty for anything that is not a question 106 """ 107 108 exchange: str 109 routing_key: str 110 envelope: Envelope 111 payload: Any 112 persistent: bool = True 113 mandatory: bool = False 114 reply_to: str = "" 115 116 def set_header(self, name: str, value: Any) -> None: 117 """Adds an application header to the message being published. 118 119 A convenience over rebuilding the envelope, which is what the three 120 commonest interceptors — a tenant, a trace context, a schema version — 121 all want to do and nothing else. 122 """ 123 self.envelope = self.envelope.with_(headers={**self.envelope.headers, name: value})
A message on its way out, before it is encoded.
Mutable on purpose, and mutable rather than rebuilt-and-returned: an interceptor that only wants to add one header should not have to reconstruct the rest, and one that reads what an earlier interceptor wrote needs to be looking at the same object rather than at a copy of an earlier version.
Parameters
- exchange: where it is going. Changing it redirects the message
- routing_key: what it is published under
- envelope: the metadata. Reserved header names are still refused when the envelope is rendered, whoever wrote them
- payload: what is about to be encoded, before the codec sees it, so an interceptor can change the message and not only its metadata
- persistent: whether the broker is asked to write it to disk
- mandatory: whether reaching no queue is an error rather than a silence
- reply_to: where an answer should be sent, written to AMQP's own
reply-toproperty. Empty for anything that is not a question
116 def set_header(self, name: str, value: Any) -> None: 117 """Adds an application header to the message being published. 118 119 A convenience over rebuilding the envelope, which is what the three 120 commonest interceptors — a tenant, a trace context, a schema version — 121 all want to do and nothing else. 122 """ 123 self.envelope = self.envelope.with_(headers={**self.envelope.headers, name: value})
Adds an application header to the message being published.
A convenience over rebuilding the envelope, which is what the three commonest interceptors — a tenant, a trace context, a schema version — all want to do and nothing else.
46class PublishError(AceMQError): 47 """A message the broker would not take, or would not route. 48 49 It carries where the message was going, which is the first thing anybody 50 wants when a routing key was built from a variable. 51 52 :param message_id: the envelope identifier it went out with 53 :param exchange: where it was sent 54 :param routing_key: what it was sent under 55 :param reason: the broker's explanation 56 :param unroutable: true when the message reached no queue rather than being 57 refused. The broker was working; nothing was listening 58 :param summary: the whole message, in place of the sentence composed from 59 the arguments above. For a failure that is about a batch rather than 60 about one message — :meth:`acemq_amqp.Publisher.send_all` raises one of 61 these naming how many messages were not confirmed and how many were, 62 in the words Java and .NET use for the same failure, so that one 63 runbook covers all of them. A batch failure leaves ``message_id`` 64 empty, because there is no single message it is about 65 """ 66 67 def __init__( 68 self, 69 message_id: str, 70 exchange: str, 71 routing_key: str, 72 reason: str, 73 *, 74 unroutable: bool = False, 75 summary: str = "", 76 ) -> None: 77 self.message_id = message_id 78 self.exchange = exchange 79 self.routing_key = routing_key 80 self.reason = reason 81 self.unroutable = unroutable 82 83 where = f"exchange {exchange!r} with key {routing_key!r}" 84 if summary: 85 super().__init__(summary) 86 elif unroutable: 87 super().__init__( 88 f"acemq: message {message_id} to {where} reached no queue: {reason}" 89 ) 90 else: 91 super().__init__(f"acemq: cannot publish message {message_id} to {where}: {reason}")
A message the broker would not take, or would not route.
It carries where the message was going, which is the first thing anybody wants when a routing key was built from a variable.
Parameters
- message_id: the envelope identifier it went out with
- exchange: where it was sent
- routing_key: what it was sent under
- reason: the broker's explanation
- unroutable: true when the message reached no queue rather than being refused. The broker was working; nothing was listening
- summary: the whole message, in place of the sentence composed from
the arguments above. For a failure that is about a batch rather than
about one message —
acemq_amqp.Publisher.send_all()raises one of these naming how many messages were not confirmed and how many were, in the words Java and .NET use for the same failure, so that one runbook covers all of them. A batch failure leavesmessage_idempty, because there is no single message it is about
67 def __init__( 68 self, 69 message_id: str, 70 exchange: str, 71 routing_key: str, 72 reason: str, 73 *, 74 unroutable: bool = False, 75 summary: str = "", 76 ) -> None: 77 self.message_id = message_id 78 self.exchange = exchange 79 self.routing_key = routing_key 80 self.reason = reason 81 self.unroutable = unroutable 82 83 where = f"exchange {exchange!r} with key {routing_key!r}" 84 if summary: 85 super().__init__(summary) 86 elif unroutable: 87 super().__init__( 88 f"acemq: message {message_id} to {where} reached no queue: {reason}" 89 ) 90 else: 91 super().__init__(f"acemq: cannot publish message {message_id} to {where}: {reason}")
114@dataclass(frozen=True, slots=True) 115class PublishResult: 116 """What the broker said about a published message. 117 118 :param message_id: the identifier it went out with 119 :param confirmed: the broker taking responsibility for the message. Without 120 publisher confirms this is false and nothing has been promised: the 121 message reached a socket, which is not the same thing 122 :param routed: false when a mandatory message reached no queue at all. A 123 message nothing is bound to receive is not an error to the broker; it is 124 dropped, silently, which is exactly the failure worth hearing about 125 :param return_reason: the broker's explanation when ``routed`` is false 126 """ 127 128 message_id: str = "" 129 confirmed: bool = False 130 routed: bool = True 131 return_reason: str = ""
What the broker said about a published message.
Parameters
- message_id: the identifier it went out with
- confirmed: the broker taking responsibility for the message. Without publisher confirms this is false and nothing has been promised: the message reached a socket, which is not the same thing
- routed: false when a mandatory message reached no queue at all. A message nothing is bound to receive is not an error to the broker; it is dropped, silently, which is exactly the failure worth hearing about
- return_reason: the broker's explanation when
routedis false
208class Publisher: 209 """Sends messages to one destination. 210 211 Cheap to keep and safe to share: build one per message type at start-up 212 rather than one per message. 213 """ 214 215 def __init__( 216 self, 217 connection: Connection, 218 exchange: str, 219 routing_key: str, 220 *, 221 codec: Codec | None = None, 222 persistent: bool = True, 223 mandatory: bool = False, 224 ) -> None: 225 self._connection = connection 226 self._exchange = exchange 227 self._routing_key = routing_key 228 self._codec = codec or connection.codec 229 self._persistent = persistent 230 self._mandatory = mandatory 231 232 async def send( 233 self, 234 payload: Any, 235 *, 236 envelope: Envelope | None = None, 237 routing_key: str | None = None, 238 reply_to: str = "", 239 ) -> PublishResult: 240 """Publishes one message. 241 242 The envelope is built here when none is given: an identifier, a 243 correlation defaulting to it, this connection's origin, and now. The 244 type falls back to the routing key, which is why a publisher bound to 245 ``order.placed`` needs no further configuration to say what it sends. 246 247 :param payload: what to send, encoded by this publisher's codec 248 :param envelope: metadata to send instead of a fresh one, for a message 249 derived from another 250 :param routing_key: what to publish under, for a publisher bound to an 251 exchange rather than to one key 252 :param reply_to: where an answer should come back to, written to AMQP's 253 own ``reply-to`` property. Only :mod:`~acemq_amqp.patterns` request 254 and reply sets it, and it sets the ``acemq-reply-to`` header to the 255 same value: the header is what survives a service that rebuilds the 256 message, the property is what the other four libraries read 257 :returns: what the broker said 258 :raises PublishError: when the publisher is mandatory and the message 259 reached no queue 260 """ 261 key = self._routing_key if routing_key is None else routing_key 262 outgoing = envelope or Envelope(origin=self._connection.origin) 263 if not outgoing.origin: 264 # A message derived from another — a reply, a pipeline hop, the next 265 # stop on a routing slip — is still published by this process, and an 266 # origin naming the service before it would be a lie in a log line 267 # somebody is going to trust. 268 outgoing = outgoing.with_(origin=self._connection.origin) 269 270 context = PublishContext( 271 exchange=self._exchange, 272 routing_key=key, 273 envelope=outgoing, 274 payload=payload, 275 persistent=self._persistent, 276 mandatory=self._mandatory, 277 reply_to=reply_to, 278 ) 279 # Read here rather than at construction, so an interceptor registered 280 # during start-up applies to the publishers that already exist. Building 281 # the chain per message costs a few closures and buys that. 282 return await publish_chain(self._connection.publish_interceptors, self._send)(context) 283 284 async def send_all(self, payloads: Iterable[Any]) -> list[PublishResult]: 285 """Publishes a batch and waits for every confirm. 286 287 What most bulk publishing actually wants: the throughput of pipelining 288 with the safety of having waited. Every message goes out before any 289 confirm is awaited, and only then are all of them checked together. 290 Awaiting each send in turn is a broker round trip per message — the 291 loop a caller could have written for themselves, and none of the 292 throughput a batch exists for. 293 294 The results come back in the order the payloads were given, whatever 295 order the broker answers in. 296 297 **This is not atomic.** AMQP has no such thing: there is no way to 298 publish a hundred messages such that all or none arrive, and a library 299 that offered one would be lying. A failure does not abandon the rest 300 either — every send is awaited whatever an earlier one did, so a batch 301 that half succeeded can say how many arrived. That count is the point: 302 a caller told only "it failed" resends the messages the broker already 303 has. 304 305 How wide the batch gets on the wire is capped, and by the connection 306 rather than by this method: 307 :attr:`Connection.max_outstanding_publishes` is a thousand by default, 308 and the thousand-and-first task waits for one of the first thousand to 309 be answered before its message is written. A list of a million is 310 therefore a million tasks and a thousand messages in flight rather than 311 a million, which is backpressure instead of a memory leak — but a 312 million tasks is still a million tasks, so chunk what comes out of a 313 database rather than handing a cursor's worth of rows to one call. 314 315 :param payloads: what to send, in order, each encoded by this 316 publisher's codec 317 :returns: what the broker said about each, in the order the payloads 318 were given 319 :raises PublishError: when any message was not confirmed. Its message 320 names how many failed and how many did not, and ``__cause__`` 321 carries the first failure in payload order — which is not 322 necessarily the first one the broker answered. A batch wide enough 323 that the broker stops keeping up fails this way too, with the first 324 failure saying so 325 """ 326 batch = list(payloads) 327 # Everything is started before anything is awaited. A task per payload, 328 # all of them created before the gather below, is what puts the whole 329 # batch on the wire while the broker is still working through the 330 # first confirm. 331 sends = [asyncio.create_task(self.send(payload)) for payload in batch] 332 # return_exceptions, and not for tidiness: without it the first failure 333 # comes out of the gather while the rest of the batch is still in 334 # flight, which loses the count this method exists to report and leaves 335 # the others' exceptions unretrieved for asyncio to complain about 336 # later. 337 settled = await asyncio.gather(*sends, return_exceptions=True) 338 339 results: list[PublishResult] = [] 340 first_failure: BaseException | None = None 341 failed = 0 342 for outcome in settled: 343 if isinstance(outcome, BaseException): 344 failed += 1 345 if first_failure is None: 346 first_failure = outcome 347 else: 348 results.append(outcome) 349 350 if first_failure is not None: 351 # Named rather than printed. A failure with nothing in its message — 352 # the CancelledError every task gets when the connection closes 353 # underneath the batch is the common one — renders as an empty 354 # string, and the sentence then ends "The first failure was: " and 355 # stops, promising an explanation it does not give. The class name 356 # is not much, but it is the difference between "the sends were 357 # cancelled" and nothing at all. 358 described = str(first_failure) or type(first_failure).__name__ 359 # The counts matter. A batch that half succeeded is the ordinary 360 # outcome of a broker problem partway through, and this sentence is 361 # word for word the one Java and .NET raise, so that an operator 362 # reading a log has one thing to recognise rather than three. 363 raise PublishError( 364 "", 365 self._exchange, 366 self._routing_key, 367 described, 368 summary=( 369 f"{failed} of {len(batch)} messages were not confirmed;" 370 f" {len(results)} were. The first failure was: {described}" 371 ), 372 ) from first_failure 373 return results 374 375 async def _send(self, context: PublishContext) -> PublishResult: 376 """The innermost work: encode what the interceptors left, and send it.""" 377 observer = self._connection.observer 378 # ``routing.key`` and not ``key``: Java and .NET both tag a publish with 379 # the fully-qualified name, and a dashboard that groups by it should 380 # read the same in all five languages. 381 labels = {TAG_EXCHANGE: context.exchange, TAG_ROUTING_KEY: context.routing_key} 382 # Started before the encode, because everything from here on is time the 383 # caller spends inside send(): the codec, the wait for a permit and the 384 # broker's confirm are all things that make a publish slow, and a timer 385 # that began after the encode would hide a slow codec completely. 386 started = time.monotonic() 387 388 def record(outcome: str) -> None: 389 """The pair, always together and always with the same labels. 390 391 Two calls rather than one because the interface has two methods, and 392 in this order because a dashboard dividing the duration's count into 393 the total should never see the total lead it. 394 """ 395 observer.observe( 396 METRIC_PUBLISH_DURATION, 397 time.monotonic() - started, 398 {**labels, TAG_OUTCOME: outcome}, 399 ) 400 observer.count(METRIC_PUBLISH_TOTAL, 1, {**labels, TAG_OUTCOME: outcome}) 401 402 # Encoded here rather than before the chain ran, so an interceptor can 403 # change the payload as well as its metadata. A codec that has already 404 # run leaves an interceptor with bytes and nothing it can do to them. 405 try: 406 body = self._codec.encode(context.payload) 407 result = await self._connection.publish_raw( 408 context.exchange, 409 context.routing_key, 410 Outbound( 411 body=body, 412 content_type=self._codec.content_type, 413 message_id=context.envelope.id, 414 reply_to=context.reply_to, 415 headers=context.envelope.to_headers(routing_key=context.routing_key), 416 persistent=context.persistent, 417 mandatory=context.mandatory, 418 ), 419 ) 420 except PublishingPausedError: 421 # Declined before anything was written: nothing lost, so not 422 # ``failed``, which is the outcome an alert on loss watches. 423 record(OUTCOME_REFUSED) 424 raise 425 except Exception: 426 record(OUTCOME_FAILED) 427 raise 428 429 # Raised rather than left in the result, because a caller who does not 430 # read the result would otherwise carry on believing the message went 431 # somewhere. Unroutable is the quietest failure AMQP has: the publish 432 # succeeds, the consumer waits, and nothing anywhere says why. 433 if context.mandatory and not result.routed: 434 record(OUTCOME_UNROUTABLE) 435 raise PublishError( 436 context.envelope.id, 437 context.exchange, 438 context.routing_key, 439 result.return_reason or "no queue is bound to receive it", 440 unroutable=True, 441 ) 442 443 record(OUTCOME_CONFIRMED) 444 return result
Sends messages to one destination.
Cheap to keep and safe to share: build one per message type at start-up rather than one per message.
215 def __init__( 216 self, 217 connection: Connection, 218 exchange: str, 219 routing_key: str, 220 *, 221 codec: Codec | None = None, 222 persistent: bool = True, 223 mandatory: bool = False, 224 ) -> None: 225 self._connection = connection 226 self._exchange = exchange 227 self._routing_key = routing_key 228 self._codec = codec or connection.codec 229 self._persistent = persistent 230 self._mandatory = mandatory
232 async def send( 233 self, 234 payload: Any, 235 *, 236 envelope: Envelope | None = None, 237 routing_key: str | None = None, 238 reply_to: str = "", 239 ) -> PublishResult: 240 """Publishes one message. 241 242 The envelope is built here when none is given: an identifier, a 243 correlation defaulting to it, this connection's origin, and now. The 244 type falls back to the routing key, which is why a publisher bound to 245 ``order.placed`` needs no further configuration to say what it sends. 246 247 :param payload: what to send, encoded by this publisher's codec 248 :param envelope: metadata to send instead of a fresh one, for a message 249 derived from another 250 :param routing_key: what to publish under, for a publisher bound to an 251 exchange rather than to one key 252 :param reply_to: where an answer should come back to, written to AMQP's 253 own ``reply-to`` property. Only :mod:`~acemq_amqp.patterns` request 254 and reply sets it, and it sets the ``acemq-reply-to`` header to the 255 same value: the header is what survives a service that rebuilds the 256 message, the property is what the other four libraries read 257 :returns: what the broker said 258 :raises PublishError: when the publisher is mandatory and the message 259 reached no queue 260 """ 261 key = self._routing_key if routing_key is None else routing_key 262 outgoing = envelope or Envelope(origin=self._connection.origin) 263 if not outgoing.origin: 264 # A message derived from another — a reply, a pipeline hop, the next 265 # stop on a routing slip — is still published by this process, and an 266 # origin naming the service before it would be a lie in a log line 267 # somebody is going to trust. 268 outgoing = outgoing.with_(origin=self._connection.origin) 269 270 context = PublishContext( 271 exchange=self._exchange, 272 routing_key=key, 273 envelope=outgoing, 274 payload=payload, 275 persistent=self._persistent, 276 mandatory=self._mandatory, 277 reply_to=reply_to, 278 ) 279 # Read here rather than at construction, so an interceptor registered 280 # during start-up applies to the publishers that already exist. Building 281 # the chain per message costs a few closures and buys that. 282 return await publish_chain(self._connection.publish_interceptors, self._send)(context)
Publishes one message.
The envelope is built here when none is given: an identifier, a
correlation defaulting to it, this connection's origin, and now. The
type falls back to the routing key, which is why a publisher bound to
order.placed needs no further configuration to say what it sends.
Parameters
- payload: what to send, encoded by this publisher's codec
- envelope: metadata to send instead of a fresh one, for a message derived from another
- routing_key: what to publish under, for a publisher bound to an exchange rather than to one key
- reply_to: where an answer should come back to, written to AMQP's
own
reply-toproperty. Only~acemq_amqp.patternsrequest and reply sets it, and it sets theacemq-reply-toheader to the same value: the header is what survives a service that rebuilds the message, the property is what the other four libraries read :returns: what the broker said
Raises
- PublishError: when the publisher is mandatory and the message reached no queue
284 async def send_all(self, payloads: Iterable[Any]) -> list[PublishResult]: 285 """Publishes a batch and waits for every confirm. 286 287 What most bulk publishing actually wants: the throughput of pipelining 288 with the safety of having waited. Every message goes out before any 289 confirm is awaited, and only then are all of them checked together. 290 Awaiting each send in turn is a broker round trip per message — the 291 loop a caller could have written for themselves, and none of the 292 throughput a batch exists for. 293 294 The results come back in the order the payloads were given, whatever 295 order the broker answers in. 296 297 **This is not atomic.** AMQP has no such thing: there is no way to 298 publish a hundred messages such that all or none arrive, and a library 299 that offered one would be lying. A failure does not abandon the rest 300 either — every send is awaited whatever an earlier one did, so a batch 301 that half succeeded can say how many arrived. That count is the point: 302 a caller told only "it failed" resends the messages the broker already 303 has. 304 305 How wide the batch gets on the wire is capped, and by the connection 306 rather than by this method: 307 :attr:`Connection.max_outstanding_publishes` is a thousand by default, 308 and the thousand-and-first task waits for one of the first thousand to 309 be answered before its message is written. A list of a million is 310 therefore a million tasks and a thousand messages in flight rather than 311 a million, which is backpressure instead of a memory leak — but a 312 million tasks is still a million tasks, so chunk what comes out of a 313 database rather than handing a cursor's worth of rows to one call. 314 315 :param payloads: what to send, in order, each encoded by this 316 publisher's codec 317 :returns: what the broker said about each, in the order the payloads 318 were given 319 :raises PublishError: when any message was not confirmed. Its message 320 names how many failed and how many did not, and ``__cause__`` 321 carries the first failure in payload order — which is not 322 necessarily the first one the broker answered. A batch wide enough 323 that the broker stops keeping up fails this way too, with the first 324 failure saying so 325 """ 326 batch = list(payloads) 327 # Everything is started before anything is awaited. A task per payload, 328 # all of them created before the gather below, is what puts the whole 329 # batch on the wire while the broker is still working through the 330 # first confirm. 331 sends = [asyncio.create_task(self.send(payload)) for payload in batch] 332 # return_exceptions, and not for tidiness: without it the first failure 333 # comes out of the gather while the rest of the batch is still in 334 # flight, which loses the count this method exists to report and leaves 335 # the others' exceptions unretrieved for asyncio to complain about 336 # later. 337 settled = await asyncio.gather(*sends, return_exceptions=True) 338 339 results: list[PublishResult] = [] 340 first_failure: BaseException | None = None 341 failed = 0 342 for outcome in settled: 343 if isinstance(outcome, BaseException): 344 failed += 1 345 if first_failure is None: 346 first_failure = outcome 347 else: 348 results.append(outcome) 349 350 if first_failure is not None: 351 # Named rather than printed. A failure with nothing in its message — 352 # the CancelledError every task gets when the connection closes 353 # underneath the batch is the common one — renders as an empty 354 # string, and the sentence then ends "The first failure was: " and 355 # stops, promising an explanation it does not give. The class name 356 # is not much, but it is the difference between "the sends were 357 # cancelled" and nothing at all. 358 described = str(first_failure) or type(first_failure).__name__ 359 # The counts matter. A batch that half succeeded is the ordinary 360 # outcome of a broker problem partway through, and this sentence is 361 # word for word the one Java and .NET raise, so that an operator 362 # reading a log has one thing to recognise rather than three. 363 raise PublishError( 364 "", 365 self._exchange, 366 self._routing_key, 367 described, 368 summary=( 369 f"{failed} of {len(batch)} messages were not confirmed;" 370 f" {len(results)} were. The first failure was: {described}" 371 ), 372 ) from first_failure 373 return results
Publishes a batch and waits for every confirm.
What most bulk publishing actually wants: the throughput of pipelining with the safety of having waited. Every message goes out before any confirm is awaited, and only then are all of them checked together. Awaiting each send in turn is a broker round trip per message — the loop a caller could have written for themselves, and none of the throughput a batch exists for.
The results come back in the order the payloads were given, whatever order the broker answers in.
This is not atomic. AMQP has no such thing: there is no way to publish a hundred messages such that all or none arrive, and a library that offered one would be lying. A failure does not abandon the rest either — every send is awaited whatever an earlier one did, so a batch that half succeeded can say how many arrived. That count is the point: a caller told only "it failed" resends the messages the broker already has.
How wide the batch gets on the wire is capped, and by the connection
rather than by this method:
Connection.max_outstanding_publishes is a thousand by default,
and the thousand-and-first task waits for one of the first thousand to
be answered before its message is written. A list of a million is
therefore a million tasks and a thousand messages in flight rather than
a million, which is backpressure instead of a memory leak — but a
million tasks is still a million tasks, so chunk what comes out of a
database rather than handing a cursor's worth of rows to one call.
Parameters
- payloads: what to send, in order, each encoded by this publisher's codec :returns: what the broker said about each, in the order the payloads were given
Raises
- PublishError: when any message was not confirmed. Its message
names how many failed and how many did not, and
__cause__carries the first failure in payload order — which is not necessarily the first one the broker answered. A batch wide enough that the broker stops keeping up fails this way too, with the first failure saying so
94class PublishingPausedError(PublishError): 95 """A publish refused because the broker has blocked this connection. 96 97 Declined, not lost. RabbitMQ blocks a connection when it is low on memory 98 or disk and stops reading the socket until the alarm clears; this library 99 refuses the publish at once rather than queueing a frame behind a broker 100 that has said it cannot take it. **Nothing was sent**, so retrying once the 101 broker unblocks cannot duplicate the message — which is what tells this 102 apart from every other :class:`PublishError`, where the message may or may 103 not have arrived. 104 105 A subclass, so an existing ``except PublishError`` still catches it. Catch 106 this one first to back off, shed load or buffer instead of counting a loss. 107 It is Go's ``PublishingPausedError``, .NET's ``ConnectionBlockedException`` 108 and the refusal Java's standing load counts as ``refused``. 109 110 ``reason`` is the broker's words when the transport kept them; the RabbitMQ 111 transport cannot (see :attr:`acemq_amqp.Connection.blocked_reason`). 112 """
A publish refused because the broker has blocked this connection.
Declined, not lost. RabbitMQ blocks a connection when it is low on memory
or disk and stops reading the socket until the alarm clears; this library
refuses the publish at once rather than queueing a frame behind a broker
that has said it cannot take it. Nothing was sent, so retrying once the
broker unblocks cannot duplicate the message — which is what tells this
apart from every other PublishError, where the message may or may
not have arrived.
A subclass, so an existing except PublishError still catches it. Catch
this one first to back off, shed load or buffer instead of counting a loss.
It is Go's PublishingPausedError, .NET's ConnectionBlockedException
and the refusal Java's standing load counts as refused.
reason is the broker's words when the transport kept them; the RabbitMQ
transport cannot (see acemq_amqp.Connection.blocked_reason).
Inherited Members
36@dataclass(frozen=True, slots=True) 37class QueueSpec: 38 """How a queue should be declared. 39 40 :param durable: survives a broker restart 41 :param auto_delete: goes away when its last consumer does 42 :param exclusive: usable only by the connection that declared it 43 :param args: broker-specific arguments, such as ``x-dead-letter-exchange`` 44 """ 45 46 durable: bool = True 47 auto_delete: bool = False 48 exclusive: bool = False 49 args: Mapping[str, Any] = field(default_factory=dict)
How a queue should be declared.
Parameters
- durable: survives a broker restart
- auto_delete: goes away when its last consumer does
- exclusive: usable only by the connection that declared it
- args: broker-specific arguments, such as
x-dead-letter-exchange
67@dataclass(frozen=True, slots=True) 68class RetryPolicy: 69 """How many times to try, how long to wait, and where. 70 71 Waits below ``broker_wait_threshold`` are spent in the consumer, which holds 72 the delivery and one prefetch slot for the duration. Waits at or above it 73 are spent in a ``{queue}.retry.{delay}`` queue whose ``x-message-ttl`` is the 74 wait and whose dead-letter target is the source queue, so the broker returns 75 the message when the time is up and a consumer restart cannot shorten it. 76 77 Splitting at a threshold rather than picking one of the two takes the 78 durability where it is worth its complexity and leaves the simplicity where 79 it is not. A queue's TTL is fixed at declaration and so cannot express a 80 jittered delay, which is why jitter applies below the threshold only; above 81 it the spread comes free, because each message's TTL starts when it enters 82 the rung, so a fleet that failed over ten seconds is released over ten 83 seconds. 84 85 :param max_attempts: total deliveries including the first. 1 means no retry 86 :param initial_delay: the wait before the second attempt 87 :param multiplier: what the delay is multiplied by each time 88 :param max_delay: the ceiling, or zero for none 89 :param jitter_factor: how far a delay may move either side, 0 to 1 90 :param max_message_age: give up on anything older, or zero for never 91 :param broker_wait_threshold: waits this long or longer are spent in the 92 broker, or zero to spend every wait in the consumer 93 """ 94 95 max_attempts: int = 1 96 initial_delay: timedelta = ZERO 97 multiplier: float = 2.0 98 max_delay: timedelta = ZERO 99 jitter_factor: float = 0.0 100 max_message_age: timedelta = ZERO 101 broker_wait_threshold: timedelta = DEFAULT_BROKER_WAIT_THRESHOLD 102 103 def give_up_after(self, age: timedelta) -> RetryPolicy: 104 """A copy that abandons a message older than ``age``. 105 106 The honest limit when a queue has been paused: attempts say nothing 107 about how long a message has been waiting, and a message four days old 108 is usually one nobody wants delivered now. 109 """ 110 return replace(self, max_message_age=age) 111 112 def with_jitter(self, factor: float) -> RetryPolicy: 113 """A copy using a different jitter factor, between 0 and 1.""" 114 return replace(self, jitter_factor=factor) 115 116 def wait_in_broker_from(self, threshold: timedelta) -> RetryPolicy: 117 """A copy that moves the line between waiting here and waiting there. 118 119 Zero is the way out: with no threshold nothing is long enough to reach 120 the broker, so every wait is spent in the consumer and no rung queue is 121 needed. That is the right setting for a service whose broker it may not 122 declare queues on, and the wrong one for a policy with delays measured 123 in minutes. 124 125 :param threshold: waits this long or longer go to a rung queue, or zero 126 for none of them 127 :returns: a new policy 128 """ 129 return replace(self, broker_wait_threshold=threshold) 130 131 def waits_in_broker(self, delay: timedelta) -> bool: 132 """Whether a wait of this length belongs on a rung queue. 133 134 :param delay: an unjittered delay, as :meth:`schedule` reports them 135 :returns: whether the broker rather than the consumer should hold it 136 """ 137 return ( 138 self.broker_wait_threshold > ZERO 139 and delay > ZERO 140 and delay >= self.broker_wait_threshold 141 ) 142 143 def broker_rungs(self) -> list[timedelta]: 144 """The delays this policy needs a rung queue for, longest last. 145 146 Exactly the entries of :meth:`schedule` that are at or above the 147 threshold, with repeats removed — a fixed policy that waits a minute 148 three times needs one queue, not three. It is a finite list because the 149 schedule is, which is what makes the queues declarable up front rather 150 than conjured by a consumer at the moment it first fails. 151 152 :returns: one delay per queue, in the order the schedule reaches them 153 """ 154 return list(dict.fromkeys(d for d in self.schedule() if self.waits_in_broker(d))) 155 156 def next_delay(self, attempt: int, message_age: timedelta = ZERO) -> timedelta | None: 157 """How long to wait before the next attempt, or ``None`` to give up. 158 159 :param attempt: the attempt that has just failed, starting at 1 160 :param message_age: how old the message is 161 :returns: the delay, or ``None`` when there should be no further attempt 162 """ 163 wait = self.next_wait(attempt, message_age) 164 return None if wait is None else wait.delay 165 166 def next_wait(self, attempt: int, message_age: timedelta = ZERO) -> Wait | None: 167 """The whole answer: how long to wait, and where. 168 169 :param attempt: the attempt that has just failed, starting at 1 170 :param message_age: how old the message is 171 :returns: the wait, or ``None`` when there should be no further attempt 172 """ 173 if attempt >= self.max_attempts: 174 return None 175 if self.max_message_age > ZERO and message_age >= self.max_message_age: 176 return None 177 178 delay = self._unjittered(attempt) 179 180 if self.waits_in_broker(delay): 181 # Deliberately not jittered. A rung queue's TTL is fixed when it is 182 # declared, so a moved delay would name a queue that does not exist; 183 # and the spread jitter buys is already there, because each message's 184 # TTL starts when it arrives rather than when the batch failed. 185 return Wait(delay, in_broker=True) 186 187 if self.jitter_factor > 0 and delay > ZERO: 188 # Both directions, so a fleet of consumers that failed together 189 # does not come back together. One-sided jitter only ever delays, 190 # which turns a thundering herd into a slower thundering herd. 191 factor = 1 + ((random.random() * 2 - 1) * self.jitter_factor) 192 delay = delay * factor 193 194 return Wait(max(delay, ZERO), in_broker=False) 195 196 def _unjittered(self, attempt: int) -> timedelta: 197 """The delay after ``attempt``, before jitter and before the threshold.""" 198 delay = self.initial_delay 199 for _ in range(1, attempt): 200 delay = delay * self.multiplier 201 if self.max_delay > ZERO and delay > self.max_delay: 202 delay = self.max_delay 203 break 204 if self.max_delay > ZERO and delay > self.max_delay: 205 delay = self.max_delay 206 return delay 207 208 def schedule(self) -> list[timedelta]: 209 """The delays this policy would use, without jitter. 210 211 What to look at when deciding whether a policy is the one you meant: 212 ``ExponentialRetry(5, 1s, 1m).schedule()`` is ``[1s, 2s, 4s, 8s]``, and 213 four numbers are easier to argue with than three parameters. 214 """ 215 delays: list[timedelta] = [] 216 delay = self.initial_delay 217 for _ in range(1, max(self.max_attempts, 0)): 218 capped = delay 219 if self.max_delay > ZERO and capped > self.max_delay: 220 capped = self.max_delay 221 delays.append(capped) 222 delay = delay * self.multiplier 223 return delays
How many times to try, how long to wait, and where.
Waits below broker_wait_threshold are spent in the consumer, which holds
the delivery and one prefetch slot for the duration. Waits at or above it
are spent in a {queue}acemq_amqp.retry.{delay} queue whose x-message-ttl is the
wait and whose dead-letter target is the source queue, so the broker returns
the message when the time is up and a consumer restart cannot shorten it.
Splitting at a threshold rather than picking one of the two takes the durability where it is worth its complexity and leaves the simplicity where it is not. A queue's TTL is fixed at declaration and so cannot express a jittered delay, which is why jitter applies below the threshold only; above it the spread comes free, because each message's TTL starts when it enters the rung, so a fleet that failed over ten seconds is released over ten seconds.
Parameters
- max_attempts: total deliveries including the first. 1 means no retry
- initial_delay: the wait before the second attempt
- multiplier: what the delay is multiplied by each time
- max_delay: the ceiling, or zero for none
- jitter_factor: how far a delay may move either side, 0 to 1
- max_message_age: give up on anything older, or zero for never
- broker_wait_threshold: waits this long or longer are spent in the broker, or zero to spend every wait in the consumer
103 def give_up_after(self, age: timedelta) -> RetryPolicy: 104 """A copy that abandons a message older than ``age``. 105 106 The honest limit when a queue has been paused: attempts say nothing 107 about how long a message has been waiting, and a message four days old 108 is usually one nobody wants delivered now. 109 """ 110 return replace(self, max_message_age=age)
A copy that abandons a message older than age.
The honest limit when a queue has been paused: attempts say nothing about how long a message has been waiting, and a message four days old is usually one nobody wants delivered now.
112 def with_jitter(self, factor: float) -> RetryPolicy: 113 """A copy using a different jitter factor, between 0 and 1.""" 114 return replace(self, jitter_factor=factor)
A copy using a different jitter factor, between 0 and 1.
116 def wait_in_broker_from(self, threshold: timedelta) -> RetryPolicy: 117 """A copy that moves the line between waiting here and waiting there. 118 119 Zero is the way out: with no threshold nothing is long enough to reach 120 the broker, so every wait is spent in the consumer and no rung queue is 121 needed. That is the right setting for a service whose broker it may not 122 declare queues on, and the wrong one for a policy with delays measured 123 in minutes. 124 125 :param threshold: waits this long or longer go to a rung queue, or zero 126 for none of them 127 :returns: a new policy 128 """ 129 return replace(self, broker_wait_threshold=threshold)
A copy that moves the line between waiting here and waiting there.
Zero is the way out: with no threshold nothing is long enough to reach the broker, so every wait is spent in the consumer and no rung queue is needed. That is the right setting for a service whose broker it may not declare queues on, and the wrong one for a policy with delays measured in minutes.
Parameters
- threshold: waits this long or longer go to a rung queue, or zero for none of them :returns: a new policy
131 def waits_in_broker(self, delay: timedelta) -> bool: 132 """Whether a wait of this length belongs on a rung queue. 133 134 :param delay: an unjittered delay, as :meth:`schedule` reports them 135 :returns: whether the broker rather than the consumer should hold it 136 """ 137 return ( 138 self.broker_wait_threshold > ZERO 139 and delay > ZERO 140 and delay >= self.broker_wait_threshold 141 )
Whether a wait of this length belongs on a rung queue.
Parameters
- delay: an unjittered delay, as
schedule()reports them :returns: whether the broker rather than the consumer should hold it
143 def broker_rungs(self) -> list[timedelta]: 144 """The delays this policy needs a rung queue for, longest last. 145 146 Exactly the entries of :meth:`schedule` that are at or above the 147 threshold, with repeats removed — a fixed policy that waits a minute 148 three times needs one queue, not three. It is a finite list because the 149 schedule is, which is what makes the queues declarable up front rather 150 than conjured by a consumer at the moment it first fails. 151 152 :returns: one delay per queue, in the order the schedule reaches them 153 """ 154 return list(dict.fromkeys(d for d in self.schedule() if self.waits_in_broker(d)))
The delays this policy needs a rung queue for, longest last.
Exactly the entries of schedule() that are at or above the
threshold, with repeats removed — a fixed policy that waits a minute
three times needs one queue, not three. It is a finite list because the
schedule is, which is what makes the queues declarable up front rather
than conjured by a consumer at the moment it first fails.
:returns: one delay per queue, in the order the schedule reaches them
156 def next_delay(self, attempt: int, message_age: timedelta = ZERO) -> timedelta | None: 157 """How long to wait before the next attempt, or ``None`` to give up. 158 159 :param attempt: the attempt that has just failed, starting at 1 160 :param message_age: how old the message is 161 :returns: the delay, or ``None`` when there should be no further attempt 162 """ 163 wait = self.next_wait(attempt, message_age) 164 return None if wait is None else wait.delay
How long to wait before the next attempt, or None to give up.
Parameters
- attempt: the attempt that has just failed, starting at 1
- message_age: how old the message is
:returns: the delay, or
Nonewhen there should be no further attempt
166 def next_wait(self, attempt: int, message_age: timedelta = ZERO) -> Wait | None: 167 """The whole answer: how long to wait, and where. 168 169 :param attempt: the attempt that has just failed, starting at 1 170 :param message_age: how old the message is 171 :returns: the wait, or ``None`` when there should be no further attempt 172 """ 173 if attempt >= self.max_attempts: 174 return None 175 if self.max_message_age > ZERO and message_age >= self.max_message_age: 176 return None 177 178 delay = self._unjittered(attempt) 179 180 if self.waits_in_broker(delay): 181 # Deliberately not jittered. A rung queue's TTL is fixed when it is 182 # declared, so a moved delay would name a queue that does not exist; 183 # and the spread jitter buys is already there, because each message's 184 # TTL starts when it arrives rather than when the batch failed. 185 return Wait(delay, in_broker=True) 186 187 if self.jitter_factor > 0 and delay > ZERO: 188 # Both directions, so a fleet of consumers that failed together 189 # does not come back together. One-sided jitter only ever delays, 190 # which turns a thundering herd into a slower thundering herd. 191 factor = 1 + ((random.random() * 2 - 1) * self.jitter_factor) 192 delay = delay * factor 193 194 return Wait(max(delay, ZERO), in_broker=False)
The whole answer: how long to wait, and where.
Parameters
- attempt: the attempt that has just failed, starting at 1
- message_age: how old the message is
:returns: the wait, or
Nonewhen there should be no further attempt
208 def schedule(self) -> list[timedelta]: 209 """The delays this policy would use, without jitter. 210 211 What to look at when deciding whether a policy is the one you meant: 212 ``ExponentialRetry(5, 1s, 1m).schedule()`` is ``[1s, 2s, 4s, 8s]``, and 213 four numbers are easier to argue with than three parameters. 214 """ 215 delays: list[timedelta] = [] 216 delay = self.initial_delay 217 for _ in range(1, max(self.max_attempts, 0)): 218 capped = delay 219 if self.max_delay > ZERO and capped > self.max_delay: 220 capped = self.max_delay 221 delays.append(capped) 222 delay = delay * self.multiplier 223 return delays
The delays this policy would use, without jitter.
What to look at when deciding whether a policy is the one you meant:
ExponentialRetry(5, 1s, 1m).schedule() is [1s, 2s, 4s, 8s], and
four numbers are easier to argue with than three parameters.
279@dataclass(frozen=True, slots=True) 280class Security: 281 """How to verify the broker, what to present to it, and who to log in as. 282 283 Every field is optional and the default is the safe one: verify against the 284 machine's trust store, present no client certificate, and take the login 285 from the URL. Pass one to :func:`acemq_amqp.connect` or 286 :func:`acemq_amqp.sync.connect`. 287 288 A ``Security`` that carries TLS settings is refused against an ``amqp://`` 289 URL rather than ignored — see this module's own documentation for why. One 290 that carries only ``credentials`` is fine on either scheme, because keeping 291 a password out of a connection string is worth doing whether or not the 292 connection is encrypted. 293 294 :param certificate_authority: a PEM file holding the authority to verify the 295 broker against. Naming one replaces the system trust store rather than 296 adding to it, which is the point: a broker with a certificate from a 297 public authority is not your broker, and the hundreds of authorities a 298 machine trusts by default are hundreds of ways to be wrong 299 :param client_certificate: a PEM certificate to present, for a broker that 300 authenticates clients by certificate rather than by password 301 :param client_key: the PEM private key for that certificate. Defaults to 302 reading the key out of ``client_certificate``, which is where it lives 303 when the two are in one file 304 :param client_key_password: the passphrase on that key, when it has one. 305 Kept out of this dataclass's :func:`repr` alongside the broker password 306 :param server_name: the name to check the certificate against, when it is 307 not the one in the URL. Needed for a broker reached by IP address, 308 through a tunnel, or at a container's internal name 309 :param credentials: the broker login, or a :data:`CredentialsSource` asked 310 for it at connection time. Overrides anything in the URL 311 :param verification: what to check about the broker's certificate 312 :param allow_development_certificates: whether to accept a certificate 313 stamped :data:`DEVELOPMENT_MARKER`. Off, and it should stay off anywhere 314 that is not a developer's machine: the signing key of a generated 315 authority is in whatever repository the certificates were committed to, 316 so trusting one means trusting everybody who can read it 317 """ 318 319 certificate_authority: FilePath | None = None 320 client_certificate: FilePath | None = None 321 client_key: FilePath | None = None 322 client_key_password: str | None = field(default=None, repr=False) 323 server_name: str | None = None 324 credentials: Credentials | CredentialsSource | None = None 325 verification: Verification = Verification.CERTIFICATE 326 allow_development_certificates: bool = False 327 328 @property 329 def describes_tls(self) -> bool: 330 """Whether this says anything about an encrypted connection. 331 332 A ``Security`` that says nothing — one carrying credentials alone — is 333 as valid against ``amqp://`` as against ``amqps://``. One that does is 334 an instruction that a plaintext connection cannot carry out. 335 """ 336 return ( 337 self.certificate_authority is not None 338 or self.client_certificate is not None 339 or self.client_key is not None 340 or self.server_name is not None 341 or self.verification is not Verification.CERTIFICATE 342 or self.allow_development_certificates 343 ) 344 345 def resolve_credentials(self) -> Credentials | None: 346 """Asks for the login, now. 347 348 :returns: the credentials, or ``None`` when this leaves the login to 349 the URL 350 :raises SecurityError: when a source was configured and could not 351 produce them 352 """ 353 if self.credentials is None: 354 return None 355 if isinstance(self.credentials, Credentials): 356 return self.credentials 357 found = self.credentials() 358 if not isinstance(found, Credentials): 359 raise SecurityError( 360 "acemq: a credentials source must return Credentials, and this one " 361 f"returned {type(found).__name__}" 362 ) 363 return found 364 365 def ssl_context(self) -> ssl.SSLContext: 366 """Builds the TLS context for a connection. 367 368 TLS 1.2 is the floor whatever the machine's ``openssl`` would otherwise 369 allow: 1.0 and 1.1 are withdrawn, and a broker still offering them is 370 not a reason to speak them. 371 372 :returns: the context, ready to hand to the transport 373 :raises SecurityError: when a certificate or key cannot be read, or when 374 one of them is a development certificate and this configuration did 375 not ask for those 376 """ 377 context = self._new_context() 378 context.minimum_version = ssl.TLSVersion.TLSv1_2 379 380 if not self.allow_development_certificates: 381 # Two checks rather than one, because a development certificate can 382 # arrive from two directions. This is the near side: the files this 383 # configuration names. The far side — what the broker actually 384 # presents — is checked at the handshake, below, where it cannot be 385 # avoided by any trust setting. 386 self._refuse_development_file(self.certificate_authority, "certificate authority") 387 self._refuse_development_file(self.client_certificate, "client certificate") 388 389 if self.verification is Verification.NOTHING_AT_ALL: 390 # Both, and in this order: check_hostname cannot be left on once 391 # verify_mode is CERT_NONE, and Python raises rather than guessing 392 # which of the two was meant. 393 context.check_hostname = False 394 context.verify_mode = ssl.CERT_NONE 395 elif self.certificate_authority is not None: 396 # One authority, and no others. Verification succeeding therefore 397 # means the chain reached the authority named here — the system 398 # trust store is never consulted, which is what makes this stronger 399 # than the default rather than merely different from it. 400 authority = os.fspath(self.certificate_authority) 401 self._load( 402 lambda: context.load_verify_locations(cafile=authority), 403 f"the certificate authority {authority!r}", 404 ) 405 else: 406 context.load_default_certs(ssl.Purpose.SERVER_AUTH) 407 408 if self.client_certificate is not None: 409 certificate = os.fspath(self.client_certificate) 410 key = None if self.client_key is None else os.fspath(self.client_key) 411 self._load( 412 lambda: context.load_cert_chain( 413 certificate, keyfile=key, password=self.client_key_password 414 ), 415 f"the client certificate {certificate!r}" 416 + ("" if key is None else f" with key {key!r}"), 417 ) 418 elif self.client_key is not None: 419 raise SecurityError( 420 "acemq: a client key was given with no client certificate; a broker " 421 "authenticates the certificate, and a key on its own proves nothing" 422 ) 423 424 return context 425 426 def _new_context(self) -> ssl.SSLContext: 427 """A client context, naming a different broker when asked to. 428 429 ``PROTOCOL_TLS_CLIENT`` rather than :func:`ssl.create_default_context` 430 because the trust store is loaded deliberately a few lines further on: 431 the default context loads the system's, and a context that has already 432 trusted three hundred authorities cannot then be narrowed to one. 433 """ 434 if self.server_name is None: 435 context = ssl.SSLContext(ssl.PROTOCOL_TLS_CLIENT) 436 else: 437 named = _ContextNamingTheBroker(ssl.PROTOCOL_TLS_CLIENT) 438 named.expected_name = self.server_name 439 context = named 440 441 if not self.allow_development_certificates: 442 # ``sslobject_class`` is the documented seam for this, and it is the 443 # only one: :mod:`ssl` has no verify callback of the sort Go's 444 # ``VerifyPeerCertificate`` and Java's ``X509TrustManager`` give, 445 # and there is nowhere else a check can be put that a trust setting 446 # cannot then turn off. Both classes are replaced because aio-pika 447 # goes through ``wrap_bio`` and a blocking client would go through 448 # ``wrap_socket``. 449 context.sslobject_class = _RefusingSSLObject 450 context.sslsocket_class = _RefusingSSLSocket 451 return context 452 453 @staticmethod 454 def _refuse_development_file(path: FilePath | None, what: str) -> None: 455 """Refuses a configured certificate file that carries the marker. 456 457 Reading the file is worth it for the error alone: at the handshake all 458 that can be said is that the broker presented one, whereas here the 459 message can name the setting and the path, which is what somebody 460 actually has to change. 461 """ 462 if path is None: 463 return 464 where = os.fspath(path) 465 try: 466 with open(where, "rb") as handle: 467 contents = handle.read() 468 except OSError: 469 # Not this method's failure to report: the loader a few lines on 470 # opens the same file and says what is wrong with it far better. 471 return 472 if is_development_certificate(contents): 473 raise SecurityError( 474 f"acemq: the {what} {where!r} is marked {DEVELOPMENT_MARKER!r}. It was " 475 "generated for development and its signing key is not a secret, so it is " 476 "not trusted here. Use a real certificate, or say so deliberately with " 477 "Security(allow_development_certificates=True)" 478 ) 479 480 @staticmethod 481 def _load(read: Callable[[], None], what: str) -> None: 482 """Runs one of :mod:`ssl`'s loaders and says what failed in words. 483 484 ``ssl`` reports a missing file as ``FileNotFoundError`` and a malformed 485 one as ``[SSL] PEM lib``, neither of which names the setting that was 486 wrong. This does. 487 """ 488 try: 489 read() 490 except (OSError, ssl.SSLError) as failure: 491 raise SecurityError(f"acemq: cannot load {what}: {failure}") from failure 492 493 def applied_to(self, url: str) -> str: 494 """The URL this configuration would actually connect to. 495 496 Credentials given here replace whatever the URL carried, which is how a 497 password stays out of a connection string. When none were given the URL 498 comes back as it was. 499 500 :param url: where the broker is 501 :returns: the URL to hand to the transport 502 :raises SecurityError: when TLS settings were given for an ``amqp://`` 503 URL, which cannot honour them 504 """ 505 scheme = urlsplit(url).scheme 506 if self.describes_tls and scheme != "amqps": 507 raise SecurityError( 508 f"acemq: this Security configures TLS but the URL is {scheme or 'unset'}://, " 509 "which is not encrypted. Nothing here can make a plaintext connection safe: " 510 "change the URL to amqps:// or take the TLS settings off" 511 ) 512 513 credentials = self.resolve_credentials() 514 if credentials is None or credentials.empty: 515 return url 516 return _with_credentials(url, credentials) 517 518 def transport_options(self, url: str) -> dict[str, Any]: 519 """What the transport needs, beyond the URL. 520 521 :param url: where the broker is, so that a plaintext connection is given 522 no context at all rather than one it would silently drop 523 :returns: keyword arguments for :func:`aio_pika.connect_robust` 524 """ 525 if urlsplit(url).scheme != "amqps": 526 return {} 527 return {"ssl_context": self.ssl_context()}
How to verify the broker, what to present to it, and who to log in as.
Every field is optional and the default is the safe one: verify against the
machine's trust store, present no client certificate, and take the login
from the URL. Pass one to acemq_amqp.connect() or
acemq_amqp.sync.connect().
A Security that carries TLS settings is refused against an amqp://
URL rather than ignored — see this module's own documentation for why. One
that carries only credentials is fine on either scheme, because keeping
a password out of a connection string is worth doing whether or not the
connection is encrypted.
Parameters
- certificate_authority: a PEM file holding the authority to verify the broker against. Naming one replaces the system trust store rather than adding to it, which is the point: a broker with a certificate from a public authority is not your broker, and the hundreds of authorities a machine trusts by default are hundreds of ways to be wrong
- client_certificate: a PEM certificate to present, for a broker that authenticates clients by certificate rather than by password
- client_key: the PEM private key for that certificate. Defaults to
reading the key out of
client_certificate, which is where it lives when the two are in one file - client_key_password: the passphrase on that key, when it has one.
Kept out of this dataclass's
repr()alongside the broker password - server_name: the name to check the certificate against, when it is not the one in the URL. Needed for a broker reached by IP address, through a tunnel, or at a container's internal name
- credentials: the broker login, or a
CredentialsSourceasked for it at connection time. Overrides anything in the URL - verification: what to check about the broker's certificate
- allow_development_certificates: whether to accept a certificate
stamped
DEVELOPMENT_MARKER. Off, and it should stay off anywhere that is not a developer's machine: the signing key of a generated authority is in whatever repository the certificates were committed to, so trusting one means trusting everybody who can read it
328 @property 329 def describes_tls(self) -> bool: 330 """Whether this says anything about an encrypted connection. 331 332 A ``Security`` that says nothing — one carrying credentials alone — is 333 as valid against ``amqp://`` as against ``amqps://``. One that does is 334 an instruction that a plaintext connection cannot carry out. 335 """ 336 return ( 337 self.certificate_authority is not None 338 or self.client_certificate is not None 339 or self.client_key is not None 340 or self.server_name is not None 341 or self.verification is not Verification.CERTIFICATE 342 or self.allow_development_certificates 343 )
Whether this says anything about an encrypted connection.
A Security that says nothing — one carrying credentials alone — is
as valid against amqp:// as against amqps://. One that does is
an instruction that a plaintext connection cannot carry out.
345 def resolve_credentials(self) -> Credentials | None: 346 """Asks for the login, now. 347 348 :returns: the credentials, or ``None`` when this leaves the login to 349 the URL 350 :raises SecurityError: when a source was configured and could not 351 produce them 352 """ 353 if self.credentials is None: 354 return None 355 if isinstance(self.credentials, Credentials): 356 return self.credentials 357 found = self.credentials() 358 if not isinstance(found, Credentials): 359 raise SecurityError( 360 "acemq: a credentials source must return Credentials, and this one " 361 f"returned {type(found).__name__}" 362 ) 363 return found
Asks for the login, now.
:returns: the credentials, or None when this leaves the login to
the URL
Raises
- SecurityError: when a source was configured and could not produce them
365 def ssl_context(self) -> ssl.SSLContext: 366 """Builds the TLS context for a connection. 367 368 TLS 1.2 is the floor whatever the machine's ``openssl`` would otherwise 369 allow: 1.0 and 1.1 are withdrawn, and a broker still offering them is 370 not a reason to speak them. 371 372 :returns: the context, ready to hand to the transport 373 :raises SecurityError: when a certificate or key cannot be read, or when 374 one of them is a development certificate and this configuration did 375 not ask for those 376 """ 377 context = self._new_context() 378 context.minimum_version = ssl.TLSVersion.TLSv1_2 379 380 if not self.allow_development_certificates: 381 # Two checks rather than one, because a development certificate can 382 # arrive from two directions. This is the near side: the files this 383 # configuration names. The far side — what the broker actually 384 # presents — is checked at the handshake, below, where it cannot be 385 # avoided by any trust setting. 386 self._refuse_development_file(self.certificate_authority, "certificate authority") 387 self._refuse_development_file(self.client_certificate, "client certificate") 388 389 if self.verification is Verification.NOTHING_AT_ALL: 390 # Both, and in this order: check_hostname cannot be left on once 391 # verify_mode is CERT_NONE, and Python raises rather than guessing 392 # which of the two was meant. 393 context.check_hostname = False 394 context.verify_mode = ssl.CERT_NONE 395 elif self.certificate_authority is not None: 396 # One authority, and no others. Verification succeeding therefore 397 # means the chain reached the authority named here — the system 398 # trust store is never consulted, which is what makes this stronger 399 # than the default rather than merely different from it. 400 authority = os.fspath(self.certificate_authority) 401 self._load( 402 lambda: context.load_verify_locations(cafile=authority), 403 f"the certificate authority {authority!r}", 404 ) 405 else: 406 context.load_default_certs(ssl.Purpose.SERVER_AUTH) 407 408 if self.client_certificate is not None: 409 certificate = os.fspath(self.client_certificate) 410 key = None if self.client_key is None else os.fspath(self.client_key) 411 self._load( 412 lambda: context.load_cert_chain( 413 certificate, keyfile=key, password=self.client_key_password 414 ), 415 f"the client certificate {certificate!r}" 416 + ("" if key is None else f" with key {key!r}"), 417 ) 418 elif self.client_key is not None: 419 raise SecurityError( 420 "acemq: a client key was given with no client certificate; a broker " 421 "authenticates the certificate, and a key on its own proves nothing" 422 ) 423 424 return context
Builds the TLS context for a connection.
TLS 1.2 is the floor whatever the machine's openssl would otherwise
allow: 1.0 and 1.1 are withdrawn, and a broker still offering them is
not a reason to speak them.
:returns: the context, ready to hand to the transport
Raises
- SecurityError: when a certificate or key cannot be read, or when one of them is a development certificate and this configuration did not ask for those
493 def applied_to(self, url: str) -> str: 494 """The URL this configuration would actually connect to. 495 496 Credentials given here replace whatever the URL carried, which is how a 497 password stays out of a connection string. When none were given the URL 498 comes back as it was. 499 500 :param url: where the broker is 501 :returns: the URL to hand to the transport 502 :raises SecurityError: when TLS settings were given for an ``amqp://`` 503 URL, which cannot honour them 504 """ 505 scheme = urlsplit(url).scheme 506 if self.describes_tls and scheme != "amqps": 507 raise SecurityError( 508 f"acemq: this Security configures TLS but the URL is {scheme or 'unset'}://, " 509 "which is not encrypted. Nothing here can make a plaintext connection safe: " 510 "change the URL to amqps:// or take the TLS settings off" 511 ) 512 513 credentials = self.resolve_credentials() 514 if credentials is None or credentials.empty: 515 return url 516 return _with_credentials(url, credentials)
The URL this configuration would actually connect to.
Credentials given here replace whatever the URL carried, which is how a password stays out of a connection string. When none were given the URL comes back as it was.
Parameters
- url: where the broker is :returns: the URL to hand to the transport
Raises
- **SecurityError: when TLS settings were given for an
amqp**: //URL, which cannot honour them
518 def transport_options(self, url: str) -> dict[str, Any]: 519 """What the transport needs, beyond the URL. 520 521 :param url: where the broker is, so that a plaintext connection is given 522 no context at all rather than one it would silently drop 523 :returns: keyword arguments for :func:`aio_pika.connect_robust` 524 """ 525 if urlsplit(url).scheme != "amqps": 526 return {} 527 return {"ssl_context": self.ssl_context()}
What the transport needs, beyond the URL.
Parameters
- url: where the broker is, so that a plaintext connection is given
no context at all rather than one it would silently drop
:returns: keyword arguments for
aio_pika.connect_robust()
35class SecurityError(AceMQError): 36 """A security setting that cannot be honoured. 37 38 Raised before anything is dialled rather than after: a missing certificate 39 file, an environment variable that was going to hold the password, or TLS 40 settings against a URL that is not encrypted are all things to hear about at 41 start-up. A connection that half-applied them and carried on is the failure 42 this exists to prevent. 43 """
A security setting that cannot be honoured.
Raised before anything is dialled rather than after: a missing certificate file, an environment variable that was going to hold the password, or TLS settings against a URL that is not encrypted are all things to hear about at start-up. A connection that half-applied them and carried on is the failure this exists to prevent.
91@dataclass(frozen=True, slots=True) 92class Settlement: 93 """What the consumer did with a delivery, once it had decided. 94 95 Reported before it is carried out rather than after, because one of them 96 takes time: a consumer-side retry sleeps out the backoff before the message 97 is republished, and a listener told afterwards would be describing the 98 delivery minutes later, on a span that had been open the whole while. 99 100 :param outcome: one of :data:`OUTCOME_ACKED`, :data:`OUTCOME_RETRIED`, 101 :data:`OUTCOME_REJECTED`, :data:`OUTCOME_DEAD_LETTERED` and 102 :data:`OUTCOME_PARKED` — the same words the other libraries put on a 103 span 104 :param reason: why it was set aside, when it was 105 :param delay: how long until the next attempt, when there is one 106 """ 107 108 outcome: str 109 reason: str | None = None 110 delay: timedelta | None = None 111 112 @property 113 def dead_lettered(self) -> bool: 114 """Whether the message went to the dead-letter queue. 115 116 True for an outright rejection as well as for an exhausted one: both 117 end up in the same queue, and the difference between the two words is 118 *who decided*, which :attr:`outcome` records and this does not. 119 120 False for :data:`OUTCOME_PARKED`, and that is the whole point of the 121 parked queue: a message nobody could read is a different problem with a 122 different answer from one that failed five times, and whoever drains 123 the dead letters should not have to sort the two apart by hand. 124 """ 125 return self.outcome in (OUTCOME_REJECTED, OUTCOME_DEAD_LETTERED) 126 127 @property 128 def parked(self) -> bool: 129 """Whether the message went to the parked queue.""" 130 return self.outcome == OUTCOME_PARKED 131 132 def __str__(self) -> str: 133 return self.outcome
What the consumer did with a delivery, once it had decided.
Reported before it is carried out rather than after, because one of them takes time: a consumer-side retry sleeps out the backoff before the message is republished, and a listener told afterwards would be describing the delivery minutes later, on a span that had been open the whole while.
Parameters
- outcome: one of
OUTCOME_ACKED,OUTCOME_RETRIED,OUTCOME_REJECTED,OUTCOME_DEAD_LETTEREDandOUTCOME_PARKED— the same words the other libraries put on a span - reason: why it was set aside, when it was
- delay: how long until the next attempt, when there is one
112 @property 113 def dead_lettered(self) -> bool: 114 """Whether the message went to the dead-letter queue. 115 116 True for an outright rejection as well as for an exhausted one: both 117 end up in the same queue, and the difference between the two words is 118 *who decided*, which :attr:`outcome` records and this does not. 119 120 False for :data:`OUTCOME_PARKED`, and that is the whole point of the 121 parked queue: a message nobody could read is a different problem with a 122 different answer from one that failed five times, and whoever drains 123 the dead letters should not have to sort the two apart by hand. 124 """ 125 return self.outcome in (OUTCOME_REJECTED, OUTCOME_DEAD_LETTERED)
Whether the message went to the dead-letter queue.
True for an outright rejection as well as for an exhausted one: both
end up in the same queue, and the difference between the two words is
who decided, which outcome records and this does not.
False for OUTCOME_PARKED, and that is the whole point of the
parked queue: a message nobody could read is a different problem with a
different answer from one that failed five times, and whoever drains
the dead letters should not have to sort the two apart by hand.
184class TextCodec: 185 """Reads and writes text. 186 187 For messages that really are text — a line of a log, a command somebody 188 typed — rather than a structure that happens to be readable. Prefer 189 :class:`JsonCodec` for anything with fields. 190 """ 191 192 @property 193 def content_type(self) -> str: 194 return TEXT_CONTENT_TYPE 195 196 def encode(self, payload: Any) -> bytes: 197 if isinstance(payload, str): 198 return payload.encode("utf-8") 199 if isinstance(payload, bytes): 200 return payload 201 raise TypeError( 202 f"acemq: TextCodec cannot encode a {type(payload).__name__}; " 203 "it takes a string or bytes" 204 ) 205 206 def decode(self, body: bytes, content_type: str | None = None) -> Any: 207 try: 208 return body.decode("utf-8") 209 except UnicodeDecodeError as failure: 210 raise FatalError(f"acemq: this message is not UTF-8 text: {failure}") from failure 211 212 def can_decode(self, content_type: str | None) -> bool: 213 """Accepts ``text/*`` and nothing else. 214 215 Unlike :class:`JsonCodec` it does not answer for a message with no 216 content type, because there is no reason to think an untyped message is 217 text — and a codec that decodes anything into a string never fails, so 218 it would quietly hand a handler the printable rendering of a body it 219 should have refused. 220 """ 221 return content_type is not None and content_type.lower().startswith("text/")
Reads and writes text.
For messages that really are text — a line of a log, a command somebody
typed — rather than a structure that happens to be readable. Prefer
JsonCodec for anything with fields.
212 def can_decode(self, content_type: str | None) -> bool: 213 """Accepts ``text/*`` and nothing else. 214 215 Unlike :class:`JsonCodec` it does not answer for a message with no 216 content type, because there is no reason to think an untyped message is 217 text — and a codec that decodes anything into a string never fails, so 218 it would quietly hand a handler the printable rendering of a body it 219 should have refused. 220 """ 221 return content_type is not None and content_type.lower().startswith("text/")
Accepts text/* and nothing else.
Unlike JsonCodec it does not answer for a message with no
content type, because there is no reason to think an untyped message is
text — and a codec that decodes anything into a string never fails, so
it would quietly hand a handler the printable rendering of a body it
should have refused.
188@dataclass 189class Topology: 190 """A description of what a service needs a broker to have. 191 192 Every method that adds something returns the topology, so a description 193 reads as one expression. The order things are added in does not matter: 194 :meth:`apply` declares exchanges first, then queues, then the bindings 195 between them, because that is the order a broker will accept them in. 196 """ 197 198 _exchanges: list[_NamedExchange] = field(default_factory=list) 199 _queues: list[_NamedQueue] = field(default_factory=list) 200 _bindings: list[Binding] = field(default_factory=list) 201 202 def exchange( 203 self, 204 name: str, 205 kind: str = "topic", 206 *, 207 durable: bool = True, 208 auto_delete: bool = False, 209 args: Mapping[str, Any] | None = None, 210 ) -> Topology: 211 """Adds an exchange. 212 213 :param name: what to call it 214 :param kind: ``direct``, ``topic``, ``fanout`` or ``headers`` 215 :param durable: survives a broker restart 216 :param auto_delete: goes away when its last binding does 217 :param args: broker-specific arguments 218 :returns: this topology 219 """ 220 if not name: 221 raise ValueError("acemq: an exchange needs a name") 222 if not kind: 223 raise ValueError(f"acemq: exchange {name!r} needs a kind (direct, topic, fanout)") 224 self._exchanges.append( 225 _NamedExchange( 226 name, 227 ExchangeSpec( 228 kind=kind, 229 durable=durable, 230 auto_delete=auto_delete, 231 args=dict(args or {}), 232 ), 233 ) 234 ) 235 return self 236 237 def queue( 238 self, 239 name: str, 240 *, 241 durable: bool = True, 242 auto_delete: bool = False, 243 exclusive: bool = False, 244 quorum: bool | None = None, 245 dead_letter: bool = False, 246 retry: RetryPolicy | None = None, 247 args: Mapping[str, Any] | None = None, 248 ) -> Topology: 249 """Adds a queue, and the queues that catch what it cannot handle. 250 251 A durable queue is a **quorum** queue unless it is asked to be something 252 else. That is what Java has always declared and what the other libraries 253 declare now, and it has to be the same everywhere: the queue type is one 254 more argument the broker compares, so a Java service and a Python 255 service that disagree about ``orders`` cannot both consume it. Pass 256 ``quorum=False`` for a classic queue, which is declared by leaving 257 ``x-queue-type`` off entirely — the spelling every AceMQ library uses, 258 and therefore the only one a broker finds equivalent to theirs. 259 260 ``exclusive``, ``auto_delete`` and ``durable=False`` all mean a queue 261 that belongs to one connection, and RabbitMQ refuses a quorum queue that 262 is any of those. Such a queue is declared classic without being asked, 263 because the alternative is a declaration the broker rejects with a 264 message that never mentions the word quorum. Asking for both — a queue 265 that is ``quorum=True`` and exclusive — is refused here instead, where 266 the contradiction is written down. 267 268 A queue whose ``args`` already name a kind keeps it: that is how 269 :func:`acemq_amqp.patterns.stream` declares a stream, and a caller who 270 wrote ``x-queue-type`` down meant it. Writing it down *and* passing 271 ``quorum`` is two answers to one question and is refused. 272 273 ``dead_letter`` declares ``{name}.dlq`` alongside it and points the 274 broker at it, so a message this queue rejects or lets expire lands 275 somewhere an operator can find by name rather than disappearing. It 276 declares ``{name}.parked`` too, because a consumer on this queue sends 277 anything it cannot decode there — and a library that parks messages into 278 a queue nobody declared has only moved the disappearance somewhere else. 279 280 Neither dead-letters itself: a dead-letter queue that dead-letters is a 281 loop, and a loop is how a poison message becomes an outage. Both are 282 reached through :data:`DEAD_LETTER_EXCHANGE`, bound on their own names, 283 which is what Java has always done and is therefore what a broker shared 284 with a Java service already has. A service that wants its own 285 dead-letter exchange instead should leave ``dead_letter`` alone and pass 286 the arguments in ``args``. 287 288 ``retry`` declares the rung queues that policy's long waits need: one 289 ``{name}.retry.{delay}`` per distinct delay at or above the policy's 290 threshold, each declared with :func:`rung_args` — the delay as 291 ``x-message-ttl``, :data:`RETRY_EXCHANGE` as the dead-letter target, and 292 this queue as the routing key. The binding that brings an expired 293 message home, ``{name}`` to :data:`RETRY_EXCHANGE` on ``{name}``, is 294 made here and not left to a caller to remember: without it every rung 295 expires into an exchange that routes nowhere, and a retry that is 296 dropped by the broker looks exactly like one that is still waiting. 297 298 It takes the policy rather than a list of delays on purpose. The rungs a 299 consumer will publish to are derived from the policy it is running, so 300 anything else here would be a second copy of the same list, free to drift 301 from the first — and the way that drift shows up is a retry published to 302 a queue nobody declared, at the moment the service is already failing. 303 One value produces both, or the topology is not a description of what the 304 service needs. 305 306 :param name: what to call it 307 :param durable: survives a broker restart 308 :param auto_delete: goes away when its last consumer does 309 :param exclusive: usable only by the connection that declared it 310 :param quorum: replicated rather than classic. ``None``, the default, 311 means quorum for a durable queue and classic for one belonging to a 312 single connection 313 :param dead_letter: also declare and wire ``{name}.dlq``, and declare 314 ``{name}.parked`` 315 :param retry: also declare the rung queues this policy's long waits use 316 :param args: broker-specific arguments 317 :returns: this topology 318 :raises ValueError: when the queue cannot be what it was asked to be 319 """ 320 if not name: 321 raise ValueError("acemq: a queue needs a name") 322 323 arguments: dict[str, Any] = dict(args or {}) 324 kind = _queue_type( 325 name, 326 arguments, 327 durable=durable, 328 auto_delete=auto_delete, 329 exclusive=exclusive, 330 quorum=quorum, 331 ) 332 if kind is not None: 333 arguments[QUEUE_TYPE_ARG] = kind 334 if dead_letter: 335 # Refused rather than resolved, because either answer would be a 336 # guess about which of two conflicting instructions was meant. 337 conflicting = sorted( 338 key 339 for key in arguments 340 if key in (DEAD_LETTER_EXCHANGE_ARG, DEAD_LETTER_ROUTING_KEY_ARG) 341 ) 342 if conflicting: 343 raise ValueError( 344 f"acemq: queue {name!r} asks for dead_letter=True and also sets " 345 + ", ".join(conflicting) 346 + "; pick one" 347 ) 348 arguments[DEAD_LETTER_EXCHANGE_ARG] = DEAD_LETTER_EXCHANGE 349 arguments[DEAD_LETTER_ROUTING_KEY_ARG] = naming.dead_letter_queue(name) 350 351 self._queues.append( 352 _NamedQueue( 353 name, 354 QueueSpec( 355 durable=durable, 356 auto_delete=auto_delete, 357 exclusive=exclusive, 358 args=arguments, 359 ), 360 ) 361 ) 362 if dead_letter: 363 self._managed_exchange(DEAD_LETTER_EXCHANGE) 364 # Classic, deliberately, whatever the queue in front of them is. 365 # Java declares both QueueType.CLASSIC and these are shared names: 366 # a dead-letter queue nobody consumes gains nothing from being 367 # replicated, and gaining it here would cost a Java service its 368 # declaration of the same queue. 369 for target in (naming.dead_letter_queue(name), naming.parked_queue(name)): 370 self._queues.append(_NamedQueue(target, QueueSpec(durable=durable))) 371 self._bindings.append( 372 Binding(queue=target, exchange=DEAD_LETTER_EXCHANGE, routing_key=target) 373 ) 374 if retry is not None: 375 rungs = retry.broker_rungs() 376 if rungs: 377 self._managed_exchange(RETRY_EXCHANGE) 378 # Classic too, and for the same reason as ``.dlq``: a rung is 379 # declared by every service consuming the source queue, Java 380 # declares it QueueType.CLASSIC, and :func:`rung_args` is the 381 # whole argument table — three keys, no ``x-queue-type``. 382 for rung in rungs: 383 self._queues.append( 384 _NamedQueue( 385 naming.retry_queue(name, rung), 386 QueueSpec(durable=durable, args=rung_args(name, rung)), 387 ) 388 ) 389 # One binding brings every expired message back to the queue it 390 # came from. Made here rather than offered as something to add, 391 # because a missing binding loses every retry silently: the 392 # broker drops what a direct exchange cannot route and says 393 # nothing, so the queue looks quiet rather than broken. 394 self._bindings.append( 395 Binding(queue=name, exchange=RETRY_EXCHANGE, routing_key=name) 396 ) 397 return self 398 399 def _managed_exchange(self, name: str) -> None: 400 """Declares one of this library's own exchanges, once. 401 402 Several queues in one topology each need :data:`RETRY_EXCHANGE` and 403 :data:`DEAD_LETTER_EXCHANGE`, and declaring either twice is what 404 :meth:`validate` refuses. A caller who has already named the same 405 exchange keeps their own spec: theirs is the deliberate one. 406 """ 407 if any(entry.name == name for entry in self._exchanges): 408 return 409 self._exchanges.append(_NamedExchange(name, _MANAGED_EXCHANGE)) 410 411 def binding(self, queue: str, exchange: str, routing_key: str = "") -> Topology: 412 """Routes messages matching a key from an exchange to a queue. 413 414 :param queue: where messages end up, which this topology must declare 415 :param exchange: where they come from, which this topology must declare 416 :param routing_key: what to match, empty for a fanout 417 :returns: this topology 418 """ 419 self._bindings.append(Binding(queue=queue, exchange=exchange, routing_key=routing_key)) 420 return self 421 422 @property 423 def queues(self) -> list[str]: 424 """The queue names this topology declares, in declaration order.""" 425 return [entry.name for entry in self._queues] 426 427 @property 428 def exchanges(self) -> list[str]: 429 """The exchange names this topology declares, in declaration order.""" 430 return [entry.name for entry in self._exchanges] 431 432 @property 433 def bindings(self) -> list[Binding]: 434 """The bindings this topology declares.""" 435 return list(self._bindings) 436 437 def validate(self) -> None: 438 """Reports what is wrong with the description itself. 439 440 A binding naming a queue the topology does not declare is the mistake 441 worth catching here. The broker would accept it if the queue happened to 442 exist already, and the service would then depend on something nothing 443 declares — which works until the day it is deployed somewhere new. 444 445 :raises ValueError: when the topology cannot be right 446 """ 447 seen_queues: set[str] = set() 448 for queue in self._queues: 449 if queue.name in seen_queues: 450 raise ValueError(f"acemq: the topology declares queue {queue.name!r} twice") 451 seen_queues.add(queue.name) 452 453 seen_exchanges: set[str] = set() 454 for exchange in self._exchanges: 455 if exchange.name in seen_exchanges: 456 raise ValueError( 457 f"acemq: the topology declares exchange {exchange.name!r} twice" 458 ) 459 seen_exchanges.add(exchange.name) 460 461 for binding in self._bindings: 462 if binding.queue not in seen_queues: 463 raise ValueError( 464 f"acemq: binding {binding} names queue {binding.queue!r}, " 465 "which this topology does not declare" 466 ) 467 if not binding.exchange: 468 raise ValueError( 469 f"acemq: binding {binding} names the default exchange, " 470 "which cannot be bound to" 471 ) 472 if binding.exchange not in seen_exchanges: 473 raise ValueError( 474 f"acemq: binding {binding} names exchange {binding.exchange!r}, " 475 "which this topology does not declare" 476 ) 477 478 def plan(self) -> list[PlanAction]: 479 """What :meth:`apply` would do, without doing it. 480 481 Deliberately not a difference against the live broker: AMQP offers no 482 way to enumerate what is there without the management API, and a plan 483 that quietly guessed would be worse than one that is honest about being 484 a statement of intent. 485 486 :returns: one action per thing that would be declared 487 :raises ValueError: when the topology cannot be right 488 """ 489 self.validate() 490 actions = [ 491 PlanAction("exchange", entry.name, _describe(entry.spec)) 492 for entry in self._exchanges 493 ] 494 actions += [ 495 PlanAction("queue", entry.name, _describe(entry.spec)) for entry in self._queues 496 ] 497 actions += [ 498 PlanAction( 499 "binding", binding.queue, f"from {binding.exchange} on {binding.routing_key}" 500 ) 501 for binding in self._bindings 502 ] 503 return actions 504 505 async def apply(self, transport: Transport) -> None: 506 """Declares everything, in the order a broker needs. 507 508 It stops at the first failure. A queue that already exists with 509 different settings is refused by the broker with ``PRECONDITION_FAILED``, 510 and that refusal is passed on rather than swallowed: it means this 511 service and the broker disagree about what the queue is, and carrying on 512 would leave the service using a queue that is not the one it asked for. 513 514 :param transport: where to declare it 515 :raises ValueError: when the topology cannot be right 516 """ 517 self.validate() 518 for exchange in self._exchanges: 519 await transport.declare_exchange(exchange.name, exchange.spec) 520 for queue in self._queues: 521 await transport.declare_queue(queue.name, queue.spec) 522 for binding in self._bindings: 523 await transport.bind(binding.queue, binding.exchange, binding.routing_key) 524 525 def __str__(self) -> str: 526 try: 527 actions = self.plan() 528 except ValueError as failure: 529 return f"Topology(invalid: {failure})" 530 header = ( 531 f"Topology: {len(self._exchanges)} exchanges, " 532 f"{len(self._queues)} queues, {len(self._bindings)} bindings" 533 ) 534 return "\n".join([header, *(f" {action}" for action in actions)])
A description of what a service needs a broker to have.
Every method that adds something returns the topology, so a description
reads as one expression. The order things are added in does not matter:
apply() declares exchanges first, then queues, then the bindings
between them, because that is the order a broker will accept them in.
202 def exchange( 203 self, 204 name: str, 205 kind: str = "topic", 206 *, 207 durable: bool = True, 208 auto_delete: bool = False, 209 args: Mapping[str, Any] | None = None, 210 ) -> Topology: 211 """Adds an exchange. 212 213 :param name: what to call it 214 :param kind: ``direct``, ``topic``, ``fanout`` or ``headers`` 215 :param durable: survives a broker restart 216 :param auto_delete: goes away when its last binding does 217 :param args: broker-specific arguments 218 :returns: this topology 219 """ 220 if not name: 221 raise ValueError("acemq: an exchange needs a name") 222 if not kind: 223 raise ValueError(f"acemq: exchange {name!r} needs a kind (direct, topic, fanout)") 224 self._exchanges.append( 225 _NamedExchange( 226 name, 227 ExchangeSpec( 228 kind=kind, 229 durable=durable, 230 auto_delete=auto_delete, 231 args=dict(args or {}), 232 ), 233 ) 234 ) 235 return self
Adds an exchange.
Parameters
- name: what to call it
- kind:
direct,topic,fanoutorheaders - durable: survives a broker restart
- auto_delete: goes away when its last binding does
- args: broker-specific arguments :returns: this topology
237 def queue( 238 self, 239 name: str, 240 *, 241 durable: bool = True, 242 auto_delete: bool = False, 243 exclusive: bool = False, 244 quorum: bool | None = None, 245 dead_letter: bool = False, 246 retry: RetryPolicy | None = None, 247 args: Mapping[str, Any] | None = None, 248 ) -> Topology: 249 """Adds a queue, and the queues that catch what it cannot handle. 250 251 A durable queue is a **quorum** queue unless it is asked to be something 252 else. That is what Java has always declared and what the other libraries 253 declare now, and it has to be the same everywhere: the queue type is one 254 more argument the broker compares, so a Java service and a Python 255 service that disagree about ``orders`` cannot both consume it. Pass 256 ``quorum=False`` for a classic queue, which is declared by leaving 257 ``x-queue-type`` off entirely — the spelling every AceMQ library uses, 258 and therefore the only one a broker finds equivalent to theirs. 259 260 ``exclusive``, ``auto_delete`` and ``durable=False`` all mean a queue 261 that belongs to one connection, and RabbitMQ refuses a quorum queue that 262 is any of those. Such a queue is declared classic without being asked, 263 because the alternative is a declaration the broker rejects with a 264 message that never mentions the word quorum. Asking for both — a queue 265 that is ``quorum=True`` and exclusive — is refused here instead, where 266 the contradiction is written down. 267 268 A queue whose ``args`` already name a kind keeps it: that is how 269 :func:`acemq_amqp.patterns.stream` declares a stream, and a caller who 270 wrote ``x-queue-type`` down meant it. Writing it down *and* passing 271 ``quorum`` is two answers to one question and is refused. 272 273 ``dead_letter`` declares ``{name}.dlq`` alongside it and points the 274 broker at it, so a message this queue rejects or lets expire lands 275 somewhere an operator can find by name rather than disappearing. It 276 declares ``{name}.parked`` too, because a consumer on this queue sends 277 anything it cannot decode there — and a library that parks messages into 278 a queue nobody declared has only moved the disappearance somewhere else. 279 280 Neither dead-letters itself: a dead-letter queue that dead-letters is a 281 loop, and a loop is how a poison message becomes an outage. Both are 282 reached through :data:`DEAD_LETTER_EXCHANGE`, bound on their own names, 283 which is what Java has always done and is therefore what a broker shared 284 with a Java service already has. A service that wants its own 285 dead-letter exchange instead should leave ``dead_letter`` alone and pass 286 the arguments in ``args``. 287 288 ``retry`` declares the rung queues that policy's long waits need: one 289 ``{name}.retry.{delay}`` per distinct delay at or above the policy's 290 threshold, each declared with :func:`rung_args` — the delay as 291 ``x-message-ttl``, :data:`RETRY_EXCHANGE` as the dead-letter target, and 292 this queue as the routing key. The binding that brings an expired 293 message home, ``{name}`` to :data:`RETRY_EXCHANGE` on ``{name}``, is 294 made here and not left to a caller to remember: without it every rung 295 expires into an exchange that routes nowhere, and a retry that is 296 dropped by the broker looks exactly like one that is still waiting. 297 298 It takes the policy rather than a list of delays on purpose. The rungs a 299 consumer will publish to are derived from the policy it is running, so 300 anything else here would be a second copy of the same list, free to drift 301 from the first — and the way that drift shows up is a retry published to 302 a queue nobody declared, at the moment the service is already failing. 303 One value produces both, or the topology is not a description of what the 304 service needs. 305 306 :param name: what to call it 307 :param durable: survives a broker restart 308 :param auto_delete: goes away when its last consumer does 309 :param exclusive: usable only by the connection that declared it 310 :param quorum: replicated rather than classic. ``None``, the default, 311 means quorum for a durable queue and classic for one belonging to a 312 single connection 313 :param dead_letter: also declare and wire ``{name}.dlq``, and declare 314 ``{name}.parked`` 315 :param retry: also declare the rung queues this policy's long waits use 316 :param args: broker-specific arguments 317 :returns: this topology 318 :raises ValueError: when the queue cannot be what it was asked to be 319 """ 320 if not name: 321 raise ValueError("acemq: a queue needs a name") 322 323 arguments: dict[str, Any] = dict(args or {}) 324 kind = _queue_type( 325 name, 326 arguments, 327 durable=durable, 328 auto_delete=auto_delete, 329 exclusive=exclusive, 330 quorum=quorum, 331 ) 332 if kind is not None: 333 arguments[QUEUE_TYPE_ARG] = kind 334 if dead_letter: 335 # Refused rather than resolved, because either answer would be a 336 # guess about which of two conflicting instructions was meant. 337 conflicting = sorted( 338 key 339 for key in arguments 340 if key in (DEAD_LETTER_EXCHANGE_ARG, DEAD_LETTER_ROUTING_KEY_ARG) 341 ) 342 if conflicting: 343 raise ValueError( 344 f"acemq: queue {name!r} asks for dead_letter=True and also sets " 345 + ", ".join(conflicting) 346 + "; pick one" 347 ) 348 arguments[DEAD_LETTER_EXCHANGE_ARG] = DEAD_LETTER_EXCHANGE 349 arguments[DEAD_LETTER_ROUTING_KEY_ARG] = naming.dead_letter_queue(name) 350 351 self._queues.append( 352 _NamedQueue( 353 name, 354 QueueSpec( 355 durable=durable, 356 auto_delete=auto_delete, 357 exclusive=exclusive, 358 args=arguments, 359 ), 360 ) 361 ) 362 if dead_letter: 363 self._managed_exchange(DEAD_LETTER_EXCHANGE) 364 # Classic, deliberately, whatever the queue in front of them is. 365 # Java declares both QueueType.CLASSIC and these are shared names: 366 # a dead-letter queue nobody consumes gains nothing from being 367 # replicated, and gaining it here would cost a Java service its 368 # declaration of the same queue. 369 for target in (naming.dead_letter_queue(name), naming.parked_queue(name)): 370 self._queues.append(_NamedQueue(target, QueueSpec(durable=durable))) 371 self._bindings.append( 372 Binding(queue=target, exchange=DEAD_LETTER_EXCHANGE, routing_key=target) 373 ) 374 if retry is not None: 375 rungs = retry.broker_rungs() 376 if rungs: 377 self._managed_exchange(RETRY_EXCHANGE) 378 # Classic too, and for the same reason as ``.dlq``: a rung is 379 # declared by every service consuming the source queue, Java 380 # declares it QueueType.CLASSIC, and :func:`rung_args` is the 381 # whole argument table — three keys, no ``x-queue-type``. 382 for rung in rungs: 383 self._queues.append( 384 _NamedQueue( 385 naming.retry_queue(name, rung), 386 QueueSpec(durable=durable, args=rung_args(name, rung)), 387 ) 388 ) 389 # One binding brings every expired message back to the queue it 390 # came from. Made here rather than offered as something to add, 391 # because a missing binding loses every retry silently: the 392 # broker drops what a direct exchange cannot route and says 393 # nothing, so the queue looks quiet rather than broken. 394 self._bindings.append( 395 Binding(queue=name, exchange=RETRY_EXCHANGE, routing_key=name) 396 ) 397 return self
Adds a queue, and the queues that catch what it cannot handle.
A durable queue is a quorum queue unless it is asked to be something
else. That is what Java has always declared and what the other libraries
declare now, and it has to be the same everywhere: the queue type is one
more argument the broker compares, so a Java service and a Python
service that disagree about orders cannot both consume it. Pass
quorum=False for a classic queue, which is declared by leaving
x-queue-type off entirely — the spelling every AceMQ library uses,
and therefore the only one a broker finds equivalent to theirs.
exclusive, auto_delete and durable=False all mean a queue
that belongs to one connection, and RabbitMQ refuses a quorum queue that
is any of those. Such a queue is declared classic without being asked,
because the alternative is a declaration the broker rejects with a
message that never mentions the word quorum. Asking for both — a queue
that is quorum=True and exclusive — is refused here instead, where
the contradiction is written down.
A queue whose args already name a kind keeps it: that is how
acemq_amqp.patterns.stream() declares a stream, and a caller who
wrote x-queue-type down meant it. Writing it down and passing
quorum is two answers to one question and is refused.
dead_letter declares {name}.dlq alongside it and points the
broker at it, so a message this queue rejects or lets expire lands
somewhere an operator can find by name rather than disappearing. It
declares {name}.parked too, because a consumer on this queue sends
anything it cannot decode there — and a library that parks messages into
a queue nobody declared has only moved the disappearance somewhere else.
Neither dead-letters itself: a dead-letter queue that dead-letters is a
loop, and a loop is how a poison message becomes an outage. Both are
reached through DEAD_LETTER_EXCHANGE, bound on their own names,
which is what Java has always done and is therefore what a broker shared
with a Java service already has. A service that wants its own
dead-letter exchange instead should leave dead_letter alone and pass
the arguments in args.
retry declares the rung queues that policy's long waits need: one
{name}acemq_amqp.retry.{delay} per distinct delay at or above the policy's
threshold, each declared with rung_args() — the delay as
x-message-ttl, RETRY_EXCHANGE as the dead-letter target, and
this queue as the routing key. The binding that brings an expired
message home, {name} to RETRY_EXCHANGE on {name}, is
made here and not left to a caller to remember: without it every rung
expires into an exchange that routes nowhere, and a retry that is
dropped by the broker looks exactly like one that is still waiting.
It takes the policy rather than a list of delays on purpose. The rungs a consumer will publish to are derived from the policy it is running, so anything else here would be a second copy of the same list, free to drift from the first — and the way that drift shows up is a retry published to a queue nobody declared, at the moment the service is already failing. One value produces both, or the topology is not a description of what the service needs.
Parameters
- name: what to call it
- durable: survives a broker restart
- auto_delete: goes away when its last consumer does
- exclusive: usable only by the connection that declared it
- quorum: replicated rather than classic.
None, the default, means quorum for a durable queue and classic for one belonging to a single connection - dead_letter: also declare and wire
{name}.dlq, and declare{name}.parked - retry: also declare the rung queues this policy's long waits use
- args: broker-specific arguments :returns: this topology
Raises
- ValueError: when the queue cannot be what it was asked to be
411 def binding(self, queue: str, exchange: str, routing_key: str = "") -> Topology: 412 """Routes messages matching a key from an exchange to a queue. 413 414 :param queue: where messages end up, which this topology must declare 415 :param exchange: where they come from, which this topology must declare 416 :param routing_key: what to match, empty for a fanout 417 :returns: this topology 418 """ 419 self._bindings.append(Binding(queue=queue, exchange=exchange, routing_key=routing_key)) 420 return self
Routes messages matching a key from an exchange to a queue.
Parameters
- queue: where messages end up, which this topology must declare
- exchange: where they come from, which this topology must declare
- routing_key: what to match, empty for a fanout :returns: this topology
422 @property 423 def queues(self) -> list[str]: 424 """The queue names this topology declares, in declaration order.""" 425 return [entry.name for entry in self._queues]
The queue names this topology declares, in declaration order.
427 @property 428 def exchanges(self) -> list[str]: 429 """The exchange names this topology declares, in declaration order.""" 430 return [entry.name for entry in self._exchanges]
The exchange names this topology declares, in declaration order.
432 @property 433 def bindings(self) -> list[Binding]: 434 """The bindings this topology declares.""" 435 return list(self._bindings)
The bindings this topology declares.
437 def validate(self) -> None: 438 """Reports what is wrong with the description itself. 439 440 A binding naming a queue the topology does not declare is the mistake 441 worth catching here. The broker would accept it if the queue happened to 442 exist already, and the service would then depend on something nothing 443 declares — which works until the day it is deployed somewhere new. 444 445 :raises ValueError: when the topology cannot be right 446 """ 447 seen_queues: set[str] = set() 448 for queue in self._queues: 449 if queue.name in seen_queues: 450 raise ValueError(f"acemq: the topology declares queue {queue.name!r} twice") 451 seen_queues.add(queue.name) 452 453 seen_exchanges: set[str] = set() 454 for exchange in self._exchanges: 455 if exchange.name in seen_exchanges: 456 raise ValueError( 457 f"acemq: the topology declares exchange {exchange.name!r} twice" 458 ) 459 seen_exchanges.add(exchange.name) 460 461 for binding in self._bindings: 462 if binding.queue not in seen_queues: 463 raise ValueError( 464 f"acemq: binding {binding} names queue {binding.queue!r}, " 465 "which this topology does not declare" 466 ) 467 if not binding.exchange: 468 raise ValueError( 469 f"acemq: binding {binding} names the default exchange, " 470 "which cannot be bound to" 471 ) 472 if binding.exchange not in seen_exchanges: 473 raise ValueError( 474 f"acemq: binding {binding} names exchange {binding.exchange!r}, " 475 "which this topology does not declare" 476 )
Reports what is wrong with the description itself.
A binding naming a queue the topology does not declare is the mistake worth catching here. The broker would accept it if the queue happened to exist already, and the service would then depend on something nothing declares — which works until the day it is deployed somewhere new.
Raises
- ValueError: when the topology cannot be right
478 def plan(self) -> list[PlanAction]: 479 """What :meth:`apply` would do, without doing it. 480 481 Deliberately not a difference against the live broker: AMQP offers no 482 way to enumerate what is there without the management API, and a plan 483 that quietly guessed would be worse than one that is honest about being 484 a statement of intent. 485 486 :returns: one action per thing that would be declared 487 :raises ValueError: when the topology cannot be right 488 """ 489 self.validate() 490 actions = [ 491 PlanAction("exchange", entry.name, _describe(entry.spec)) 492 for entry in self._exchanges 493 ] 494 actions += [ 495 PlanAction("queue", entry.name, _describe(entry.spec)) for entry in self._queues 496 ] 497 actions += [ 498 PlanAction( 499 "binding", binding.queue, f"from {binding.exchange} on {binding.routing_key}" 500 ) 501 for binding in self._bindings 502 ] 503 return actions
What apply() would do, without doing it.
Deliberately not a difference against the live broker: AMQP offers no way to enumerate what is there without the management API, and a plan that quietly guessed would be worse than one that is honest about being a statement of intent.
:returns: one action per thing that would be declared
Raises
- ValueError: when the topology cannot be right
505 async def apply(self, transport: Transport) -> None: 506 """Declares everything, in the order a broker needs. 507 508 It stops at the first failure. A queue that already exists with 509 different settings is refused by the broker with ``PRECONDITION_FAILED``, 510 and that refusal is passed on rather than swallowed: it means this 511 service and the broker disagree about what the queue is, and carrying on 512 would leave the service using a queue that is not the one it asked for. 513 514 :param transport: where to declare it 515 :raises ValueError: when the topology cannot be right 516 """ 517 self.validate() 518 for exchange in self._exchanges: 519 await transport.declare_exchange(exchange.name, exchange.spec) 520 for queue in self._queues: 521 await transport.declare_queue(queue.name, queue.spec) 522 for binding in self._bindings: 523 await transport.bind(binding.queue, binding.exchange, binding.routing_key)
Declares everything, in the order a broker needs.
It stops at the first failure. A queue that already exists with
different settings is refused by the broker with PRECONDITION_FAILED,
and that refusal is passed on rather than swallowed: it means this
service and the broker disagree about what the queue is, and carrying on
would leave the service using a queue that is not the one it asked for.
Parameters
- transport: where to declare it
Raises
- ValueError: when the topology cannot be right
194@runtime_checkable 195class Transport(Protocol): 196 """A connection to a broker. 197 198 Async because that is the shape a broker client has: every one of these is a 199 round trip, and a library that hid them behind blocking calls would make one 200 slow broker stall a whole process. The blocking API in 201 :mod:`acemq_amqp.sync` is built on top of this rather than beside it. 202 """ 203 204 async def declare_queue(self, name: str, spec: QueueSpec) -> None: 205 """Creates a queue if it is not already there.""" 206 207 async def declare_exchange(self, name: str, spec: ExchangeSpec) -> None: 208 """Creates an exchange if it is not already there.""" 209 210 async def bind(self, queue: str, exchange: str, routing_key: str) -> None: 211 """Routes messages matching a key from an exchange to a queue.""" 212 213 async def publish( 214 self, exchange: str, routing_key: str, message: Outbound 215 ) -> PublishResult: 216 """Sends one message and reports what the broker said about it.""" 217 218 async def consume( 219 self, 220 queue: str, 221 spec: ConsumeSpec, 222 deliver: Callable[[Delivery], Awaitable[None]], 223 ) -> Subscription: 224 """Delivers messages to ``deliver`` until the subscription is closed.""" 225 226 async def close(self) -> None: 227 """Releases the connection."""
A connection to a broker.
Async because that is the shape a broker client has: every one of these is a
round trip, and a library that hid them behind blocking calls would make one
slow broker stall a whole process. The blocking API in
acemq_amqp.sync is built on top of this rather than beside it.
1771def _no_init_or_replace_init(self, *args, **kwargs): 1772 cls = type(self) 1773 1774 if cls._is_protocol: 1775 raise TypeError('Protocols cannot be instantiated') 1776 1777 # Already using a custom `__init__`. No need to calculate correct 1778 # `__init__` to call. This can lead to RecursionError. See bpo-45121. 1779 if cls.__init__ is not _no_init_or_replace_init: 1780 return 1781 1782 # Initially, `__init__` of a protocol subclass is set to `_no_init_or_replace_init`. 1783 # The first instantiation of the subclass will call `_no_init_or_replace_init` which 1784 # searches for a proper new `__init__` in the MRO. The new `__init__` 1785 # replaces the subclass' old `__init__` (ie `_no_init_or_replace_init`). Subsequent 1786 # instantiation of the protocol subclass will thus use the new 1787 # `__init__` and no longer call `_no_init_or_replace_init`. 1788 for base in cls.__mro__: 1789 init = base.__dict__.get('__init__', _no_init_or_replace_init) 1790 if init is not _no_init_or_replace_init: 1791 cls.__init__ = init 1792 break 1793 else: 1794 # should not happen 1795 cls.__init__ = object.__init__ 1796 1797 cls.__init__(self, *args, **kwargs)
204 async def declare_queue(self, name: str, spec: QueueSpec) -> None: 205 """Creates a queue if it is not already there."""
Creates a queue if it is not already there.
207 async def declare_exchange(self, name: str, spec: ExchangeSpec) -> None: 208 """Creates an exchange if it is not already there."""
Creates an exchange if it is not already there.
210 async def bind(self, queue: str, exchange: str, routing_key: str) -> None: 211 """Routes messages matching a key from an exchange to a queue."""
Routes messages matching a key from an exchange to a queue.
213 async def publish( 214 self, exchange: str, routing_key: str, message: Outbound 215 ) -> PublishResult: 216 """Sends one message and reports what the broker said about it."""
Sends one message and reports what the broker said about it.
218 async def consume( 219 self, 220 queue: str, 221 spec: ConsumeSpec, 222 deliver: Callable[[Delivery], Awaitable[None]], 223 ) -> Subscription: 224 """Delivers messages to ``deliver`` until the subscription is closed."""
Delivers messages to deliver until the subscription is closed.
262class Verification(Enum): 263 """What an encrypted connection checks about the broker on the other end. 264 265 Named rather than left as a boolean because a ``verify=False`` in a 266 configuration file says nothing about what is being given up. These say it. 267 """ 268 269 #: Check the certificate chain and the name on it. The only setting to use 270 #: against a broker holding anything that matters. 271 CERTIFICATE = "certificate" 272 273 #: Check nothing. Encrypted, and to whoever answered. 274 #: :func:`without_verifying_the_broker` is the way to ask for this, and its 275 #: docstring says what it costs. 276 NOTHING_AT_ALL = "nothing-at-all"
What an encrypted connection checks about the broker on the other end.
Named rather than left as a boolean because a verify=False in a
configuration file says nothing about what is being given up. These say it.
50@dataclass(frozen=True, slots=True) 51class Wait: 52 """How long before the next attempt, and where the message spends it. 53 54 Two answers rather than one because they cannot be worked out separately: 55 jitter applies only to a wait spent in the consumer, so a caller given a 56 delay alone could not tell whether it had already been moved — and a 57 jittered delay does not name a rung queue. 58 59 :param delay: how long the message waits 60 :param in_broker: whether it waits on a rung queue rather than here 61 """ 62 63 delay: timedelta 64 in_broker: bool
How long before the next attempt, and where the message spends it.
Two answers rather than one because they cannot be worked out separately: jitter applies only to a wait spent in the consumer, so a caller given a delay alone could not tell whether it had already been moved — and a jittered delay does not name a rung queue.
Parameters
- delay: how long the message waits
- in_broker: whether it waits on a rung queue rather than here
136def accept() -> Ack: 137 """Confirms the message. It will not be delivered again.""" 138 return Ack(Action.ACCEPT)
Confirms the message. It will not be delivered again.
604async def aggregate_health(*checks: HealthCheck, timeout: float = 5.0) -> HealthReport: 605 """Runs every check at once and combines the answers. 606 607 The combined status is the worst of them: one thing being down makes the 608 whole report down, because a service that cannot reach its broker is not 609 ready however healthy the rest of it is. 610 611 At once rather than in turn, so a slow check does not add its latency to the 612 others, and under a deadline, so one that ignores the instruction above 613 still cannot hang the probe. A check that times out is reported as down with 614 the reason, which is the honest reading: something that will not answer in 615 five seconds is not something a request can depend on. 616 617 :param checks: what to ask 618 :param timeout: how long to give all of them together 619 :returns: the combined report 620 """ 621 if not checks: 622 return HealthReport(HealthStatus.UP) 623 624 async def run(check: HealthCheck) -> tuple[str, HealthReport]: 625 try: 626 return check.name, await check.check() 627 except Exception as failure: 628 return check.name, HealthReport( 629 HealthStatus.DOWN, f"the check itself failed: {failure}" 630 ) 631 632 tasks = [asyncio.ensure_future(run(check)) for check in checks] 633 try: 634 done, pending = await asyncio.wait(tasks, timeout=timeout) 635 except asyncio.CancelledError: 636 for task in tasks: 637 task.cancel() 638 raise 639 640 parts: dict[str, Any] = {} 641 worst = HealthStatus.UP 642 reasons: list[str] = [] 643 644 for task in pending: 645 task.cancel() 646 for index, task in enumerate(tasks): 647 if task in done: 648 name, report = task.result() 649 else: 650 name = checks[index].name 651 report = HealthReport(HealthStatus.DOWN, f"did not answer within {timeout}s") 652 parts[name] = report 653 if report.status is HealthStatus.DOWN: 654 worst = HealthStatus.DOWN 655 reasons.append(name) 656 elif report.status is HealthStatus.DEGRADED: 657 if worst is not HealthStatus.DOWN: 658 worst = HealthStatus.DEGRADED 659 reasons.append(name) 660 661 return HealthReport(worst, ", ".join(reasons), parts=parts)
Runs every check at once and combines the answers.
The combined status is the worst of them: one thing being down makes the whole report down, because a service that cannot reach its broker is not ready however healthy the rest of it is.
At once rather than in turn, so a slow check does not add its latency to the others, and under a deadline, so one that ignores the instruction above still cannot hang the probe. A check that times out is reported as down with the reason, which is the honest reading: something that will not answer in five seconds is not something a request can depend on.
Parameters
- checks: what to ask
- timeout: how long to give all of them together :returns: the combined report
319def codec_by_name(name: str) -> Codec: 320 """Builds the codec registered under a name. 321 322 :param name: the registered name 323 :returns: a new instance 324 :raises KeyError: when nothing is registered under that name 325 """ 326 with _registry_lock: 327 build = _registry.get(name) 328 if build is None: 329 raise KeyError( 330 f"acemq: no codec named {name!r} is registered; known: {', '.join(codec_names())}" 331 ) 332 return build()
Builds the codec registered under a name.
Parameters
- name: the registered name :returns: a new instance
Raises
- KeyError: when nothing is registered under that name
335def codec_names() -> list[str]: 336 """The registered codec names, sorted.""" 337 with _registry_lock: 338 return sorted(_registry)
The registered codec names, sorted.
2012async def connect( 2013 url: str, 2014 *, 2015 codec: Codec | None = None, 2016 origin: str | None = None, 2017 retry: RetryPolicy | None = None, 2018 prefetch: int = DEFAULT_PREFETCH, 2019 max_outstanding_publishes: int = DEFAULT_MAX_OUTSTANDING_PUBLISHES, 2020 confirm_timeout: timedelta = DEFAULT_CONFIRM_TIMEOUT, 2021 security: Security | None = None, 2022 on_publish: Sequence[PublishInterceptor] = (), 2023 on_consume: Sequence[ConsumeInterceptor] = (), 2024 observer: Observer | None = None, 2025 **transport_options: Any, 2026) -> Connection: 2027 """Opens a connection to a broker. 2028 2029 The URL's scheme picks the transport, and says whether the connection is 2030 encrypted: ``amqps://`` is, ``amqp://`` is not. Both need aio-pika, which is 2031 why it is imported here rather than by the package: a program that only 2032 reads AceMQ envelopes should not be made to install an AMQP client to import 2033 this library. 2034 2035 An ``amqps://`` URL is verified against the machine's trust store and will 2036 not speak anything older than TLS 1.2, with or without a ``security``. Pass 2037 one to name a different authority, to present a client certificate, or to 2038 supply the login separately from the URL — which is how a password stays out 2039 of a connection string that ends up in a log. 2040 2041 :param url: where the broker is 2042 :param codec: what publishers and consumers use unless they say otherwise 2043 :param origin: what to stamp on published messages 2044 :param retry: what consumers use unless they say otherwise 2045 :param prefetch: how many unacknowledged messages a consumer holds 2046 :param max_outstanding_publishes: how many publishes may be waiting for the 2047 broker at once, a thousand by default. See :class:`Connection` 2048 :param confirm_timeout: how long a publish waits for room before raising 2049 :param security: how to verify the broker and who to log in as. See 2050 :class:`~acemq_amqp.Security` 2051 :param on_publish: what to wrap every publish in, outermost first. See 2052 :mod:`acemq_amqp.interceptors` 2053 :param on_consume: what to wrap every handler in 2054 :param observer: where the numbers go. See :mod:`acemq_amqp.telemetry` 2055 :param transport_options: passed to the transport, which for RabbitMQ is 2056 :func:`aio_pika.connect_robust` 2057 :returns: the connection 2058 :raises SecurityError: when the security settings cannot be honoured, before 2059 anything is dialled 2060 """ 2061 scheme = urlsplit(url).scheme 2062 if scheme not in ("amqp", "amqps"): 2063 raise AceMQError( 2064 f"acemq: no transport knows how to reach {scheme or url!r}; " 2065 "this library speaks amqp:// and amqps://" 2066 ) 2067 2068 # Resolved here rather than in the transport, so that a missing certificate 2069 # or an unset password variable is an exception from connect() with the 2070 # setting named in it, rather than a handshake failure from somewhere in 2071 # aio-pika that says only that the socket closed. 2072 settings = security or Security() 2073 url = settings.applied_to(url) 2074 # The caller's own options win: transport_options is the escape hatch for 2075 # everything this library does not model, and a Security built from defaults 2076 # should not quietly overrule an ssl_context somebody passed by hand. 2077 transport_options = {**settings.transport_options(url), **transport_options} 2078 2079 try: 2080 from .rabbitmq import RabbitMQTransport 2081 except ImportError as missing: # pragma: no cover - depends on how it was installed 2082 raise AceMQError( 2083 "acemq: reaching a broker needs aio-pika, which is an optional extra. " 2084 'Install it with pip install "acemq-amqp[rabbitmq]"' 2085 ) from missing 2086 2087 transport = await RabbitMQTransport.connect(url, **transport_options) 2088 return Connection( 2089 transport, 2090 codec=codec, 2091 origin=origin, 2092 retry=retry, 2093 prefetch=prefetch, 2094 max_outstanding_publishes=max_outstanding_publishes, 2095 confirm_timeout=confirm_timeout, 2096 on_publish=on_publish, 2097 on_consume=on_consume, 2098 observer=observer, 2099 )
Opens a connection to a broker.
The URL's scheme picks the transport, and says whether the connection is
encrypted: amqps:// is, amqp:// is not. Both need aio-pika, which is
why it is imported here rather than by the package: a program that only
reads AceMQ envelopes should not be made to install an AMQP client to import
this library.
An amqps:// URL is verified against the machine's trust store and will
not speak anything older than TLS 1.2, with or without a security. Pass
one to name a different authority, to present a client certificate, or to
supply the login separately from the URL — which is how a password stays out
of a connection string that ends up in a log.
Parameters
- url: where the broker is
- codec: what publishers and consumers use unless they say otherwise
- origin: what to stamp on published messages
- retry: what consumers use unless they say otherwise
- prefetch: how many unacknowledged messages a consumer holds
- max_outstanding_publishes: how many publishes may be waiting for the
broker at once, a thousand by default. See
Connection - confirm_timeout: how long a publish waits for room before raising
- security: how to verify the broker and who to log in as. See
~acemq_amqp.Security - on_publish: what to wrap every publish in, outermost first. See
acemq_amqp.interceptors - on_consume: what to wrap every handler in
- observer: where the numbers go. See
acemq_amqp.telemetry - transport_options: passed to the transport, which for RabbitMQ is
aio_pika.connect_robust():returns: the connection
Raises
- SecurityError: when the security settings cannot be honoured, before anything is dialled
248def consume_chain( 249 interceptors: Sequence[ConsumeInterceptor], handle: ConsumeNext 250) -> ConsumeNext: 251 """Folds interceptors around a handler, first registered outermost. 252 253 :param interceptors: what to wrap it in, in registration order 254 :param handle: the innermost work, which runs the handler 255 :returns: the whole thing as one callable, or ``handle`` when there is 256 nothing to wrap it in 257 """ 258 composed = handle 259 for interceptor in reversed(interceptors): 260 composed = _consume_step(interceptor, composed) 261 return composed
Folds interceptors around a handler, first registered outermost.
Parameters
- interceptors: what to wrap it in, in registration order
- handle: the innermost work, which runs the handler
:returns: the whole thing as one callable, or
handlewhen there is nothing to wrap it in
189def credentials_from_environment( 190 username_variable: str = "ACEMQ_USERNAME", 191 password_variable: str = "ACEMQ_PASSWORD", 192) -> CredentialsSource: 193 """Reads two environment variables, each time it is asked. 194 195 :param username_variable: the variable holding the account name 196 :param password_variable: the variable holding the password 197 :returns: a source that reads them 198 :raises SecurityError: when either variable is unset, naming which one 199 """ 200 201 def read() -> Credentials: 202 username = os.environ.get(username_variable) 203 if username is None: 204 raise SecurityError( 205 f"acemq: {username_variable} is not set, so there is no broker username" 206 ) 207 password = os.environ.get(password_variable) 208 if password is None: 209 raise SecurityError( 210 f"acemq: {password_variable} is not set, so there is no broker password" 211 ) 212 return Credentials(username, password) 213 214 return read
Reads two environment variables, each time it is asked.
Parameters
- username_variable: the variable holding the account name
- password_variable: the variable holding the password :returns: a source that reads them
Raises
- SecurityError: when either variable is unset, naming which one
217def credentials_from_file(path: FilePath, *, username: str = "") -> CredentialsSource: 218 """Reads a secret from a file, each time it is asked. 219 220 This is how a mounted Kubernetes secret and a Docker secret arrive, and 221 reading it per connection rather than once at import is what lets a rotated 222 file take effect without a restart. 223 224 The file holds the password alone when ``username`` is given, and 225 ``username:password`` when it is not. Trailing whitespace is trimmed, 226 because a file written by an editor almost always ends in a newline and a 227 password with a newline on the end fails in a way nobody enjoys diagnosing. 228 229 :param path: the file to read 230 :param username: the account name, when the file holds only the secret 231 :returns: a source that reads the file 232 :raises SecurityError: when the file is missing, empty, or holds no username 233 and none was given 234 """ 235 236 def read() -> Credentials: 237 where = os.fspath(path) 238 try: 239 with open(path, encoding="utf-8") as handle: 240 contents = handle.read().rstrip(" \t\r\n") 241 except OSError as unreadable: 242 raise SecurityError( 243 f"acemq: cannot read the credentials file {where!r}: {unreadable}" 244 ) from unreadable 245 246 if not contents: 247 raise SecurityError(f"acemq: the credentials file {where!r} is empty") 248 if username: 249 return Credentials(username, contents) 250 account, separator, secret = contents.partition(":") 251 if not separator: 252 raise SecurityError( 253 f"acemq: the credentials file {where!r} holds no username, and none was " 254 "given; write it as username:password, or pass username= to " 255 "credentials_from_file" 256 ) 257 return Credentials(account, secret) 258 259 return read
Reads a secret from a file, each time it is asked.
This is how a mounted Kubernetes secret and a Docker secret arrive, and reading it per connection rather than once at import is what lets a rotated file take effect without a restart.
The file holds the password alone when username is given, and
username:password when it is not. Trailing whitespace is trimmed,
because a file written by an editor almost always ends in a newline and a
password with a newline on the end fails in a way nobody enjoys diagnosing.
Parameters
- path: the file to read
- username: the account name, when the file holds only the secret :returns: a source that reads the file
Raises
- SecurityError: when the file is missing, empty, or holds no username and none was given
35def dead_letter_queue(queue: str) -> str: 36 """``orders.new`` becomes ``orders.new.dlq``.""" 37 return queue + DEAD_LETTER_SUFFIX
orders.new becomes orders.new.dlq.
604async def declare_where_failures_go( 605 transport: Transport, queue: str, retry: RetryPolicy | None = None 606) -> None: 607 """Declares the queues a consumer of ``queue`` will need when it fails. 608 609 ``acemq.dlx``, ``{queue}.dlq``, ``{queue}.parked`` and the two bindings that 610 reach them; and, when ``retry`` has waits long enough to be held by the 611 broker, ``acemq.retry``, one rung queue per distinct wait, and the binding 612 that brings an expired message back to ``queue``. Exactly what 613 :meth:`Topology.queue` writes for the same arguments, and exactly what Java's 614 consumer declares when it starts. 615 616 Called by :meth:`acemq_amqp.Connection.consume` before it subscribes, which 617 is the point of it: everything here is somewhere a message goes when the 618 handler has given up, and a message republished to a queue that is not there 619 is discarded by the broker without a word. Applying a topology first is still 620 the right way to run a service — it is reviewable, and it declares the source 621 queue with the ``x-dead-letter-exchange`` this cannot — but a service that was 622 deployed without one should lose a message to a bug rather than to a missing 623 queue. 624 625 The source queue is deliberately not declared. It belongs to whoever set the 626 service up, who chose its type and its arguments, and guessing wrong is a 627 ``PRECONDITION_FAILED`` that stops the consumer starting at all. 628 629 Safe to call repeatedly, and safe to call after :meth:`Topology.apply` or 630 before it: every declaration here carries the same arguments the topology 631 would send, so the second one to arrive is a no-op rather than a refusal. 632 633 :param transport: where to declare it 634 :param queue: the queue being consumed 635 :param retry: the policy the consumer is running, whose long waits decide 636 which rungs exist. ``None``, or a policy that holds every wait in the 637 consumer, declares only the dead-letter half 638 """ 639 await transport.declare_exchange(DEAD_LETTER_EXCHANGE, _MANAGED_EXCHANGE) 640 for target in (naming.dead_letter_queue(queue), naming.parked_queue(queue)): 641 # Classic, whatever the queue in front of them is, for the reason 642 # :meth:`Topology.queue` gives: these are shared names, and a Java 643 # service declaring the same queue declares it classic. 644 await transport.declare_queue(target, QueueSpec(durable=True)) 645 await transport.bind(target, DEAD_LETTER_EXCHANGE, target) 646 647 # Keyed by name rather than by delay. Two delays can render to the same name, 648 # and a second queue by that name with a different time-to-live is not a 649 # second rung — it is the PRECONDITION_FAILED that stops this consumer 650 # starting. The first delay to reach a name owns it, which is what Java does. 651 rungs: dict[str, timedelta] = {} 652 for delay in retry.broker_rungs() if retry is not None else []: 653 rungs.setdefault(naming.retry_queue(queue, delay), delay) 654 if not rungs: 655 return 656 657 await transport.declare_exchange(RETRY_EXCHANGE, _MANAGED_EXCHANGE) 658 for name, delay in rungs.items(): 659 await transport.declare_queue( 660 name, QueueSpec(durable=True, args=rung_args(queue, delay)) 661 ) 662 # One binding brings every expired message home. Without it a rung expires 663 # into an exchange that routes nowhere, and the broker drops what it cannot 664 # route in silence: the retry looks like one still waiting rather than one 665 # that is gone. 666 await transport.bind(queue, RETRY_EXCHANGE, queue)
Declares the queues a consumer of queue will need when it fails.
acemq.dlx, {queue}.dlq, {queue}.parked and the two bindings that
reach them; and, when retry has waits long enough to be held by the
broker, acemq.retry, one rung queue per distinct wait, and the binding
that brings an expired message back to queue. Exactly what
Topology.queue() writes for the same arguments, and exactly what Java's
consumer declares when it starts.
Called by acemq_amqp.Connection.consume() before it subscribes, which
is the point of it: everything here is somewhere a message goes when the
handler has given up, and a message republished to a queue that is not there
is discarded by the broker without a word. Applying a topology first is still
the right way to run a service — it is reviewable, and it declares the source
queue with the x-dead-letter-exchange this cannot — but a service that was
deployed without one should lose a message to a bug rather than to a missing
queue.
The source queue is deliberately not declared. It belongs to whoever set the
service up, who chose its type and its arguments, and guessing wrong is a
PRECONDITION_FAILED that stops the consumer starting at all.
Safe to call repeatedly, and safe to call after Topology.apply() or
before it: every declaration here carries the same arguments the topology
would send, so the second one to arrive is a no-op rather than a refusal.
Parameters
- transport: where to declare it
- queue: the queue being consumed
- retry: the policy the consumer is running, whose long waits decide
which rungs exist.
None, or a policy that holds every wait in the consumer, declares only the dead-letter half
231def exponential_retry( 232 max_attempts: int, 233 initial_delay: timedelta, 234 max_delay: timedelta = ZERO, 235) -> RetryPolicy: 236 """Doubling delays with 20% jitter, which is the sane default. 237 238 :param max_attempts: total deliveries including the first 239 :param initial_delay: the wait before the second attempt 240 :param max_delay: the ceiling, or zero for none 241 """ 242 return RetryPolicy( 243 max_attempts=max_attempts, 244 initial_delay=initial_delay, 245 multiplier=2.0, 246 max_delay=max_delay, 247 jitter_factor=0.2, 248 )
Doubling delays with 20% jitter, which is the sane default.
Parameters
- max_attempts: total deliveries including the first
- initial_delay: the wait before the second attempt
- max_delay: the ceiling, or zero for none
251def fixed_retry(max_attempts: int, delay: timedelta) -> RetryPolicy: 252 """The same wait every time, with no jitter.""" 253 return RetryPolicy(max_attempts=max_attempts, initial_delay=delay, multiplier=1.0)
The same wait every time, with no jitter.
179def in_progress(delay: timedelta = DEFAULT_IN_PROGRESS_DELAY) -> Ack: 180 """Puts the message back because somebody else is still working on it. 181 182 What :func:`~acemq_amqp.patterns.idempotent` answers when a redelivery finds 183 a claim that is live but not yet confirmed. Accepting it would lose the 184 message if that other handler then died; running it would do the work 185 twice. So it goes back on its own queue after ``delay``, with its attempt 186 **unchanged**: a claim held elsewhere says nothing about whether this 187 message can be handled, so it neither spends a retry nor can exhaust the 188 policy and be dead-lettered. When the claim is confirmed the next look is a 189 duplicate and is accepted; when its lease runs out, it is taken over. 190 191 The wait is spent in the consumer, holding the delivery, as a short retry's 192 is; keep it short beside the store's claim timeout. 193 194 :param delay: how long until the message is looked at again 195 """ 196 return Ack(Action.IN_PROGRESS, delay=delay)
Puts the message back because somebody else is still working on it.
What ~acemq_amqp.patterns.idempotent() answers when a redelivery finds
a claim that is live but not yet confirmed. Accepting it would lose the
message if that other handler then died; running it would do the work
twice. So it goes back on its own queue after delay, with its attempt
unchanged: a claim held elsewhere says nothing about whether this
message can be handled, so it neither spends a retry nor can exhaust the
policy and be dead-lettered. When the claim is confirmed the next look is a
duplicate and is accepted; when its lease runs out, it is taken over.
The wait is spent in the consumer, holding the delivery, as a short retry's is; keep it short beside the store's claim timeout.
Parameters
- delay: how long until the message is looked at again
97def is_development_certificate(certificate: bytes) -> bool: 98 """Whether a certificate carries :data:`DEVELOPMENT_MARKER`. 99 100 Reads DER, and reads PEM by decoding it first — the marker is plain ASCII 101 inside a DER certificate and is nowhere to be seen in the Base64 that wraps 102 it, so a PEM file searched as it stands would come back clean every time. A 103 PEM holding several certificates is read in full, so a chain file is checked 104 entry by entry. 105 106 The marker is looked for in the encoded certificate as a whole rather than in 107 a parsed subject, which is deliberate and is what lets this live in the 108 dependency-free core: an X.509 parser is not in the standard library, and the 109 answer would be the same one. Any distinguished name carrying the marker — 110 subject or issuer, ``O=`` as Go and .NET write it or ``OU=`` as Java does — 111 puts these bytes in the encoding, and there is nothing else a certificate 112 could plausibly contain them for. 113 114 Erring towards refusing is the right direction for this particular question. 115 A certificate that somehow contains the marker without being a development 116 one is refused, which is an inconvenience; the failure in the other direction 117 is a production system trusting an authority whose private key is on GitHub. 118 119 :param certificate: the certificate, DER-encoded or PEM-encoded 120 :returns: whether it is marked as a development certificate 121 """ 122 if _MARKER_BYTES_LOWER in certificate.lower(): 123 return True 124 return any(_MARKER_BYTES_LOWER in block.lower() for block in _der_blocks_in(certificate))
Whether a certificate carries DEVELOPMENT_MARKER.
Reads DER, and reads PEM by decoding it first — the marker is plain ASCII inside a DER certificate and is nowhere to be seen in the Base64 that wraps it, so a PEM file searched as it stands would come back clean every time. A PEM holding several certificates is read in full, so a chain file is checked entry by entry.
The marker is looked for in the encoded certificate as a whole rather than in
a parsed subject, which is deliberate and is what lets this live in the
dependency-free core: an X.509 parser is not in the standard library, and the
answer would be the same one. Any distinguished name carrying the marker —
subject or issuer, O= as Go and .NET write it or OU= as Java does —
puts these bytes in the encoding, and there is nothing else a certificate
could plausibly contain them for.
Erring towards refusing is the right direction for this particular question. A certificate that somehow contains the marker without being a development one is refused, which is an inconvenience; the failure in the other direction is a production system trusting an authority whose private key is on GitHub.
Parameters
- certificate: the certificate, DER-encoded or PEM-encoded :returns: whether it is marked as a development certificate
226def no_retry() -> RetryPolicy: 227 """One delivery, no second chance.""" 228 return RetryPolicy(max_attempts=1)
One delivery, no second chance.
162def park(error: BaseException | None = None) -> Ack: 163 """Sets the message aside on ``{queue}.parked`` without trying again. 164 165 For when the message is not readable at all: a body in a format nothing 166 here understands, a version this service was never taught. The engine 167 already parks a body its codec cannot decode; this is how a handler that 168 worked out the same thing one layer further in says so. 169 170 :func:`reject` is the neighbouring word and the difference is worth 171 keeping. A rejected message is a *bad request* — understood, and refused — 172 and it belongs with the other dead letters. A parked one was never 173 understood, and mixing the two means whoever drains the dead-letter queue 174 has to sort them by hand to find the producer that is emitting rubbish. 175 """ 176 return Ack(Action.PARK, error)
Sets the message aside on {queue}.parked without trying again.
For when the message is not readable at all: a body in a format nothing here understands, a version this service was never taught. The engine already parks a body its codec cannot decode; this is how a handler that worked out the same thing one layer further in says so.
reject() is the neighbouring word and the difference is worth
keeping. A rejected message is a bad request — understood, and refused —
and it belongs with the other dead letters. A parked one was never
understood, and mixing the two means whoever drains the dead-letter queue
has to sort them by hand to find the producer that is emitting rubbish.
40def parked_queue(queue: str) -> str: 41 """``orders.new`` becomes ``orders.new.parked``.""" 42 return queue + PARKED_SUFFIX
orders.new becomes orders.new.parked.
454def prometheus_text(metrics: Metrics) -> str: 455 """Renders a :class:`Metrics` in the Prometheus text format. 456 457 Written out rather than through a client library, so a service that only 458 wants a scrape endpoint needs nothing installed. The format is small and 459 stable: a TYPE line, then samples, with dots in the metric name — and in the 460 label names, where ``routing.key`` and ``message.type`` have the same 461 problem — replaced by underscores, because Prometheus allows them in 462 neither. 463 464 A real registry — exemplars, native histograms, a shared process collector — 465 is what :mod:`acemq_amqp.prometheus` is for, and it needs the extra. 466 467 :param metrics: what to render 468 :returns: the body of a ``/acemq-metrics`` response 469 """ 470 lines: list[str] = [] 471 for name, value in sorted(metrics.counts.items()): 472 metric, labels = _split_key(name) 473 base = _prometheus_name(metric) 474 lines += [f"# TYPE {base} counter", f"{base}{labels} {value}"] 475 for name, value in sorted(metrics.gauges.items()): 476 metric, labels = _split_key(name) 477 base = _prometheus_name(metric) 478 lines += [f"# TYPE {base} gauge", f"{base}{labels} {value}"] 479 for name, summary in sorted(metrics.durations.items()): 480 metric, labels = _split_key(name) 481 base = _prometheus_name(metric) 482 lines += [ 483 f"# TYPE {base} summary", 484 f"{base}_count{labels} {summary.count}", 485 f"{base}_sum{labels} {summary.total}", 486 ] 487 return "\n".join(lines) + ("\n" if lines else "")
Renders a Metrics in the Prometheus text format.
Written out rather than through a client library, so a service that only
wants a scrape endpoint needs nothing installed. The format is small and
stable: a TYPE line, then samples, with dots in the metric name — and in the
label names, where routing.key and message.type have the same
problem — replaced by underscores, because Prometheus allows them in
neither.
A real registry — exemplars, native histograms, a shared process collector —
is what acemq_amqp.prometheus is for, and it needs the extra.
Parameters
- metrics: what to render
:returns: the body of a
/acemq-metricsresponse
228def publish_chain( 229 interceptors: Sequence[PublishInterceptor], publish: PublishNext 230) -> PublishNext: 231 """Folds interceptors around a publish, first registered outermost. 232 233 Built backwards because that is what nesting is: the last interceptor wraps 234 the publish, the one before wraps that, and the first ends up on the 235 outside, seeing the message before anybody and its result after everybody. 236 237 :param interceptors: what to wrap it in, in registration order 238 :param publish: the innermost work, which actually sends the message 239 :returns: the whole thing as one callable, or ``publish`` when there is 240 nothing to wrap it in 241 """ 242 composed = publish 243 for interceptor in reversed(interceptors): 244 composed = _publish_step(interceptor, composed) 245 return composed
Folds interceptors around a publish, first registered outermost.
Built backwards because that is what nesting is: the last interceptor wraps the publish, the one before wraps that, and the first ends up on the outside, seeing the message before anybody and its result after everybody.
Parameters
- interceptors: what to wrap it in, in registration order
- publish: the innermost work, which actually sends the message
:returns: the whole thing as one callable, or
publishwhen there is nothing to wrap it in
305def register_codec(name: str, build: Callable[[], Codec]) -> None: 306 """Makes a codec available by name. 307 308 So that configuration can name a format without the code that reads the 309 configuration importing every format it might name. Registering the same 310 name twice replaces the first, which is what lets a test override a default. 311 312 :param name: what configuration will call it, such as ``"json"`` 313 :param build: makes an instance 314 """ 315 with _registry_lock: 316 _registry[name] = build
Makes a codec available by name.
So that configuration can name a format without the code that reads the configuration importing every format it might name. Registering the same name twice replaces the first, which is what lets a test override a default.
Parameters
- name: what configuration will call it, such as
"json" - build: makes an instance
152def reject(error: BaseException | None = None) -> Ack: 153 """Dead-letters the message without trying again. 154 155 For when the message itself is the problem — a field that cannot be missing 156 is missing, a reference points at nothing — rather than when the world is 157 temporarily unhelpful. 158 """ 159 return Ack(Action.REJECT, error)
Dead-letters the message without trying again.
For when the message itself is the problem — a field that cannot be missing is missing, a reference points at nothing — rather than when the world is temporarily unhelpful.
141def retry(error: BaseException | None = None) -> Ack: 142 """Returns the message to be tried again. 143 144 The retry policy decides whether there is another attempt left; when there 145 is not, the message is dead-lettered with ``error`` as the reason. A 146 :class:`Fatal` error skips the remaining attempts, because they would all 147 fail the same way. 148 """ 149 return Ack(Action.RETRY, error)
Returns the message to be tried again.
The retry policy decides whether there is another attempt left; when there
is not, the message is dead-lettered with error as the reason. A
Fatal error skips the remaining attempts, because they would all
fail the same way.
45def retry_queue(queue: str, delay: timedelta) -> str: 46 """``orders.new`` and 30s become ``orders.new.retry.30s``. 47 48 The delay is in the name because a delay queue is per-delay: its 49 ``x-message-ttl`` is fixed at declaration, so a policy with four different 50 waits needs four queues, and an operator should be able to tell which is 51 which without reading its arguments. 52 """ 53 return f"{queue}.retry.{_short(delay)}"
orders.new and 30s become orders.new.retry.30s.
The delay is in the name because a delay queue is per-delay: its
x-message-ttl is fixed at declaration, so a policy with four different
waits needs four queues, and an operator should be able to tell which is
which without reading its arguments.
120def rung_args(source: str, delay: timedelta) -> dict[str, Any]: 121 """The arguments a ``{source}.retry.{delay}`` queue must be declared with. 122 123 Exactly three keys, and the same three in every AceMQ library: the delay as 124 ``x-message-ttl``, :data:`RETRY_EXCHANGE` as the dead-letter target, and the 125 source queue as the routing key it is sent under. Nothing consumes a rung; 126 the time-to-live is the only thing that ever takes a message out of one, and 127 the binding made by :meth:`Topology.queue` is what brings it home. 128 129 The TTL is on the queue rather than on each message, which is the whole 130 reason a policy needs several queues instead of one. RabbitMQ expires 131 messages only from the head of a queue, so a single queue holding 132 per-message TTLs releases nothing while a message with a long one sits at 133 the front: a thirty-second wait queued behind a ten-minute wait becomes a 134 ten-minute wait. The delays that come out would bear no relation to the ones 135 that went in, and the bug would only appear under the load that puts two 136 different waits on the queue at once. 137 138 :param source: the queue being consumed, which is where an expired message 139 goes back to 140 :param delay: how long the rung holds a message 141 :returns: the argument table, which is a contract and is pinned by a test 142 """ 143 return { 144 MESSAGE_TTL_ARG: int(delay.total_seconds() * 1000), 145 DEAD_LETTER_EXCHANGE_ARG: RETRY_EXCHANGE, 146 DEAD_LETTER_ROUTING_KEY_ARG: source, 147 }
The arguments a {source}acemq_amqp.retry.{delay} queue must be declared with.
Exactly three keys, and the same three in every AceMQ library: the delay as
x-message-ttl, RETRY_EXCHANGE as the dead-letter target, and the
source queue as the routing key it is sent under. Nothing consumes a rung;
the time-to-live is the only thing that ever takes a message out of one, and
the binding made by Topology.queue() is what brings it home.
The TTL is on the queue rather than on each message, which is the whole reason a policy needs several queues instead of one. RabbitMQ expires messages only from the head of a queue, so a single queue holding per-message TTLs releases nothing while a message with a long one sits at the front: a thirty-second wait queued behind a ten-minute wait becomes a ten-minute wait. The delays that come out would bear no relation to the ones that went in, and the bug would only appear under the load that puts two different waits on the queue at once.
Parameters
- source: the queue being consumed, which is where an expired message goes back to
- delay: how long the rung holds a message :returns: the argument table, which is a contract and is pinned by a test
648def without_verifying_the_broker( 649 *, 650 client_certificate: FilePath | None = None, 651 client_key: FilePath | None = None, 652 client_key_password: str | None = None, 653 credentials: Credentials | CredentialsSource | None = None, 654) -> Security: 655 """Encrypts the connection and checks nothing about who is on the other end. 656 657 **This gives up most of what TLS is for, and it is worth being exact about 658 which part.** The traffic cannot be read by somebody watching the network. 659 Nothing whatsoever stops that somebody from *being* the broker: any 660 certificate is accepted, from any issuer, for any name, so an attacker who 661 can answer on the address in your URL receives every message you publish and 662 every password you log in with, over a connection that looks encrypted in 663 every log and every metric. 664 665 It exists because there is one situation where the alternative is worse: a 666 broker with a self-signed certificate on a laptop or in a test fixture, 667 where the choice is otherwise between this and turning encryption off 668 entirely. It is deliberately a function with a long name rather than a 669 ``verify=False``, because it should be impossible to end up here by pasting 670 a keyword argument and difficult to leave in a file nobody rereads. 671 672 **It is wrong in production, always.** Against a real broker, name the 673 authority instead — ``Security(certificate_authority="ca.crt")`` — which is 674 one more line and gives back everything this gives up. Self-signed 675 certificates are fine there too: put the broker's own certificate in the 676 file and it becomes the one authority you trust. 677 678 :param client_certificate: a PEM certificate to present 679 :param client_key: the PEM private key for it 680 :param client_key_password: the passphrase on that key, when it has one 681 :param credentials: the broker login, or a source asked for it 682 :returns: the configuration 683 """ 684 return Security( 685 client_certificate=client_certificate, 686 client_key=client_key, 687 client_key_password=client_key_password, 688 credentials=credentials, 689 verification=Verification.NOTHING_AT_ALL, 690 )
Encrypts the connection and checks nothing about who is on the other end.
This gives up most of what TLS is for, and it is worth being exact about which part. The traffic cannot be read by somebody watching the network. Nothing whatsoever stops that somebody from being the broker: any certificate is accepted, from any issuer, for any name, so an attacker who can answer on the address in your URL receives every message you publish and every password you log in with, over a connection that looks encrypted in every log and every metric.
It exists because there is one situation where the alternative is worse: a
broker with a self-signed certificate on a laptop or in a test fixture,
where the choice is otherwise between this and turning encryption off
entirely. It is deliberately a function with a long name rather than a
verify=False, because it should be impossible to end up here by pasting
a keyword argument and difficult to leave in a file nobody rereads.
It is wrong in production, always. Against a real broker, name the
authority instead — Security(certificate_authority="ca.crt") — which is
one more line and gives back everything this gives up. Self-signed
certificates are fine there too: put the broker's own certificate in the
file and it becomes the one authority you trust.
Parameters
- client_certificate: a PEM certificate to present
- client_key: the PEM private key for it
- client_key_password: the passphrase on that key, when it has one
- credentials: the broker login, or a source asked for it :returns: the configuration