acemq_amqp

AceMQ for Python.

A client for AceMQ messaging over AMQP, speaking the same wire contract as the Java , Go and .NET libraries: the same reserved headers, the same defaults, the same retry semantics. A Python consumer reads what a Java producer writes, and the fixtures generated from the Java implementation pin that rather than leaving it to be discovered in production.

The API shape is Python's, deliberately. The contract is portable; the ergonomics are native.

Importing this package needs nothing installed. connect() reaches for aio-pika at the moment it is called rather than at import, so a program that only reads an envelope off a message somebody else delivered never has to have an AMQP client at all. The blocking API lives in acemq_amqp.sync for programs that are not running an event loop.

Reaching a real broker means reaching it over TLS: an amqps:// URL is verified against the machine's trust store with no further configuration, and Security in acemq_amqp.security covers a private certificate authority, a client certificate, and a login supplied separately from the URL so that a password never has to be written into a connection string.

  1# Copyright 2026 AceMQ.
  2#
  3# Licensed under the Apache License, Version 2.0 (the "License");
  4# you may not use this file except in compliance with the License.
  5# You may obtain a copy of the License at
  6#
  7#     https://www.apache.org/licenses/LICENSE-2.0
  8#
  9# Unless required by applicable law or agreed to in writing, software
 10# distributed under the License is distributed on an "AS IS" BASIS,
 11# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 12# See the License for the specific language governing permissions and
 13# limitations under the License.
 14
 15"""AceMQ for Python.
 16
 17A client for AceMQ messaging over AMQP, speaking the same wire contract as the
 18`Java <https://github.com/AceMQ-Company/acemq-java-amqp>`_, `Go
 19<https://github.com/AceMQ-Company/acemq-go-amqp>`_ and `.NET
 20<https://github.com/AceMQ-Company/acemq-dotnet-amqp>`_ libraries: the same
 21reserved headers, the same defaults, the same retry semantics. A Python
 22consumer reads what a Java producer writes, and the fixtures generated from the
 23Java implementation pin that rather than leaving it to be discovered in
 24production.
 25
 26The API shape is Python's, deliberately. The contract is portable; the
 27ergonomics are native.
 28
 29Importing this package needs nothing installed. :func:`connect` reaches for
 30aio-pika at the moment it is called rather than at import, so a program that
 31only reads an envelope off a message somebody else delivered never has to have
 32an AMQP client at all. The blocking API lives in :mod:`acemq_amqp.sync` for
 33programs that are not running an event loop.
 34
 35Reaching a real broker means reaching it over TLS: an ``amqps://`` URL is
 36verified against the machine's trust store with no further configuration, and
 37:class:`Security` in :mod:`acemq_amqp.security` covers a private certificate
 38authority, a client certificate, and a login supplied separately from the URL so
 39that a password never has to be written into a connection string.
 40"""
 41
 42from __future__ import annotations
 43
 44from . import headers
 45from .ack import Ack, Action, FatalError, Settlement, accept, in_progress, park, reject
 46from .ack import retry as _retry
 47from .codec import (
 48    BYTES_CONTENT_TYPE,
 49    JSON_CONTENT_TYPE,
 50    TEXT_CONTENT_TYPE,
 51    BytesCodec,
 52    Codec,
 53    CompositeCodec,
 54    JsonCodec,
 55    TextCodec,
 56    codec_by_name,
 57    codec_names,
 58    register_codec,
 59)
 60from .connection import (
 61    BLOCKED_DETAIL,
 62    DEFAULT_DRAIN_TIMEOUT,
 63    DEFAULT_HEALTH_TIMEOUT,
 64    AsyncHandler,
 65    BrokerHealth,
 66    Connection,
 67    Consumer,
 68    Handler,
 69    Message,
 70    Publisher,
 71    connect,
 72)
 73from .envelope import Envelope
 74from .errors import AceMQError, PublishError, PublishingPausedError, SecurityError
 75from .interceptors import (
 76    ConsumeContext,
 77    ConsumeInterceptor,
 78    ConsumeNext,
 79    PublishContext,
 80    PublishInterceptor,
 81    PublishNext,
 82    SettlementListener,
 83    consume_chain,
 84    publish_chain,
 85)
 86from .naming import dead_letter_queue, parked_queue, retry_queue
 87from .retry import (
 88    DEFAULT_BROKER_WAIT_THRESHOLD,
 89    RetryPolicy,
 90    Wait,
 91    exponential_retry,
 92    fixed_retry,
 93    no_retry,
 94)
 95from .security import (
 96    DEVELOPMENT_MARKER,
 97    Credentials,
 98    CredentialsSource,
 99    Security,
100    Verification,
101    credentials_from_environment,
102    credentials_from_file,
103    is_development_certificate,
104    without_verifying_the_broker,
105)
106from .telemetry import (
107    METRIC_CONSUME_ATTEMPTS,
108    METRIC_CONSUME_DURATION,
109    METRIC_CONSUME_IN_FLIGHT,
110    METRIC_CONSUME_TOTAL,
111    METRIC_DEAD_LETTERED_TOTAL,
112    METRIC_OUTBOX_LAG,
113    METRIC_OUTBOX_TOTAL,
114    METRIC_PIPELINE_RUN_DURATION,
115    METRIC_PIPELINE_RUN_TOTAL,
116    METRIC_PUBLISH_DURATION,
117    METRIC_PUBLISH_TOTAL,
118    METRIC_REQUEST_DURATION,
119    METRIC_REQUEST_TOTAL,
120    METRIC_RETRIED_TOTAL,
121    METRIC_RUNG_MISSING,
122    METRIC_SET_ASIDE_FAILED,
123    DurationSummary,
124    HealthCheck,
125    HealthReport,
126    HealthStatus,
127    Metrics,
128    NullObserver,
129    Observer,
130    aggregate_health,
131    prometheus_text,
132)
133from .topology import (
134    DEAD_LETTER_EXCHANGE,
135    QUEUE_TYPE_ARG,
136    QUORUM_QUEUE_TYPE,
137    RETRY_EXCHANGE,
138    Topology,
139    declare_where_failures_go,
140    rung_args,
141)
142from .transport import (
143    BlockedState,
144    ConsumeSpec,
145    Delivery,
146    ExchangeSpec,
147    Outbound,
148    PublishResult,
149    QueueSpec,
150    Transport,
151)
152
153#: Returns a message to be tried again. See :func:`acemq_amqp.ack.retry`.
154#:
155#: Bound here rather than imported by name, because importing the retry module a
156#: few lines above binds ``acemq_amqp.retry`` to the module and the last write
157#: wins. A module is not what ``from acemq_amqp import retry`` should hand
158#: somebody when the other four libraries hand them the acknowledgement, and
159#: ``from acemq_amqp.retry import RetryPolicy`` still reaches the module.
160retry = _retry
161
162__version__ = "0.7.12"
163
164__all__ = [
165    "BLOCKED_DETAIL",
166    "BYTES_CONTENT_TYPE",
167    "DEAD_LETTER_EXCHANGE",
168    "DEFAULT_BROKER_WAIT_THRESHOLD",
169    "DEFAULT_DRAIN_TIMEOUT",
170    "DEFAULT_HEALTH_TIMEOUT",
171    "DEVELOPMENT_MARKER",
172    "JSON_CONTENT_TYPE",
173    "METRIC_CONSUME_ATTEMPTS",
174    "METRIC_CONSUME_DURATION",
175    "METRIC_CONSUME_IN_FLIGHT",
176    "METRIC_CONSUME_TOTAL",
177    "METRIC_DEAD_LETTERED_TOTAL",
178    "METRIC_OUTBOX_LAG",
179    "METRIC_OUTBOX_TOTAL",
180    "METRIC_PIPELINE_RUN_DURATION",
181    "METRIC_PIPELINE_RUN_TOTAL",
182    "METRIC_PUBLISH_DURATION",
183    "METRIC_PUBLISH_TOTAL",
184    "METRIC_REQUEST_DURATION",
185    "METRIC_REQUEST_TOTAL",
186    "METRIC_RETRIED_TOTAL",
187    "METRIC_RUNG_MISSING",
188    "METRIC_SET_ASIDE_FAILED",
189    "QUEUE_TYPE_ARG",
190    "QUORUM_QUEUE_TYPE",
191    "RETRY_EXCHANGE",
192    "TEXT_CONTENT_TYPE",
193    "AceMQError",
194    "Ack",
195    "Action",
196    "AsyncHandler",
197    "BlockedState",
198    "BrokerHealth",
199    "BytesCodec",
200    "Codec",
201    "CompositeCodec",
202    "Connection",
203    "ConsumeContext",
204    "ConsumeInterceptor",
205    "ConsumeNext",
206    "ConsumeSpec",
207    "Consumer",
208    "Credentials",
209    "CredentialsSource",
210    "Delivery",
211    "DurationSummary",
212    "Envelope",
213    "ExchangeSpec",
214    "FatalError",
215    "Handler",
216    "HealthCheck",
217    "HealthReport",
218    "HealthStatus",
219    "JsonCodec",
220    "Message",
221    "Metrics",
222    "NullObserver",
223    "Observer",
224    "Outbound",
225    "PublishContext",
226    "PublishError",
227    "PublishInterceptor",
228    "PublishNext",
229    "PublishResult",
230    "Publisher",
231    "PublishingPausedError",
232    "QueueSpec",
233    "RetryPolicy",
234    "Security",
235    "SecurityError",
236    "Settlement",
237    "SettlementListener",
238    "TextCodec",
239    "Topology",
240    "Transport",
241    "Verification",
242    "Wait",
243    "__version__",
244    "accept",
245    "aggregate_health",
246    "codec_by_name",
247    "codec_names",
248    "connect",
249    "consume_chain",
250    "credentials_from_environment",
251    "credentials_from_file",
252    "dead_letter_queue",
253    "declare_where_failures_go",
254    "exponential_retry",
255    "fixed_retry",
256    "headers",
257    "in_progress",
258    "is_development_certificate",
259    "no_retry",
260    "park",
261    "parked_queue",
262    "prometheus_text",
263    "publish_chain",
264    "register_codec",
265    "reject",
266    "retry",
267    "retry_queue",
268    "rung_args",
269    "without_verifying_the_broker",
270]
BLOCKED_DETAIL = 'the broker has blocked this connection; publishing is paused'
BYTES_CONTENT_TYPE = 'application/octet-stream'
DEAD_LETTER_EXCHANGE = 'acemq.dlx'
DEFAULT_BROKER_WAIT_THRESHOLD = datetime.timedelta(seconds=30)
DEFAULT_DRAIN_TIMEOUT = 20.0
DEFAULT_HEALTH_TIMEOUT = 3.0
DEVELOPMENT_MARKER = 'ACEMQ DEVELOPMENT ONLY - DO NOT TRUST'
JSON_CONTENT_TYPE = 'application/json'
METRIC_CONSUME_ATTEMPTS = 'acemq.consume.attempts'
METRIC_CONSUME_DURATION = 'acemq.consume.duration'
METRIC_CONSUME_IN_FLIGHT = 'acemq.consume.in.flight'
METRIC_CONSUME_TOTAL = 'acemq.consume.total'
METRIC_DEAD_LETTERED_TOTAL = 'acemq.messages.dead.lettered.total'
METRIC_OUTBOX_LAG = 'acemq.outbox.lag'
METRIC_OUTBOX_TOTAL = 'acemq.outbox.total'
METRIC_PIPELINE_RUN_DURATION = 'acemq.pipeline.run.duration'
METRIC_PIPELINE_RUN_TOTAL = 'acemq.pipeline.run.total'
METRIC_PUBLISH_DURATION = 'acemq.publish.duration'
METRIC_PUBLISH_TOTAL = 'acemq.publish.total'
METRIC_REQUEST_DURATION = 'acemq.request.duration'
METRIC_REQUEST_TOTAL = 'acemq.request.total'
METRIC_RETRIED_TOTAL = 'acemq.messages.retried.total'
METRIC_RUNG_MISSING = 'acemq.retry.rung.missing'
METRIC_SET_ASIDE_FAILED = 'acemq.messages.set.aside.failed'
QUEUE_TYPE_ARG = 'x-queue-type'
QUORUM_QUEUE_TYPE = 'quorum'
RETRY_EXCHANGE = 'acemq.retry'
TEXT_CONTENT_TYPE = 'text/plain; charset=utf-8'
class AceMQError(builtins.Exception):
31class AceMQError(Exception):
32    """Something this library refused to do."""

Something this library refused to do.

@dataclass(frozen=True, slots=True)
class Ack:
70@dataclass(frozen=True, slots=True)
71class Ack:
72    """A handler's decision: it worked, try it again, never again, or nobody
73    can read it.
74
75    Returned rather than performed, so a handler that forgets to decide is a
76    handler that returns ``None`` and is caught immediately, rather than one
77    whose message sits unacknowledged until the connection drops and then comes
78    back — usually to the same handler, with the same outcome.
79    """
80
81    action: Action
82    error: BaseException | None = None
83    #: How long to wait before the message is looked at again. Read for
84    #: :attr:`Action.IN_PROGRESS` only; a retry's wait is the policy's.
85    delay: timedelta | None = None
86
87    def __str__(self) -> str:
88        return self.action.value

A handler's decision: it worked, try it again, never again, or nobody can read it.

Returned rather than performed, so a handler that forgets to decide is a handler that returns None and is caught immediately, rather than one whose message sits unacknowledged until the connection drops and then comes back — usually to the same handler, with the same outcome.

Ack( action: Action, error: BaseException | None = None, delay: datetime.timedelta | None = None)
action: Action
error: BaseException | None
delay: datetime.timedelta | None
class Action(enum.Enum):
60class Action(Enum):
61    """The things that can be done with a delivered message."""
62
63    ACCEPT = "accept"
64    RETRY = "retry"
65    REJECT = "reject"
66    PARK = "park"
67    IN_PROGRESS = "in_progress"

The things that can be done with a delivered message.

ACCEPT = <Action.ACCEPT: 'accept'>
RETRY = <Action.RETRY: 'retry'>
REJECT = <Action.REJECT: 'reject'>
PARK = <Action.PARK: 'park'>
IN_PROGRESS = <Action.IN_PROGRESS: 'in_progress'>
AsyncHandler = collections.abc.Callable[[Message], collections.abc.Awaitable[Ack]]
@runtime_checkable
class BlockedState(typing.Protocol):
230@runtime_checkable
231class BlockedState(Protocol):
232    """A transport that can say whether the broker has blocked its connection.
233
234    RabbitMQ sends ``connection.blocked`` when it is low on memory or disk and
235    ``connection.unblocked`` when the alarm clears. In between it stops reading
236    from the socket, so everything sent on that connection — a publish, a
237    declaration, a health probe — waits rather than failing. From outside that
238    is indistinguishable from a broker that has silently gone away, and the two
239    want opposite responses: a blocked broker is one to wait for, a dead one is
240    one to fail over from.
241
242    Separate from :class:`Transport` because not every transport has an answer.
243    A fake in a test is never blocked and can say so; a client library that
244    keeps the state to itself cannot be asked at all, which is what ``None``
245    below is for. It is the same shape as Java's ``isBlocked()`` and
246    ``blockedReason()``, .NET's ``IsBlocked``/``BlockedReason`` and Go's
247    ``BlockedReason()``.
248    """
249
250    @property
251    def blocked(self) -> bool | None:
252        """Whether the broker has blocked this connection.
253
254        ``None`` rather than ``False`` when the question could not be asked at
255        all — a client that does not expose the state, a connection that is
256        between reconnections. The two are different facts and a report that
257        folds them together says a broker is fine when nobody looked.
258        """
259
260    @property
261    def blocked_reason(self) -> str | None:
262        """What the broker said when it blocked this connection.
263
264        ``None`` when it is not blocked, and also ``None`` when the client read
265        the reason and did not keep it — which is what aio-pika does, and why
266        this is separate from :attr:`blocked` rather than a string that is empty
267        when there is nothing to say. A reason invented here would read exactly
268        like one RabbitMQ sent.
269        """

A transport that can say whether the broker has blocked its connection.

RabbitMQ sends connection.blocked when it is low on memory or disk and connection.unblocked when the alarm clears. In between it stops reading from the socket, so everything sent on that connection — a publish, a declaration, a health probe — waits rather than failing. From outside that is indistinguishable from a broker that has silently gone away, and the two want opposite responses: a blocked broker is one to wait for, a dead one is one to fail over from.

Separate from Transport because not every transport has an answer. A fake in a test is never blocked and can say so; a client library that keeps the state to itself cannot be asked at all, which is what None below is for. It is the same shape as Java's isBlocked() and blockedReason(), .NET's IsBlocked/BlockedReason and Go's BlockedReason().

BlockedState(*args, **kwargs)
1771def _no_init_or_replace_init(self, *args, **kwargs):
1772    cls = type(self)
1773
1774    if cls._is_protocol:
1775        raise TypeError('Protocols cannot be instantiated')
1776
1777    # Already using a custom `__init__`. No need to calculate correct
1778    # `__init__` to call. This can lead to RecursionError. See bpo-45121.
1779    if cls.__init__ is not _no_init_or_replace_init:
1780        return
1781
1782    # Initially, `__init__` of a protocol subclass is set to `_no_init_or_replace_init`.
1783    # The first instantiation of the subclass will call `_no_init_or_replace_init` which
1784    # searches for a proper new `__init__` in the MRO. The new `__init__`
1785    # replaces the subclass' old `__init__` (ie `_no_init_or_replace_init`). Subsequent
1786    # instantiation of the protocol subclass will thus use the new
1787    # `__init__` and no longer call `_no_init_or_replace_init`.
1788    for base in cls.__mro__:
1789        init = base.__dict__.get('__init__', _no_init_or_replace_init)
1790        if init is not _no_init_or_replace_init:
1791            cls.__init__ = init
1792            break
1793    else:
1794        # should not happen
1795        cls.__init__ = object.__init__
1796
1797    cls.__init__(self, *args, **kwargs)
blocked: bool | None
250    @property
251    def blocked(self) -> bool | None:
252        """Whether the broker has blocked this connection.
253
254        ``None`` rather than ``False`` when the question could not be asked at
255        all — a client that does not expose the state, a connection that is
256        between reconnections. The two are different facts and a report that
257        folds them together says a broker is fine when nobody looked.
258        """

Whether the broker has blocked this connection.

None rather than False when the question could not be asked at all — a client that does not expose the state, a connection that is between reconnections. The two are different facts and a report that folds them together says a broker is fine when nobody looked.

blocked_reason: str | None
260    @property
261    def blocked_reason(self) -> str | None:
262        """What the broker said when it blocked this connection.
263
264        ``None`` when it is not blocked, and also ``None`` when the client read
265        the reason and did not keep it — which is what aio-pika does, and why
266        this is separate from :attr:`blocked` rather than a string that is empty
267        when there is nothing to say. A reason invented here would read exactly
268        like one RabbitMQ sent.
269        """

What the broker said when it blocked this connection.

None when it is not blocked, and also None when the client read the reason and did not keep it — which is what aio-pika does, and why this is separate from blocked rather than a string that is empty when there is nothing to say. A reason invented here would read exactly like one RabbitMQ sent.

@dataclass(frozen=True, slots=True)
class BrokerHealth:
1977@dataclass(frozen=True, slots=True)
1978class BrokerHealth:
1979    """A connection as a :class:`~acemq_amqp.telemetry.HealthCheck`.
1980
1981    So that a broker check and an application's own — a database, a downstream
1982    service — can be handed to
1983    :func:`~acemq_amqp.telemetry.aggregate_health` together, which is what a
1984    readiness endpoint actually needs.
1985
1986    A blocked broker comes back through here as **up, with the reason**, and
1987    nothing in between folds it into anything else: the aggregate takes the
1988    worst status it is given, so a check that reported back pressure as degraded
1989    would quietly degrade a report that had decided otherwise. The deadline is
1990    the same argument — :attr:`timeout` is shorter than the aggregate's, so that
1991    a broker which has stopped answering is described by this check rather than
1992    by the aggregate giving up on it.
1993
1994    :param connection: what to ask
1995    :param label: what to call it in a combined report
1996    :param timeout: how long to give the broker's answer. See
1997        :meth:`Connection.health`
1998    """
1999
2000    connection: Connection
2001    label: str = "broker"
2002    timeout: float = DEFAULT_HEALTH_TIMEOUT
2003
2004    @property
2005    def name(self) -> str:
2006        return self.label
2007
2008    async def check(self) -> HealthReport:
2009        return await self.connection.health(self.timeout)

A connection as a ~acemq_amqp.telemetry.HealthCheck.

So that a broker check and an application's own — a database, a downstream service — can be handed to ~acemq_amqp.telemetry.aggregate_health() together, which is what a readiness endpoint actually needs.

A blocked broker comes back through here as up, with the reason, and nothing in between folds it into anything else: the aggregate takes the worst status it is given, so a check that reported back pressure as degraded would quietly degrade a report that had decided otherwise. The deadline is the same argument — timeout is shorter than the aggregate's, so that a broker which has stopped answering is described by this check rather than by the aggregate giving up on it.

Parameters
  • connection: what to ask
  • label: what to call it in a combined report
  • timeout: how long to give the broker's answer. See Connection.health()
BrokerHealth( connection: Connection, label: str = 'broker', timeout: float = 3.0)
connection: Connection
label: str
timeout: float
name: str
2004    @property
2005    def name(self) -> str:
2006        return self.label
async def check(self) -> HealthReport:
2008    async def check(self) -> HealthReport:
2009        return await self.connection.health(self.timeout)
class BytesCodec:
145class BytesCodec:
146    """Passes bodies through without touching them.
147
148    For a payload that is already encoded — an image, something another system's
149    serializer produced — and for reading a message this process has no type
150    for. Replaying a dead-lettered message uses it: the bytes that were
151    committed are the bytes that should go back, and re-encoding through a class
152    that has since gained a field would produce something else.
153
154    It answers for every content type, which is why it has to be asked for
155    rather than found: put it last in a :class:`CompositeCodec`, or nothing
156    after it is ever reached.
157    """
158
159    @property
160    def content_type(self) -> str:
161        return BYTES_CONTENT_TYPE
162
163    def encode(self, payload: Any) -> bytes:
164        if payload is None:
165            return b""
166        if isinstance(payload, bytes):
167            return payload
168        if isinstance(payload, bytearray | memoryview):
169            return bytes(payload)
170        if isinstance(payload, str):
171            return payload.encode("utf-8")
172        raise TypeError(
173            f"acemq: BytesCodec cannot encode a {type(payload).__name__}; "
174            "it takes bytes or a string"
175        )
176
177    def decode(self, body: bytes, content_type: str | None = None) -> Any:
178        return bytes(body)
179
180    def can_decode(self, content_type: str | None) -> bool:
181        return True

Passes bodies through without touching them.

For a payload that is already encoded — an image, something another system's serializer produced — and for reading a message this process has no type for. Replaying a dead-lettered message uses it: the bytes that were committed are the bytes that should go back, and re-encoding through a class that has since gained a field would produce something else.

It answers for every content type, which is why it has to be asked for rather than found: put it last in a CompositeCodec, or nothing after it is ever reached.

content_type: str
159    @property
160    def content_type(self) -> str:
161        return BYTES_CONTENT_TYPE
def encode(self, payload: Any) -> bytes:
163    def encode(self, payload: Any) -> bytes:
164        if payload is None:
165            return b""
166        if isinstance(payload, bytes):
167            return payload
168        if isinstance(payload, bytearray | memoryview):
169            return bytes(payload)
170        if isinstance(payload, str):
171            return payload.encode("utf-8")
172        raise TypeError(
173            f"acemq: BytesCodec cannot encode a {type(payload).__name__}; "
174            "it takes bytes or a string"
175        )
def decode(self, body: bytes, content_type: str | None = None) -> Any:
177    def decode(self, body: bytes, content_type: str | None = None) -> Any:
178        return bytes(body)
def can_decode(self, content_type: str | None) -> bool:
180    def can_decode(self, content_type: str | None) -> bool:
181        return True
@runtime_checkable
class Codec(typing.Protocol):
53@runtime_checkable
54class Codec(Protocol):
55    """Reads and writes message bodies in one format."""
56
57    @property
58    def content_type(self) -> str:
59        """What this codec writes onto a message it encodes."""
60
61    def encode(self, payload: Any) -> bytes:
62        """Turns a payload into bytes.
63
64        :param payload: what the application wants to send
65        :returns: the body
66        :raises TypeError: when the payload is not something this codec writes
67        """
68
69    def decode(self, body: bytes, content_type: str | None = None) -> Any:
70        """Turns bytes back into a payload.
71
72        :param body: the message body
73        :param content_type: what the sender said it was, or ``None`` when the
74            sender said nothing
75        :returns: the payload
76        :raises FatalError: when the body cannot be read, which is fatal because
77            the same bytes will not read any better on a second attempt
78        """
79
80    def can_decode(self, content_type: str | None) -> bool:
81        """Whether this codec should be offered a message.
82
83        :param content_type: the sender's content type, or ``None`` when there
84            was none
85        :returns: whether this codec claims it
86        """

Reads and writes message bodies in one format.

Codec(*args, **kwargs)
1771def _no_init_or_replace_init(self, *args, **kwargs):
1772    cls = type(self)
1773
1774    if cls._is_protocol:
1775        raise TypeError('Protocols cannot be instantiated')
1776
1777    # Already using a custom `__init__`. No need to calculate correct
1778    # `__init__` to call. This can lead to RecursionError. See bpo-45121.
1779    if cls.__init__ is not _no_init_or_replace_init:
1780        return
1781
1782    # Initially, `__init__` of a protocol subclass is set to `_no_init_or_replace_init`.
1783    # The first instantiation of the subclass will call `_no_init_or_replace_init` which
1784    # searches for a proper new `__init__` in the MRO. The new `__init__`
1785    # replaces the subclass' old `__init__` (ie `_no_init_or_replace_init`). Subsequent
1786    # instantiation of the protocol subclass will thus use the new
1787    # `__init__` and no longer call `_no_init_or_replace_init`.
1788    for base in cls.__mro__:
1789        init = base.__dict__.get('__init__', _no_init_or_replace_init)
1790        if init is not _no_init_or_replace_init:
1791            cls.__init__ = init
1792            break
1793    else:
1794        # should not happen
1795        cls.__init__ = object.__init__
1796
1797    cls.__init__(self, *args, **kwargs)
content_type: str
57    @property
58    def content_type(self) -> str:
59        """What this codec writes onto a message it encodes."""

What this codec writes onto a message it encodes.

def encode(self, payload: Any) -> bytes:
61    def encode(self, payload: Any) -> bytes:
62        """Turns a payload into bytes.
63
64        :param payload: what the application wants to send
65        :returns: the body
66        :raises TypeError: when the payload is not something this codec writes
67        """

Turns a payload into bytes.

Parameters
  • payload: what the application wants to send :returns: the body
Raises
  • TypeError: when the payload is not something this codec writes
def decode(self, body: bytes, content_type: str | None = None) -> Any:
69    def decode(self, body: bytes, content_type: str | None = None) -> Any:
70        """Turns bytes back into a payload.
71
72        :param body: the message body
73        :param content_type: what the sender said it was, or ``None`` when the
74            sender said nothing
75        :returns: the payload
76        :raises FatalError: when the body cannot be read, which is fatal because
77            the same bytes will not read any better on a second attempt
78        """

Turns bytes back into a payload.

Parameters
  • body: the message body
  • content_type: what the sender said it was, or None when the sender said nothing :returns: the payload
Raises
  • FatalError: when the body cannot be read, which is fatal because the same bytes will not read any better on a second attempt
def can_decode(self, content_type: str | None) -> bool:
80    def can_decode(self, content_type: str | None) -> bool:
81        """Whether this codec should be offered a message.
82
83        :param content_type: the sender's content type, or ``None`` when there
84            was none
85        :returns: whether this codec claims it
86        """

Whether this codec should be offered a message.

Parameters
  • content_type: the sender's content type, or None when there was none :returns: whether this codec claims it
class CompositeCodec:
224class CompositeCodec:
225    """Picks a codec by the message's content type.
226
227    For a queue carrying more than one format — during a migration, or where
228    several producers were written at different times::
229
230        codec = CompositeCodec(JsonCodec(), TextCodec())
231
232    The first codec is what it writes. All of them are offered a message to
233    read, in the order they were given, and the first that claims the content
234    type *and* succeeds wins. Order matters when two overlap: :class:`BytesCodec`
235    answers for everything, so anything after it would never be reached.
236    """
237
238    def __init__(self, *codecs: Codec) -> None:
239        if not codecs:
240            raise ValueError("acemq: a CompositeCodec needs at least one codec in it")
241        self._codecs: tuple[Codec, ...] = codecs
242
243    @property
244    def codecs(self) -> tuple[Codec, ...]:
245        """The codecs, in the order they are tried."""
246        return self._codecs
247
248    @property
249    def content_type(self) -> str:
250        """The first codec's, since that is the one that encodes."""
251        return self._codecs[0].content_type
252
253    def encode(self, payload: Any) -> bytes:
254        return self._codecs[0].encode(payload)
255
256    def decode(self, body: bytes, content_type: str | None = None) -> Any:
257        """Reads a body with whichever codec recognises the content type.
258
259        With no content type nothing can be ruled out, so every codec is a
260        candidate rather than the first being assumed right. Candidates are
261        tried in turn and the first that reads the body wins — so a message that
262        claims one format and is written in another is still read — and only
263        when every one has refused is anything raised, carrying all the reasons
264        rather than just the last.
265        """
266        if content_type is None:
267            candidates = list(self._codecs)
268        else:
269            candidates = [codec for codec in self._codecs if codec.can_decode(content_type)]
270
271        if not candidates:
272            raise FatalError(
273                f"acemq: no codec here reads {content_type!r}; "
274                f"this one holds {self._describe()}"
275            )
276
277        refusals: list[str] = []
278        for codec in candidates:
279            try:
280                return codec.decode(body, content_type)
281            except Exception as failure:
282                # Collected rather than raised: a codec refusing says nothing
283                # about the next one, and the message that matters at the end is
284                # every reason together rather than whichever came last.
285                refusals.append(f"{type(codec).__name__}: {failure}")
286
287        raise FatalError(
288            "acemq: no codec could read this message. Tried " + "; ".join(refusals)
289        )
290
291    def can_decode(self, content_type: str | None) -> bool:
292        return any(codec.can_decode(content_type) for codec in self._codecs)
293
294    def _describe(self) -> str:
295        return ", ".join(codec.content_type for codec in self._codecs)
296
297    def __repr__(self) -> str:
298        return f"CompositeCodec({self._describe()})"

Picks a codec by the message's content type.

For a queue carrying more than one format — during a migration, or where several producers were written at different times::

codec = CompositeCodec(JsonCodec(), TextCodec())

The first codec is what it writes. All of them are offered a message to read, in the order they were given, and the first that claims the content type and succeeds wins. Order matters when two overlap: BytesCodec answers for everything, so anything after it would never be reached.

CompositeCodec(*codecs: Codec)
238    def __init__(self, *codecs: Codec) -> None:
239        if not codecs:
240            raise ValueError("acemq: a CompositeCodec needs at least one codec in it")
241        self._codecs: tuple[Codec, ...] = codecs
codecs: tuple[Codec, ...]
243    @property
244    def codecs(self) -> tuple[Codec, ...]:
245        """The codecs, in the order they are tried."""
246        return self._codecs

The codecs, in the order they are tried.

content_type: str
248    @property
249    def content_type(self) -> str:
250        """The first codec's, since that is the one that encodes."""
251        return self._codecs[0].content_type

The first codec's, since that is the one that encodes.

def encode(self, payload: Any) -> bytes:
253    def encode(self, payload: Any) -> bytes:
254        return self._codecs[0].encode(payload)
def decode(self, body: bytes, content_type: str | None = None) -> Any:
256    def decode(self, body: bytes, content_type: str | None = None) -> Any:
257        """Reads a body with whichever codec recognises the content type.
258
259        With no content type nothing can be ruled out, so every codec is a
260        candidate rather than the first being assumed right. Candidates are
261        tried in turn and the first that reads the body wins — so a message that
262        claims one format and is written in another is still read — and only
263        when every one has refused is anything raised, carrying all the reasons
264        rather than just the last.
265        """
266        if content_type is None:
267            candidates = list(self._codecs)
268        else:
269            candidates = [codec for codec in self._codecs if codec.can_decode(content_type)]
270
271        if not candidates:
272            raise FatalError(
273                f"acemq: no codec here reads {content_type!r}; "
274                f"this one holds {self._describe()}"
275            )
276
277        refusals: list[str] = []
278        for codec in candidates:
279            try:
280                return codec.decode(body, content_type)
281            except Exception as failure:
282                # Collected rather than raised: a codec refusing says nothing
283                # about the next one, and the message that matters at the end is
284                # every reason together rather than whichever came last.
285                refusals.append(f"{type(codec).__name__}: {failure}")
286
287        raise FatalError(
288            "acemq: no codec could read this message. Tried " + "; ".join(refusals)
289        )

Reads a body with whichever codec recognises the content type.

With no content type nothing can be ruled out, so every codec is a candidate rather than the first being assumed right. Candidates are tried in turn and the first that reads the body wins — so a message that claims one format and is written in another is still read — and only when every one has refused is anything raised, carrying all the reasons rather than just the last.

def can_decode(self, content_type: str | None) -> bool:
291    def can_decode(self, content_type: str | None) -> bool:
292        return any(codec.can_decode(content_type) for codec in self._codecs)
class Connection:
1221class Connection:
1222    """A connection to a broker, and the defaults everything on it inherits.
1223
1224    Built by :func:`connect` in the ordinary case. Taking a transport directly
1225    is for a fake in a test, or for a connection made with settings a URL cannot
1226    carry.
1227
1228    :param transport: how to reach the broker
1229    :param codec: what publishers and consumers use unless they say otherwise
1230    :param origin: what to stamp on published messages, conventionally
1231        ``service@host``
1232    :param retry: what consumers use unless they say otherwise. The default is
1233        one delivery and no second chance, so a handler that calls
1234        ``retry()`` on a connection with no policy dead-letters the message and
1235        says so in the reason. That is deliberate: the alternative default is an
1236        immediate requeue, which is a hot loop against a broker that nobody
1237        asked for
1238    :param prefetch: how many unacknowledged messages a consumer holds
1239    :param max_outstanding_publishes: how many publishes may be waiting for the
1240        broker at once. The ceiling that turns a memory leak into backpressure:
1241        without one, a caller publishing faster than the broker answers
1242        accumulates unconfirmed messages until the process dies, which looks
1243        like throughput right up to the moment it does not. A thousand by
1244        default, which is what Java's ``maxOutstandingPublishes`` and .NET's
1245        ``MaxOutstandingPublishes`` default to
1246    :param confirm_timeout: how long a publish waits for one of those permits,
1247        and then for the broker's confirm, before giving up. Ten seconds by
1248        default, the same as Java's ``confirmTimeout``. Reaching it raises
1249        rather than waiting on, because a publisher stalled for ever behind a
1250        broker that has stopped answering is the failure this bound exists to make visible
1251    :param on_publish: cross-cutting behaviour to wrap every publish on this
1252        connection in, outermost first. See :mod:`acemq_amqp.interceptors`
1253    :param on_consume: the same around every handler.
1254        :meth:`intercept_publish` and :meth:`intercept_consume` add more later
1255    :param observer: where the numbers go. Nothing by default, so a service
1256        that wants none pays for none. See :mod:`acemq_amqp.telemetry`
1257    """
1258
1259    def __init__(
1260        self,
1261        transport: Transport,
1262        *,
1263        codec: Codec | None = None,
1264        origin: str | None = None,
1265        retry: RetryPolicy | None = None,
1266        prefetch: int = DEFAULT_PREFETCH,
1267        max_outstanding_publishes: int = DEFAULT_MAX_OUTSTANDING_PUBLISHES,
1268        confirm_timeout: timedelta = DEFAULT_CONFIRM_TIMEOUT,
1269        on_publish: Sequence[PublishInterceptor] = (),
1270        on_consume: Sequence[ConsumeInterceptor] = (),
1271        observer: Observer | None = None,
1272    ) -> None:
1273        if prefetch < 0:
1274            raise ValueError(f"acemq: prefetch must not be negative, got {prefetch}")
1275        if max_outstanding_publishes < 1:
1276            raise ValueError(
1277                "acemq: max_outstanding_publishes must be at least 1, got "
1278                f"{max_outstanding_publishes}"
1279            )
1280        if confirm_timeout <= timedelta(0):
1281            raise ValueError(
1282                f"acemq: confirm_timeout must be positive, got {confirm_timeout}"
1283            )
1284        self._transport = transport
1285        self._observer: Observer = observer or NullObserver()
1286        self._codec = codec or JsonCodec()
1287        self._origin = origin or default_origin()
1288        self._retry = retry or no_retry()
1289        self._prefetch = prefetch
1290        self._max_outstanding = max_outstanding_publishes
1291        self._confirm_timeout = confirm_timeout
1292        # Built here rather than on first use. An asyncio.Semaphore has bound no
1293        # loop at construction since 3.10, so it is safe to make one outside a
1294        # running loop, and making it here means every publish on this connection
1295        # shares the one counter however many tasks are publishing.
1296        self._outstanding = asyncio.Semaphore(max_outstanding_publishes)
1297        self._in_flight_publishes = 0
1298        self._consumers: list[Consumer] = []
1299        self._publishing: list[PublishInterceptor] = list(on_publish)
1300        self._consuming: list[ConsumeInterceptor] = list(on_consume)
1301        self._closed = False
1302
1303    @property
1304    def transport(self) -> Transport:
1305        """The transport underneath, for anything this library does not wrap."""
1306        return self._transport
1307
1308    @property
1309    def codec(self) -> Codec:
1310        """The codec publishers and consumers use unless told otherwise."""
1311        return self._codec
1312
1313    @property
1314    def origin(self) -> str:
1315        """What published messages are stamped with."""
1316        return self._origin
1317
1318    @property
1319    def retry(self) -> RetryPolicy:
1320        """The retry policy consumers use unless told otherwise."""
1321        return self._retry
1322
1323    @property
1324    def observer(self) -> Observer:
1325        """Where this connection's numbers go."""
1326        return self._observer
1327
1328    @property
1329    def max_outstanding_publishes(self) -> int:
1330        """How many publishes may be waiting for the broker at once."""
1331        return self._max_outstanding
1332
1333    @property
1334    def confirm_timeout(self) -> timedelta:
1335        """How long a publish waits for room before giving up."""
1336        return self._confirm_timeout
1337
1338    @property
1339    def outstanding_publishes(self) -> int:
1340        """How many publishes are waiting for the broker right now.
1341
1342        What a stalled publisher looks like from outside: a number pinned at
1343        :attr:`max_outstanding_publishes` is a broker that has stopped answering,
1344        and it says so before the first :class:`~acemq_amqp.PublishError` does.
1345        """
1346        # Counted here rather than read off the semaphore, which keeps its count
1347        # private. Raised after the permit is taken and lowered before it is
1348        # given back, both inside publish_raw, so it never reads high.
1349        return self._in_flight_publishes
1350
1351    @property
1352    def consumers(self) -> tuple[Consumer, ...]:
1353        """The consumers running on this connection.
1354
1355        A snapshot, and public because a health check that has to say whether
1356        the consumers are alive should not need private access to find them —
1357        the same rule an interceptor follows.
1358        """
1359        return tuple(self._consumers)
1360
1361    @property
1362    def closed(self) -> bool:
1363        """Whether this connection has been closed."""
1364        return self._closed
1365
1366    @property
1367    def blocked(self) -> bool | None:
1368        """Whether the broker has blocked this connection, or ``None`` when the
1369        transport cannot be asked.
1370
1371        RabbitMQ blocks a connection when it is low on memory or disk, and while
1372        it lasts it stops reading that connection's socket: everything sent —
1373        a publish, a declaration, :meth:`health`'s own probe — waits rather than
1374        failing. It is the broker protecting itself and it is not an error, so
1375        it is worth telling apart from the broker having gone away, which looks
1376        identical from outside and wants the opposite response.
1377
1378        ``None`` means the question could not be asked at all, which is a
1379        different fact from ``False`` and is reported as one: a transport that
1380        does not implement
1381        :class:`~acemq_amqp.transport.BlockedState`, or one whose client has
1382        moved the state out of reach. See :attr:`blocked_reason` for why there
1383        may be a boolean and no reason to go with it.
1384        """
1385        transport = self._transport
1386        return transport.blocked if isinstance(transport, BlockedState) else None
1387
1388    @property
1389    def blocked_reason(self) -> str | None:
1390        """What the broker said when it blocked this connection, if anything.
1391
1392        ``None`` when the connection is not blocked, and also ``None`` when the
1393        transport is blocked but cannot say why — which is what the RabbitMQ
1394        transport answers, because aio-pika reads RabbitMQ's reason and does not
1395        keep it. Reporting the block without the reason is the honest half;
1396        inventing a reason would read exactly like one the broker sent.
1397        """
1398        transport = self._transport
1399        return transport.blocked_reason if isinstance(transport, BlockedState) else None
1400
1401    async def health(self, timeout: float = DEFAULT_HEALTH_TIMEOUT) -> HealthReport:
1402        """Whether this connection can reach its broker, and its consumers run.
1403
1404        The broker half is a declaration of a temporary queue, which is the
1405        cheapest thing AMQP offers that actually proves the connection works. A
1406        TCP connection that is open but wedged — the broker paused, the network
1407        black-holing — answers a socket-level check exactly as a healthy one
1408        does, right up until something is asked of it.
1409
1410        The consumer half is the one a probe usually wants and nothing outside
1411        can see. A consumer whose workers have died without it being closed is
1412        one the broker is still sending messages to and nothing is reading, and
1413        from outside that is indistinguishable from a quiet queue.
1414
1415        **A blocked connection is up, with the reason.** The broker blocks a
1416        connection when it is low on disk or memory, and an application that
1417        fails its own readiness check for it is one an orchestrator restarts
1418        into the same blocked broker, having thrown away whatever it was
1419        holding. ``parts["blocked"]`` carries the state either way, and is
1420        ``None`` when the transport could not be asked — see :attr:`blocked`.
1421
1422        **The probe is bounded.** A blocked broker stops reading the socket, so
1423        the round trip this makes is precisely the thing that does not come back
1424        when something is wrong; without a deadline of its own a health check
1425        hangs, and a readiness endpoint that hangs takes the instance out of
1426        rotation with no report at all. A probe that runs out of time is
1427        abandoned rather than waited for, because a request that cannot be
1428        answered cannot reliably be cancelled either.
1429
1430        It costs a round trip, so it is not something to call per request. Wire
1431        it to a readiness probe and let the probe's interval decide how often.
1432
1433        :param timeout: how long to give the broker's answer, three seconds by
1434            default. Shorter than the deadline
1435            :func:`~acemq_amqp.telemetry.aggregate_health` puts around a whole
1436            set of checks, so that this report rather than that one is what an
1437            operator reads when a broker stops answering
1438        :returns: the report. Degraded means working but worth an alert
1439        """
1440        checked = datetime.now(timezone.utc)
1441        consumers = self.consumers
1442        stalled = [consumer.queue for consumer in consumers if not consumer.running]
1443        blocked = self.blocked
1444        parts: dict[str, Any] = {
1445            "consumers": len(consumers),
1446            "in-flight": sum(consumer.in_flight for consumer in consumers),
1447            "blocked": blocked,
1448        }
1449        reason = self.blocked_reason
1450        if reason:
1451            parts["blocked-reason"] = reason
1452
1453        if self._closed:
1454            return HealthReport(
1455                HealthStatus.DOWN, "the connection has been closed", checked, parts
1456            )
1457
1458        if blocked:
1459            # Asked before the probe rather than after it, because the probe is
1460            # what a blocked broker will not answer: a round trip started here
1461            # would spend the whole deadline arriving at the answer this already
1462            # has.
1463            return self._blocked_report(checked, parts, stalled, reason)
1464
1465        # Asking whether a queue nothing has ever declared exists, rather than
1466        # declaring a temporary one. It is the same round trip and it proves the
1467        # same thing, and it creates nothing at all: an exclusive queue is only
1468        # released when the channel that declared it closes, so a probe that
1469        # declared one would leave a queue on the broker for the life of the
1470        # connection and a new one behind every restart.
1471        probe = f"acemq-health-{uuid.uuid4().hex}"
1472        started = time.monotonic()
1473        try:
1474            await self._probe_within(probe, timeout)
1475        except asyncio.TimeoutError:
1476            # A round trip that does not come back is the ordinary way a block
1477            # shows itself, and the notification may have arrived while this was
1478            # waiting. Asked again rather than assumed, because "the broker is
1479            # protecting itself" and "the broker has gone" are the same silence.
1480            blocked = self.blocked
1481            parts["blocked"] = blocked
1482            reason = self.blocked_reason
1483            if reason:
1484                parts["blocked-reason"] = reason
1485            if blocked:
1486                return self._blocked_report(checked, parts, stalled, reason)
1487            return HealthReport(
1488                HealthStatus.DOWN,
1489                f"the broker did not answer within {timeout:g}s",
1490                checked,
1491                parts,
1492            )
1493        except Exception as failure:
1494            parts["error"] = _describe(failure)
1495            return HealthReport(
1496                HealthStatus.DOWN, f"the broker did not answer: {failure}", checked, parts
1497            )
1498        parts["round-trip"] = round(time.monotonic() - started, 4)
1499
1500        if stalled:
1501            # Degraded rather than down: the connection works, and a replacement
1502            # instance would almost certainly stall the same way. Worth waking
1503            # somebody; not worth taking this one out of rotation.
1504            return HealthReport(
1505                HealthStatus.DEGRADED,
1506                _stalled_detail(stalled),
1507                checked,
1508                {**parts, "stalled": sorted(stalled)},
1509            )
1510        return HealthReport(HealthStatus.UP, "", checked, parts)
1511
1512    def _blocked_report(
1513        self,
1514        checked: datetime,
1515        parts: dict[str, Any],
1516        stalled: list[str],
1517        reason: str | None,
1518    ) -> HealthReport:
1519        """What a blocked connection reports: up, with the reason.
1520
1521        Up for the block itself, and still degraded when a consumer of this
1522        process has stopped reading — that is this instance's own failure rather
1523        than the broker's, it outlives the alarm, and a block is not a reason to
1524        stop saying so.
1525        """
1526        detail = self._blocked_detail(reason)
1527        if stalled:
1528            return HealthReport(
1529                HealthStatus.DEGRADED,
1530                f"{_stalled_detail(stalled)}; and {detail}",
1531                checked,
1532                {**parts, "stalled": sorted(stalled)},
1533            )
1534        return HealthReport(HealthStatus.UP, detail, checked, parts)
1535
1536    @staticmethod
1537    def _blocked_detail(reason: str | None) -> str:
1538        """The blocked sentence, with the broker's reason when there is one.
1539
1540        The reason follows a colon, which is how Go and Ruby write it too —
1541        ``blockedPrefix + ": " + reason`` and ``"#{BLOCKED}: #{reason}"``. One
1542        shape across the family means one alert rule reads a blocked broker
1543        whatever language the service happens to be written in.
1544        """
1545        return BLOCKED_DETAIL if not reason else f"{BLOCKED_DETAIL}: {reason}"
1546
1547    async def _probe_within(self, name: str, timeout: float) -> None:
1548        """One round trip to the broker, given ``timeout`` to come back.
1549
1550        The probe runs as a task that is abandoned rather than awaited when the
1551        deadline passes, which is the difference between a bounded check and one
1552        that merely looks bounded: cancelling a request to a broker that has
1553        stopped reading its socket means waiting for a cancellation that travels
1554        the same way the request did.
1555
1556        :raises asyncio.TimeoutError: when the broker did not answer in time
1557        """
1558        probe = asyncio.ensure_future(self._probe(name))
1559        try:
1560            done, _ = await asyncio.wait({probe}, timeout=timeout)
1561        except asyncio.CancelledError:
1562            # The caller gave up on the whole check. Waiting on a task is not
1563            # cancelling it, so it is cancelled here rather than left running
1564            # against a broker nobody is listening to any more.
1565            probe.cancel()
1566            probe.add_done_callback(_retrieved)
1567            raise
1568        if not done:
1569            probe.cancel()
1570            # Whatever it ends as is nobody's news now, and an abandoned task
1571            # whose exception is never retrieved is a warning at collection time
1572            # about a failure that has already been reported as a timeout.
1573            probe.add_done_callback(_retrieved)
1574            raise asyncio.TimeoutError
1575        # Raises what the probe raised, which is what the caller reports.
1576        probe.result()
1577
1578    async def _probe(self, name: str) -> None:
1579        """One round trip to the broker that leaves nothing behind.
1580
1581        A transport that cannot be asked whether a queue exists gets the next
1582        cheapest thing: a temporary queue, removed again straight away. Both
1583        prove the same thing, which is that the connection answers rather than
1584        merely being open.
1585
1586        The temporary queue is classic, and has to be. It is exclusive and
1587        auto-deleting so that it goes when the channel does, and RabbitMQ
1588        refuses a quorum queue that is either — a health check that could not be
1589        declared would report a healthy broker as down.
1590        """
1591        if isinstance(self._transport, QueueAdmin):
1592            await self._transport.queue_exists(name)
1593            return
1594        await self._transport.declare_queue(
1595            name, QueueSpec(durable=False, auto_delete=True, exclusive=True)
1596        )
1597
1598    @property
1599    def publish_interceptors(self) -> tuple[PublishInterceptor, ...]:
1600        """What wraps every publish, outermost first.
1601
1602        A snapshot rather than the list itself, so a publish in progress cannot
1603        be handed a chain that is being edited underneath it.
1604        """
1605        return tuple(self._publishing)
1606
1607    @property
1608    def consume_interceptors(self) -> tuple[ConsumeInterceptor, ...]:
1609        """What wraps every handler, outermost first."""
1610        return tuple(self._consuming)
1611
1612    def intercept_publish(self, interceptor: PublishInterceptor) -> Connection:
1613        """Wraps every publish on this connection in ``interceptor``.
1614
1615        Registered ones run in the order they were added, so this one is
1616        innermost until the next is added. Read at publish time rather than
1617        copied into each publisher, so an interceptor registered during start-up
1618        applies to publishers that already exist — which is what makes it usable
1619        from a framework's wiring rather than only from the line that connects.
1620
1621        Adding at start-up is still what to do. A message already in flight will
1622        not see it, and an interceptor whose whole job is to stamp every message
1623        is not doing it if the first few went without.
1624
1625        :param interceptor: what to wrap the publish in. See
1626            :mod:`acemq_amqp.interceptors`
1627        :returns: this connection, so registrations read as one expression
1628        """
1629        self._publishing.append(interceptor)
1630        return self
1631
1632    def intercept_consume(self, interceptor: ConsumeInterceptor) -> Connection:
1633        """Wraps every handler on this connection in ``interceptor``.
1634
1635        :param interceptor: what to wrap the handler in
1636        :returns: this connection
1637        """
1638        self._consuming.append(interceptor)
1639        return self
1640
1641    async def declare(self, topology: Topology) -> None:
1642        """Applies a topology to this connection's broker.
1643
1644        :param topology: what the broker should have
1645        """
1646        await topology.apply(self._transport)
1647
1648    def publisher(
1649        self,
1650        exchange: str = "",
1651        routing_key: str = "",
1652        *,
1653        codec: Codec | None = None,
1654        persistent: bool = True,
1655        mandatory: bool = False,
1656    ) -> Publisher:
1657        """Builds a publisher for an exchange and routing key.
1658
1659        Publish straight to a queue by leaving the exchange empty: the default
1660        exchange routes to the queue whose name matches the routing key.
1661
1662        :param exchange: where to publish, empty for the default exchange
1663        :param routing_key: what to publish under, or a queue name
1664        :param codec: a codec other than this connection's
1665        :param persistent: ask the broker to write these messages to disk
1666        :param mandatory: fail when a message reaches no queue at all, rather
1667            than letting the broker drop it silently
1668        :returns: the publisher
1669        """
1670        return Publisher(
1671            self,
1672            exchange,
1673            routing_key,
1674            codec=codec,
1675            persistent=persistent,
1676            mandatory=mandatory,
1677        )
1678
1679    async def consume(
1680        self,
1681        queue: str,
1682        handler: Handler,
1683        *,
1684        codec: Codec | None = None,
1685        retry: RetryPolicy | None = None,
1686        prefetch: int | None = None,
1687        concurrency: int = 1,
1688        tag: str = "",
1689        args: Mapping[str, Any] | None = None,
1690        declare: bool = True,
1691    ) -> Consumer:
1692        """Reads messages from a queue until the returned consumer is closed.
1693
1694        Before it subscribes it declares the queues it will need when a message
1695        fails: ``{queue}.dlq``, ``{queue}.parked``, ``acemq.dlx`` and, when the
1696        retry policy has waits the broker holds, ``acemq.retry`` and the rungs.
1697        See :func:`acemq_amqp.topology.declare_where_failures_go` for exactly
1698        what and why. The source queue is not among them, and neither is
1699        anything a producer needs, so this is not a replacement for declaring a
1700        topology — it is the floor underneath a service that was deployed
1701        without one.
1702
1703        :param queue: what to read
1704        :param handler: what to do with a message; returning an
1705            :class:`~acemq_amqp.Ack` is the decision
1706        :param codec: a codec other than this connection's
1707        :param retry: a policy other than this connection's
1708        :param prefetch: how many unacknowledged messages to hold
1709        :param concurrency: how many messages to work on at once. One by
1710            default, which keeps a queue's messages in order; raising it trades
1711            that order for throughput, which is the right trade for handlers
1712            that spend their time waiting on something else
1713        :param tag: what to call this consumer to the broker
1714        :param args: broker-specific consumer arguments
1715        :param declare: declare the queues above before subscribing. On by
1716            default, because the alternative default loses messages silently.
1717            Turn it off for a login with no ``configure`` permission on the
1718            vhost, which is refused by the broker rather than ignored, and for a
1719            tool draining a queue it does not own — a one-off reader of
1720            ``orders.dlq`` has no business creating ``orders.dlq.dlq``. A
1721            consumer started this way still publishes to those queues, so
1722            somebody else has to have declared them
1723        :returns: the running consumer
1724        """
1725        if self._closed:
1726            raise AceMQError("acemq: this connection is closed")
1727
1728        consumer = Consumer(
1729            self,
1730            queue,
1731            handler,
1732            codec=codec or self._codec,
1733            retry=retry or self._retry,
1734            concurrency=concurrency,
1735            declare=declare,
1736        )
1737        self._consumers.append(consumer)
1738        try:
1739            await consumer._start(
1740                self._prefetch if prefetch is None else prefetch,
1741                tag,
1742                args or {},
1743            )
1744        except BaseException:
1745            self._untrack(consumer)
1746            raise
1747        return consumer
1748
1749    async def publish_raw(
1750        self, exchange: str, routing_key: str, message: Outbound
1751    ) -> PublishResult:
1752        """Sends bytes that are already encoded, with headers already rendered.
1753
1754        For the retry and dead-letter hops, and for anything else replaying a
1755        message recorded earlier: the payload's class may not exist any more,
1756        and re-encoding it would produce different bytes from the ones that were
1757        committed. Ordinary publishing goes through :class:`Publisher`.
1758
1759        **This is the one place the outstanding-publish bound is taken.** Every
1760        publish in the library comes through here — a :class:`Publisher`, a
1761        retry rung, a dead letter, a replay — so one permit counts them all, and
1762        :meth:`Publisher.send_all` cannot put more messages on the wire at once
1763        than :attr:`max_outstanding_publishes` allows however long the list it
1764        was given is.
1765
1766        :param exchange: where to publish, empty for the default exchange
1767        :param routing_key: what to publish under
1768        :param message: the message, encoded
1769        :returns: what the broker said
1770        :raises PublishingPausedError: when the broker has blocked this
1771            connection. Nothing was sent; retry once it unblocks
1772        :raises PublishError: when every permit is taken and none came free
1773            within :attr:`confirm_timeout`, or when the broker did not confirm
1774            this publish within it. The second may or may not have arrived
1775        """
1776        # Refused rather than queued, before a permit or a byte is spent. A
1777        # blocked broker stops reading the socket, so a publish sent now sits in
1778        # the client's write buffer with no confirm coming until the alarm
1779        # clears, then times out as a failure that may or may not have arrived.
1780        # Go and .NET refuse at the same point for the same reason. ``None``
1781        # means the state could not be read and is not treated as blocked.
1782        if self.blocked is True:
1783            reason = self.blocked_reason
1784            raise PublishingPausedError(
1785                message.message_id,
1786                exchange,
1787                routing_key,
1788                (
1789                    f"{self._blocked_detail(reason)}. Nothing was sent; back off and "
1790                    "retry once the broker unblocks"
1791                ),
1792            )
1793
1794        # Taken before the message is written and not after, which is the whole
1795        # point: a bound applied afterwards has already let the message into
1796        # memory. The permit is given back in the ``finally`` below on every
1797        # path there is — the publish returning, the publish raising, and this
1798        # task being cancelled while the broker is still thinking — because a
1799        # permit lost on any one of those is a connection that publishes a
1800        # thousand more messages and then stops for ever.
1801        try:
1802            await asyncio.wait_for(
1803                self._outstanding.acquire(), self._confirm_timeout.total_seconds()
1804            )
1805        except asyncio.TimeoutError as stalled:
1806            # Said rather than waited out. A publisher parked behind a broker
1807            # that has stopped answering is indistinguishable from a quiet
1808            # service, and this sentence is the one Java raises word for word so
1809            # that an operator reading a log has one thing to recognise.
1810            raise PublishError(
1811                message.message_id,
1812                exchange,
1813                routing_key,
1814                (
1815                    f"{self._max_outstanding} publishes are already waiting for a "
1816                    f"confirm and none completed within {self._confirm_timeout}. "
1817                    "The broker is not keeping up; publish more slowly rather than "
1818                    "buffering more."
1819                ),
1820            ) from stalled
1821
1822        self._in_flight_publishes += 1
1823        try:
1824            # Bounded as well, by the same timeout. RabbitMQ says a connection is
1825            # blocked only once it publishes, so the first publish after an alarm
1826            # is written and then held with no confirm until the alarm clears --
1827            # minutes, or never. Said here as a failure, not a refusal: it was
1828            # written, so it may have arrived. The publish after it sees the
1829            # block and is refused above. .NET bounds the same wait the same way.
1830            result = await asyncio.wait_for(
1831                self._transport.publish(exchange, routing_key, message),
1832                self._confirm_timeout.total_seconds(),
1833            )
1834        except asyncio.TimeoutError as unconfirmed:
1835            raise PublishError(
1836                message.message_id,
1837                exchange,
1838                routing_key,
1839                (
1840                    f"the broker did not confirm this publish within "
1841                    f"{self._confirm_timeout}. It may or may not have been stored, "
1842                    "so treat it as neither sent nor lost"
1843                ),
1844            ) from unconfirmed
1845        finally:
1846            self._in_flight_publishes -= 1
1847            self._outstanding.release()
1848
1849        # A nack is the broker declining to keep a message it was asked to take, and
1850        # it is raised here rather than returned.
1851        #
1852        # It used to be returned, and `confirmed` was read nowhere in this library
1853        # except a telemetry label -- so a caller who did not inspect the result
1854        # carried on believing the message had gone somewhere, and Java, .NET, Go and
1855        # Ruby all raised on the same event. Worse than the wrong belief: the outbox
1856        # relay removes a record because the publish did not raise, so a nack deleted
1857        # a record for a message the broker had refused. At-most-once, in the pattern
1858        # whose whole purpose is not losing anything.
1859        #
1860        # Checked here because this is the one place every publish in the library
1861        # comes through -- a Publisher, a retry rung, a dead letter, a replay, the
1862        # outbox -- so one check covers all of them, and anything that settles or
1863        # deletes on the strength of a publish returning now hears the refusal.
1864        #
1865        # Distinct from unroutable, which the Publisher path reports separately and
1866        # which sets `unroutable`: the broker could route this one and would not keep
1867        # it, and the two want different responses.
1868        #
1869        # So a returned message is passed back rather than raised here. The
1870        # RabbitMQ transport reports a Basic.Return as not confirmed as well as
1871        # not routed, and raising it as a nack hid ``routed`` from every caller
1872        # that reads it: a mandatory Publisher said "refused" rather than
1873        # "reached no queue", and the retry and dead-letter hops never reached
1874        # the fallbacks written for a queue that is not there.
1875        if not result.confirmed and result.routed:
1876            raise PublishError(
1877                message.message_id,
1878                exchange,
1879                routing_key,
1880                (
1881                    "the broker refused to confirm it. The message reached the broker "
1882                    "and the broker has not taken responsibility for it, so it must be "
1883                    "treated as not sent"
1884                ),
1885            )
1886
1887        return result
1888
1889    async def pull(self, queue: str) -> Delivery | None:
1890        """Takes one message off a queue, or ``None`` when there is none waiting.
1891
1892        For tools rather than for services: it is a round trip per message where
1893        :meth:`consume` gets a stream. What it can do that a consumer cannot is
1894        say that a queue is empty, which is what anything working through a
1895        backlog and then stopping — a replay, a drain — has to know.
1896
1897        The delivery comes back unsettled, so the caller decides whether it is
1898        gone or goes back.
1899
1900        :param queue: what to read from
1901        :returns: the delivery, or ``None``
1902        """
1903        return await self._source().pull(queue)
1904
1905    async def queue_exists(self, queue: str) -> bool:
1906        """Whether a queue is on the broker, creating nothing."""
1907        return await self._admin().queue_exists(queue)
1908
1909    async def message_count(self, queue: str) -> int:
1910        """How many messages are waiting on a queue."""
1911        return await self._admin().message_count(queue)
1912
1913    async def delete_queue(self, queue: str) -> None:
1914        """Removes a queue and every message still on it."""
1915        await self._admin().delete_queue(queue)
1916
1917    def _admin(self) -> QueueAdmin:
1918        if not isinstance(self._transport, QueueAdmin):
1919            raise AceMQError(
1920                f"acemq: the {type(self._transport).__name__} transport cannot manage queues"
1921            )
1922        return self._transport
1923
1924    def _source(self) -> MessageSource:
1925        if not isinstance(self._transport, MessageSource):
1926            raise AceMQError(
1927                f"acemq: the {type(self._transport).__name__} transport cannot be "
1928                "asked for one message at a time"
1929            )
1930        return self._transport
1931
1932    def _untrack(self, consumer: Consumer) -> None:
1933        if consumer in self._consumers:
1934            self._consumers.remove(consumer)
1935
1936    async def close(self, *, timeout: float | None = DEFAULT_DRAIN_TIMEOUT) -> bool:
1937        """Stops every consumer on this connection and releases it.
1938
1939        Consumers first, and their handlers given up to ``timeout`` to finish,
1940        so a message being worked on when this is called is settled rather than
1941        returned to the queue for somebody else to redo. The bound is one
1942        deadline for every handler of every consumer, not one each: they are
1943        all drained at once. A handler still running at the deadline is
1944        cancelled and its message left unsettled, for the broker to redeliver —
1945        see :meth:`Consumer.close`.
1946
1947        The connection is released whatever happens, including when the caller
1948        cancels closing; that cancellation is raised again afterwards.
1949
1950        :param timeout: seconds to wait for the running handlers; ``None``
1951            waits for as long as they take
1952        :returns: ``True`` if every handler finished, ``False`` if the time ran
1953            out first
1954        """
1955        if self._closed:
1956            return True
1957        self._closed = True
1958        try:
1959            drained = await asyncio.gather(
1960                *(consumer.close(timeout=timeout) for consumer in list(self._consumers)),
1961                return_exceptions=True,
1962            )
1963        finally:
1964            await self._transport.close()
1965        for outcome in drained:
1966            if isinstance(outcome, BaseException):
1967                raise outcome
1968        return all(drained)
1969
1970    async def __aenter__(self) -> Connection:
1971        return self
1972
1973    async def __aexit__(self, *_: object) -> None:
1974        await self.close()

A connection to a broker, and the defaults everything on it inherits.

Built by connect() in the ordinary case. Taking a transport directly is for a fake in a test, or for a connection made with settings a URL cannot carry.

Parameters
  • transport: how to reach the broker
  • codec: what publishers and consumers use unless they say otherwise
  • origin: what to stamp on published messages, conventionally service@host
  • retry: what consumers use unless they say otherwise. The default is one delivery and no second chance, so a handler that calls retry() on a connection with no policy dead-letters the message and says so in the reason. That is deliberate: the alternative default is an immediate requeue, which is a hot loop against a broker that nobody asked for
  • prefetch: how many unacknowledged messages a consumer holds
  • max_outstanding_publishes: how many publishes may be waiting for the broker at once. The ceiling that turns a memory leak into backpressure: without one, a caller publishing faster than the broker answers accumulates unconfirmed messages until the process dies, which looks like throughput right up to the moment it does not. A thousand by default, which is what Java's maxOutstandingPublishes and .NET's MaxOutstandingPublishes default to
  • confirm_timeout: how long a publish waits for one of those permits, and then for the broker's confirm, before giving up. Ten seconds by default, the same as Java's confirmTimeout. Reaching it raises rather than waiting on, because a publisher stalled for ever behind a broker that has stopped answering is the failure this bound exists to make visible
  • on_publish: cross-cutting behaviour to wrap every publish on this connection in, outermost first. See acemq_amqp.interceptors
  • on_consume: the same around every handler. intercept_publish() and intercept_consume() add more later
  • observer: where the numbers go. Nothing by default, so a service that wants none pays for none. See acemq_amqp.telemetry
Connection( transport: Transport, *, codec: Codec | None = None, origin: str | None = None, retry: RetryPolicy | None = None, prefetch: int = 20, max_outstanding_publishes: int = 1000, confirm_timeout: datetime.timedelta = datetime.timedelta(seconds=10), on_publish: Sequence[Callable[[PublishContext, Callable[[PublishContext], Awaitable[PublishResult]]], Awaitable[PublishResult]]] = (), on_consume: Sequence[Callable[[ConsumeContext, Callable[[ConsumeContext], Awaitable[Ack]]], Awaitable[Ack]]] = (), observer: Observer | None = None)
1259    def __init__(
1260        self,
1261        transport: Transport,
1262        *,
1263        codec: Codec | None = None,
1264        origin: str | None = None,
1265        retry: RetryPolicy | None = None,
1266        prefetch: int = DEFAULT_PREFETCH,
1267        max_outstanding_publishes: int = DEFAULT_MAX_OUTSTANDING_PUBLISHES,
1268        confirm_timeout: timedelta = DEFAULT_CONFIRM_TIMEOUT,
1269        on_publish: Sequence[PublishInterceptor] = (),
1270        on_consume: Sequence[ConsumeInterceptor] = (),
1271        observer: Observer | None = None,
1272    ) -> None:
1273        if prefetch < 0:
1274            raise ValueError(f"acemq: prefetch must not be negative, got {prefetch}")
1275        if max_outstanding_publishes < 1:
1276            raise ValueError(
1277                "acemq: max_outstanding_publishes must be at least 1, got "
1278                f"{max_outstanding_publishes}"
1279            )
1280        if confirm_timeout <= timedelta(0):
1281            raise ValueError(
1282                f"acemq: confirm_timeout must be positive, got {confirm_timeout}"
1283            )
1284        self._transport = transport
1285        self._observer: Observer = observer or NullObserver()
1286        self._codec = codec or JsonCodec()
1287        self._origin = origin or default_origin()
1288        self._retry = retry or no_retry()
1289        self._prefetch = prefetch
1290        self._max_outstanding = max_outstanding_publishes
1291        self._confirm_timeout = confirm_timeout
1292        # Built here rather than on first use. An asyncio.Semaphore has bound no
1293        # loop at construction since 3.10, so it is safe to make one outside a
1294        # running loop, and making it here means every publish on this connection
1295        # shares the one counter however many tasks are publishing.
1296        self._outstanding = asyncio.Semaphore(max_outstanding_publishes)
1297        self._in_flight_publishes = 0
1298        self._consumers: list[Consumer] = []
1299        self._publishing: list[PublishInterceptor] = list(on_publish)
1300        self._consuming: list[ConsumeInterceptor] = list(on_consume)
1301        self._closed = False
transport: Transport
1303    @property
1304    def transport(self) -> Transport:
1305        """The transport underneath, for anything this library does not wrap."""
1306        return self._transport

The transport underneath, for anything this library does not wrap.

codec: Codec
1308    @property
1309    def codec(self) -> Codec:
1310        """The codec publishers and consumers use unless told otherwise."""
1311        return self._codec

The codec publishers and consumers use unless told otherwise.

origin: str
1313    @property
1314    def origin(self) -> str:
1315        """What published messages are stamped with."""
1316        return self._origin

What published messages are stamped with.

retry: RetryPolicy
1318    @property
1319    def retry(self) -> RetryPolicy:
1320        """The retry policy consumers use unless told otherwise."""
1321        return self._retry

The retry policy consumers use unless told otherwise.

observer: Observer
1323    @property
1324    def observer(self) -> Observer:
1325        """Where this connection's numbers go."""
1326        return self._observer

Where this connection's numbers go.

max_outstanding_publishes: int
1328    @property
1329    def max_outstanding_publishes(self) -> int:
1330        """How many publishes may be waiting for the broker at once."""
1331        return self._max_outstanding

How many publishes may be waiting for the broker at once.

confirm_timeout: datetime.timedelta
1333    @property
1334    def confirm_timeout(self) -> timedelta:
1335        """How long a publish waits for room before giving up."""
1336        return self._confirm_timeout

How long a publish waits for room before giving up.

outstanding_publishes: int
1338    @property
1339    def outstanding_publishes(self) -> int:
1340        """How many publishes are waiting for the broker right now.
1341
1342        What a stalled publisher looks like from outside: a number pinned at
1343        :attr:`max_outstanding_publishes` is a broker that has stopped answering,
1344        and it says so before the first :class:`~acemq_amqp.PublishError` does.
1345        """
1346        # Counted here rather than read off the semaphore, which keeps its count
1347        # private. Raised after the permit is taken and lowered before it is
1348        # given back, both inside publish_raw, so it never reads high.
1349        return self._in_flight_publishes

How many publishes are waiting for the broker right now.

What a stalled publisher looks like from outside: a number pinned at max_outstanding_publishes is a broker that has stopped answering, and it says so before the first ~acemq_amqp.PublishError does.

consumers: tuple[Consumer, ...]
1351    @property
1352    def consumers(self) -> tuple[Consumer, ...]:
1353        """The consumers running on this connection.
1354
1355        A snapshot, and public because a health check that has to say whether
1356        the consumers are alive should not need private access to find them —
1357        the same rule an interceptor follows.
1358        """
1359        return tuple(self._consumers)

The consumers running on this connection.

A snapshot, and public because a health check that has to say whether the consumers are alive should not need private access to find them — the same rule an interceptor follows.

closed: bool
1361    @property
1362    def closed(self) -> bool:
1363        """Whether this connection has been closed."""
1364        return self._closed

Whether this connection has been closed.

blocked: bool | None
1366    @property
1367    def blocked(self) -> bool | None:
1368        """Whether the broker has blocked this connection, or ``None`` when the
1369        transport cannot be asked.
1370
1371        RabbitMQ blocks a connection when it is low on memory or disk, and while
1372        it lasts it stops reading that connection's socket: everything sent —
1373        a publish, a declaration, :meth:`health`'s own probe — waits rather than
1374        failing. It is the broker protecting itself and it is not an error, so
1375        it is worth telling apart from the broker having gone away, which looks
1376        identical from outside and wants the opposite response.
1377
1378        ``None`` means the question could not be asked at all, which is a
1379        different fact from ``False`` and is reported as one: a transport that
1380        does not implement
1381        :class:`~acemq_amqp.transport.BlockedState`, or one whose client has
1382        moved the state out of reach. See :attr:`blocked_reason` for why there
1383        may be a boolean and no reason to go with it.
1384        """
1385        transport = self._transport
1386        return transport.blocked if isinstance(transport, BlockedState) else None

Whether the broker has blocked this connection, or None when the transport cannot be asked.

RabbitMQ blocks a connection when it is low on memory or disk, and while it lasts it stops reading that connection's socket: everything sent — a publish, a declaration, health()'s own probe — waits rather than failing. It is the broker protecting itself and it is not an error, so it is worth telling apart from the broker having gone away, which looks identical from outside and wants the opposite response.

None means the question could not be asked at all, which is a different fact from False and is reported as one: a transport that does not implement ~acemq_amqp.transport.BlockedState, or one whose client has moved the state out of reach. See blocked_reason for why there may be a boolean and no reason to go with it.

blocked_reason: str | None
1388    @property
1389    def blocked_reason(self) -> str | None:
1390        """What the broker said when it blocked this connection, if anything.
1391
1392        ``None`` when the connection is not blocked, and also ``None`` when the
1393        transport is blocked but cannot say why — which is what the RabbitMQ
1394        transport answers, because aio-pika reads RabbitMQ's reason and does not
1395        keep it. Reporting the block without the reason is the honest half;
1396        inventing a reason would read exactly like one the broker sent.
1397        """
1398        transport = self._transport
1399        return transport.blocked_reason if isinstance(transport, BlockedState) else None

What the broker said when it blocked this connection, if anything.

None when the connection is not blocked, and also None when the transport is blocked but cannot say why — which is what the RabbitMQ transport answers, because aio-pika reads RabbitMQ's reason and does not keep it. Reporting the block without the reason is the honest half; inventing a reason would read exactly like one the broker sent.

async def health(self, timeout: float = 3.0) -> HealthReport:
1401    async def health(self, timeout: float = DEFAULT_HEALTH_TIMEOUT) -> HealthReport:
1402        """Whether this connection can reach its broker, and its consumers run.
1403
1404        The broker half is a declaration of a temporary queue, which is the
1405        cheapest thing AMQP offers that actually proves the connection works. A
1406        TCP connection that is open but wedged — the broker paused, the network
1407        black-holing — answers a socket-level check exactly as a healthy one
1408        does, right up until something is asked of it.
1409
1410        The consumer half is the one a probe usually wants and nothing outside
1411        can see. A consumer whose workers have died without it being closed is
1412        one the broker is still sending messages to and nothing is reading, and
1413        from outside that is indistinguishable from a quiet queue.
1414
1415        **A blocked connection is up, with the reason.** The broker blocks a
1416        connection when it is low on disk or memory, and an application that
1417        fails its own readiness check for it is one an orchestrator restarts
1418        into the same blocked broker, having thrown away whatever it was
1419        holding. ``parts["blocked"]`` carries the state either way, and is
1420        ``None`` when the transport could not be asked — see :attr:`blocked`.
1421
1422        **The probe is bounded.** A blocked broker stops reading the socket, so
1423        the round trip this makes is precisely the thing that does not come back
1424        when something is wrong; without a deadline of its own a health check
1425        hangs, and a readiness endpoint that hangs takes the instance out of
1426        rotation with no report at all. A probe that runs out of time is
1427        abandoned rather than waited for, because a request that cannot be
1428        answered cannot reliably be cancelled either.
1429
1430        It costs a round trip, so it is not something to call per request. Wire
1431        it to a readiness probe and let the probe's interval decide how often.
1432
1433        :param timeout: how long to give the broker's answer, three seconds by
1434            default. Shorter than the deadline
1435            :func:`~acemq_amqp.telemetry.aggregate_health` puts around a whole
1436            set of checks, so that this report rather than that one is what an
1437            operator reads when a broker stops answering
1438        :returns: the report. Degraded means working but worth an alert
1439        """
1440        checked = datetime.now(timezone.utc)
1441        consumers = self.consumers
1442        stalled = [consumer.queue for consumer in consumers if not consumer.running]
1443        blocked = self.blocked
1444        parts: dict[str, Any] = {
1445            "consumers": len(consumers),
1446            "in-flight": sum(consumer.in_flight for consumer in consumers),
1447            "blocked": blocked,
1448        }
1449        reason = self.blocked_reason
1450        if reason:
1451            parts["blocked-reason"] = reason
1452
1453        if self._closed:
1454            return HealthReport(
1455                HealthStatus.DOWN, "the connection has been closed", checked, parts
1456            )
1457
1458        if blocked:
1459            # Asked before the probe rather than after it, because the probe is
1460            # what a blocked broker will not answer: a round trip started here
1461            # would spend the whole deadline arriving at the answer this already
1462            # has.
1463            return self._blocked_report(checked, parts, stalled, reason)
1464
1465        # Asking whether a queue nothing has ever declared exists, rather than
1466        # declaring a temporary one. It is the same round trip and it proves the
1467        # same thing, and it creates nothing at all: an exclusive queue is only
1468        # released when the channel that declared it closes, so a probe that
1469        # declared one would leave a queue on the broker for the life of the
1470        # connection and a new one behind every restart.
1471        probe = f"acemq-health-{uuid.uuid4().hex}"
1472        started = time.monotonic()
1473        try:
1474            await self._probe_within(probe, timeout)
1475        except asyncio.TimeoutError:
1476            # A round trip that does not come back is the ordinary way a block
1477            # shows itself, and the notification may have arrived while this was
1478            # waiting. Asked again rather than assumed, because "the broker is
1479            # protecting itself" and "the broker has gone" are the same silence.
1480            blocked = self.blocked
1481            parts["blocked"] = blocked
1482            reason = self.blocked_reason
1483            if reason:
1484                parts["blocked-reason"] = reason
1485            if blocked:
1486                return self._blocked_report(checked, parts, stalled, reason)
1487            return HealthReport(
1488                HealthStatus.DOWN,
1489                f"the broker did not answer within {timeout:g}s",
1490                checked,
1491                parts,
1492            )
1493        except Exception as failure:
1494            parts["error"] = _describe(failure)
1495            return HealthReport(
1496                HealthStatus.DOWN, f"the broker did not answer: {failure}", checked, parts
1497            )
1498        parts["round-trip"] = round(time.monotonic() - started, 4)
1499
1500        if stalled:
1501            # Degraded rather than down: the connection works, and a replacement
1502            # instance would almost certainly stall the same way. Worth waking
1503            # somebody; not worth taking this one out of rotation.
1504            return HealthReport(
1505                HealthStatus.DEGRADED,
1506                _stalled_detail(stalled),
1507                checked,
1508                {**parts, "stalled": sorted(stalled)},
1509            )
1510        return HealthReport(HealthStatus.UP, "", checked, parts)

Whether this connection can reach its broker, and its consumers run.

The broker half is a declaration of a temporary queue, which is the cheapest thing AMQP offers that actually proves the connection works. A TCP connection that is open but wedged — the broker paused, the network black-holing — answers a socket-level check exactly as a healthy one does, right up until something is asked of it.

The consumer half is the one a probe usually wants and nothing outside can see. A consumer whose workers have died without it being closed is one the broker is still sending messages to and nothing is reading, and from outside that is indistinguishable from a quiet queue.

A blocked connection is up, with the reason. The broker blocks a connection when it is low on disk or memory, and an application that fails its own readiness check for it is one an orchestrator restarts into the same blocked broker, having thrown away whatever it was holding. parts["blocked"] carries the state either way, and is None when the transport could not be asked — see blocked.

The probe is bounded. A blocked broker stops reading the socket, so the round trip this makes is precisely the thing that does not come back when something is wrong; without a deadline of its own a health check hangs, and a readiness endpoint that hangs takes the instance out of rotation with no report at all. A probe that runs out of time is abandoned rather than waited for, because a request that cannot be answered cannot reliably be cancelled either.

It costs a round trip, so it is not something to call per request. Wire it to a readiness probe and let the probe's interval decide how often.

Parameters
  • timeout: how long to give the broker's answer, three seconds by default. Shorter than the deadline ~acemq_amqp.telemetry.aggregate_health() puts around a whole set of checks, so that this report rather than that one is what an operator reads when a broker stops answering :returns: the report. Degraded means working but worth an alert
publish_interceptors: tuple[Callable[[PublishContext, Callable[[PublishContext], Awaitable[PublishResult]]], Awaitable[PublishResult]], ...]
1598    @property
1599    def publish_interceptors(self) -> tuple[PublishInterceptor, ...]:
1600        """What wraps every publish, outermost first.
1601
1602        A snapshot rather than the list itself, so a publish in progress cannot
1603        be handed a chain that is being edited underneath it.
1604        """
1605        return tuple(self._publishing)

What wraps every publish, outermost first.

A snapshot rather than the list itself, so a publish in progress cannot be handed a chain that is being edited underneath it.

consume_interceptors: tuple[Callable[[ConsumeContext, Callable[[ConsumeContext], Awaitable[Ack]]], Awaitable[Ack]], ...]
1607    @property
1608    def consume_interceptors(self) -> tuple[ConsumeInterceptor, ...]:
1609        """What wraps every handler, outermost first."""
1610        return tuple(self._consuming)

What wraps every handler, outermost first.

def intercept_publish( self, interceptor: Callable[[PublishContext, Callable[[PublishContext], Awaitable[PublishResult]]], Awaitable[PublishResult]]) -> Connection:
1612    def intercept_publish(self, interceptor: PublishInterceptor) -> Connection:
1613        """Wraps every publish on this connection in ``interceptor``.
1614
1615        Registered ones run in the order they were added, so this one is
1616        innermost until the next is added. Read at publish time rather than
1617        copied into each publisher, so an interceptor registered during start-up
1618        applies to publishers that already exist — which is what makes it usable
1619        from a framework's wiring rather than only from the line that connects.
1620
1621        Adding at start-up is still what to do. A message already in flight will
1622        not see it, and an interceptor whose whole job is to stamp every message
1623        is not doing it if the first few went without.
1624
1625        :param interceptor: what to wrap the publish in. See
1626            :mod:`acemq_amqp.interceptors`
1627        :returns: this connection, so registrations read as one expression
1628        """
1629        self._publishing.append(interceptor)
1630        return self

Wraps every publish on this connection in interceptor.

Registered ones run in the order they were added, so this one is innermost until the next is added. Read at publish time rather than copied into each publisher, so an interceptor registered during start-up applies to publishers that already exist — which is what makes it usable from a framework's wiring rather than only from the line that connects.

Adding at start-up is still what to do. A message already in flight will not see it, and an interceptor whose whole job is to stamp every message is not doing it if the first few went without.

Parameters
  • interceptor: what to wrap the publish in. See acemq_amqp.interceptors :returns: this connection, so registrations read as one expression
def intercept_consume( self, interceptor: Callable[[ConsumeContext, Callable[[ConsumeContext], Awaitable[Ack]]], Awaitable[Ack]]) -> Connection:
1632    def intercept_consume(self, interceptor: ConsumeInterceptor) -> Connection:
1633        """Wraps every handler on this connection in ``interceptor``.
1634
1635        :param interceptor: what to wrap the handler in
1636        :returns: this connection
1637        """
1638        self._consuming.append(interceptor)
1639        return self

Wraps every handler on this connection in interceptor.

Parameters
  • interceptor: what to wrap the handler in :returns: this connection
async def declare(self, topology: Topology) -> None:
1641    async def declare(self, topology: Topology) -> None:
1642        """Applies a topology to this connection's broker.
1643
1644        :param topology: what the broker should have
1645        """
1646        await topology.apply(self._transport)

Applies a topology to this connection's broker.

Parameters
  • topology: what the broker should have
def publisher( self, exchange: str = '', routing_key: str = '', *, codec: Codec | None = None, persistent: bool = True, mandatory: bool = False) -> Publisher:
1648    def publisher(
1649        self,
1650        exchange: str = "",
1651        routing_key: str = "",
1652        *,
1653        codec: Codec | None = None,
1654        persistent: bool = True,
1655        mandatory: bool = False,
1656    ) -> Publisher:
1657        """Builds a publisher for an exchange and routing key.
1658
1659        Publish straight to a queue by leaving the exchange empty: the default
1660        exchange routes to the queue whose name matches the routing key.
1661
1662        :param exchange: where to publish, empty for the default exchange
1663        :param routing_key: what to publish under, or a queue name
1664        :param codec: a codec other than this connection's
1665        :param persistent: ask the broker to write these messages to disk
1666        :param mandatory: fail when a message reaches no queue at all, rather
1667            than letting the broker drop it silently
1668        :returns: the publisher
1669        """
1670        return Publisher(
1671            self,
1672            exchange,
1673            routing_key,
1674            codec=codec,
1675            persistent=persistent,
1676            mandatory=mandatory,
1677        )

Builds a publisher for an exchange and routing key.

Publish straight to a queue by leaving the exchange empty: the default exchange routes to the queue whose name matches the routing key.

Parameters
  • exchange: where to publish, empty for the default exchange
  • routing_key: what to publish under, or a queue name
  • codec: a codec other than this connection's
  • persistent: ask the broker to write these messages to disk
  • mandatory: fail when a message reaches no queue at all, rather than letting the broker drop it silently :returns: the publisher
async def consume( self, queue: str, handler: Callable[Message, Ack | Awaitable[Ack]], *, codec: Codec | None = None, retry: RetryPolicy | None = None, prefetch: int | None = None, concurrency: int = 1, tag: str = '', args: Mapping[str, typing.Any] | None = None, declare: bool = True) -> Consumer:
1679    async def consume(
1680        self,
1681        queue: str,
1682        handler: Handler,
1683        *,
1684        codec: Codec | None = None,
1685        retry: RetryPolicy | None = None,
1686        prefetch: int | None = None,
1687        concurrency: int = 1,
1688        tag: str = "",
1689        args: Mapping[str, Any] | None = None,
1690        declare: bool = True,
1691    ) -> Consumer:
1692        """Reads messages from a queue until the returned consumer is closed.
1693
1694        Before it subscribes it declares the queues it will need when a message
1695        fails: ``{queue}.dlq``, ``{queue}.parked``, ``acemq.dlx`` and, when the
1696        retry policy has waits the broker holds, ``acemq.retry`` and the rungs.
1697        See :func:`acemq_amqp.topology.declare_where_failures_go` for exactly
1698        what and why. The source queue is not among them, and neither is
1699        anything a producer needs, so this is not a replacement for declaring a
1700        topology — it is the floor underneath a service that was deployed
1701        without one.
1702
1703        :param queue: what to read
1704        :param handler: what to do with a message; returning an
1705            :class:`~acemq_amqp.Ack` is the decision
1706        :param codec: a codec other than this connection's
1707        :param retry: a policy other than this connection's
1708        :param prefetch: how many unacknowledged messages to hold
1709        :param concurrency: how many messages to work on at once. One by
1710            default, which keeps a queue's messages in order; raising it trades
1711            that order for throughput, which is the right trade for handlers
1712            that spend their time waiting on something else
1713        :param tag: what to call this consumer to the broker
1714        :param args: broker-specific consumer arguments
1715        :param declare: declare the queues above before subscribing. On by
1716            default, because the alternative default loses messages silently.
1717            Turn it off for a login with no ``configure`` permission on the
1718            vhost, which is refused by the broker rather than ignored, and for a
1719            tool draining a queue it does not own — a one-off reader of
1720            ``orders.dlq`` has no business creating ``orders.dlq.dlq``. A
1721            consumer started this way still publishes to those queues, so
1722            somebody else has to have declared them
1723        :returns: the running consumer
1724        """
1725        if self._closed:
1726            raise AceMQError("acemq: this connection is closed")
1727
1728        consumer = Consumer(
1729            self,
1730            queue,
1731            handler,
1732            codec=codec or self._codec,
1733            retry=retry or self._retry,
1734            concurrency=concurrency,
1735            declare=declare,
1736        )
1737        self._consumers.append(consumer)
1738        try:
1739            await consumer._start(
1740                self._prefetch if prefetch is None else prefetch,
1741                tag,
1742                args or {},
1743            )
1744        except BaseException:
1745            self._untrack(consumer)
1746            raise
1747        return consumer

Reads messages from a queue until the returned consumer is closed.

Before it subscribes it declares the queues it will need when a message fails: {queue}.dlq, {queue}.parked, acemq.dlx and, when the retry policy has waits the broker holds, acemq.retry and the rungs. See acemq_amqp.topology.declare_where_failures_go() for exactly what and why. The source queue is not among them, and neither is anything a producer needs, so this is not a replacement for declaring a topology — it is the floor underneath a service that was deployed without one.

Parameters
  • queue: what to read
  • handler: what to do with a message; returning an ~acemq_amqp.Ack is the decision
  • codec: a codec other than this connection's
  • retry: a policy other than this connection's
  • prefetch: how many unacknowledged messages to hold
  • concurrency: how many messages to work on at once. One by default, which keeps a queue's messages in order; raising it trades that order for throughput, which is the right trade for handlers that spend their time waiting on something else
  • tag: what to call this consumer to the broker
  • args: broker-specific consumer arguments
  • declare: declare the queues above before subscribing. On by default, because the alternative default loses messages silently. Turn it off for a login with no configure permission on the vhost, which is refused by the broker rather than ignored, and for a tool draining a queue it does not own — a one-off reader of orders.dlq has no business creating orders.dlq.dlq. A consumer started this way still publishes to those queues, so somebody else has to have declared them :returns: the running consumer
async def publish_raw( self, exchange: str, routing_key: str, message: Outbound) -> PublishResult:
1749    async def publish_raw(
1750        self, exchange: str, routing_key: str, message: Outbound
1751    ) -> PublishResult:
1752        """Sends bytes that are already encoded, with headers already rendered.
1753
1754        For the retry and dead-letter hops, and for anything else replaying a
1755        message recorded earlier: the payload's class may not exist any more,
1756        and re-encoding it would produce different bytes from the ones that were
1757        committed. Ordinary publishing goes through :class:`Publisher`.
1758
1759        **This is the one place the outstanding-publish bound is taken.** Every
1760        publish in the library comes through here — a :class:`Publisher`, a
1761        retry rung, a dead letter, a replay — so one permit counts them all, and
1762        :meth:`Publisher.send_all` cannot put more messages on the wire at once
1763        than :attr:`max_outstanding_publishes` allows however long the list it
1764        was given is.
1765
1766        :param exchange: where to publish, empty for the default exchange
1767        :param routing_key: what to publish under
1768        :param message: the message, encoded
1769        :returns: what the broker said
1770        :raises PublishingPausedError: when the broker has blocked this
1771            connection. Nothing was sent; retry once it unblocks
1772        :raises PublishError: when every permit is taken and none came free
1773            within :attr:`confirm_timeout`, or when the broker did not confirm
1774            this publish within it. The second may or may not have arrived
1775        """
1776        # Refused rather than queued, before a permit or a byte is spent. A
1777        # blocked broker stops reading the socket, so a publish sent now sits in
1778        # the client's write buffer with no confirm coming until the alarm
1779        # clears, then times out as a failure that may or may not have arrived.
1780        # Go and .NET refuse at the same point for the same reason. ``None``
1781        # means the state could not be read and is not treated as blocked.
1782        if self.blocked is True:
1783            reason = self.blocked_reason
1784            raise PublishingPausedError(
1785                message.message_id,
1786                exchange,
1787                routing_key,
1788                (
1789                    f"{self._blocked_detail(reason)}. Nothing was sent; back off and "
1790                    "retry once the broker unblocks"
1791                ),
1792            )
1793
1794        # Taken before the message is written and not after, which is the whole
1795        # point: a bound applied afterwards has already let the message into
1796        # memory. The permit is given back in the ``finally`` below on every
1797        # path there is — the publish returning, the publish raising, and this
1798        # task being cancelled while the broker is still thinking — because a
1799        # permit lost on any one of those is a connection that publishes a
1800        # thousand more messages and then stops for ever.
1801        try:
1802            await asyncio.wait_for(
1803                self._outstanding.acquire(), self._confirm_timeout.total_seconds()
1804            )
1805        except asyncio.TimeoutError as stalled:
1806            # Said rather than waited out. A publisher parked behind a broker
1807            # that has stopped answering is indistinguishable from a quiet
1808            # service, and this sentence is the one Java raises word for word so
1809            # that an operator reading a log has one thing to recognise.
1810            raise PublishError(
1811                message.message_id,
1812                exchange,
1813                routing_key,
1814                (
1815                    f"{self._max_outstanding} publishes are already waiting for a "
1816                    f"confirm and none completed within {self._confirm_timeout}. "
1817                    "The broker is not keeping up; publish more slowly rather than "
1818                    "buffering more."
1819                ),
1820            ) from stalled
1821
1822        self._in_flight_publishes += 1
1823        try:
1824            # Bounded as well, by the same timeout. RabbitMQ says a connection is
1825            # blocked only once it publishes, so the first publish after an alarm
1826            # is written and then held with no confirm until the alarm clears --
1827            # minutes, or never. Said here as a failure, not a refusal: it was
1828            # written, so it may have arrived. The publish after it sees the
1829            # block and is refused above. .NET bounds the same wait the same way.
1830            result = await asyncio.wait_for(
1831                self._transport.publish(exchange, routing_key, message),
1832                self._confirm_timeout.total_seconds(),
1833            )
1834        except asyncio.TimeoutError as unconfirmed:
1835            raise PublishError(
1836                message.message_id,
1837                exchange,
1838                routing_key,
1839                (
1840                    f"the broker did not confirm this publish within "
1841                    f"{self._confirm_timeout}. It may or may not have been stored, "
1842                    "so treat it as neither sent nor lost"
1843                ),
1844            ) from unconfirmed
1845        finally:
1846            self._in_flight_publishes -= 1
1847            self._outstanding.release()
1848
1849        # A nack is the broker declining to keep a message it was asked to take, and
1850        # it is raised here rather than returned.
1851        #
1852        # It used to be returned, and `confirmed` was read nowhere in this library
1853        # except a telemetry label -- so a caller who did not inspect the result
1854        # carried on believing the message had gone somewhere, and Java, .NET, Go and
1855        # Ruby all raised on the same event. Worse than the wrong belief: the outbox
1856        # relay removes a record because the publish did not raise, so a nack deleted
1857        # a record for a message the broker had refused. At-most-once, in the pattern
1858        # whose whole purpose is not losing anything.
1859        #
1860        # Checked here because this is the one place every publish in the library
1861        # comes through -- a Publisher, a retry rung, a dead letter, a replay, the
1862        # outbox -- so one check covers all of them, and anything that settles or
1863        # deletes on the strength of a publish returning now hears the refusal.
1864        #
1865        # Distinct from unroutable, which the Publisher path reports separately and
1866        # which sets `unroutable`: the broker could route this one and would not keep
1867        # it, and the two want different responses.
1868        #
1869        # So a returned message is passed back rather than raised here. The
1870        # RabbitMQ transport reports a Basic.Return as not confirmed as well as
1871        # not routed, and raising it as a nack hid ``routed`` from every caller
1872        # that reads it: a mandatory Publisher said "refused" rather than
1873        # "reached no queue", and the retry and dead-letter hops never reached
1874        # the fallbacks written for a queue that is not there.
1875        if not result.confirmed and result.routed:
1876            raise PublishError(
1877                message.message_id,
1878                exchange,
1879                routing_key,
1880                (
1881                    "the broker refused to confirm it. The message reached the broker "
1882                    "and the broker has not taken responsibility for it, so it must be "
1883                    "treated as not sent"
1884                ),
1885            )
1886
1887        return result

Sends bytes that are already encoded, with headers already rendered.

For the retry and dead-letter hops, and for anything else replaying a message recorded earlier: the payload's class may not exist any more, and re-encoding it would produce different bytes from the ones that were committed. Ordinary publishing goes through Publisher.

This is the one place the outstanding-publish bound is taken. Every publish in the library comes through here — a Publisher, a retry rung, a dead letter, a replay — so one permit counts them all, and Publisher.send_all() cannot put more messages on the wire at once than max_outstanding_publishes allows however long the list it was given is.

Parameters
  • exchange: where to publish, empty for the default exchange
  • routing_key: what to publish under
  • message: the message, encoded :returns: what the broker said
Raises
  • PublishingPausedError: when the broker has blocked this connection. Nothing was sent; retry once it unblocks
  • PublishError: when every permit is taken and none came free within confirm_timeout, or when the broker did not confirm this publish within it. The second may or may not have arrived
async def pull(self, queue: str) -> Delivery | None:
1889    async def pull(self, queue: str) -> Delivery | None:
1890        """Takes one message off a queue, or ``None`` when there is none waiting.
1891
1892        For tools rather than for services: it is a round trip per message where
1893        :meth:`consume` gets a stream. What it can do that a consumer cannot is
1894        say that a queue is empty, which is what anything working through a
1895        backlog and then stopping — a replay, a drain — has to know.
1896
1897        The delivery comes back unsettled, so the caller decides whether it is
1898        gone or goes back.
1899
1900        :param queue: what to read from
1901        :returns: the delivery, or ``None``
1902        """
1903        return await self._source().pull(queue)

Takes one message off a queue, or None when there is none waiting.

For tools rather than for services: it is a round trip per message where consume() gets a stream. What it can do that a consumer cannot is say that a queue is empty, which is what anything working through a backlog and then stopping — a replay, a drain — has to know.

The delivery comes back unsettled, so the caller decides whether it is gone or goes back.

Parameters
  • queue: what to read from :returns: the delivery, or None
async def queue_exists(self, queue: str) -> bool:
1905    async def queue_exists(self, queue: str) -> bool:
1906        """Whether a queue is on the broker, creating nothing."""
1907        return await self._admin().queue_exists(queue)

Whether a queue is on the broker, creating nothing.

async def message_count(self, queue: str) -> int:
1909    async def message_count(self, queue: str) -> int:
1910        """How many messages are waiting on a queue."""
1911        return await self._admin().message_count(queue)

How many messages are waiting on a queue.

async def delete_queue(self, queue: str) -> None:
1913    async def delete_queue(self, queue: str) -> None:
1914        """Removes a queue and every message still on it."""
1915        await self._admin().delete_queue(queue)

Removes a queue and every message still on it.

async def close(self, *, timeout: float | None = 20.0) -> bool:
1936    async def close(self, *, timeout: float | None = DEFAULT_DRAIN_TIMEOUT) -> bool:
1937        """Stops every consumer on this connection and releases it.
1938
1939        Consumers first, and their handlers given up to ``timeout`` to finish,
1940        so a message being worked on when this is called is settled rather than
1941        returned to the queue for somebody else to redo. The bound is one
1942        deadline for every handler of every consumer, not one each: they are
1943        all drained at once. A handler still running at the deadline is
1944        cancelled and its message left unsettled, for the broker to redeliver —
1945        see :meth:`Consumer.close`.
1946
1947        The connection is released whatever happens, including when the caller
1948        cancels closing; that cancellation is raised again afterwards.
1949
1950        :param timeout: seconds to wait for the running handlers; ``None``
1951            waits for as long as they take
1952        :returns: ``True`` if every handler finished, ``False`` if the time ran
1953            out first
1954        """
1955        if self._closed:
1956            return True
1957        self._closed = True
1958        try:
1959            drained = await asyncio.gather(
1960                *(consumer.close(timeout=timeout) for consumer in list(self._consumers)),
1961                return_exceptions=True,
1962            )
1963        finally:
1964            await self._transport.close()
1965        for outcome in drained:
1966            if isinstance(outcome, BaseException):
1967                raise outcome
1968        return all(drained)

Stops every consumer on this connection and releases it.

Consumers first, and their handlers given up to timeout to finish, so a message being worked on when this is called is settled rather than returned to the queue for somebody else to redo. The bound is one deadline for every handler of every consumer, not one each: they are all drained at once. A handler still running at the deadline is cancelled and its message left unsettled, for the broker to redeliver — see Consumer.close().

The connection is released whatever happens, including when the caller cancels closing; that cancellation is raised again afterwards.

Parameters
  • timeout: seconds to wait for the running handlers; None waits for as long as they take :returns: True if every handler finished, False if the time ran out first
@dataclass(slots=True)
class ConsumeContext:
126@dataclass(slots=True)
127class ConsumeContext:
128    """A message that has arrived and been decoded, before the handler sees it.
129
130    :param queue: what it was read from
131    :param envelope: the metadata, as it will reach the handler. Changing it
132        changes what the handler is given
133    :param payload: the decoded body, likewise
134    :param body: the undecoded bytes, for an interceptor that wants to see what
135        actually arrived
136    :param content_type: what the sender said the body was, or ``None``
137    :param routing_key: the key it arrived under
138    :param redelivered: the broker saying it has handed this one over before
139    :param state: somewhere for an interceptor to leave something for a later
140        one, or for its own way out. Empty, and this library never reads it
141    :param reports_settlement: whether whatever is driving this delivery will
142        say how it was settled. True when a consumer is, and false when an
143        interceptor chain is being run by something else — a test, or a caller
144        composing the chain by hand
145    :param reply_to: AMQP's own ``reply-to`` property as it arrived, empty when
146        the sender set none
147    """
148
149    queue: str
150    envelope: Envelope
151    payload: Any
152    body: bytes
153    content_type: str | None
154    routing_key: str
155    redelivered: bool
156    state: dict[str, Any] = field(default_factory=dict)
157    reports_settlement: bool = False
158    reply_to: str = ""
159    _settlement_listeners: list[SettlementListener] = field(
160        default_factory=list, repr=False
161    )
162
163    def when_settled(self, listener: SettlementListener) -> bool:
164        """Asks to be told how this delivery was settled.
165
166        What the handler answered is not what happened to the message: a
167        handler asking for a retry with no attempts left is dead-lettered, and
168        an interceptor that read only the :class:`~acemq_amqp.ack.Ack` would
169        report a retry that never happened. That gap is exactly what somebody
170        querying a trace backend for dead letters falls into, so the consumer
171        reports its decision here and the tracing adapter waits for it.
172
173        The listener is called once, on the consumer's task, after the decision
174        and before it is carried out. It should not block and must not raise;
175        one that does is logged and otherwise ignored, because a delivery still
176        has to be settled when whoever was watching it breaks.
177
178        :param listener: called with the :class:`~acemq_amqp.ack.Settlement`
179        :returns: whether anything will ever call it. ``False`` when this
180            delivery is not being driven by a consumer, in which case the
181            caller should not wait for an answer that is not coming
182        """
183        if not self.reports_settlement:
184            return False
185        self._settlement_listeners.append(listener)
186        return True
187
188    def settled(self, settlement: Settlement) -> None:
189        """Tells the listeners what happened. Called by the consumer.
190
191        :param settlement: what the consumer decided to do with the delivery
192        """
193        for listener in self._settlement_listeners:
194            try:
195                listener(settlement)
196            except Exception:
197                # The delivery still has to be settled. A listener is an
198                # observer, and an observer that breaks must not take the
199                # message with it.
200                log.exception(
201                    "acemq: a settlement listener failed for a delivery from %s", self.queue
202                )

A message that has arrived and been decoded, before the handler sees it.

Parameters
  • queue: what it was read from
  • envelope: the metadata, as it will reach the handler. Changing it changes what the handler is given
  • payload: the decoded body, likewise
  • body: the undecoded bytes, for an interceptor that wants to see what actually arrived
  • content_type: what the sender said the body was, or None
  • routing_key: the key it arrived under
  • redelivered: the broker saying it has handed this one over before
  • state: somewhere for an interceptor to leave something for a later one, or for its own way out. Empty, and this library never reads it
  • reports_settlement: whether whatever is driving this delivery will say how it was settled. True when a consumer is, and false when an interceptor chain is being run by something else — a test, or a caller composing the chain by hand
  • reply_to: AMQP's own reply-to property as it arrived, empty when the sender set none
ConsumeContext( queue: str, envelope: Envelope, payload: Any, body: bytes, content_type: str | None, routing_key: str, redelivered: bool, state: dict[str, typing.Any] = <factory>, reports_settlement: bool = False, reply_to: str = '', _settlement_listeners: list[Callable[[Settlement], None]] = <factory>)
queue: str
envelope: Envelope
payload: Any
body: bytes
content_type: str | None
routing_key: str
redelivered: bool
state: dict[str, typing.Any]
reports_settlement: bool
reply_to: str
def when_settled(self, listener: Callable[[Settlement], None]) -> bool:
163    def when_settled(self, listener: SettlementListener) -> bool:
164        """Asks to be told how this delivery was settled.
165
166        What the handler answered is not what happened to the message: a
167        handler asking for a retry with no attempts left is dead-lettered, and
168        an interceptor that read only the :class:`~acemq_amqp.ack.Ack` would
169        report a retry that never happened. That gap is exactly what somebody
170        querying a trace backend for dead letters falls into, so the consumer
171        reports its decision here and the tracing adapter waits for it.
172
173        The listener is called once, on the consumer's task, after the decision
174        and before it is carried out. It should not block and must not raise;
175        one that does is logged and otherwise ignored, because a delivery still
176        has to be settled when whoever was watching it breaks.
177
178        :param listener: called with the :class:`~acemq_amqp.ack.Settlement`
179        :returns: whether anything will ever call it. ``False`` when this
180            delivery is not being driven by a consumer, in which case the
181            caller should not wait for an answer that is not coming
182        """
183        if not self.reports_settlement:
184            return False
185        self._settlement_listeners.append(listener)
186        return True

Asks to be told how this delivery was settled.

What the handler answered is not what happened to the message: a handler asking for a retry with no attempts left is dead-lettered, and an interceptor that read only the ~acemq_amqp.ack.Ack would report a retry that never happened. That gap is exactly what somebody querying a trace backend for dead letters falls into, so the consumer reports its decision here and the tracing adapter waits for it.

The listener is called once, on the consumer's task, after the decision and before it is carried out. It should not block and must not raise; one that does is logged and otherwise ignored, because a delivery still has to be settled when whoever was watching it breaks.

Parameters
  • listener: called with the ~acemq_amqp.ack.Settlement :returns: whether anything will ever call it. False when this delivery is not being driven by a consumer, in which case the caller should not wait for an answer that is not coming
def settled(self, settlement: Settlement) -> None:
188    def settled(self, settlement: Settlement) -> None:
189        """Tells the listeners what happened. Called by the consumer.
190
191        :param settlement: what the consumer decided to do with the delivery
192        """
193        for listener in self._settlement_listeners:
194            try:
195                listener(settlement)
196            except Exception:
197                # The delivery still has to be settled. A listener is an
198                # observer, and an observer that breaks must not take the
199                # message with it.
200                log.exception(
201                    "acemq: a settlement listener failed for a delivery from %s", self.queue
202                )

Tells the listeners what happened. Called by the consumer.

Parameters
  • settlement: what the consumer decided to do with the delivery
ConsumeInterceptor = collections.abc.Callable[[ConsumeContext, collections.abc.Callable[[ConsumeContext], collections.abc.Awaitable[Ack]]], collections.abc.Awaitable[Ack]]
ConsumeNext = collections.abc.Callable[[ConsumeContext], collections.abc.Awaitable[Ack]]
@dataclass(frozen=True, slots=True)
class ConsumeSpec:
68@dataclass(frozen=True, slots=True)
69class ConsumeSpec:
70    """How a consumer should be set up.
71
72    :param prefetch: how many unacknowledged messages the broker will send at
73        once, or zero for the transport's default
74    :param tag: what to call this consumer to the broker, which is what an
75        operator sees when working out who is holding a message
76    :param args: broker-specific consumer arguments, such as the
77        ``x-stream-offset`` a stream consumer needs to say where it starts
78    """
79
80    prefetch: int = 0
81    tag: str = ""
82    args: Mapping[str, Any] = field(default_factory=dict)

How a consumer should be set up.

Parameters
  • prefetch: how many unacknowledged messages the broker will send at once, or zero for the transport's default
  • tag: what to call this consumer to the broker, which is what an operator sees when working out who is holding a message
  • args: broker-specific consumer arguments, such as the x-stream-offset a stream consumer needs to say where it starts
ConsumeSpec( prefetch: int = 0, tag: str = '', args: Mapping[str, typing.Any] = <factory>)
prefetch: int
tag: str
args: Mapping[str, typing.Any]
class Consumer:
 451class Consumer:
 452    """A running subscription. Close it to stop.
 453
 454    One is built by :meth:`Connection.consume` rather than directly, because it
 455    has to be started before it is any use and a half-built consumer that looks
 456    finished is a thing somebody will hold on to.
 457
 458    Starting one declares the queues it will need when a message fails — its
 459    dead-letter queue, its parking lot and its rungs — unless it was told not
 460    to. See :func:`acemq_amqp.topology.declare_where_failures_go`.
 461    """
 462
 463    def __init__(
 464        self,
 465        connection: Connection,
 466        queue: str,
 467        handler: Handler,
 468        *,
 469        codec: Codec,
 470        retry: RetryPolicy,
 471        concurrency: int,
 472        declare: bool = True,
 473    ) -> None:
 474        if concurrency < 1:
 475            raise ValueError(f"acemq: concurrency must be at least 1, got {concurrency}")
 476        self._connection = connection
 477        self._queue = queue
 478        self._handler = handler
 479        self._codec = codec
 480        self._retry = retry
 481        self._concurrency = concurrency
 482        self._declare = declare
 483        self._work: asyncio.Queue[Delivery | None] = asyncio.Queue()
 484        self._workers: list[asyncio.Task[None]] = []
 485        self._subscription: Subscription | None = None
 486        self._in_flight = 0
 487        self._closed = False
 488
 489    @property
 490    def queue(self) -> str:
 491        """The queue this consumer reads."""
 492        return self._queue
 493
 494    @property
 495    def closed(self) -> bool:
 496        """Whether this consumer has been stopped."""
 497        return self._closed
 498
 499    @property
 500    def running(self) -> bool:
 501        """Whether this consumer is subscribed and still has workers to hand a
 502        delivery to.
 503
 504        What a health check reads. A consumer whose workers have all finished
 505        without it being closed is a consumer the broker is still sending
 506        messages to and nothing is reading — which looks identical to a quiet
 507        queue from outside, and is the failure worth telling a probe about.
 508        """
 509        return (
 510            not self._closed
 511            and self._subscription is not None
 512            and any(not worker.done() for worker in self._workers)
 513        )
 514
 515    @property
 516    def in_flight(self) -> int:
 517        """How many messages this consumer is working on right now."""
 518        return self._in_flight
 519
 520    @property
 521    def _labels(self) -> dict[str, str]:
 522        return {TAG_QUEUE: self._queue}
 523
 524    @property
 525    def _observer(self) -> Observer:
 526        return self._connection.observer
 527
 528    async def _start(self, prefetch: int, tag: str, args: Mapping[str, Any]) -> None:
 529        if self._declare:
 530            # Before the subscription rather than after it, so that the first
 531            # message this consumer gives up on already has somewhere to go. A
 532            # broker that refuses one of these declarations is a consumer that
 533            # does not start, which is the honest failure: it would otherwise
 534            # start and lose its first dead letter.
 535            await declare_where_failures_go(
 536                self._connection.transport, self._queue, self._retry
 537            )
 538
 539        self._workers = [
 540            asyncio.create_task(self._work_loop(), name=f"acemq-consumer-{self._queue}-{n}")
 541            for n in range(self._concurrency)
 542        ]
 543        try:
 544            self._subscription = await self._connection.transport.consume(
 545                self._queue,
 546                ConsumeSpec(prefetch=prefetch, tag=tag, args=dict(args)),
 547                self._deliver,
 548            )
 549        except BaseException:
 550            # Nothing was subscribed, so nothing will arrive; the workers would
 551            # otherwise sit on an empty queue for the life of the process.
 552            for _ in self._workers:
 553                self._work.put_nowait(None)
 554            await asyncio.gather(*self._workers)
 555            raise
 556
 557    async def _deliver(self, delivery: Delivery) -> None:
 558        """Hands a delivery to the workers.
 559
 560        Queued rather than handled here, so that ``concurrency`` means what it
 561        says: aio-pika starts a task per delivery and does not wait for the last
 562        one, so without a queue of our own a prefetch of twenty would run twenty
 563        handlers at once whatever the caller asked for. The queue needs no bound
 564        because prefetch is one: the broker will not send a twenty-first message
 565        until one of the twenty has been settled.
 566        """
 567        self._work.put_nowait(delivery)
 568
 569    async def _work_loop(self) -> None:
 570        while True:
 571            delivery = await self._work.get()
 572            if delivery is None:
 573                return
 574            try:
 575                await self._handle(delivery)
 576            except Exception:
 577                # A failure here is this library's, not the handler's, and the
 578                # message is left unsettled on purpose: the broker will hand it
 579                # back rather than it disappearing because our own code broke.
 580                log.exception("acemq: could not settle a delivery from %s", self._queue)
 581
 582    async def _handle(self, delivery: Delivery) -> None:
 583        envelope = Envelope.from_headers(delivery.headers, delivery.routing_key)
 584        # No counter for "a message arrived": every delivery is counted once
 585        # when it is settled, on ``acemq.consume.total`` with the outcome, and
 586        # the sum across the outcomes is how many arrived. One counter that
 587        # leads the others by however many messages are in flight is a counter
 588        # that makes an operator wonder which one is lying.
 589        #
 590        # The attempt is a distribution rather than a counter, and it is recorded
 591        # here on arrival rather than at the settlement: it is a fact about the
 592        # delivery that is true before the handler runs, and recording it after
 593        # would drop every message this consumer is still working on when the
 594        # process stops. A rising distribution is a dependency in trouble, and it
 595        # says so before anything else does — the dead letters only move once the
 596        # attempts run out.
 597        self._observer.observe(
 598            METRIC_CONSUME_ATTEMPTS, float(envelope.attempt), self._labels
 599        )
 600        self._in_flight += 1
 601        self._observer.gauge(METRIC_CONSUME_IN_FLIGHT, self._in_flight, self._labels)
 602        try:
 603            await self._handle_one(delivery, envelope)
 604        finally:
 605            self._in_flight -= 1
 606            self._observer.gauge(METRIC_CONSUME_IN_FLIGHT, self._in_flight, self._labels)
 607
 608    async def _handle_one(self, delivery: Delivery, envelope: Envelope) -> None:
 609        try:
 610            payload = self._codec.decode(delivery.body, delivery.content_type)
 611        except Exception as failure:
 612            # A body that will not decode decodes no better next time, so it
 613            # does not go round the retry schedule until it ages out.
 614            #
 615            # Parked rather than dead-lettered: a message that failed five times
 616            # and a message nothing could read are different problems with
 617            # different answers — one is usually the world, the other is usually
 618            # a producer — and whoever drains the dead letters should not have
 619            # to sort them by hand.
 620            #
 621            # Counted as a settled delivery here rather than in ``_carry_out``,
 622            # which this path never reaches: no handler ran, so there is no
 623            # duration to record, but the message did arrive and was dealt with
 624            # and a total that missed it would not add up.
 625            self._observer.count(
 626                METRIC_CONSUME_TOTAL, 1, {**self._labels, TAG_OUTCOME: OUTCOME_PARKED}
 627            )
 628            await self._park(
 629                delivery, envelope, f"could not be decoded: {_describe(failure)}"
 630            )
 631            return
 632
 633        context = ConsumeContext(
 634            queue=self._queue,
 635            envelope=envelope,
 636            payload=payload,
 637            body=delivery.body,
 638            content_type=delivery.content_type,
 639            routing_key=delivery.routing_key,
 640            redelivered=delivery.redelivered,
 641            # This delivery is being driven by a consumer, so how it ends will
 642            # be reported. An interceptor that wants to know can wait for it.
 643            reports_settlement=True,
 644            reply_to=delivery.reply_to,
 645        )
 646
 647        started = time.monotonic()
 648        try:
 649            decision = await consume_chain(
 650                self._connection.consume_interceptors, self._run_handler
 651            )(context)
 652        except Exception as failure:
 653            # An exception is how Python says a thing failed, so a handler that
 654            # raises is asking for the retry policy rather than confessing a
 655            # bug — which is what Java does with the same situation, and what a
 656            # handler written without reading the documentation will expect.
 657            # Raising FatalError is how it says the opposite, and the retry path
 658            # below reads that mark before it reads the request.
 659            #
 660            # An interceptor that raises lands here too, and deliberately: a
 661            # refused message is retried and then dead-lettered rather than
 662            # acknowledged as though something had processed it.
 663            decision = Ack(Action.RETRY, failure)
 664
 665        # Timed around the interceptors as well as the handler, because what an
 666        # operator wants to know is how long a message takes to deal with, and
 667        # an interceptor that opens a transaction is part of dealing with it.
 668        # Stopped here and recorded a few lines below rather than in a
 669        # ``finally``, because the duration is tagged with the outcome and the
 670        # outcome is not known until ``_decide`` has run: how long a message
 671        # took and what happened to it are one question, and a p99 that mixes
 672        # the messages that worked with the ones that timed out answers neither
 673        # half of it.
 674        elapsed = time.monotonic() - started
 675
 676        # Whatever the interceptors left, rather than what arrived: one that
 677        # rewrote the envelope on the way in meant that rewrite for the retry
 678        # and the dead letter as much as for the handler.
 679        envelope = context.envelope
 680
 681        if isinstance(decision, Ack):
 682            settlement, wait = self._decide(envelope, decision)
 683        else:
 684            settlement, wait = (
 685                Settlement(
 686                    OUTCOME_DEAD_LETTERED,
 687                    reason=(
 688                        f"the handler returned {type(decision).__name__} instead of an Ack"
 689                    ),
 690                ),
 691                None,
 692            )
 693
 694        self._observer.observe(
 695            METRIC_CONSUME_DURATION,
 696            elapsed,
 697            {**self._labels, TAG_OUTCOME: settlement.outcome},
 698        )
 699
 700        # Announced before it is carried out, and so before a consumer-side
 701        # retry sleeps out its backoff. A listener told afterwards would be
 702        # describing the delivery minutes after the decision, and the span it is
 703        # writing on would have stayed open for a wait that is not work.
 704        context.settled(settlement)
 705        await self._carry_out(delivery, envelope, settlement, wait)
 706
 707    async def _run_handler(self, context: ConsumeContext) -> Ack:
 708        """The innermost work: build the message the interceptors left, and run
 709        the handler on it.
 710
 711        The message is built here rather than before the chain, so that an
 712        interceptor which changed the envelope or the payload changed what the
 713        handler is given rather than a copy nobody reads.
 714        """
 715        message = Message(
 716            payload=context.payload,
 717            envelope=context.envelope,
 718            routing_key=context.routing_key,
 719            content_type=context.content_type,
 720            redelivered=context.redelivered,
 721            body=context.body,
 722            reply_to=context.reply_to,
 723        )
 724        returned = self._handler(message)
 725        # Not an Ack is a mistake worth reporting, but it is not this method's
 726        # to report: it is returned as it stands and _handle dead-letters it
 727        # with a reason, which is where every other unhandleable delivery goes.
 728        # An interceptor between here and there sees the same thing the handler
 729        # returned, which is the honest thing to show it.
 730        decision: Ack = await returned if inspect.isawaitable(returned) else returned
 731        return decision
 732
 733    def _decide(self, envelope: Envelope, decision: Ack) -> tuple[Settlement, Wait | None]:
 734        """What is going to happen to this delivery, before anything happens to
 735        it.
 736
 737        Worked out on its own, and without touching the broker, so that it can
 738        be announced first. The gap this closes is a small one to describe and a
 739        wide one to fall into: a handler asking for a retry it has no attempts
 740        left for is dead-lettered, and anything reading the handler's answer
 741        instead of this one records a retry that never happened.
 742
 743        :param envelope: the message's metadata, for the attempt and the age
 744        :param decision: what the handler answered
 745        :returns: the settlement, and the wait when there is another attempt —
 746            which the settlement does not carry, because whether a wait is spent
 747            here or on a rung queue is this consumer's business and not a
 748            listener's
 749        """
 750        if decision.action is Action.ACCEPT:
 751            return Settlement(OUTCOME_ACKED), None
 752
 753        if decision.action is Action.REJECT:
 754            return (
 755                Settlement(
 756                    OUTCOME_REJECTED,
 757                    reason=f"the handler rejected it: {_describe(decision.error)}",
 758                ),
 759                None,
 760            )
 761
 762        if decision.action is Action.PARK:
 763            # The handler read the message and found nothing it could read. The
 764            # engine reaches the same conclusion for a body its codec refuses,
 765            # and both end up in the same queue for the same reason: a message
 766            # nobody can read is a producer problem, and it is only findable if
 767            # it is not filed with the messages that merely ran out of luck.
 768            return (
 769                Settlement(
 770                    OUTCOME_PARKED,
 771                    reason=f"the handler parked it: {_describe(decision.error)}",
 772                ),
 773                None,
 774            )
 775
 776        if decision.action is Action.IN_PROGRESS:
 777            # Somebody else holds the message. Not a failure of this one, so the
 778            # retry policy is not asked: it can neither spend an attempt nor run
 779            # out and dead-letter a message nobody has failed to handle.
 780            delay = decision.delay if decision.delay is not None else ZERO
 781            return Settlement(OUTCOME_IN_PROGRESS, delay=delay), Wait(delay, in_broker=False)
 782
 783        if isinstance(decision.error, FatalError):
 784            # The handler asked for a retry but marked the reason as one that
 785            # will not change. Honouring the mark rather than the request is the
 786            # point of having it.
 787            return (
 788                Settlement(
 789                    OUTCOME_DEAD_LETTERED,
 790                    reason=(
 791                        "the handler reported an unprocessable message: "
 792                        f"{_describe(decision.error)}"
 793                    ),
 794                ),
 795                None,
 796            )
 797
 798        wait = self._retry.next_wait(envelope.attempt, envelope.age)
 799        if wait is None:
 800            return (
 801                Settlement(
 802                    OUTCOME_DEAD_LETTERED,
 803                    reason=f"{self._exhausted(envelope)}: {_describe(decision.error)}",
 804                ),
 805                None,
 806            )
 807
 808        return Settlement(OUTCOME_RETRIED, delay=wait.delay), wait
 809
 810    async def _carry_out(
 811        self,
 812        delivery: Delivery,
 813        envelope: Envelope,
 814        settlement: Settlement,
 815        wait: Wait | None,
 816    ) -> None:
 817        """Does what :meth:`_decide` decided, and counts it as decided.
 818
 819        Every counter here is chosen from the :class:`~acemq_amqp.ack.Settlement`
 820        and never from the handler's :class:`~acemq_amqp.ack.Ack`, which is the
 821        same rule the delivery's span follows. A handler asking for a retry it
 822        has no attempts left for is dead-lettered, and a counter that read the
 823        request rather than the answer would report a retry for a message
 824        nothing will ever try again — so the dead letters would be undercounted
 825        by exactly the messages an operator most wants to find.
 826
 827        One ``acemq.consume.total`` per delivery, carrying the outcome, and then
 828        whichever of the standalone counters applies. ``retried`` and
 829        ``dead_lettered`` are both an outcome here and a counter of their own,
 830        which is what Java does and is not a redundancy: the outcome says what
 831        was decided about a delivery, and the counter says how many messages are
 832        going round again or have been set aside, which is the number an alert
 833        is written against.
 834        """
 835        self._observer.count(
 836            METRIC_CONSUME_TOTAL, 1, {**self._labels, TAG_OUTCOME: settlement.outcome}
 837        )
 838
 839        if settlement.outcome == OUTCOME_ACKED:
 840            await delivery.ack()
 841            return
 842
 843        if settlement.outcome == OUTCOME_PARKED:
 844            # ``_park`` counts it, exactly as it does for a body that would not
 845            # decode. There is one parked counter and one parked queue whether
 846            # the codec or the handler was the one that could not read it.
 847            await self._park(delivery, envelope, settlement.reason or "")
 848            return
 849
 850        if wait is None:
 851            await self._dead_letter(delivery, envelope, settlement.reason or "")
 852            return
 853
 854        if settlement.outcome == OUTCOME_IN_PROGRESS:
 855            await self._put_back(delivery, envelope, wait.delay)
 856            return
 857
 858        if wait.in_broker and await self._retry_in_broker(delivery, envelope, wait.delay):
 859            return
 860
 861        if wait.delay > ZERO:
 862            # Waiting here holds the delivery, and so holds one of this
 863            # consumer's prefetch slots. For a wait of a few seconds that is the
 864            # cheaper of the two costs; for a longer one it is not, which is why
 865            # the policy has a threshold and the long waits went to a rung queue
 866            # a few lines above.
 867            await asyncio.sleep(wait.delay.total_seconds())
 868
 869        await self._retry_again(delivery, envelope, wait.delay)
 870
 871    def _exhausted(self, envelope: Envelope) -> str:
 872        """Why there is no next attempt, in words an operator can act on."""
 873        if envelope.attempt >= self._retry.max_attempts:
 874            attempts = self._retry.max_attempts
 875            return f"exhausted {attempts} attempt{'' if attempts == 1 else 's'}"
 876        return f"exceeded the maximum message age of {self._retry.max_message_age}"
 877
 878    async def _retry_in_broker(
 879        self, delivery: Delivery, envelope: Envelope, delay: timedelta
 880    ) -> bool:
 881        """Parks the message on ``{queue}.retry.{delay}`` and lets the broker
 882        return it.
 883
 884        The rung's ``x-message-ttl`` is the delay and its dead-letter target is
 885        this queue, so the wait costs nothing here: no delivery held, no prefetch
 886        slot spent, and — the reason it exists — nothing lost when this process
 887        restarts halfway through. A consumer sleeping on a five-minute backoff
 888        that dies at minute one does not resume at minute one; the broker
 889        redelivers the unacknowledged message immediately, and the policy that
 890        said five minutes delivers in none.
 891
 892        The obvious simplification is one rung queue and a per-message
 893        ``expiration`` instead of several with fixed TTLs. It does not work.
 894        RabbitMQ only ever expires the message at the *head* of a queue, so a
 895        message with a ten-minute TTL at the front holds back every thirty-second
 896        one behind it, and the delays that come out bear no resemblance to the
 897        ones that went in. Never set a per-message TTL for this.
 898
 899        The attempt advances here exactly as it does on an immediate retry: the
 900        counter belongs to the message, and a message that has been round the
 901        broker is no less on its second attempt than one that waited here.
 902
 903        :returns: whether the rung took it. ``False`` means no such queue, and
 904            the caller should fall back to waiting here
 905        """
 906        rung = naming.retry_queue(self._queue, delay)
 907        next_attempt = envelope.with_(attempt=envelope.attempt + 1)
 908        try:
 909            routed = await self._republish(delivery, rung, next_attempt)
 910        except _HopRefusedError:
 911            # The rung is there and would not take it. Requeued at once, with
 912            # nothing else on this consumer settled, rather than waited out
 913            # here: the broker is the one saying no, and holding the delivery
 914            # for the whole delay would stall this consumer for nothing.
 915            self._observer.count(METRIC_RETRIED_TOTAL, 1, {**self._labels, "where": "requeued"})
 916            await delivery.nack(True)
 917            return True
 918        if not routed:
 919            # Kept even though a consumer declares its own rungs when it starts,
 920            # because the rung can still be missing: a consumer started with
 921            # ``declare=False`` declares nothing, and a rung deleted under a
 922            # running consumer is gone whoever declared it. What has changed is
 923            # what it means — it used to be an ordinary topology mistake and is
 924            # now either a deliberate opt-out or somebody removing queues.
 925            #
 926            # Degraded rather than fatal: the message is still deliverable, and
 927            # waiting for it here is what this library did before there were
 928            # rungs. Counted as well as logged, because this is the one failure
 929            # here that nothing else shows: the message is still retried and the
 930            # wait still happens, so a dashboard reads as normal while the reason
 931            # the rung exists is quietly gone.
 932            self._observer.count(METRIC_RUNG_MISSING, 1, {**self._labels, "rung": rung})
 933            log.error(
 934                "acemq: %s is not on the broker, so %s will wait %s in this consumer "
 935                "instead; declare it with Topology().queue(%r, retry=policy), or let "
 936                "the consumer declare it by leaving declare=True",
 937                rung,
 938                envelope.id,
 939                delay,
 940                self._queue,
 941            )
 942            return False
 943
 944        self._observer.count(METRIC_RETRIED_TOTAL, 1, {**self._labels, "where": "broker"})
 945        log.info(
 946            "acemq: retrying %s from %s, attempt %d of %d, after %s in %s",
 947            envelope.id,
 948            self._queue,
 949            next_attempt.attempt,
 950            self._retry.max_attempts,
 951            delay,
 952            rung,
 953        )
 954        await delivery.ack()
 955        return True
 956
 957    async def _retry_again(
 958        self, delivery: Delivery, envelope: Envelope, delay: timedelta
 959    ) -> None:
 960        """Puts the message back on its own queue, one attempt further on.
 961
 962        Republished rather than requeued, because a requeue returns the bytes
 963        the broker was given: the attempt header would still read what the
 964        publisher wrote however many times the message had come round, and the
 965        count would live only in this process's memory, which is the one place
 966        it is lost when the process that has been failing restarts.
 967
 968        The cost is that the message goes to the back of the queue rather than
 969        the front, so a retry is no longer in order with its neighbours. For a
 970        message that has already failed once, that is the better trade.
 971        """
 972        next_attempt = envelope.with_(attempt=envelope.attempt + 1)
 973        try:
 974            delivered = await self._republish(delivery, self._queue, next_attempt)
 975        except _HopRefusedError:
 976            delivered = False
 977        if not delivered:
 978            # The queue we are consuming has gone. Requeued rather than
 979            # acknowledged, because dropping it here would lose a message over a
 980            # broker change nobody told this consumer about.
 981            #
 982            # Counted as a retry all the same, and it is one: the settlement
 983            # said ``retried``, the span says ``retried``, and the broker is
 984            # about to hand the message back. Leaving it uncounted was the one
 985            # place where the counters and the span disagreed about the same
 986            # delivery. The label says which path it took, because a requeue
 987            # keeps the attempt header it arrived with and the other two do not.
 988            self._observer.count(METRIC_RETRIED_TOTAL, 1, {**self._labels, "where": "requeued"})
 989            log.error(
 990                "acemq: cannot republish %s onto %s for attempt %d; returning it to the broker",
 991                envelope.id,
 992                self._queue,
 993                next_attempt.attempt,
 994            )
 995            await delivery.nack(True)
 996            return
 997
 998        self._observer.count(METRIC_RETRIED_TOTAL, 1, {**self._labels, "where": "consumer"})
 999        log.info(
1000            "acemq: retrying %s from %s, attempt %d of %d, after %s",
1001            envelope.id,
1002            self._queue,
1003            next_attempt.attempt,
1004            self._retry.max_attempts,
1005            delay,
1006        )
1007        await delivery.ack()
1008
1009    async def _put_back(
1010        self, delivery: Delivery, envelope: Envelope, delay: timedelta
1011    ) -> None:
1012        """Returns a message somebody else is working on, attempt unchanged.
1013
1014        :meth:`_retry_again` without the increment: the attempt counts this
1015        message's failures, and a claim held elsewhere is not one of them.
1016        """
1017        if delay > ZERO:
1018            await asyncio.sleep(delay.total_seconds())
1019        try:
1020            delivered = await self._republish(delivery, self._queue, envelope)
1021        except _HopRefusedError:
1022            delivered = False
1023        if delivered:
1024            await delivery.ack()
1025            return
1026        log.error(
1027            "acemq: cannot put %s back onto %s; returning it to the broker",
1028            envelope.id,
1029            self._queue,
1030        )
1031        await delivery.nack(True)
1032
1033    async def _park(self, delivery: Delivery, envelope: Envelope, reason: str) -> None:
1034        """Sends the message to ``{queue}.parked`` with the reason attached.
1035
1036        Where a message goes when it never reached the handler at all. Somebody
1037        has to look at it, and what they need to know first is that it was
1038        unreadable rather than unlucky.
1039        """
1040        # The same counter a dead-lettering moves, separated by the outcome
1041        # rather than by a metric of its own — which is what Java settled on, and
1042        # what makes "how much is this queue giving up on" one number that can
1043        # then be split into the two problems it is made of.
1044        self._observer.count(
1045            METRIC_DEAD_LETTERED_TOTAL, 1, {**self._labels, TAG_OUTCOME: OUTCOME_PARKED}
1046        )
1047        await self._set_aside(delivery, envelope, naming.parked_queue(self._queue), reason)
1048
1049    async def _dead_letter(self, delivery: Delivery, envelope: Envelope, reason: str) -> None:
1050        """Sends the message to ``{queue}.dlq`` with the reason attached, and
1051        acknowledges the original.
1052
1053        Acknowledging a message that failed looks wrong and is what makes this
1054        reliable: the message has already been safely republished somewhere
1055        else, so acknowledging the original is removing the copy that has been
1056        dealt with. Rejecting it instead would either requeue it into a hot loop
1057        or, with a dead-letter exchange configured on the queue, send it
1058        somewhere this consumer did not choose and without the reason.
1059
1060        The reason travels as an envelope field, so a consumer of the
1061        dead-letter queue reads it back through the API rather than having to
1062        know the wire header name.
1063        """
1064        self._observer.count(
1065            METRIC_DEAD_LETTERED_TOTAL,
1066            1,
1067            {**self._labels, TAG_OUTCOME: OUTCOME_DEAD_LETTERED},
1068        )
1069        await self._set_aside(
1070            delivery, envelope, naming.dead_letter_queue(self._queue), reason
1071        )
1072
1073    async def _set_aside(
1074        self, delivery: Delivery, envelope: Envelope, target: str, reason: str
1075    ) -> None:
1076        """Republishes to ``target`` with the reason, then acknowledges."""
1077        failed = envelope.with_(error=reason)
1078        try:
1079            delivered = await self._republish(delivery, target, failed)
1080        except _HopRefusedError:
1081            delivered = False
1082        if not delivered:
1083            # Rejected rather than acknowledged: without a dead-letter queue to
1084            # put it in, the broker's own dead-lettering is the last thing left
1085            # between this message and nothing.
1086            self._observer.count(
1087                METRIC_SET_ASIDE_FAILED, 1, {**self._labels, TAG_TARGET: target}
1088            )
1089            log.error(
1090                "acemq: cannot move %s to %s (%s); rejecting it to the broker instead",
1091                envelope.id,
1092                target,
1093                reason,
1094            )
1095            await delivery.nack(False)
1096            return
1097
1098        log.warning(
1099            "acemq: set aside %s from %s after %d attempts, into %s: %s",
1100            envelope.id,
1101            self._queue,
1102            envelope.attempt,
1103            target,
1104            reason,
1105        )
1106        await delivery.ack()
1107
1108    async def _republish(self, delivery: Delivery, queue: str, envelope: Envelope) -> bool:
1109        """Sends the original bytes to a queue by name, and says whether they
1110        arrived.
1111
1112        Through the default exchange, which routes to the queue whose name
1113        matches the routing key, and mandatory so that a queue that is not there
1114        is an answer rather than a silence. The body goes back exactly as it
1115        came: re-encoding through a class that has since changed would replace
1116        what was committed with something else.
1117        """
1118        try:
1119            result = await self._connection.publish_raw(
1120                "",
1121                queue,
1122                Outbound(
1123                    body=delivery.body,
1124                    content_type=delivery.content_type or "",
1125                    message_id=envelope.id,
1126                    headers=envelope.to_headers(routing_key=delivery.routing_key),
1127                    persistent=True,
1128                    mandatory=True,
1129                ),
1130            )
1131        except PublishError as refused:
1132            # The broker nacked it, or would not take it while blocked. Raised
1133            # as the caller's to settle, because what to do differs: a refused
1134            # rung means requeue now, where a missing rung means wait here.
1135            # Escaping further left the delivery unsettled for the life of the
1136            # channel, which is what Ruby did before 0.7.12.
1137            log.error("acemq: the broker refused %s onto %s: %s", envelope.id, queue, refused)
1138            raise _HopRefusedError(str(refused)) from refused
1139        return result.routed
1140
1141    async def close(self, *, timeout: float | None = DEFAULT_DRAIN_TIMEOUT) -> bool:
1142        """Stops the consumer and waits, for up to ``timeout``, for the handlers
1143        already running.
1144
1145        A message being worked on when this is called is finished and settled,
1146        rather than abandoned for the broker to hand to somebody else. A message
1147        that had been delivered but not started is given back instead: it is the
1148        broker's to hand to another consumer, and holding it here only to work
1149        through a retry delay would make closing take as long as the schedule.
1150
1151        A handler still running when the time is up is cancelled, and its message
1152        is left unsettled — neither acknowledged nor rejected — so the broker
1153        hands it out again once the channel goes. Shutdown is never the reason a
1154        message is lost or dead-lettered. The same happens when the caller
1155        cancels closing, ``asyncio.wait_for`` included, and that cancellation is
1156        raised again once the consumer has been cleaned up.
1157
1158        The subscription is released last, after everything has been settled,
1159        because a settlement travels on the channel its delivery arrived on.
1160
1161        :param timeout: seconds to wait for every running handler together;
1162            ``None`` waits for as long as they take
1163        :returns: ``True`` if every handler finished, ``False`` if the time ran
1164            out first and some were cut off — the consumer is stopped either way
1165        """
1166        if self._closed:
1167            return True
1168        self._closed = True
1169
1170        running: set[asyncio.Task[None]] = set()
1171        try:
1172            if self._subscription is not None:
1173                await self._subscription.stop()
1174
1175            while True:
1176                try:
1177                    pending = self._work.get_nowait()
1178                except asyncio.QueueEmpty:
1179                    break
1180                if pending is not None:
1181                    await pending.nack(True)
1182
1183            for _ in self._workers:
1184                self._work.put_nowait(None)
1185            if self._workers:
1186                _, running = await asyncio.wait(self._workers, timeout=timeout)
1187        finally:
1188            # Reached with workers still running for two reasons: the deadline
1189            # passed, or the caller cancelled us. Either way they are cancelled
1190            # before the channel goes, so nothing settles a message on a channel
1191            # that is closing — and a cancellation that came from the caller is
1192            # not caught here, so it carries on out once this has run.
1193            stragglers = [worker for worker in self._workers if not worker.done()]
1194            for worker in stragglers:
1195                worker.cancel()
1196            if stragglers:
1197                await asyncio.wait(stragglers)
1198            if self._subscription is not None:
1199                await self._subscription.close()
1200            self._connection._untrack(self)
1201
1202        for worker in self._workers:
1203            # A worker that was already cancelled — a task group unwinding, a
1204            # shutdown that cancelled the loop's tasks before closing the
1205            # connection — is not news to report to a caller who is merely
1206            # closing. One that died of its own accord is.
1207            failure = None if worker.cancelled() else worker.exception()
1208            if failure is not None:
1209                raise failure
1210        return not running
1211
1212    async def __aenter__(self) -> Consumer:
1213        return self
1214
1215    async def __aexit__(self, *_: object) -> None:
1216        # Not returned: a true value from here tells the interpreter that the
1217        # exception leaving the block was handled.
1218        await self.close()

A running subscription. Close it to stop.

One is built by Connection.consume() rather than directly, because it has to be started before it is any use and a half-built consumer that looks finished is a thing somebody will hold on to.

Starting one declares the queues it will need when a message fails — its dead-letter queue, its parking lot and its rungs — unless it was told not to. See acemq_amqp.topology.declare_where_failures_go().

Consumer( connection: Connection, queue: str, handler: Callable[Message, Ack | Awaitable[Ack]], *, codec: Codec, retry: RetryPolicy, concurrency: int, declare: bool = True)
463    def __init__(
464        self,
465        connection: Connection,
466        queue: str,
467        handler: Handler,
468        *,
469        codec: Codec,
470        retry: RetryPolicy,
471        concurrency: int,
472        declare: bool = True,
473    ) -> None:
474        if concurrency < 1:
475            raise ValueError(f"acemq: concurrency must be at least 1, got {concurrency}")
476        self._connection = connection
477        self._queue = queue
478        self._handler = handler
479        self._codec = codec
480        self._retry = retry
481        self._concurrency = concurrency
482        self._declare = declare
483        self._work: asyncio.Queue[Delivery | None] = asyncio.Queue()
484        self._workers: list[asyncio.Task[None]] = []
485        self._subscription: Subscription | None = None
486        self._in_flight = 0
487        self._closed = False
queue: str
489    @property
490    def queue(self) -> str:
491        """The queue this consumer reads."""
492        return self._queue

The queue this consumer reads.

closed: bool
494    @property
495    def closed(self) -> bool:
496        """Whether this consumer has been stopped."""
497        return self._closed

Whether this consumer has been stopped.

running: bool
499    @property
500    def running(self) -> bool:
501        """Whether this consumer is subscribed and still has workers to hand a
502        delivery to.
503
504        What a health check reads. A consumer whose workers have all finished
505        without it being closed is a consumer the broker is still sending
506        messages to and nothing is reading — which looks identical to a quiet
507        queue from outside, and is the failure worth telling a probe about.
508        """
509        return (
510            not self._closed
511            and self._subscription is not None
512            and any(not worker.done() for worker in self._workers)
513        )

Whether this consumer is subscribed and still has workers to hand a delivery to.

What a health check reads. A consumer whose workers have all finished without it being closed is a consumer the broker is still sending messages to and nothing is reading — which looks identical to a quiet queue from outside, and is the failure worth telling a probe about.

in_flight: int
515    @property
516    def in_flight(self) -> int:
517        """How many messages this consumer is working on right now."""
518        return self._in_flight

How many messages this consumer is working on right now.

async def close(self, *, timeout: float | None = 20.0) -> bool:
1141    async def close(self, *, timeout: float | None = DEFAULT_DRAIN_TIMEOUT) -> bool:
1142        """Stops the consumer and waits, for up to ``timeout``, for the handlers
1143        already running.
1144
1145        A message being worked on when this is called is finished and settled,
1146        rather than abandoned for the broker to hand to somebody else. A message
1147        that had been delivered but not started is given back instead: it is the
1148        broker's to hand to another consumer, and holding it here only to work
1149        through a retry delay would make closing take as long as the schedule.
1150
1151        A handler still running when the time is up is cancelled, and its message
1152        is left unsettled — neither acknowledged nor rejected — so the broker
1153        hands it out again once the channel goes. Shutdown is never the reason a
1154        message is lost or dead-lettered. The same happens when the caller
1155        cancels closing, ``asyncio.wait_for`` included, and that cancellation is
1156        raised again once the consumer has been cleaned up.
1157
1158        The subscription is released last, after everything has been settled,
1159        because a settlement travels on the channel its delivery arrived on.
1160
1161        :param timeout: seconds to wait for every running handler together;
1162            ``None`` waits for as long as they take
1163        :returns: ``True`` if every handler finished, ``False`` if the time ran
1164            out first and some were cut off — the consumer is stopped either way
1165        """
1166        if self._closed:
1167            return True
1168        self._closed = True
1169
1170        running: set[asyncio.Task[None]] = set()
1171        try:
1172            if self._subscription is not None:
1173                await self._subscription.stop()
1174
1175            while True:
1176                try:
1177                    pending = self._work.get_nowait()
1178                except asyncio.QueueEmpty:
1179                    break
1180                if pending is not None:
1181                    await pending.nack(True)
1182
1183            for _ in self._workers:
1184                self._work.put_nowait(None)
1185            if self._workers:
1186                _, running = await asyncio.wait(self._workers, timeout=timeout)
1187        finally:
1188            # Reached with workers still running for two reasons: the deadline
1189            # passed, or the caller cancelled us. Either way they are cancelled
1190            # before the channel goes, so nothing settles a message on a channel
1191            # that is closing — and a cancellation that came from the caller is
1192            # not caught here, so it carries on out once this has run.
1193            stragglers = [worker for worker in self._workers if not worker.done()]
1194            for worker in stragglers:
1195                worker.cancel()
1196            if stragglers:
1197                await asyncio.wait(stragglers)
1198            if self._subscription is not None:
1199                await self._subscription.close()
1200            self._connection._untrack(self)
1201
1202        for worker in self._workers:
1203            # A worker that was already cancelled — a task group unwinding, a
1204            # shutdown that cancelled the loop's tasks before closing the
1205            # connection — is not news to report to a caller who is merely
1206            # closing. One that died of its own accord is.
1207            failure = None if worker.cancelled() else worker.exception()
1208            if failure is not None:
1209                raise failure
1210        return not running

Stops the consumer and waits, for up to timeout, for the handlers already running.

A message being worked on when this is called is finished and settled, rather than abandoned for the broker to hand to somebody else. A message that had been delivered but not started is given back instead: it is the broker's to hand to another consumer, and holding it here only to work through a retry delay would make closing take as long as the schedule.

A handler still running when the time is up is cancelled, and its message is left unsettled — neither acknowledged nor rejected — so the broker hands it out again once the channel goes. Shutdown is never the reason a message is lost or dead-lettered. The same happens when the caller cancels closing, asyncio.wait_for included, and that cancellation is raised again once the consumer has been cleaned up.

The subscription is released last, after everything has been settled, because a settlement travels on the channel its delivery arrived on.

Parameters
  • timeout: seconds to wait for every running handler together; None waits for as long as they take :returns: True if every handler finished, False if the time ran out first and some were cut off — the consumer is stopped either way
@dataclass(frozen=True, slots=True, repr=False)
class Credentials:
151@dataclass(frozen=True, slots=True, repr=False)
152class Credentials:
153    """A username and a secret for the broker.
154
155    The secret never appears in :func:`repr` or :func:`str`, so a credential
156    logged by accident — in a dataclass dump, in an exception, in a debugger's
157    variables pane — prints the username and nothing else. That is the whole
158    reason this is a type rather than two strings, and a test pins it.
159
160    :param username: the account name
161    :param secret: the password, or the token for a broker that takes one
162    """
163
164    username: str = ""
165    secret: str = ""
166
167    @property
168    def empty(self) -> bool:
169        """Whether these credentials carry nothing at all."""
170        return not self.username and not self.secret
171
172    def __repr__(self) -> str:
173        if not self.username:
174            return "Credentials(<secret>)"
175        return f"Credentials(username={self.username!r}, secret=<secret>)"
176
177    __str__ = __repr__

A username and a secret for the broker.

The secret never appears in repr() or str(), so a credential logged by accident — in a dataclass dump, in an exception, in a debugger's variables pane — prints the username and nothing else. That is the whole reason this is a type rather than two strings, and a test pins it.

Parameters
  • username: the account name
  • secret: the password, or the token for a broker that takes one
Credentials(username: str = '', secret: str = '')
username: str
secret: str
empty: bool
167    @property
168    def empty(self) -> bool:
169        """Whether these credentials carry nothing at all."""
170        return not self.username and not self.secret

Whether these credentials carry nothing at all.

CredentialsSource = collections.abc.Callable[[], 'Credentials']
@dataclass(frozen=True, slots=True)
class Delivery:
134@dataclass(frozen=True, slots=True)
135class Delivery:
136    """A message as it arrived, before any codec has looked at it.
137
138    :param body: the bytes, undecoded
139    :param content_type: what the sender said the body is, or ``None`` when the
140        sender said nothing — which is a different thing from an empty string
141        and is why the codecs distinguish them
142    :param routing_key: the key it arrived under
143    :param message_id: the AMQP message id
144    :param headers: everything the sender wrote, reserved names included
145    :param redelivered: the broker saying it has handed this message over
146        before. It is the broker's own view and says nothing about which attempt
147        this is: a requeue returns the bytes the broker was given, so the
148        attempt header still reads whatever the publisher wrote
149    :param ack: confirms the message and removes it from the queue
150    :param nack: returns the message, requeued or not
151    :param reply_to: AMQP's own ``reply-to`` property, empty when the sender did
152        not set one. A transport that cannot report it leaves it empty and the
153        responder falls back to the header, which is the only reason this has a
154        default at all
155    """
156
157    body: bytes
158    content_type: str | None
159    routing_key: str
160    message_id: str
161    headers: Mapping[str, Any]
162    redelivered: bool
163    ack: Callable[[], Awaitable[None]]
164    nack: Callable[[bool], Awaitable[None]]
165    reply_to: str = ""

A message as it arrived, before any codec has looked at it.

Parameters
  • body: the bytes, undecoded
  • content_type: what the sender said the body is, or None when the sender said nothing — which is a different thing from an empty string and is why the codecs distinguish them
  • routing_key: the key it arrived under
  • message_id: the AMQP message id
  • headers: everything the sender wrote, reserved names included
  • redelivered: the broker saying it has handed this message over before. It is the broker's own view and says nothing about which attempt this is: a requeue returns the bytes the broker was given, so the attempt header still reads whatever the publisher wrote
  • ack: confirms the message and removes it from the queue
  • nack: returns the message, requeued or not
  • reply_to: AMQP's own reply-to property, empty when the sender did not set one. A transport that cannot report it leaves it empty and the responder falls back to the header, which is the only reason this has a default at all
Delivery( body: bytes, content_type: str | None, routing_key: str, message_id: str, headers: Mapping[str, typing.Any], redelivered: bool, ack: Callable[[], Awaitable[None]], nack: Callable[[bool], Awaitable[None]], reply_to: str = '')
body: bytes
content_type: str | None
routing_key: str
message_id: str
headers: Mapping[str, typing.Any]
redelivered: bool
ack: Callable[[], Awaitable[None]]
nack: Callable[[bool], Awaitable[None]]
reply_to: str
@dataclass(frozen=True, slots=True)
class DurationSummary:
350@dataclass(frozen=True, slots=True)
351class DurationSummary:
352    """What a :class:`Metrics` knows about a timing.
353
354    Deliberately not percentiles. Computing those needs either every sample kept
355    or a sketch, and a library that quietly did either would be making a
356    decision about memory that belongs to the application.
357    """
358
359    count: int = 0
360    total: float = 0.0
361    fastest: float = 0.0
362    slowest: float = 0.0
363
364    @property
365    def mean(self) -> float:
366        """The average, or zero when nothing has been recorded."""
367        return self.total / self.count if self.count else 0.0

What a Metrics knows about a timing.

Deliberately not percentiles. Computing those needs either every sample kept or a sketch, and a library that quietly did either would be making a decision about memory that belongs to the application.

DurationSummary( count: int = 0, total: float = 0.0, fastest: float = 0.0, slowest: float = 0.0)
count: int
total: float
fastest: float
slowest: float
mean: float
364    @property
365    def mean(self) -> float:
366        """The average, or zero when nothing has been recorded."""
367        return self.total / self.count if self.count else 0.0

The average, or zero when nothing has been recorded.

@dataclass(frozen=True, slots=True)
class Envelope:
 45@dataclass(frozen=True, slots=True)
 46class Envelope:
 47    """The metadata AceMQ carries with every message.
 48
 49    Frozen, because an envelope describes a message that has already been
 50    published or received: changing one in place would change what a log line
 51    said after it was written. :meth:`with_` returns a modified copy for the
 52    cases where a new message derives from an old one.
 53
 54    Defaults are applied when the envelope is built rather than when it is
 55    read, so two libraries reading the same message agree without having to
 56    agree on a second set of rules. ``correlation_id`` defaults to ``id``, an
 57    attempt starts at 1, a version starts at 1, and ``type`` falls back to the
 58    routing key the message was published under.
 59    """
 60
 61    id: str = field(default_factory=lambda: str(uuid.uuid4()))
 62    type: str = ""
 63    version: int = 1
 64    correlation_id: str = ""
 65    causation_id: str = ""
 66    attempt: int = 1
 67    first_seen: datetime = field(default_factory=_now)
 68    origin: str = ""
 69    error: str = ""
 70    claim: str = ""
 71
 72    #: The steps of a declared route, by name, comma-separated and in order —
 73    #: the form Java's ``Pipeline`` writes. Empty for a message that is not
 74    #: travelling one.
 75    #:
 76    #: A field rather than an application header for the same reason ``claim``
 77    #: is one: the name is reserved, so the application map refuses it, and a
 78    #: pattern that needs it off a delivery has nowhere else to read it from.
 79    #: Carried by :meth:`with_`, so a retry, a dead-letter and a replay all keep
 80    #: it, which is what makes replaying a message *resume* a route rather than
 81    #: restart it. :mod:`acemq_amqp.patterns.routingslip` is what reads it.
 82    route: str = ""
 83
 84    #: Which step of :attr:`route` this message is for, counting from zero.
 85    route_position: int = 0
 86
 87    #: One run through a route, carried by every hop of it.
 88    route_id: str = ""
 89
 90    #: The application's own headers. Never contains a reserved name.
 91    headers: Mapping[str, Any] = field(default_factory=dict)
 92
 93    def __post_init__(self) -> None:
 94        if not self.correlation_id:
 95            object.__setattr__(self, "correlation_id", self.id)
 96        if self.attempt < 1:
 97            object.__setattr__(self, "attempt", 1)
 98        if self.version < 1:
 99            object.__setattr__(self, "version", 1)
100        if self.route_position < 0:
101            # A position before the start would send the message to an
102            # arbitrary step. Starting over is the only defensible reading, and
103            # it is the one Java takes for a position it cannot parse.
104            object.__setattr__(self, "route_position", 0)
105        # Reserved names in the application's map would be written twice and
106        # read back inconsistently, so they are refused rather than dropped:
107        # silently discarding a header somebody set is worse than saying no.
108        offending = sorted(name for name in self.headers if headers.is_reserved(name))
109        if offending:
110            raise ValueError(
111                "these header names belong to AceMQ and cannot be set by hand: "
112                + ", ".join(offending)
113            )
114
115    @property
116    def age(self) -> timedelta:
117        """How long since the message was first published.
118
119        The basis for giving up on age rather than on attempts, which is the
120        honest limit when a queue has been paused: a message can be on attempt
121        two and four days old.
122        """
123        return _now() - self.first_seen
124
125    def with_(self, **changes: Any) -> Envelope:
126        """A copy with fields changed.
127
128        :param changes: the fields to replace
129        :returns: a new envelope
130        """
131        return replace(self, **changes)
132
133    def to_headers(self, routing_key: str = "") -> dict[str, Any]:
134        """The AMQP headers for this envelope.
135
136        :param routing_key: what the message is published under, used when no
137            type was given; when that is empty too, the type is the literal
138            ``message`` (AceMQ TCK requirement AMQP-ENV-006), rather than an
139            empty string no consumer can dispatch on
140        :returns: the reserved headers, plus the application's own
141        """
142        written: dict[str, Any] = {
143            headers.ID: self.id,
144            headers.TYPE: self.type or routing_key or DEFAULT_TYPE,
145            headers.VERSION: self.version,
146            headers.CORRELATION: self.correlation_id or self.id,
147            headers.ATTEMPT: self.attempt,
148            headers.FIRST_SEEN: _millis(self.first_seen),
149        }
150        # Absent rather than empty. A header carrying "" is a header somebody
151        # has to write a special case for at the other end.
152        for name, value in (
153            (headers.CAUSATION, self.causation_id),
154            (headers.ORIGIN, self.origin),
155            (headers.ERROR, self.error),
156            (headers.CLAIM, self.claim),
157        ):
158            if value:
159                written[name] = value
160
161        # The position is written whenever there is a route, including the zero
162        # a first hop carries: a route with no position reads as position zero
163        # everywhere, and leaving it off would make the first hop the only one
164        # whose slip is incomplete.
165        if self.route:
166            written[headers.ROUTE] = self.route
167            written[headers.ROUTE_POSITION] = self.route_position
168            if self.route_id:
169                written[headers.ROUTE_ID] = self.route_id
170
171        written.update(self.headers)
172        return written
173
174    @classmethod
175    def from_headers(cls, raw: Mapping[str, Any] | None, routing_key: str = "") -> Envelope:
176        """Reads an envelope off a delivery.
177
178        Anything missing takes its default, and anything unreadable takes its
179        default too: a message from a producer that wrote ``x-acemq-attempt``
180        as a string is still a message, and refusing to deliver it would hand
181        the application an outage rather than a message.
182
183        :param raw: the delivery's headers
184        :param routing_key: what it arrived on, used when no type was set
185        :returns: the envelope
186        """
187        raw = raw or {}
188        application = {
189            name: value for name, value in raw.items() if not headers.is_reserved(name)
190        }
191
192        identifier = _text(raw.get(headers.ID)) or str(uuid.uuid4())
193        first_seen = raw.get(headers.FIRST_SEEN)
194
195        return cls(
196            id=identifier,
197            type=_text(raw.get(headers.TYPE)) or routing_key or DEFAULT_TYPE,
198            version=_number(raw.get(headers.VERSION), 1),
199            correlation_id=_text(raw.get(headers.CORRELATION)) or identifier,
200            causation_id=_text(raw.get(headers.CAUSATION)),
201            attempt=_number(raw.get(headers.ATTEMPT), 1),
202            first_seen=_from_millis(_number(first_seen, _millis(_now()))),
203            origin=_text(raw.get(headers.ORIGIN)),
204            error=_text(raw.get(headers.ERROR)),
205            claim=_text(raw.get(headers.CLAIM)),
206            route=_text(raw.get(headers.ROUTE)),
207            # A position that will not read is taken as the start rather than
208            # dropped: a route whose position is unreadable would otherwise send
209            # the message to whichever step happened to be first in the list.
210            route_position=_number(raw.get(headers.ROUTE_POSITION), 0),
211            route_id=_text(raw.get(headers.ROUTE_ID)),
212            headers=application,
213        )

The metadata AceMQ carries with every message.

Frozen, because an envelope describes a message that has already been published or received: changing one in place would change what a log line said after it was written. with_() returns a modified copy for the cases where a new message derives from an old one.

Defaults are applied when the envelope is built rather than when it is read, so two libraries reading the same message agree without having to agree on a second set of rules. correlation_id defaults to id, an attempt starts at 1, a version starts at 1, and type falls back to the routing key the message was published under.

Envelope( id: str = <factory>, type: str = '', version: int = 1, correlation_id: str = '', causation_id: str = '', attempt: int = 1, first_seen: datetime.datetime = <factory>, origin: str = '', error: str = '', claim: str = '', route: str = '', route_position: int = 0, route_id: str = '', headers: Mapping[str, typing.Any] = <factory>)
id: str
type: str
version: int
correlation_id: str
causation_id: str
attempt: int
first_seen: datetime.datetime
origin: str
error: str
claim: str
route: str
route_position: int
route_id: str
headers: Mapping[str, typing.Any]
age: datetime.timedelta
115    @property
116    def age(self) -> timedelta:
117        """How long since the message was first published.
118
119        The basis for giving up on age rather than on attempts, which is the
120        honest limit when a queue has been paused: a message can be on attempt
121        two and four days old.
122        """
123        return _now() - self.first_seen

How long since the message was first published.

The basis for giving up on age rather than on attempts, which is the honest limit when a queue has been paused: a message can be on attempt two and four days old.

def with_(self, **changes: Any) -> Envelope:
125    def with_(self, **changes: Any) -> Envelope:
126        """A copy with fields changed.
127
128        :param changes: the fields to replace
129        :returns: a new envelope
130        """
131        return replace(self, **changes)

A copy with fields changed.

Parameters
  • changes: the fields to replace :returns: a new envelope
def to_headers(self, routing_key: str = '') -> dict[str, typing.Any]:
133    def to_headers(self, routing_key: str = "") -> dict[str, Any]:
134        """The AMQP headers for this envelope.
135
136        :param routing_key: what the message is published under, used when no
137            type was given; when that is empty too, the type is the literal
138            ``message`` (AceMQ TCK requirement AMQP-ENV-006), rather than an
139            empty string no consumer can dispatch on
140        :returns: the reserved headers, plus the application's own
141        """
142        written: dict[str, Any] = {
143            headers.ID: self.id,
144            headers.TYPE: self.type or routing_key or DEFAULT_TYPE,
145            headers.VERSION: self.version,
146            headers.CORRELATION: self.correlation_id or self.id,
147            headers.ATTEMPT: self.attempt,
148            headers.FIRST_SEEN: _millis(self.first_seen),
149        }
150        # Absent rather than empty. A header carrying "" is a header somebody
151        # has to write a special case for at the other end.
152        for name, value in (
153            (headers.CAUSATION, self.causation_id),
154            (headers.ORIGIN, self.origin),
155            (headers.ERROR, self.error),
156            (headers.CLAIM, self.claim),
157        ):
158            if value:
159                written[name] = value
160
161        # The position is written whenever there is a route, including the zero
162        # a first hop carries: a route with no position reads as position zero
163        # everywhere, and leaving it off would make the first hop the only one
164        # whose slip is incomplete.
165        if self.route:
166            written[headers.ROUTE] = self.route
167            written[headers.ROUTE_POSITION] = self.route_position
168            if self.route_id:
169                written[headers.ROUTE_ID] = self.route_id
170
171        written.update(self.headers)
172        return written

The AMQP headers for this envelope.

Parameters
  • routing_key: what the message is published under, used when no type was given; when that is empty too, the type is the literal message (AceMQ TCK requirement AMQP-ENV-006), rather than an empty string no consumer can dispatch on :returns: the reserved headers, plus the application's own
@classmethod
def from_headers( cls, raw: Mapping[str, typing.Any] | None, routing_key: str = '') -> Envelope:
174    @classmethod
175    def from_headers(cls, raw: Mapping[str, Any] | None, routing_key: str = "") -> Envelope:
176        """Reads an envelope off a delivery.
177
178        Anything missing takes its default, and anything unreadable takes its
179        default too: a message from a producer that wrote ``x-acemq-attempt``
180        as a string is still a message, and refusing to deliver it would hand
181        the application an outage rather than a message.
182
183        :param raw: the delivery's headers
184        :param routing_key: what it arrived on, used when no type was set
185        :returns: the envelope
186        """
187        raw = raw or {}
188        application = {
189            name: value for name, value in raw.items() if not headers.is_reserved(name)
190        }
191
192        identifier = _text(raw.get(headers.ID)) or str(uuid.uuid4())
193        first_seen = raw.get(headers.FIRST_SEEN)
194
195        return cls(
196            id=identifier,
197            type=_text(raw.get(headers.TYPE)) or routing_key or DEFAULT_TYPE,
198            version=_number(raw.get(headers.VERSION), 1),
199            correlation_id=_text(raw.get(headers.CORRELATION)) or identifier,
200            causation_id=_text(raw.get(headers.CAUSATION)),
201            attempt=_number(raw.get(headers.ATTEMPT), 1),
202            first_seen=_from_millis(_number(first_seen, _millis(_now()))),
203            origin=_text(raw.get(headers.ORIGIN)),
204            error=_text(raw.get(headers.ERROR)),
205            claim=_text(raw.get(headers.CLAIM)),
206            route=_text(raw.get(headers.ROUTE)),
207            # A position that will not read is taken as the start rather than
208            # dropped: a route whose position is unreadable would otherwise send
209            # the message to whichever step happened to be first in the list.
210            route_position=_number(raw.get(headers.ROUTE_POSITION), 0),
211            route_id=_text(raw.get(headers.ROUTE_ID)),
212            headers=application,
213        )

Reads an envelope off a delivery.

Anything missing takes its default, and anything unreadable takes its default too: a message from a producer that wrote x-acemq-attempt as a string is still a message, and refusing to deliver it would hand the application an outage rather than a message.

Parameters
  • raw: the delivery's headers
  • routing_key: what it arrived on, used when no type was set :returns: the envelope
@dataclass(frozen=True, slots=True)
class ExchangeSpec:
52@dataclass(frozen=True, slots=True)
53class ExchangeSpec:
54    """How an exchange should be declared.
55
56    :param kind: ``direct``, ``topic``, ``fanout`` or ``headers``
57    :param durable: survives a broker restart
58    :param auto_delete: goes away when its last binding does
59    :param args: broker-specific arguments
60    """
61
62    kind: str = "topic"
63    durable: bool = True
64    auto_delete: bool = False
65    args: Mapping[str, Any] = field(default_factory=dict)

How an exchange should be declared.

Parameters
  • kind: direct, topic, fanout or headers
  • durable: survives a broker restart
  • auto_delete: goes away when its last binding does
  • args: broker-specific arguments
ExchangeSpec( kind: str = 'topic', durable: bool = True, auto_delete: bool = False, args: Mapping[str, typing.Any] = <factory>)
kind: str
durable: bool
auto_delete: bool
args: Mapping[str, typing.Any]
class FatalError(builtins.Exception):
199class FatalError(Exception):
200    """An error no number of retries will fix.
201
202    Wrapping a cause in this says "stop now" without the handler having to know
203    how many attempts remain, which is knowledge handlers should not need.
204
205    The other languages call this ``Fatal``; the name is Python's because a
206    class name is ergonomics rather than wire contract, and Python expects an
207    exception to say that it is one.
208    """

An error no number of retries will fix.

Wrapping a cause in this says "stop now" without the handler having to know how many attempts remain, which is knowledge handlers should not need.

The other languages call this Fatal; the name is Python's because a class name is ergonomics rather than wire contract, and Python expects an exception to say that it is one.

Handler = collections.abc.Callable[[Message], 'Ack | Awaitable[Ack]']
@runtime_checkable
class HealthCheck(typing.Protocol):
583@runtime_checkable
584class HealthCheck(Protocol):
585    """Anything that can report on itself.
586
587    The library provides one for the connection; an application adds its own — a
588    database, a downstream service — and :func:`aggregate_health` combines them.
589    """
590
591    @property
592    def name(self) -> str:
593        """What this check is called in a combined report."""
594
595    async def check(self) -> HealthReport:
596        """The current state.
597
598        It must return quickly and must not raise: a check that hangs makes a
599        readiness probe hang with it, and a probe that hangs is a pod that never
600        comes back.
601        """

Anything that can report on itself.

The library provides one for the connection; an application adds its own — a database, a downstream service — and aggregate_health() combines them.

HealthCheck(*args, **kwargs)
1771def _no_init_or_replace_init(self, *args, **kwargs):
1772    cls = type(self)
1773
1774    if cls._is_protocol:
1775        raise TypeError('Protocols cannot be instantiated')
1776
1777    # Already using a custom `__init__`. No need to calculate correct
1778    # `__init__` to call. This can lead to RecursionError. See bpo-45121.
1779    if cls.__init__ is not _no_init_or_replace_init:
1780        return
1781
1782    # Initially, `__init__` of a protocol subclass is set to `_no_init_or_replace_init`.
1783    # The first instantiation of the subclass will call `_no_init_or_replace_init` which
1784    # searches for a proper new `__init__` in the MRO. The new `__init__`
1785    # replaces the subclass' old `__init__` (ie `_no_init_or_replace_init`). Subsequent
1786    # instantiation of the protocol subclass will thus use the new
1787    # `__init__` and no longer call `_no_init_or_replace_init`.
1788    for base in cls.__mro__:
1789        init = base.__dict__.get('__init__', _no_init_or_replace_init)
1790        if init is not _no_init_or_replace_init:
1791            cls.__init__ = init
1792            break
1793    else:
1794        # should not happen
1795        cls.__init__ = object.__init__
1796
1797    cls.__init__(self, *args, **kwargs)
name: str
591    @property
592    def name(self) -> str:
593        """What this check is called in a combined report."""

What this check is called in a combined report.

async def check(self) -> HealthReport:
595    async def check(self) -> HealthReport:
596        """The current state.
597
598        It must return quickly and must not raise: a check that hangs makes a
599        readiness probe hang with it, and a probe that hangs is a pod that never
600        comes back.
601        """

The current state.

It must return quickly and must not raise: a check that hangs makes a readiness probe hang with it, and a probe that hangs is a pod that never comes back.

@dataclass(frozen=True, slots=True)
class HealthReport:
553@dataclass(frozen=True, slots=True)
554class HealthReport:
555    """What a check found.
556
557    :param status: up, down or degraded
558    :param detail: why, when it is not simply up
559    :param checked: when this was worked out, in UTC
560    :param parts: whatever the check wants to show — a consumer count, a
561        round-trip time, the reports of the checks that were combined
562    """
563
564    status: HealthStatus
565    detail: str = ""
566    checked: datetime = field(default_factory=lambda: datetime.now(timezone.utc))
567    parts: Mapping[str, Any] = field(default_factory=dict)
568
569    @property
570    def healthy(self) -> bool:
571        """Whether a readiness probe should pass.
572
573        Degraded passes. An instance that is slow is still an instance that
574        works, and taking it out of rotation moves the traffic to one that will
575        be just as slow with more of it.
576        """
577        return self.status is not HealthStatus.DOWN
578
579    def __str__(self) -> str:
580        return self.status.value if not self.detail else f"{self.status.value}: {self.detail}"

What a check found.

Parameters
  • status: up, down or degraded
  • detail: why, when it is not simply up
  • checked: when this was worked out, in UTC
  • parts: whatever the check wants to show — a consumer count, a round-trip time, the reports of the checks that were combined
HealthReport( status: HealthStatus, detail: str = '', checked: datetime.datetime = <factory>, parts: Mapping[str, typing.Any] = <factory>)
status: HealthStatus
detail: str
checked: datetime.datetime
parts: Mapping[str, typing.Any]
healthy: bool
569    @property
570    def healthy(self) -> bool:
571        """Whether a readiness probe should pass.
572
573        Degraded passes. An instance that is slow is still an instance that
574        works, and taking it out of rotation moves the traffic to one that will
575        be just as slow with more of it.
576        """
577        return self.status is not HealthStatus.DOWN

Whether a readiness probe should pass.

Degraded passes. An instance that is slow is still an instance that works, and taking it out of rotation moves the traffic to one that will be just as slow with more of it.

class HealthStatus(builtins.str, enum.Enum):
538class HealthStatus(str, Enum):
539    """How a check turned out."""
540
541    #: Working.
542    UP = "up"
543
544    #: Not working. A readiness probe should fail on this.
545    DOWN = "down"
546
547    #: Working, but not as well as it should. Worth an alert; not worth taking
548    #: the instance out of rotation, because the replacement will almost
549    #: certainly be degraded too.
550    DEGRADED = "degraded"

How a check turned out.

UP = <HealthStatus.UP: 'up'>
DOWN = <HealthStatus.DOWN: 'down'>
DEGRADED = <HealthStatus.DEGRADED: 'degraded'>
class JsonCodec:
 89class JsonCodec:
 90    """Reads and writes JSON. The default, and the format every AceMQ library
 91    has without an extra dependency.
 92
 93    Dataclasses are encoded through :func:`dataclasses.asdict`, so a payload
 94    written as a dataclass goes on the wire as an object with its field names.
 95    Everything else is handed to :mod:`json` unchanged, which covers dicts,
 96    lists and the scalars.
 97
 98    Decoding produces dicts and lists rather than a class, because the wire says
 99    nothing about which class was meant. An application that wants its own type
100    back constructs it from what it gets, where it can decide what a missing
101    field means.
102    """
103
104    @property
105    def content_type(self) -> str:
106        return JSON_CONTENT_TYPE
107
108    def encode(self, payload: Any) -> bytes:
109        if dataclasses.is_dataclass(payload) and not isinstance(payload, type):
110            payload = dataclasses.asdict(payload)
111        try:
112            return json.dumps(payload).encode("utf-8")
113        except TypeError as failure:
114            raise TypeError(
115                f"acemq: a {type(payload).__name__} is not something JSON can carry: {failure}"
116            ) from failure
117
118    def decode(self, body: bytes, content_type: str | None = None) -> Any:
119        try:
120            return json.loads(body)
121        except (ValueError, UnicodeDecodeError) as failure:
122            # Fatal rather than retryable: a body that is not JSON is not JSON
123            # the next time either, and retrying it only holds a queue open
124            # until the message ages out.
125            raise FatalError(f"acemq: this message is not JSON: {failure}") from failure
126
127    def can_decode(self, content_type: str | None) -> bool:
128        """Accepts JSON, any ``+json`` media type, and a message with none.
129
130        Answering for a message whose sender set no content type is deliberate
131        and is what the other libraries do: JSON is the default format, so an
132        untyped message is far more likely to be JSON than anything else, and
133        something has to read it.
134        """
135        if content_type is None or content_type == "":
136            return True
137        lowered = content_type.lower()
138        return (
139            lowered.startswith(JSON_CONTENT_TYPE)
140            or lowered.startswith("text/json")
141            or "+json" in lowered
142        )

Reads and writes JSON. The default, and the format every AceMQ library has without an extra dependency.

Dataclasses are encoded through dataclasses.asdict(), so a payload written as a dataclass goes on the wire as an object with its field names. Everything else is handed to json unchanged, which covers dicts, lists and the scalars.

Decoding produces dicts and lists rather than a class, because the wire says nothing about which class was meant. An application that wants its own type back constructs it from what it gets, where it can decide what a missing field means.

content_type: str
104    @property
105    def content_type(self) -> str:
106        return JSON_CONTENT_TYPE
def encode(self, payload: Any) -> bytes:
108    def encode(self, payload: Any) -> bytes:
109        if dataclasses.is_dataclass(payload) and not isinstance(payload, type):
110            payload = dataclasses.asdict(payload)
111        try:
112            return json.dumps(payload).encode("utf-8")
113        except TypeError as failure:
114            raise TypeError(
115                f"acemq: a {type(payload).__name__} is not something JSON can carry: {failure}"
116            ) from failure
def decode(self, body: bytes, content_type: str | None = None) -> Any:
118    def decode(self, body: bytes, content_type: str | None = None) -> Any:
119        try:
120            return json.loads(body)
121        except (ValueError, UnicodeDecodeError) as failure:
122            # Fatal rather than retryable: a body that is not JSON is not JSON
123            # the next time either, and retrying it only holds a queue open
124            # until the message ages out.
125            raise FatalError(f"acemq: this message is not JSON: {failure}") from failure
def can_decode(self, content_type: str | None) -> bool:
127    def can_decode(self, content_type: str | None) -> bool:
128        """Accepts JSON, any ``+json`` media type, and a message with none.
129
130        Answering for a message whose sender set no content type is deliberate
131        and is what the other libraries do: JSON is the default format, so an
132        untyped message is far more likely to be JSON than anything else, and
133        something has to read it.
134        """
135        if content_type is None or content_type == "":
136            return True
137        lowered = content_type.lower()
138        return (
139            lowered.startswith(JSON_CONTENT_TYPE)
140            or lowered.startswith("text/json")
141            or "+json" in lowered
142        )

Accepts JSON, any +json media type, and a message with none.

Answering for a message whose sender set no content type is deliberate and is what the other libraries do: JSON is the default format, so an untyped message is far more likely to be JSON than anything else, and something has to read it.

@dataclass(frozen=True, slots=True)
class Message:
166@dataclass(frozen=True, slots=True)
167class Message:
168    """A delivery that has been decoded.
169
170    :param payload: the body, read through the codec
171    :param envelope: the metadata that travelled with it
172    :param routing_key: the key it arrived under
173    :param content_type: what the sender said the body was, or ``None``
174    :param redelivered: the broker saying it has handed this one over before
175    :param body: the undecoded bytes, for a handler that wants to see them
176    :param reply_to: AMQP's own ``reply-to`` property, empty when the sender set
177        none. A responder reads it as the fallback behind the
178        ``acemq-reply-to`` header, which is what lets a Java or .NET caller —
179        neither of which writes the header — be answered by a Python responder
180    """
181
182    payload: Any
183    envelope: Envelope
184    routing_key: str
185    content_type: str | None
186    redelivered: bool
187    body: bytes
188    reply_to: str = ""

A delivery that has been decoded.

Parameters
  • payload: the body, read through the codec
  • envelope: the metadata that travelled with it
  • routing_key: the key it arrived under
  • content_type: what the sender said the body was, or None
  • redelivered: the broker saying it has handed this one over before
  • body: the undecoded bytes, for a handler that wants to see them
  • reply_to: AMQP's own reply-to property, empty when the sender set none. A responder reads it as the fallback behind the acemq-reply-to header, which is what lets a Java or .NET caller — neither of which writes the header — be answered by a Python responder
Message( payload: Any, envelope: Envelope, routing_key: str, content_type: str | None, redelivered: bool, body: bytes, reply_to: str = '')
payload: Any
envelope: Envelope
routing_key: str
content_type: str | None
redelivered: bool
body: bytes
reply_to: str
class Metrics:
370class Metrics:
371    """An :class:`Observer` that keeps the numbers in memory.
372
373    Enough to serve from a health endpoint, assert on in a test, or print on a
374    signal. It is not a substitute for a real metrics system: no histograms, no
375    percentiles, and nothing is exported anywhere. :func:`prometheus_text`
376    renders what it has for a scraper, which needs no dependency at all.
377
378    Locked rather than relying on the event loop, because the blocking API in
379    :mod:`acemq_amqp.sync` runs handlers on a thread pool and a counter read
380    from two threads at once is a counter that loses increments.
381    """
382
383    def __init__(self) -> None:
384        self._lock = threading.Lock()
385        self._counts: dict[str, int] = {}
386        self._gauges: dict[str, int] = {}
387        self._durations: dict[str, DurationSummary] = {}
388
389    def count(self, metric: str, delta: int, labels: Mapping[str, str]) -> None:
390        key = metric_key(metric, labels)
391        with self._lock:
392            self._counts[key] = self._counts.get(key, 0) + delta
393
394    def gauge(self, metric: str, value: int, labels: Mapping[str, str]) -> None:
395        key = metric_key(metric, labels)
396        with self._lock:
397            self._gauges[key] = value
398
399    def observe(self, metric: str, seconds: float, labels: Mapping[str, str]) -> None:
400        key = metric_key(metric, labels)
401        with self._lock:
402            existing = self._durations.get(key)
403            if existing is None:
404                self._durations[key] = DurationSummary(1, seconds, seconds, seconds)
405                return
406            self._durations[key] = DurationSummary(
407                count=existing.count + 1,
408                total=existing.total + seconds,
409                fastest=min(existing.fastest, seconds),
410                slowest=max(existing.slowest, seconds),
411            )
412
413    @property
414    def counts(self) -> dict[str, int]:
415        """Every counter, keyed by metric and labels."""
416        with self._lock:
417            return dict(self._counts)
418
419    @property
420    def gauges(self) -> dict[str, int]:
421        """Every gauge."""
422        with self._lock:
423            return dict(self._gauges)
424
425    @property
426    def durations(self) -> dict[str, DurationSummary]:
427        """A summary per timing: how many, the total, the fastest, the slowest."""
428        with self._lock:
429            return dict(self._durations)
430
431    def __str__(self) -> str:
432        lines = [f"{name} {value}" for name, value in sorted(self.counts.items())]
433        lines += [f"{name} {value}" for name, value in sorted(self.gauges.items())]
434        lines += [
435            f"{name} count={summary.count} mean={summary.mean:.4f}s"
436            for name, summary in sorted(self.durations.items())
437        ]
438        return "\n".join(lines) or "nothing recorded yet"

An Observer that keeps the numbers in memory.

Enough to serve from a health endpoint, assert on in a test, or print on a signal. It is not a substitute for a real metrics system: no histograms, no percentiles, and nothing is exported anywhere. prometheus_text() renders what it has for a scraper, which needs no dependency at all.

Locked rather than relying on the event loop, because the blocking API in acemq_amqp.sync runs handlers on a thread pool and a counter read from two threads at once is a counter that loses increments.

def count(self, metric: str, delta: int, labels: Mapping[str, str]) -> None:
389    def count(self, metric: str, delta: int, labels: Mapping[str, str]) -> None:
390        key = metric_key(metric, labels)
391        with self._lock:
392            self._counts[key] = self._counts.get(key, 0) + delta
def gauge(self, metric: str, value: int, labels: Mapping[str, str]) -> None:
394    def gauge(self, metric: str, value: int, labels: Mapping[str, str]) -> None:
395        key = metric_key(metric, labels)
396        with self._lock:
397            self._gauges[key] = value
def observe(self, metric: str, seconds: float, labels: Mapping[str, str]) -> None:
399    def observe(self, metric: str, seconds: float, labels: Mapping[str, str]) -> None:
400        key = metric_key(metric, labels)
401        with self._lock:
402            existing = self._durations.get(key)
403            if existing is None:
404                self._durations[key] = DurationSummary(1, seconds, seconds, seconds)
405                return
406            self._durations[key] = DurationSummary(
407                count=existing.count + 1,
408                total=existing.total + seconds,
409                fastest=min(existing.fastest, seconds),
410                slowest=max(existing.slowest, seconds),
411            )
counts: dict[str, int]
413    @property
414    def counts(self) -> dict[str, int]:
415        """Every counter, keyed by metric and labels."""
416        with self._lock:
417            return dict(self._counts)

Every counter, keyed by metric and labels.

gauges: dict[str, int]
419    @property
420    def gauges(self) -> dict[str, int]:
421        """Every gauge."""
422        with self._lock:
423            return dict(self._gauges)

Every gauge.

durations: dict[str, DurationSummary]
425    @property
426    def durations(self) -> dict[str, DurationSummary]:
427        """A summary per timing: how many, the total, the fastest, the slowest."""
428        with self._lock:
429            return dict(self._durations)

A summary per timing: how many, the total, the fastest, the slowest.

class NullObserver:
332class NullObserver:
333    """An observer that does nothing, which is what a connection has by default.
334
335    A real object rather than ``None`` checked for at each call site, because
336    the alternative is a branch on every publish and every delivery, and one of
337    them is eventually written the wrong way round.
338    """
339
340    def count(self, metric: str, delta: int, labels: Mapping[str, str]) -> None:
341        return None
342
343    def observe(self, metric: str, seconds: float, labels: Mapping[str, str]) -> None:
344        return None
345
346    def gauge(self, metric: str, value: int, labels: Mapping[str, str]) -> None:
347        return None

An observer that does nothing, which is what a connection has by default.

A real object rather than None checked for at each call site, because the alternative is a branch on every publish and every delivery, and one of them is eventually written the wrong way round.

def count(self, metric: str, delta: int, labels: Mapping[str, str]) -> None:
340    def count(self, metric: str, delta: int, labels: Mapping[str, str]) -> None:
341        return None
def observe(self, metric: str, seconds: float, labels: Mapping[str, str]) -> None:
343    def observe(self, metric: str, seconds: float, labels: Mapping[str, str]) -> None:
344        return None
def gauge(self, metric: str, value: int, labels: Mapping[str, str]) -> None:
346    def gauge(self, metric: str, value: int, labels: Mapping[str, str]) -> None:
347        return None
@runtime_checkable
class Observer(typing.Protocol):
292@runtime_checkable
293class Observer(Protocol):
294    """Told what the library is doing.
295
296    Every method is called on the path a message takes, so none of them may
297    block and all of them have to be safe to call from several tasks at once.
298    An observer that talks to the network on each call is an observer that makes
299    every publish as slow as the network it talks to; buffer, and flush
300    somewhere else.
301
302    Nothing here is awaitable, deliberately. A metrics call that could suspend
303    would reorder the code around it and turn "count this" into a scheduling
304    decision, and no metrics library needs it.
305    """
306
307    def count(self, metric: str, delta: int, labels: Mapping[str, str]) -> None:
308        """Adds to a counter."""
309
310    def observe(self, metric: str, seconds: float, labels: Mapping[str, str]) -> None:
311        """Records one sample of a distribution.
312
313        A duration in seconds for every metric but one. The exception is
314        :data:`METRIC_CONSUME_ATTEMPTS`, which is a count of attempts and is
315        recorded here because this is the only distribution the interface has —
316        Java uses a Micrometer ``DistributionSummary`` for it and a ``Timer`` for
317        the rest, and this interface has one method where Java has two.
318
319        The argument keeps its name because renaming it would break every
320        observer that passes it by keyword, and because it is a duration in every
321        case but that one. An implementation that puts durations into histogram
322        buckets should give that metric its own: seconds-shaped buckets over a
323        value of 1, 2 or 3 report an attempt count as a handful of very fast
324        handlers. :class:`~acemq_amqp.prometheus.PrometheusObserver` does exactly
325        that, with :data:`~acemq_amqp.prometheus.DEFAULT_ATTEMPT_BUCKETS`.
326        """
327
328    def gauge(self, metric: str, value: int, labels: Mapping[str, str]) -> None:
329        """Sets a current value."""

Told what the library is doing.

Every method is called on the path a message takes, so none of them may block and all of them have to be safe to call from several tasks at once. An observer that talks to the network on each call is an observer that makes every publish as slow as the network it talks to; buffer, and flush somewhere else.

Nothing here is awaitable, deliberately. A metrics call that could suspend would reorder the code around it and turn "count this" into a scheduling decision, and no metrics library needs it.

Observer(*args, **kwargs)
1771def _no_init_or_replace_init(self, *args, **kwargs):
1772    cls = type(self)
1773
1774    if cls._is_protocol:
1775        raise TypeError('Protocols cannot be instantiated')
1776
1777    # Already using a custom `__init__`. No need to calculate correct
1778    # `__init__` to call. This can lead to RecursionError. See bpo-45121.
1779    if cls.__init__ is not _no_init_or_replace_init:
1780        return
1781
1782    # Initially, `__init__` of a protocol subclass is set to `_no_init_or_replace_init`.
1783    # The first instantiation of the subclass will call `_no_init_or_replace_init` which
1784    # searches for a proper new `__init__` in the MRO. The new `__init__`
1785    # replaces the subclass' old `__init__` (ie `_no_init_or_replace_init`). Subsequent
1786    # instantiation of the protocol subclass will thus use the new
1787    # `__init__` and no longer call `_no_init_or_replace_init`.
1788    for base in cls.__mro__:
1789        init = base.__dict__.get('__init__', _no_init_or_replace_init)
1790        if init is not _no_init_or_replace_init:
1791            cls.__init__ = init
1792            break
1793    else:
1794        # should not happen
1795        cls.__init__ = object.__init__
1796
1797    cls.__init__(self, *args, **kwargs)
def count(self, metric: str, delta: int, labels: Mapping[str, str]) -> None:
307    def count(self, metric: str, delta: int, labels: Mapping[str, str]) -> None:
308        """Adds to a counter."""

Adds to a counter.

def observe(self, metric: str, seconds: float, labels: Mapping[str, str]) -> None:
310    def observe(self, metric: str, seconds: float, labels: Mapping[str, str]) -> None:
311        """Records one sample of a distribution.
312
313        A duration in seconds for every metric but one. The exception is
314        :data:`METRIC_CONSUME_ATTEMPTS`, which is a count of attempts and is
315        recorded here because this is the only distribution the interface has —
316        Java uses a Micrometer ``DistributionSummary`` for it and a ``Timer`` for
317        the rest, and this interface has one method where Java has two.
318
319        The argument keeps its name because renaming it would break every
320        observer that passes it by keyword, and because it is a duration in every
321        case but that one. An implementation that puts durations into histogram
322        buckets should give that metric its own: seconds-shaped buckets over a
323        value of 1, 2 or 3 report an attempt count as a handful of very fast
324        handlers. :class:`~acemq_amqp.prometheus.PrometheusObserver` does exactly
325        that, with :data:`~acemq_amqp.prometheus.DEFAULT_ATTEMPT_BUCKETS`.
326        """

Records one sample of a distribution.

A duration in seconds for every metric but one. The exception is METRIC_CONSUME_ATTEMPTS, which is a count of attempts and is recorded here because this is the only distribution the interface has — Java uses a Micrometer DistributionSummary for it and a Timer for the rest, and this interface has one method where Java has two.

The argument keeps its name because renaming it would break every observer that passes it by keyword, and because it is a duration in every case but that one. An implementation that puts durations into histogram buckets should give that metric its own: seconds-shaped buckets over a value of 1, 2 or 3 report an attempt count as a handful of very fast handlers. ~acemq_amqp.prometheus.PrometheusObserver does exactly that, with ~acemq_amqp.prometheus.DEFAULT_ATTEMPT_BUCKETS.

def gauge(self, metric: str, value: int, labels: Mapping[str, str]) -> None:
328    def gauge(self, metric: str, value: int, labels: Mapping[str, str]) -> None:
329        """Sets a current value."""

Sets a current value.

@dataclass(frozen=True, slots=True)
class Outbound:
 85@dataclass(frozen=True, slots=True)
 86class Outbound:
 87    """A message on its way to the broker, already encoded.
 88
 89    :param body: the encoded payload
 90    :param content_type: what the body is, for the consumer's codec
 91    :param message_id: the AMQP message id, conventionally the envelope's
 92    :param reply_to: AMQP's own ``reply-to`` property, naming where an answer
 93        should be sent. Empty for the overwhelming majority of messages, which
 94        are not questions. It is here as well as in the headers because the four
 95        other libraries read the native property, and a Python responder that
 96        only ever wrote a header could not be answered by a Java or .NET caller
 97    :param headers: the envelope's headers plus the application's
 98    :param persistent: asks the broker to write the message to disk, which is
 99        not a guarantee on its own — a persistent message on a queue that is not
100        durable still dies with the broker
101    :param mandatory: asks the broker to return the message rather than drop it
102        when no queue is bound to receive it
103    """
104
105    body: bytes
106    content_type: str = ""
107    message_id: str = ""
108    reply_to: str = ""
109    headers: Mapping[str, Any] = field(default_factory=dict)
110    persistent: bool = True
111    mandatory: bool = False

A message on its way to the broker, already encoded.

Parameters
  • body: the encoded payload
  • content_type: what the body is, for the consumer's codec
  • message_id: the AMQP message id, conventionally the envelope's
  • reply_to: AMQP's own reply-to property, naming where an answer should be sent. Empty for the overwhelming majority of messages, which are not questions. It is here as well as in the headers because the four other libraries read the native property, and a Python responder that only ever wrote a header could not be answered by a Java or .NET caller
  • headers: the envelope's headers plus the application's
  • persistent: asks the broker to write the message to disk, which is not a guarantee on its own — a persistent message on a queue that is not durable still dies with the broker
  • mandatory: asks the broker to return the message rather than drop it when no queue is bound to receive it
Outbound( body: bytes, content_type: str = '', message_id: str = '', reply_to: str = '', headers: Mapping[str, typing.Any] = <factory>, persistent: bool = True, mandatory: bool = False)
body: bytes
content_type: str
message_id: str
reply_to: str
headers: Mapping[str, typing.Any]
persistent: bool
mandatory: bool
@dataclass(slots=True)
class PublishContext:
 86@dataclass(slots=True)
 87class PublishContext:
 88    """A message on its way out, before it is encoded.
 89
 90    Mutable on purpose, and mutable rather than rebuilt-and-returned: an
 91    interceptor that only wants to add one header should not have to reconstruct
 92    the rest, and one that reads what an earlier interceptor wrote needs to be
 93    looking at the same object rather than at a copy of an earlier version.
 94
 95    :param exchange: where it is going. Changing it redirects the message
 96    :param routing_key: what it is published under
 97    :param envelope: the metadata. Reserved header names are still refused when
 98        the envelope is rendered, whoever wrote them
 99    :param payload: what is about to be encoded, before the codec sees it, so an
100        interceptor can change the message and not only its metadata
101    :param persistent: whether the broker is asked to write it to disk
102    :param mandatory: whether reaching no queue is an error rather than a
103        silence
104    :param reply_to: where an answer should be sent, written to AMQP's own
105        ``reply-to`` property. Empty for anything that is not a question
106    """
107
108    exchange: str
109    routing_key: str
110    envelope: Envelope
111    payload: Any
112    persistent: bool = True
113    mandatory: bool = False
114    reply_to: str = ""
115
116    def set_header(self, name: str, value: Any) -> None:
117        """Adds an application header to the message being published.
118
119        A convenience over rebuilding the envelope, which is what the three
120        commonest interceptors — a tenant, a trace context, a schema version —
121        all want to do and nothing else.
122        """
123        self.envelope = self.envelope.with_(headers={**self.envelope.headers, name: value})

A message on its way out, before it is encoded.

Mutable on purpose, and mutable rather than rebuilt-and-returned: an interceptor that only wants to add one header should not have to reconstruct the rest, and one that reads what an earlier interceptor wrote needs to be looking at the same object rather than at a copy of an earlier version.

Parameters
  • exchange: where it is going. Changing it redirects the message
  • routing_key: what it is published under
  • envelope: the metadata. Reserved header names are still refused when the envelope is rendered, whoever wrote them
  • payload: what is about to be encoded, before the codec sees it, so an interceptor can change the message and not only its metadata
  • persistent: whether the broker is asked to write it to disk
  • mandatory: whether reaching no queue is an error rather than a silence
  • reply_to: where an answer should be sent, written to AMQP's own reply-to property. Empty for anything that is not a question
PublishContext( exchange: str, routing_key: str, envelope: Envelope, payload: Any, persistent: bool = True, mandatory: bool = False, reply_to: str = '')
exchange: str
routing_key: str
envelope: Envelope
payload: Any
persistent: bool
mandatory: bool
reply_to: str
def set_header(self, name: str, value: Any) -> None:
116    def set_header(self, name: str, value: Any) -> None:
117        """Adds an application header to the message being published.
118
119        A convenience over rebuilding the envelope, which is what the three
120        commonest interceptors — a tenant, a trace context, a schema version —
121        all want to do and nothing else.
122        """
123        self.envelope = self.envelope.with_(headers={**self.envelope.headers, name: value})

Adds an application header to the message being published.

A convenience over rebuilding the envelope, which is what the three commonest interceptors — a tenant, a trace context, a schema version — all want to do and nothing else.

class PublishError(acemq_amqp.AceMQError):
46class PublishError(AceMQError):
47    """A message the broker would not take, or would not route.
48
49    It carries where the message was going, which is the first thing anybody
50    wants when a routing key was built from a variable.
51
52    :param message_id: the envelope identifier it went out with
53    :param exchange: where it was sent
54    :param routing_key: what it was sent under
55    :param reason: the broker's explanation
56    :param unroutable: true when the message reached no queue rather than being
57        refused. The broker was working; nothing was listening
58    :param summary: the whole message, in place of the sentence composed from
59        the arguments above. For a failure that is about a batch rather than
60        about one message — :meth:`acemq_amqp.Publisher.send_all` raises one of
61        these naming how many messages were not confirmed and how many were,
62        in the words Java and .NET use for the same failure, so that one
63        runbook covers all of them. A batch failure leaves ``message_id``
64        empty, because there is no single message it is about
65    """
66
67    def __init__(
68        self,
69        message_id: str,
70        exchange: str,
71        routing_key: str,
72        reason: str,
73        *,
74        unroutable: bool = False,
75        summary: str = "",
76    ) -> None:
77        self.message_id = message_id
78        self.exchange = exchange
79        self.routing_key = routing_key
80        self.reason = reason
81        self.unroutable = unroutable
82
83        where = f"exchange {exchange!r} with key {routing_key!r}"
84        if summary:
85            super().__init__(summary)
86        elif unroutable:
87            super().__init__(
88                f"acemq: message {message_id} to {where} reached no queue: {reason}"
89            )
90        else:
91            super().__init__(f"acemq: cannot publish message {message_id} to {where}: {reason}")

A message the broker would not take, or would not route.

It carries where the message was going, which is the first thing anybody wants when a routing key was built from a variable.

Parameters
  • message_id: the envelope identifier it went out with
  • exchange: where it was sent
  • routing_key: what it was sent under
  • reason: the broker's explanation
  • unroutable: true when the message reached no queue rather than being refused. The broker was working; nothing was listening
  • summary: the whole message, in place of the sentence composed from the arguments above. For a failure that is about a batch rather than about one message — acemq_amqp.Publisher.send_all() raises one of these naming how many messages were not confirmed and how many were, in the words Java and .NET use for the same failure, so that one runbook covers all of them. A batch failure leaves message_id empty, because there is no single message it is about
PublishError( message_id: str, exchange: str, routing_key: str, reason: str, *, unroutable: bool = False, summary: str = '')
67    def __init__(
68        self,
69        message_id: str,
70        exchange: str,
71        routing_key: str,
72        reason: str,
73        *,
74        unroutable: bool = False,
75        summary: str = "",
76    ) -> None:
77        self.message_id = message_id
78        self.exchange = exchange
79        self.routing_key = routing_key
80        self.reason = reason
81        self.unroutable = unroutable
82
83        where = f"exchange {exchange!r} with key {routing_key!r}"
84        if summary:
85            super().__init__(summary)
86        elif unroutable:
87            super().__init__(
88                f"acemq: message {message_id} to {where} reached no queue: {reason}"
89            )
90        else:
91            super().__init__(f"acemq: cannot publish message {message_id} to {where}: {reason}")
message_id
exchange
routing_key
reason
unroutable
PublishInterceptor = collections.abc.Callable[[PublishContext, collections.abc.Callable[[PublishContext], collections.abc.Awaitable[PublishResult]]], collections.abc.Awaitable[PublishResult]]
PublishNext = collections.abc.Callable[[PublishContext], collections.abc.Awaitable[PublishResult]]
@dataclass(frozen=True, slots=True)
class PublishResult:
114@dataclass(frozen=True, slots=True)
115class PublishResult:
116    """What the broker said about a published message.
117
118    :param message_id: the identifier it went out with
119    :param confirmed: the broker taking responsibility for the message. Without
120        publisher confirms this is false and nothing has been promised: the
121        message reached a socket, which is not the same thing
122    :param routed: false when a mandatory message reached no queue at all. A
123        message nothing is bound to receive is not an error to the broker; it is
124        dropped, silently, which is exactly the failure worth hearing about
125    :param return_reason: the broker's explanation when ``routed`` is false
126    """
127
128    message_id: str = ""
129    confirmed: bool = False
130    routed: bool = True
131    return_reason: str = ""

What the broker said about a published message.

Parameters
  • message_id: the identifier it went out with
  • confirmed: the broker taking responsibility for the message. Without publisher confirms this is false and nothing has been promised: the message reached a socket, which is not the same thing
  • routed: false when a mandatory message reached no queue at all. A message nothing is bound to receive is not an error to the broker; it is dropped, silently, which is exactly the failure worth hearing about
  • return_reason: the broker's explanation when routed is false
PublishResult( message_id: str = '', confirmed: bool = False, routed: bool = True, return_reason: str = '')
message_id: str
confirmed: bool
routed: bool
return_reason: str
class Publisher:
208class Publisher:
209    """Sends messages to one destination.
210
211    Cheap to keep and safe to share: build one per message type at start-up
212    rather than one per message.
213    """
214
215    def __init__(
216        self,
217        connection: Connection,
218        exchange: str,
219        routing_key: str,
220        *,
221        codec: Codec | None = None,
222        persistent: bool = True,
223        mandatory: bool = False,
224    ) -> None:
225        self._connection = connection
226        self._exchange = exchange
227        self._routing_key = routing_key
228        self._codec = codec or connection.codec
229        self._persistent = persistent
230        self._mandatory = mandatory
231
232    async def send(
233        self,
234        payload: Any,
235        *,
236        envelope: Envelope | None = None,
237        routing_key: str | None = None,
238        reply_to: str = "",
239    ) -> PublishResult:
240        """Publishes one message.
241
242        The envelope is built here when none is given: an identifier, a
243        correlation defaulting to it, this connection's origin, and now. The
244        type falls back to the routing key, which is why a publisher bound to
245        ``order.placed`` needs no further configuration to say what it sends.
246
247        :param payload: what to send, encoded by this publisher's codec
248        :param envelope: metadata to send instead of a fresh one, for a message
249            derived from another
250        :param routing_key: what to publish under, for a publisher bound to an
251            exchange rather than to one key
252        :param reply_to: where an answer should come back to, written to AMQP's
253            own ``reply-to`` property. Only :mod:`~acemq_amqp.patterns` request
254            and reply sets it, and it sets the ``acemq-reply-to`` header to the
255            same value: the header is what survives a service that rebuilds the
256            message, the property is what the other four libraries read
257        :returns: what the broker said
258        :raises PublishError: when the publisher is mandatory and the message
259            reached no queue
260        """
261        key = self._routing_key if routing_key is None else routing_key
262        outgoing = envelope or Envelope(origin=self._connection.origin)
263        if not outgoing.origin:
264            # A message derived from another — a reply, a pipeline hop, the next
265            # stop on a routing slip — is still published by this process, and an
266            # origin naming the service before it would be a lie in a log line
267            # somebody is going to trust.
268            outgoing = outgoing.with_(origin=self._connection.origin)
269
270        context = PublishContext(
271            exchange=self._exchange,
272            routing_key=key,
273            envelope=outgoing,
274            payload=payload,
275            persistent=self._persistent,
276            mandatory=self._mandatory,
277            reply_to=reply_to,
278        )
279        # Read here rather than at construction, so an interceptor registered
280        # during start-up applies to the publishers that already exist. Building
281        # the chain per message costs a few closures and buys that.
282        return await publish_chain(self._connection.publish_interceptors, self._send)(context)
283
284    async def send_all(self, payloads: Iterable[Any]) -> list[PublishResult]:
285        """Publishes a batch and waits for every confirm.
286
287        What most bulk publishing actually wants: the throughput of pipelining
288        with the safety of having waited. Every message goes out before any
289        confirm is awaited, and only then are all of them checked together.
290        Awaiting each send in turn is a broker round trip per message — the
291        loop a caller could have written for themselves, and none of the
292        throughput a batch exists for.
293
294        The results come back in the order the payloads were given, whatever
295        order the broker answers in.
296
297        **This is not atomic.** AMQP has no such thing: there is no way to
298        publish a hundred messages such that all or none arrive, and a library
299        that offered one would be lying. A failure does not abandon the rest
300        either — every send is awaited whatever an earlier one did, so a batch
301        that half succeeded can say how many arrived. That count is the point:
302        a caller told only "it failed" resends the messages the broker already
303        has.
304
305        How wide the batch gets on the wire is capped, and by the connection
306        rather than by this method:
307        :attr:`Connection.max_outstanding_publishes` is a thousand by default,
308        and the thousand-and-first task waits for one of the first thousand to
309        be answered before its message is written. A list of a million is
310        therefore a million tasks and a thousand messages in flight rather than
311        a million, which is backpressure instead of a memory leak — but a
312        million tasks is still a million tasks, so chunk what comes out of a
313        database rather than handing a cursor's worth of rows to one call.
314
315        :param payloads: what to send, in order, each encoded by this
316            publisher's codec
317        :returns: what the broker said about each, in the order the payloads
318            were given
319        :raises PublishError: when any message was not confirmed. Its message
320            names how many failed and how many did not, and ``__cause__``
321            carries the first failure in payload order — which is not
322            necessarily the first one the broker answered. A batch wide enough
323            that the broker stops keeping up fails this way too, with the first
324            failure saying so
325        """
326        batch = list(payloads)
327        # Everything is started before anything is awaited. A task per payload,
328        # all of them created before the gather below, is what puts the whole
329        # batch on the wire while the broker is still working through the
330        # first confirm.
331        sends = [asyncio.create_task(self.send(payload)) for payload in batch]
332        # return_exceptions, and not for tidiness: without it the first failure
333        # comes out of the gather while the rest of the batch is still in
334        # flight, which loses the count this method exists to report and leaves
335        # the others' exceptions unretrieved for asyncio to complain about
336        # later.
337        settled = await asyncio.gather(*sends, return_exceptions=True)
338
339        results: list[PublishResult] = []
340        first_failure: BaseException | None = None
341        failed = 0
342        for outcome in settled:
343            if isinstance(outcome, BaseException):
344                failed += 1
345                if first_failure is None:
346                    first_failure = outcome
347            else:
348                results.append(outcome)
349
350        if first_failure is not None:
351            # Named rather than printed. A failure with nothing in its message —
352            # the CancelledError every task gets when the connection closes
353            # underneath the batch is the common one — renders as an empty
354            # string, and the sentence then ends "The first failure was: " and
355            # stops, promising an explanation it does not give. The class name
356            # is not much, but it is the difference between "the sends were
357            # cancelled" and nothing at all.
358            described = str(first_failure) or type(first_failure).__name__
359            # The counts matter. A batch that half succeeded is the ordinary
360            # outcome of a broker problem partway through, and this sentence is
361            # word for word the one Java and .NET raise, so that an operator
362            # reading a log has one thing to recognise rather than three.
363            raise PublishError(
364                "",
365                self._exchange,
366                self._routing_key,
367                described,
368                summary=(
369                    f"{failed} of {len(batch)} messages were not confirmed;"
370                    f" {len(results)} were. The first failure was: {described}"
371                ),
372            ) from first_failure
373        return results
374
375    async def _send(self, context: PublishContext) -> PublishResult:
376        """The innermost work: encode what the interceptors left, and send it."""
377        observer = self._connection.observer
378        # ``routing.key`` and not ``key``: Java and .NET both tag a publish with
379        # the fully-qualified name, and a dashboard that groups by it should
380        # read the same in all five languages.
381        labels = {TAG_EXCHANGE: context.exchange, TAG_ROUTING_KEY: context.routing_key}
382        # Started before the encode, because everything from here on is time the
383        # caller spends inside send(): the codec, the wait for a permit and the
384        # broker's confirm are all things that make a publish slow, and a timer
385        # that began after the encode would hide a slow codec completely.
386        started = time.monotonic()
387
388        def record(outcome: str) -> None:
389            """The pair, always together and always with the same labels.
390
391            Two calls rather than one because the interface has two methods, and
392            in this order because a dashboard dividing the duration's count into
393            the total should never see the total lead it.
394            """
395            observer.observe(
396                METRIC_PUBLISH_DURATION,
397                time.monotonic() - started,
398                {**labels, TAG_OUTCOME: outcome},
399            )
400            observer.count(METRIC_PUBLISH_TOTAL, 1, {**labels, TAG_OUTCOME: outcome})
401
402        # Encoded here rather than before the chain ran, so an interceptor can
403        # change the payload as well as its metadata. A codec that has already
404        # run leaves an interceptor with bytes and nothing it can do to them.
405        try:
406            body = self._codec.encode(context.payload)
407            result = await self._connection.publish_raw(
408                context.exchange,
409                context.routing_key,
410                Outbound(
411                    body=body,
412                    content_type=self._codec.content_type,
413                    message_id=context.envelope.id,
414                    reply_to=context.reply_to,
415                    headers=context.envelope.to_headers(routing_key=context.routing_key),
416                    persistent=context.persistent,
417                    mandatory=context.mandatory,
418                ),
419            )
420        except PublishingPausedError:
421            # Declined before anything was written: nothing lost, so not
422            # ``failed``, which is the outcome an alert on loss watches.
423            record(OUTCOME_REFUSED)
424            raise
425        except Exception:
426            record(OUTCOME_FAILED)
427            raise
428
429        # Raised rather than left in the result, because a caller who does not
430        # read the result would otherwise carry on believing the message went
431        # somewhere. Unroutable is the quietest failure AMQP has: the publish
432        # succeeds, the consumer waits, and nothing anywhere says why.
433        if context.mandatory and not result.routed:
434            record(OUTCOME_UNROUTABLE)
435            raise PublishError(
436                context.envelope.id,
437                context.exchange,
438                context.routing_key,
439                result.return_reason or "no queue is bound to receive it",
440                unroutable=True,
441            )
442
443        record(OUTCOME_CONFIRMED)
444        return result

Sends messages to one destination.

Cheap to keep and safe to share: build one per message type at start-up rather than one per message.

Publisher( connection: Connection, exchange: str, routing_key: str, *, codec: Codec | None = None, persistent: bool = True, mandatory: bool = False)
215    def __init__(
216        self,
217        connection: Connection,
218        exchange: str,
219        routing_key: str,
220        *,
221        codec: Codec | None = None,
222        persistent: bool = True,
223        mandatory: bool = False,
224    ) -> None:
225        self._connection = connection
226        self._exchange = exchange
227        self._routing_key = routing_key
228        self._codec = codec or connection.codec
229        self._persistent = persistent
230        self._mandatory = mandatory
async def send( self, payload: Any, *, envelope: Envelope | None = None, routing_key: str | None = None, reply_to: str = '') -> PublishResult:
232    async def send(
233        self,
234        payload: Any,
235        *,
236        envelope: Envelope | None = None,
237        routing_key: str | None = None,
238        reply_to: str = "",
239    ) -> PublishResult:
240        """Publishes one message.
241
242        The envelope is built here when none is given: an identifier, a
243        correlation defaulting to it, this connection's origin, and now. The
244        type falls back to the routing key, which is why a publisher bound to
245        ``order.placed`` needs no further configuration to say what it sends.
246
247        :param payload: what to send, encoded by this publisher's codec
248        :param envelope: metadata to send instead of a fresh one, for a message
249            derived from another
250        :param routing_key: what to publish under, for a publisher bound to an
251            exchange rather than to one key
252        :param reply_to: where an answer should come back to, written to AMQP's
253            own ``reply-to`` property. Only :mod:`~acemq_amqp.patterns` request
254            and reply sets it, and it sets the ``acemq-reply-to`` header to the
255            same value: the header is what survives a service that rebuilds the
256            message, the property is what the other four libraries read
257        :returns: what the broker said
258        :raises PublishError: when the publisher is mandatory and the message
259            reached no queue
260        """
261        key = self._routing_key if routing_key is None else routing_key
262        outgoing = envelope or Envelope(origin=self._connection.origin)
263        if not outgoing.origin:
264            # A message derived from another — a reply, a pipeline hop, the next
265            # stop on a routing slip — is still published by this process, and an
266            # origin naming the service before it would be a lie in a log line
267            # somebody is going to trust.
268            outgoing = outgoing.with_(origin=self._connection.origin)
269
270        context = PublishContext(
271            exchange=self._exchange,
272            routing_key=key,
273            envelope=outgoing,
274            payload=payload,
275            persistent=self._persistent,
276            mandatory=self._mandatory,
277            reply_to=reply_to,
278        )
279        # Read here rather than at construction, so an interceptor registered
280        # during start-up applies to the publishers that already exist. Building
281        # the chain per message costs a few closures and buys that.
282        return await publish_chain(self._connection.publish_interceptors, self._send)(context)

Publishes one message.

The envelope is built here when none is given: an identifier, a correlation defaulting to it, this connection's origin, and now. The type falls back to the routing key, which is why a publisher bound to order.placed needs no further configuration to say what it sends.

Parameters
  • payload: what to send, encoded by this publisher's codec
  • envelope: metadata to send instead of a fresh one, for a message derived from another
  • routing_key: what to publish under, for a publisher bound to an exchange rather than to one key
  • reply_to: where an answer should come back to, written to AMQP's own reply-to property. Only ~acemq_amqp.patterns request and reply sets it, and it sets the acemq-reply-to header to the same value: the header is what survives a service that rebuilds the message, the property is what the other four libraries read :returns: what the broker said
Raises
  • PublishError: when the publisher is mandatory and the message reached no queue
async def send_all( self, payloads: Iterable[typing.Any]) -> list[PublishResult]:
284    async def send_all(self, payloads: Iterable[Any]) -> list[PublishResult]:
285        """Publishes a batch and waits for every confirm.
286
287        What most bulk publishing actually wants: the throughput of pipelining
288        with the safety of having waited. Every message goes out before any
289        confirm is awaited, and only then are all of them checked together.
290        Awaiting each send in turn is a broker round trip per message — the
291        loop a caller could have written for themselves, and none of the
292        throughput a batch exists for.
293
294        The results come back in the order the payloads were given, whatever
295        order the broker answers in.
296
297        **This is not atomic.** AMQP has no such thing: there is no way to
298        publish a hundred messages such that all or none arrive, and a library
299        that offered one would be lying. A failure does not abandon the rest
300        either — every send is awaited whatever an earlier one did, so a batch
301        that half succeeded can say how many arrived. That count is the point:
302        a caller told only "it failed" resends the messages the broker already
303        has.
304
305        How wide the batch gets on the wire is capped, and by the connection
306        rather than by this method:
307        :attr:`Connection.max_outstanding_publishes` is a thousand by default,
308        and the thousand-and-first task waits for one of the first thousand to
309        be answered before its message is written. A list of a million is
310        therefore a million tasks and a thousand messages in flight rather than
311        a million, which is backpressure instead of a memory leak — but a
312        million tasks is still a million tasks, so chunk what comes out of a
313        database rather than handing a cursor's worth of rows to one call.
314
315        :param payloads: what to send, in order, each encoded by this
316            publisher's codec
317        :returns: what the broker said about each, in the order the payloads
318            were given
319        :raises PublishError: when any message was not confirmed. Its message
320            names how many failed and how many did not, and ``__cause__``
321            carries the first failure in payload order — which is not
322            necessarily the first one the broker answered. A batch wide enough
323            that the broker stops keeping up fails this way too, with the first
324            failure saying so
325        """
326        batch = list(payloads)
327        # Everything is started before anything is awaited. A task per payload,
328        # all of them created before the gather below, is what puts the whole
329        # batch on the wire while the broker is still working through the
330        # first confirm.
331        sends = [asyncio.create_task(self.send(payload)) for payload in batch]
332        # return_exceptions, and not for tidiness: without it the first failure
333        # comes out of the gather while the rest of the batch is still in
334        # flight, which loses the count this method exists to report and leaves
335        # the others' exceptions unretrieved for asyncio to complain about
336        # later.
337        settled = await asyncio.gather(*sends, return_exceptions=True)
338
339        results: list[PublishResult] = []
340        first_failure: BaseException | None = None
341        failed = 0
342        for outcome in settled:
343            if isinstance(outcome, BaseException):
344                failed += 1
345                if first_failure is None:
346                    first_failure = outcome
347            else:
348                results.append(outcome)
349
350        if first_failure is not None:
351            # Named rather than printed. A failure with nothing in its message —
352            # the CancelledError every task gets when the connection closes
353            # underneath the batch is the common one — renders as an empty
354            # string, and the sentence then ends "The first failure was: " and
355            # stops, promising an explanation it does not give. The class name
356            # is not much, but it is the difference between "the sends were
357            # cancelled" and nothing at all.
358            described = str(first_failure) or type(first_failure).__name__
359            # The counts matter. A batch that half succeeded is the ordinary
360            # outcome of a broker problem partway through, and this sentence is
361            # word for word the one Java and .NET raise, so that an operator
362            # reading a log has one thing to recognise rather than three.
363            raise PublishError(
364                "",
365                self._exchange,
366                self._routing_key,
367                described,
368                summary=(
369                    f"{failed} of {len(batch)} messages were not confirmed;"
370                    f" {len(results)} were. The first failure was: {described}"
371                ),
372            ) from first_failure
373        return results

Publishes a batch and waits for every confirm.

What most bulk publishing actually wants: the throughput of pipelining with the safety of having waited. Every message goes out before any confirm is awaited, and only then are all of them checked together. Awaiting each send in turn is a broker round trip per message — the loop a caller could have written for themselves, and none of the throughput a batch exists for.

The results come back in the order the payloads were given, whatever order the broker answers in.

This is not atomic. AMQP has no such thing: there is no way to publish a hundred messages such that all or none arrive, and a library that offered one would be lying. A failure does not abandon the rest either — every send is awaited whatever an earlier one did, so a batch that half succeeded can say how many arrived. That count is the point: a caller told only "it failed" resends the messages the broker already has.

How wide the batch gets on the wire is capped, and by the connection rather than by this method: Connection.max_outstanding_publishes is a thousand by default, and the thousand-and-first task waits for one of the first thousand to be answered before its message is written. A list of a million is therefore a million tasks and a thousand messages in flight rather than a million, which is backpressure instead of a memory leak — but a million tasks is still a million tasks, so chunk what comes out of a database rather than handing a cursor's worth of rows to one call.

Parameters
  • payloads: what to send, in order, each encoded by this publisher's codec :returns: what the broker said about each, in the order the payloads were given
Raises
  • PublishError: when any message was not confirmed. Its message names how many failed and how many did not, and __cause__ carries the first failure in payload order — which is not necessarily the first one the broker answered. A batch wide enough that the broker stops keeping up fails this way too, with the first failure saying so
class PublishingPausedError(acemq_amqp.PublishError):
 94class PublishingPausedError(PublishError):
 95    """A publish refused because the broker has blocked this connection.
 96
 97    Declined, not lost. RabbitMQ blocks a connection when it is low on memory
 98    or disk and stops reading the socket until the alarm clears; this library
 99    refuses the publish at once rather than queueing a frame behind a broker
100    that has said it cannot take it. **Nothing was sent**, so retrying once the
101    broker unblocks cannot duplicate the message — which is what tells this
102    apart from every other :class:`PublishError`, where the message may or may
103    not have arrived.
104
105    A subclass, so an existing ``except PublishError`` still catches it. Catch
106    this one first to back off, shed load or buffer instead of counting a loss.
107    It is Go's ``PublishingPausedError``, .NET's ``ConnectionBlockedException``
108    and the refusal Java's standing load counts as ``refused``.
109
110    ``reason`` is the broker's words when the transport kept them; the RabbitMQ
111    transport cannot (see :attr:`acemq_amqp.Connection.blocked_reason`).
112    """

A publish refused because the broker has blocked this connection.

Declined, not lost. RabbitMQ blocks a connection when it is low on memory or disk and stops reading the socket until the alarm clears; this library refuses the publish at once rather than queueing a frame behind a broker that has said it cannot take it. Nothing was sent, so retrying once the broker unblocks cannot duplicate the message — which is what tells this apart from every other PublishError, where the message may or may not have arrived.

A subclass, so an existing except PublishError still catches it. Catch this one first to back off, shed load or buffer instead of counting a loss. It is Go's PublishingPausedError, .NET's ConnectionBlockedException and the refusal Java's standing load counts as refused.

reason is the broker's words when the transport kept them; the RabbitMQ transport cannot (see acemq_amqp.Connection.blocked_reason).

@dataclass(frozen=True, slots=True)
class QueueSpec:
36@dataclass(frozen=True, slots=True)
37class QueueSpec:
38    """How a queue should be declared.
39
40    :param durable: survives a broker restart
41    :param auto_delete: goes away when its last consumer does
42    :param exclusive: usable only by the connection that declared it
43    :param args: broker-specific arguments, such as ``x-dead-letter-exchange``
44    """
45
46    durable: bool = True
47    auto_delete: bool = False
48    exclusive: bool = False
49    args: Mapping[str, Any] = field(default_factory=dict)

How a queue should be declared.

Parameters
  • durable: survives a broker restart
  • auto_delete: goes away when its last consumer does
  • exclusive: usable only by the connection that declared it
  • args: broker-specific arguments, such as x-dead-letter-exchange
QueueSpec( durable: bool = True, auto_delete: bool = False, exclusive: bool = False, args: Mapping[str, typing.Any] = <factory>)
durable: bool
auto_delete: bool
exclusive: bool
args: Mapping[str, typing.Any]
@dataclass(frozen=True, slots=True)
class RetryPolicy:
 67@dataclass(frozen=True, slots=True)
 68class RetryPolicy:
 69    """How many times to try, how long to wait, and where.
 70
 71    Waits below ``broker_wait_threshold`` are spent in the consumer, which holds
 72    the delivery and one prefetch slot for the duration. Waits at or above it
 73    are spent in a ``{queue}.retry.{delay}`` queue whose ``x-message-ttl`` is the
 74    wait and whose dead-letter target is the source queue, so the broker returns
 75    the message when the time is up and a consumer restart cannot shorten it.
 76
 77    Splitting at a threshold rather than picking one of the two takes the
 78    durability where it is worth its complexity and leaves the simplicity where
 79    it is not. A queue's TTL is fixed at declaration and so cannot express a
 80    jittered delay, which is why jitter applies below the threshold only; above
 81    it the spread comes free, because each message's TTL starts when it enters
 82    the rung, so a fleet that failed over ten seconds is released over ten
 83    seconds.
 84
 85    :param max_attempts: total deliveries including the first. 1 means no retry
 86    :param initial_delay: the wait before the second attempt
 87    :param multiplier: what the delay is multiplied by each time
 88    :param max_delay: the ceiling, or zero for none
 89    :param jitter_factor: how far a delay may move either side, 0 to 1
 90    :param max_message_age: give up on anything older, or zero for never
 91    :param broker_wait_threshold: waits this long or longer are spent in the
 92        broker, or zero to spend every wait in the consumer
 93    """
 94
 95    max_attempts: int = 1
 96    initial_delay: timedelta = ZERO
 97    multiplier: float = 2.0
 98    max_delay: timedelta = ZERO
 99    jitter_factor: float = 0.0
100    max_message_age: timedelta = ZERO
101    broker_wait_threshold: timedelta = DEFAULT_BROKER_WAIT_THRESHOLD
102
103    def give_up_after(self, age: timedelta) -> RetryPolicy:
104        """A copy that abandons a message older than ``age``.
105
106        The honest limit when a queue has been paused: attempts say nothing
107        about how long a message has been waiting, and a message four days old
108        is usually one nobody wants delivered now.
109        """
110        return replace(self, max_message_age=age)
111
112    def with_jitter(self, factor: float) -> RetryPolicy:
113        """A copy using a different jitter factor, between 0 and 1."""
114        return replace(self, jitter_factor=factor)
115
116    def wait_in_broker_from(self, threshold: timedelta) -> RetryPolicy:
117        """A copy that moves the line between waiting here and waiting there.
118
119        Zero is the way out: with no threshold nothing is long enough to reach
120        the broker, so every wait is spent in the consumer and no rung queue is
121        needed. That is the right setting for a service whose broker it may not
122        declare queues on, and the wrong one for a policy with delays measured
123        in minutes.
124
125        :param threshold: waits this long or longer go to a rung queue, or zero
126            for none of them
127        :returns: a new policy
128        """
129        return replace(self, broker_wait_threshold=threshold)
130
131    def waits_in_broker(self, delay: timedelta) -> bool:
132        """Whether a wait of this length belongs on a rung queue.
133
134        :param delay: an unjittered delay, as :meth:`schedule` reports them
135        :returns: whether the broker rather than the consumer should hold it
136        """
137        return (
138            self.broker_wait_threshold > ZERO
139            and delay > ZERO
140            and delay >= self.broker_wait_threshold
141        )
142
143    def broker_rungs(self) -> list[timedelta]:
144        """The delays this policy needs a rung queue for, longest last.
145
146        Exactly the entries of :meth:`schedule` that are at or above the
147        threshold, with repeats removed — a fixed policy that waits a minute
148        three times needs one queue, not three. It is a finite list because the
149        schedule is, which is what makes the queues declarable up front rather
150        than conjured by a consumer at the moment it first fails.
151
152        :returns: one delay per queue, in the order the schedule reaches them
153        """
154        return list(dict.fromkeys(d for d in self.schedule() if self.waits_in_broker(d)))
155
156    def next_delay(self, attempt: int, message_age: timedelta = ZERO) -> timedelta | None:
157        """How long to wait before the next attempt, or ``None`` to give up.
158
159        :param attempt: the attempt that has just failed, starting at 1
160        :param message_age: how old the message is
161        :returns: the delay, or ``None`` when there should be no further attempt
162        """
163        wait = self.next_wait(attempt, message_age)
164        return None if wait is None else wait.delay
165
166    def next_wait(self, attempt: int, message_age: timedelta = ZERO) -> Wait | None:
167        """The whole answer: how long to wait, and where.
168
169        :param attempt: the attempt that has just failed, starting at 1
170        :param message_age: how old the message is
171        :returns: the wait, or ``None`` when there should be no further attempt
172        """
173        if attempt >= self.max_attempts:
174            return None
175        if self.max_message_age > ZERO and message_age >= self.max_message_age:
176            return None
177
178        delay = self._unjittered(attempt)
179
180        if self.waits_in_broker(delay):
181            # Deliberately not jittered. A rung queue's TTL is fixed when it is
182            # declared, so a moved delay would name a queue that does not exist;
183            # and the spread jitter buys is already there, because each message's
184            # TTL starts when it arrives rather than when the batch failed.
185            return Wait(delay, in_broker=True)
186
187        if self.jitter_factor > 0 and delay > ZERO:
188            # Both directions, so a fleet of consumers that failed together
189            # does not come back together. One-sided jitter only ever delays,
190            # which turns a thundering herd into a slower thundering herd.
191            factor = 1 + ((random.random() * 2 - 1) * self.jitter_factor)
192            delay = delay * factor
193
194        return Wait(max(delay, ZERO), in_broker=False)
195
196    def _unjittered(self, attempt: int) -> timedelta:
197        """The delay after ``attempt``, before jitter and before the threshold."""
198        delay = self.initial_delay
199        for _ in range(1, attempt):
200            delay = delay * self.multiplier
201            if self.max_delay > ZERO and delay > self.max_delay:
202                delay = self.max_delay
203                break
204        if self.max_delay > ZERO and delay > self.max_delay:
205            delay = self.max_delay
206        return delay
207
208    def schedule(self) -> list[timedelta]:
209        """The delays this policy would use, without jitter.
210
211        What to look at when deciding whether a policy is the one you meant:
212        ``ExponentialRetry(5, 1s, 1m).schedule()`` is ``[1s, 2s, 4s, 8s]``, and
213        four numbers are easier to argue with than three parameters.
214        """
215        delays: list[timedelta] = []
216        delay = self.initial_delay
217        for _ in range(1, max(self.max_attempts, 0)):
218            capped = delay
219            if self.max_delay > ZERO and capped > self.max_delay:
220                capped = self.max_delay
221            delays.append(capped)
222            delay = delay * self.multiplier
223        return delays

How many times to try, how long to wait, and where.

Waits below broker_wait_threshold are spent in the consumer, which holds the delivery and one prefetch slot for the duration. Waits at or above it are spent in a {queue}acemq_amqp.retry.{delay} queue whose x-message-ttl is the wait and whose dead-letter target is the source queue, so the broker returns the message when the time is up and a consumer restart cannot shorten it.

Splitting at a threshold rather than picking one of the two takes the durability where it is worth its complexity and leaves the simplicity where it is not. A queue's TTL is fixed at declaration and so cannot express a jittered delay, which is why jitter applies below the threshold only; above it the spread comes free, because each message's TTL starts when it enters the rung, so a fleet that failed over ten seconds is released over ten seconds.

Parameters
  • max_attempts: total deliveries including the first. 1 means no retry
  • initial_delay: the wait before the second attempt
  • multiplier: what the delay is multiplied by each time
  • max_delay: the ceiling, or zero for none
  • jitter_factor: how far a delay may move either side, 0 to 1
  • max_message_age: give up on anything older, or zero for never
  • broker_wait_threshold: waits this long or longer are spent in the broker, or zero to spend every wait in the consumer
RetryPolicy( max_attempts: int = 1, initial_delay: datetime.timedelta = datetime.timedelta(0), multiplier: float = 2.0, max_delay: datetime.timedelta = datetime.timedelta(0), jitter_factor: float = 0.0, max_message_age: datetime.timedelta = datetime.timedelta(0), broker_wait_threshold: datetime.timedelta = datetime.timedelta(seconds=30))
max_attempts: int
initial_delay: datetime.timedelta
multiplier: float
max_delay: datetime.timedelta
jitter_factor: float
max_message_age: datetime.timedelta
broker_wait_threshold: datetime.timedelta
def give_up_after(self, age: datetime.timedelta) -> RetryPolicy:
103    def give_up_after(self, age: timedelta) -> RetryPolicy:
104        """A copy that abandons a message older than ``age``.
105
106        The honest limit when a queue has been paused: attempts say nothing
107        about how long a message has been waiting, and a message four days old
108        is usually one nobody wants delivered now.
109        """
110        return replace(self, max_message_age=age)

A copy that abandons a message older than age.

The honest limit when a queue has been paused: attempts say nothing about how long a message has been waiting, and a message four days old is usually one nobody wants delivered now.

def with_jitter(self, factor: float) -> RetryPolicy:
112    def with_jitter(self, factor: float) -> RetryPolicy:
113        """A copy using a different jitter factor, between 0 and 1."""
114        return replace(self, jitter_factor=factor)

A copy using a different jitter factor, between 0 and 1.

def wait_in_broker_from(self, threshold: datetime.timedelta) -> RetryPolicy:
116    def wait_in_broker_from(self, threshold: timedelta) -> RetryPolicy:
117        """A copy that moves the line between waiting here and waiting there.
118
119        Zero is the way out: with no threshold nothing is long enough to reach
120        the broker, so every wait is spent in the consumer and no rung queue is
121        needed. That is the right setting for a service whose broker it may not
122        declare queues on, and the wrong one for a policy with delays measured
123        in minutes.
124
125        :param threshold: waits this long or longer go to a rung queue, or zero
126            for none of them
127        :returns: a new policy
128        """
129        return replace(self, broker_wait_threshold=threshold)

A copy that moves the line between waiting here and waiting there.

Zero is the way out: with no threshold nothing is long enough to reach the broker, so every wait is spent in the consumer and no rung queue is needed. That is the right setting for a service whose broker it may not declare queues on, and the wrong one for a policy with delays measured in minutes.

Parameters
  • threshold: waits this long or longer go to a rung queue, or zero for none of them :returns: a new policy
def waits_in_broker(self, delay: datetime.timedelta) -> bool:
131    def waits_in_broker(self, delay: timedelta) -> bool:
132        """Whether a wait of this length belongs on a rung queue.
133
134        :param delay: an unjittered delay, as :meth:`schedule` reports them
135        :returns: whether the broker rather than the consumer should hold it
136        """
137        return (
138            self.broker_wait_threshold > ZERO
139            and delay > ZERO
140            and delay >= self.broker_wait_threshold
141        )

Whether a wait of this length belongs on a rung queue.

Parameters
  • delay: an unjittered delay, as schedule() reports them :returns: whether the broker rather than the consumer should hold it
def broker_rungs(self) -> list[datetime.timedelta]:
143    def broker_rungs(self) -> list[timedelta]:
144        """The delays this policy needs a rung queue for, longest last.
145
146        Exactly the entries of :meth:`schedule` that are at or above the
147        threshold, with repeats removed — a fixed policy that waits a minute
148        three times needs one queue, not three. It is a finite list because the
149        schedule is, which is what makes the queues declarable up front rather
150        than conjured by a consumer at the moment it first fails.
151
152        :returns: one delay per queue, in the order the schedule reaches them
153        """
154        return list(dict.fromkeys(d for d in self.schedule() if self.waits_in_broker(d)))

The delays this policy needs a rung queue for, longest last.

Exactly the entries of schedule() that are at or above the threshold, with repeats removed — a fixed policy that waits a minute three times needs one queue, not three. It is a finite list because the schedule is, which is what makes the queues declarable up front rather than conjured by a consumer at the moment it first fails.

:returns: one delay per queue, in the order the schedule reaches them

def next_delay( self, attempt: int, message_age: datetime.timedelta = datetime.timedelta(0)) -> datetime.timedelta | None:
156    def next_delay(self, attempt: int, message_age: timedelta = ZERO) -> timedelta | None:
157        """How long to wait before the next attempt, or ``None`` to give up.
158
159        :param attempt: the attempt that has just failed, starting at 1
160        :param message_age: how old the message is
161        :returns: the delay, or ``None`` when there should be no further attempt
162        """
163        wait = self.next_wait(attempt, message_age)
164        return None if wait is None else wait.delay

How long to wait before the next attempt, or None to give up.

Parameters
  • attempt: the attempt that has just failed, starting at 1
  • message_age: how old the message is :returns: the delay, or None when there should be no further attempt
def next_wait( self, attempt: int, message_age: datetime.timedelta = datetime.timedelta(0)) -> Wait | None:
166    def next_wait(self, attempt: int, message_age: timedelta = ZERO) -> Wait | None:
167        """The whole answer: how long to wait, and where.
168
169        :param attempt: the attempt that has just failed, starting at 1
170        :param message_age: how old the message is
171        :returns: the wait, or ``None`` when there should be no further attempt
172        """
173        if attempt >= self.max_attempts:
174            return None
175        if self.max_message_age > ZERO and message_age >= self.max_message_age:
176            return None
177
178        delay = self._unjittered(attempt)
179
180        if self.waits_in_broker(delay):
181            # Deliberately not jittered. A rung queue's TTL is fixed when it is
182            # declared, so a moved delay would name a queue that does not exist;
183            # and the spread jitter buys is already there, because each message's
184            # TTL starts when it arrives rather than when the batch failed.
185            return Wait(delay, in_broker=True)
186
187        if self.jitter_factor > 0 and delay > ZERO:
188            # Both directions, so a fleet of consumers that failed together
189            # does not come back together. One-sided jitter only ever delays,
190            # which turns a thundering herd into a slower thundering herd.
191            factor = 1 + ((random.random() * 2 - 1) * self.jitter_factor)
192            delay = delay * factor
193
194        return Wait(max(delay, ZERO), in_broker=False)

The whole answer: how long to wait, and where.

Parameters
  • attempt: the attempt that has just failed, starting at 1
  • message_age: how old the message is :returns: the wait, or None when there should be no further attempt
def schedule(self) -> list[datetime.timedelta]:
208    def schedule(self) -> list[timedelta]:
209        """The delays this policy would use, without jitter.
210
211        What to look at when deciding whether a policy is the one you meant:
212        ``ExponentialRetry(5, 1s, 1m).schedule()`` is ``[1s, 2s, 4s, 8s]``, and
213        four numbers are easier to argue with than three parameters.
214        """
215        delays: list[timedelta] = []
216        delay = self.initial_delay
217        for _ in range(1, max(self.max_attempts, 0)):
218            capped = delay
219            if self.max_delay > ZERO and capped > self.max_delay:
220                capped = self.max_delay
221            delays.append(capped)
222            delay = delay * self.multiplier
223        return delays

The delays this policy would use, without jitter.

What to look at when deciding whether a policy is the one you meant: ExponentialRetry(5, 1s, 1m).schedule() is [1s, 2s, 4s, 8s], and four numbers are easier to argue with than three parameters.

@dataclass(frozen=True, slots=True)
class Security:
279@dataclass(frozen=True, slots=True)
280class Security:
281    """How to verify the broker, what to present to it, and who to log in as.
282
283    Every field is optional and the default is the safe one: verify against the
284    machine's trust store, present no client certificate, and take the login
285    from the URL. Pass one to :func:`acemq_amqp.connect` or
286    :func:`acemq_amqp.sync.connect`.
287
288    A ``Security`` that carries TLS settings is refused against an ``amqp://``
289    URL rather than ignored — see this module's own documentation for why. One
290    that carries only ``credentials`` is fine on either scheme, because keeping
291    a password out of a connection string is worth doing whether or not the
292    connection is encrypted.
293
294    :param certificate_authority: a PEM file holding the authority to verify the
295        broker against. Naming one replaces the system trust store rather than
296        adding to it, which is the point: a broker with a certificate from a
297        public authority is not your broker, and the hundreds of authorities a
298        machine trusts by default are hundreds of ways to be wrong
299    :param client_certificate: a PEM certificate to present, for a broker that
300        authenticates clients by certificate rather than by password
301    :param client_key: the PEM private key for that certificate. Defaults to
302        reading the key out of ``client_certificate``, which is where it lives
303        when the two are in one file
304    :param client_key_password: the passphrase on that key, when it has one.
305        Kept out of this dataclass's :func:`repr` alongside the broker password
306    :param server_name: the name to check the certificate against, when it is
307        not the one in the URL. Needed for a broker reached by IP address,
308        through a tunnel, or at a container's internal name
309    :param credentials: the broker login, or a :data:`CredentialsSource` asked
310        for it at connection time. Overrides anything in the URL
311    :param verification: what to check about the broker's certificate
312    :param allow_development_certificates: whether to accept a certificate
313        stamped :data:`DEVELOPMENT_MARKER`. Off, and it should stay off anywhere
314        that is not a developer's machine: the signing key of a generated
315        authority is in whatever repository the certificates were committed to,
316        so trusting one means trusting everybody who can read it
317    """
318
319    certificate_authority: FilePath | None = None
320    client_certificate: FilePath | None = None
321    client_key: FilePath | None = None
322    client_key_password: str | None = field(default=None, repr=False)
323    server_name: str | None = None
324    credentials: Credentials | CredentialsSource | None = None
325    verification: Verification = Verification.CERTIFICATE
326    allow_development_certificates: bool = False
327
328    @property
329    def describes_tls(self) -> bool:
330        """Whether this says anything about an encrypted connection.
331
332        A ``Security`` that says nothing — one carrying credentials alone — is
333        as valid against ``amqp://`` as against ``amqps://``. One that does is
334        an instruction that a plaintext connection cannot carry out.
335        """
336        return (
337            self.certificate_authority is not None
338            or self.client_certificate is not None
339            or self.client_key is not None
340            or self.server_name is not None
341            or self.verification is not Verification.CERTIFICATE
342            or self.allow_development_certificates
343        )
344
345    def resolve_credentials(self) -> Credentials | None:
346        """Asks for the login, now.
347
348        :returns: the credentials, or ``None`` when this leaves the login to
349            the URL
350        :raises SecurityError: when a source was configured and could not
351            produce them
352        """
353        if self.credentials is None:
354            return None
355        if isinstance(self.credentials, Credentials):
356            return self.credentials
357        found = self.credentials()
358        if not isinstance(found, Credentials):
359            raise SecurityError(
360                "acemq: a credentials source must return Credentials, and this one "
361                f"returned {type(found).__name__}"
362            )
363        return found
364
365    def ssl_context(self) -> ssl.SSLContext:
366        """Builds the TLS context for a connection.
367
368        TLS 1.2 is the floor whatever the machine's ``openssl`` would otherwise
369        allow: 1.0 and 1.1 are withdrawn, and a broker still offering them is
370        not a reason to speak them.
371
372        :returns: the context, ready to hand to the transport
373        :raises SecurityError: when a certificate or key cannot be read, or when
374            one of them is a development certificate and this configuration did
375            not ask for those
376        """
377        context = self._new_context()
378        context.minimum_version = ssl.TLSVersion.TLSv1_2
379
380        if not self.allow_development_certificates:
381            # Two checks rather than one, because a development certificate can
382            # arrive from two directions. This is the near side: the files this
383            # configuration names. The far side — what the broker actually
384            # presents — is checked at the handshake, below, where it cannot be
385            # avoided by any trust setting.
386            self._refuse_development_file(self.certificate_authority, "certificate authority")
387            self._refuse_development_file(self.client_certificate, "client certificate")
388
389        if self.verification is Verification.NOTHING_AT_ALL:
390            # Both, and in this order: check_hostname cannot be left on once
391            # verify_mode is CERT_NONE, and Python raises rather than guessing
392            # which of the two was meant.
393            context.check_hostname = False
394            context.verify_mode = ssl.CERT_NONE
395        elif self.certificate_authority is not None:
396            # One authority, and no others. Verification succeeding therefore
397            # means the chain reached the authority named here — the system
398            # trust store is never consulted, which is what makes this stronger
399            # than the default rather than merely different from it.
400            authority = os.fspath(self.certificate_authority)
401            self._load(
402                lambda: context.load_verify_locations(cafile=authority),
403                f"the certificate authority {authority!r}",
404            )
405        else:
406            context.load_default_certs(ssl.Purpose.SERVER_AUTH)
407
408        if self.client_certificate is not None:
409            certificate = os.fspath(self.client_certificate)
410            key = None if self.client_key is None else os.fspath(self.client_key)
411            self._load(
412                lambda: context.load_cert_chain(
413                    certificate, keyfile=key, password=self.client_key_password
414                ),
415                f"the client certificate {certificate!r}"
416                + ("" if key is None else f" with key {key!r}"),
417            )
418        elif self.client_key is not None:
419            raise SecurityError(
420                "acemq: a client key was given with no client certificate; a broker "
421                "authenticates the certificate, and a key on its own proves nothing"
422            )
423
424        return context
425
426    def _new_context(self) -> ssl.SSLContext:
427        """A client context, naming a different broker when asked to.
428
429        ``PROTOCOL_TLS_CLIENT`` rather than :func:`ssl.create_default_context`
430        because the trust store is loaded deliberately a few lines further on:
431        the default context loads the system's, and a context that has already
432        trusted three hundred authorities cannot then be narrowed to one.
433        """
434        if self.server_name is None:
435            context = ssl.SSLContext(ssl.PROTOCOL_TLS_CLIENT)
436        else:
437            named = _ContextNamingTheBroker(ssl.PROTOCOL_TLS_CLIENT)
438            named.expected_name = self.server_name
439            context = named
440
441        if not self.allow_development_certificates:
442            # ``sslobject_class`` is the documented seam for this, and it is the
443            # only one: :mod:`ssl` has no verify callback of the sort Go's
444            # ``VerifyPeerCertificate`` and Java's ``X509TrustManager`` give,
445            # and there is nowhere else a check can be put that a trust setting
446            # cannot then turn off. Both classes are replaced because aio-pika
447            # goes through ``wrap_bio`` and a blocking client would go through
448            # ``wrap_socket``.
449            context.sslobject_class = _RefusingSSLObject
450            context.sslsocket_class = _RefusingSSLSocket
451        return context
452
453    @staticmethod
454    def _refuse_development_file(path: FilePath | None, what: str) -> None:
455        """Refuses a configured certificate file that carries the marker.
456
457        Reading the file is worth it for the error alone: at the handshake all
458        that can be said is that the broker presented one, whereas here the
459        message can name the setting and the path, which is what somebody
460        actually has to change.
461        """
462        if path is None:
463            return
464        where = os.fspath(path)
465        try:
466            with open(where, "rb") as handle:
467                contents = handle.read()
468        except OSError:
469            # Not this method's failure to report: the loader a few lines on
470            # opens the same file and says what is wrong with it far better.
471            return
472        if is_development_certificate(contents):
473            raise SecurityError(
474                f"acemq: the {what} {where!r} is marked {DEVELOPMENT_MARKER!r}. It was "
475                "generated for development and its signing key is not a secret, so it is "
476                "not trusted here. Use a real certificate, or say so deliberately with "
477                "Security(allow_development_certificates=True)"
478            )
479
480    @staticmethod
481    def _load(read: Callable[[], None], what: str) -> None:
482        """Runs one of :mod:`ssl`'s loaders and says what failed in words.
483
484        ``ssl`` reports a missing file as ``FileNotFoundError`` and a malformed
485        one as ``[SSL] PEM lib``, neither of which names the setting that was
486        wrong. This does.
487        """
488        try:
489            read()
490        except (OSError, ssl.SSLError) as failure:
491            raise SecurityError(f"acemq: cannot load {what}: {failure}") from failure
492
493    def applied_to(self, url: str) -> str:
494        """The URL this configuration would actually connect to.
495
496        Credentials given here replace whatever the URL carried, which is how a
497        password stays out of a connection string. When none were given the URL
498        comes back as it was.
499
500        :param url: where the broker is
501        :returns: the URL to hand to the transport
502        :raises SecurityError: when TLS settings were given for an ``amqp://``
503            URL, which cannot honour them
504        """
505        scheme = urlsplit(url).scheme
506        if self.describes_tls and scheme != "amqps":
507            raise SecurityError(
508                f"acemq: this Security configures TLS but the URL is {scheme or 'unset'}://, "
509                "which is not encrypted. Nothing here can make a plaintext connection safe: "
510                "change the URL to amqps:// or take the TLS settings off"
511            )
512
513        credentials = self.resolve_credentials()
514        if credentials is None or credentials.empty:
515            return url
516        return _with_credentials(url, credentials)
517
518    def transport_options(self, url: str) -> dict[str, Any]:
519        """What the transport needs, beyond the URL.
520
521        :param url: where the broker is, so that a plaintext connection is given
522            no context at all rather than one it would silently drop
523        :returns: keyword arguments for :func:`aio_pika.connect_robust`
524        """
525        if urlsplit(url).scheme != "amqps":
526            return {}
527        return {"ssl_context": self.ssl_context()}

How to verify the broker, what to present to it, and who to log in as.

Every field is optional and the default is the safe one: verify against the machine's trust store, present no client certificate, and take the login from the URL. Pass one to acemq_amqp.connect() or acemq_amqp.sync.connect().

A Security that carries TLS settings is refused against an amqp:// URL rather than ignored — see this module's own documentation for why. One that carries only credentials is fine on either scheme, because keeping a password out of a connection string is worth doing whether or not the connection is encrypted.

Parameters
  • certificate_authority: a PEM file holding the authority to verify the broker against. Naming one replaces the system trust store rather than adding to it, which is the point: a broker with a certificate from a public authority is not your broker, and the hundreds of authorities a machine trusts by default are hundreds of ways to be wrong
  • client_certificate: a PEM certificate to present, for a broker that authenticates clients by certificate rather than by password
  • client_key: the PEM private key for that certificate. Defaults to reading the key out of client_certificate, which is where it lives when the two are in one file
  • client_key_password: the passphrase on that key, when it has one. Kept out of this dataclass's repr() alongside the broker password
  • server_name: the name to check the certificate against, when it is not the one in the URL. Needed for a broker reached by IP address, through a tunnel, or at a container's internal name
  • credentials: the broker login, or a CredentialsSource asked for it at connection time. Overrides anything in the URL
  • verification: what to check about the broker's certificate
  • allow_development_certificates: whether to accept a certificate stamped DEVELOPMENT_MARKER. Off, and it should stay off anywhere that is not a developer's machine: the signing key of a generated authority is in whatever repository the certificates were committed to, so trusting one means trusting everybody who can read it
Security( certificate_authority: 'FilePath | None' = None, client_certificate: 'FilePath | None' = None, client_key: 'FilePath | None' = None, client_key_password: str | None = None, server_name: str | None = None, credentials: Credentials | Callable[Credentials] | None = None, verification: Verification = <Verification.CERTIFICATE: 'certificate'>, allow_development_certificates: bool = False)
certificate_authority: 'FilePath | None'
client_certificate: 'FilePath | None'
client_key: 'FilePath | None'
client_key_password: str | None
server_name: str | None
credentials: Credentials | Callable[Credentials] | None
verification: Verification
allow_development_certificates: bool
describes_tls: bool
328    @property
329    def describes_tls(self) -> bool:
330        """Whether this says anything about an encrypted connection.
331
332        A ``Security`` that says nothing — one carrying credentials alone — is
333        as valid against ``amqp://`` as against ``amqps://``. One that does is
334        an instruction that a plaintext connection cannot carry out.
335        """
336        return (
337            self.certificate_authority is not None
338            or self.client_certificate is not None
339            or self.client_key is not None
340            or self.server_name is not None
341            or self.verification is not Verification.CERTIFICATE
342            or self.allow_development_certificates
343        )

Whether this says anything about an encrypted connection.

A Security that says nothing — one carrying credentials alone — is as valid against amqp:// as against amqps://. One that does is an instruction that a plaintext connection cannot carry out.

def resolve_credentials(self) -> Credentials | None:
345    def resolve_credentials(self) -> Credentials | None:
346        """Asks for the login, now.
347
348        :returns: the credentials, or ``None`` when this leaves the login to
349            the URL
350        :raises SecurityError: when a source was configured and could not
351            produce them
352        """
353        if self.credentials is None:
354            return None
355        if isinstance(self.credentials, Credentials):
356            return self.credentials
357        found = self.credentials()
358        if not isinstance(found, Credentials):
359            raise SecurityError(
360                "acemq: a credentials source must return Credentials, and this one "
361                f"returned {type(found).__name__}"
362            )
363        return found

Asks for the login, now.

:returns: the credentials, or None when this leaves the login to the URL

Raises
  • SecurityError: when a source was configured and could not produce them
def ssl_context(self) -> ssl.SSLContext:
365    def ssl_context(self) -> ssl.SSLContext:
366        """Builds the TLS context for a connection.
367
368        TLS 1.2 is the floor whatever the machine's ``openssl`` would otherwise
369        allow: 1.0 and 1.1 are withdrawn, and a broker still offering them is
370        not a reason to speak them.
371
372        :returns: the context, ready to hand to the transport
373        :raises SecurityError: when a certificate or key cannot be read, or when
374            one of them is a development certificate and this configuration did
375            not ask for those
376        """
377        context = self._new_context()
378        context.minimum_version = ssl.TLSVersion.TLSv1_2
379
380        if not self.allow_development_certificates:
381            # Two checks rather than one, because a development certificate can
382            # arrive from two directions. This is the near side: the files this
383            # configuration names. The far side — what the broker actually
384            # presents — is checked at the handshake, below, where it cannot be
385            # avoided by any trust setting.
386            self._refuse_development_file(self.certificate_authority, "certificate authority")
387            self._refuse_development_file(self.client_certificate, "client certificate")
388
389        if self.verification is Verification.NOTHING_AT_ALL:
390            # Both, and in this order: check_hostname cannot be left on once
391            # verify_mode is CERT_NONE, and Python raises rather than guessing
392            # which of the two was meant.
393            context.check_hostname = False
394            context.verify_mode = ssl.CERT_NONE
395        elif self.certificate_authority is not None:
396            # One authority, and no others. Verification succeeding therefore
397            # means the chain reached the authority named here — the system
398            # trust store is never consulted, which is what makes this stronger
399            # than the default rather than merely different from it.
400            authority = os.fspath(self.certificate_authority)
401            self._load(
402                lambda: context.load_verify_locations(cafile=authority),
403                f"the certificate authority {authority!r}",
404            )
405        else:
406            context.load_default_certs(ssl.Purpose.SERVER_AUTH)
407
408        if self.client_certificate is not None:
409            certificate = os.fspath(self.client_certificate)
410            key = None if self.client_key is None else os.fspath(self.client_key)
411            self._load(
412                lambda: context.load_cert_chain(
413                    certificate, keyfile=key, password=self.client_key_password
414                ),
415                f"the client certificate {certificate!r}"
416                + ("" if key is None else f" with key {key!r}"),
417            )
418        elif self.client_key is not None:
419            raise SecurityError(
420                "acemq: a client key was given with no client certificate; a broker "
421                "authenticates the certificate, and a key on its own proves nothing"
422            )
423
424        return context

Builds the TLS context for a connection.

TLS 1.2 is the floor whatever the machine's openssl would otherwise allow: 1.0 and 1.1 are withdrawn, and a broker still offering them is not a reason to speak them.

:returns: the context, ready to hand to the transport

Raises
  • SecurityError: when a certificate or key cannot be read, or when one of them is a development certificate and this configuration did not ask for those
def applied_to(self, url: str) -> str:
493    def applied_to(self, url: str) -> str:
494        """The URL this configuration would actually connect to.
495
496        Credentials given here replace whatever the URL carried, which is how a
497        password stays out of a connection string. When none were given the URL
498        comes back as it was.
499
500        :param url: where the broker is
501        :returns: the URL to hand to the transport
502        :raises SecurityError: when TLS settings were given for an ``amqp://``
503            URL, which cannot honour them
504        """
505        scheme = urlsplit(url).scheme
506        if self.describes_tls and scheme != "amqps":
507            raise SecurityError(
508                f"acemq: this Security configures TLS but the URL is {scheme or 'unset'}://, "
509                "which is not encrypted. Nothing here can make a plaintext connection safe: "
510                "change the URL to amqps:// or take the TLS settings off"
511            )
512
513        credentials = self.resolve_credentials()
514        if credentials is None or credentials.empty:
515            return url
516        return _with_credentials(url, credentials)

The URL this configuration would actually connect to.

Credentials given here replace whatever the URL carried, which is how a password stays out of a connection string. When none were given the URL comes back as it was.

Parameters
  • url: where the broker is :returns: the URL to hand to the transport
Raises
  • **SecurityError: when TLS settings were given for an amqp**: // URL, which cannot honour them
def transport_options(self, url: str) -> dict[str, typing.Any]:
518    def transport_options(self, url: str) -> dict[str, Any]:
519        """What the transport needs, beyond the URL.
520
521        :param url: where the broker is, so that a plaintext connection is given
522            no context at all rather than one it would silently drop
523        :returns: keyword arguments for :func:`aio_pika.connect_robust`
524        """
525        if urlsplit(url).scheme != "amqps":
526            return {}
527        return {"ssl_context": self.ssl_context()}

What the transport needs, beyond the URL.

Parameters
  • url: where the broker is, so that a plaintext connection is given no context at all rather than one it would silently drop :returns: keyword arguments for aio_pika.connect_robust()
class SecurityError(acemq_amqp.AceMQError):
35class SecurityError(AceMQError):
36    """A security setting that cannot be honoured.
37
38    Raised before anything is dialled rather than after: a missing certificate
39    file, an environment variable that was going to hold the password, or TLS
40    settings against a URL that is not encrypted are all things to hear about at
41    start-up. A connection that half-applied them and carried on is the failure
42    this exists to prevent.
43    """

A security setting that cannot be honoured.

Raised before anything is dialled rather than after: a missing certificate file, an environment variable that was going to hold the password, or TLS settings against a URL that is not encrypted are all things to hear about at start-up. A connection that half-applied them and carried on is the failure this exists to prevent.

@dataclass(frozen=True, slots=True)
class Settlement:
 91@dataclass(frozen=True, slots=True)
 92class Settlement:
 93    """What the consumer did with a delivery, once it had decided.
 94
 95    Reported before it is carried out rather than after, because one of them
 96    takes time: a consumer-side retry sleeps out the backoff before the message
 97    is republished, and a listener told afterwards would be describing the
 98    delivery minutes later, on a span that had been open the whole while.
 99
100    :param outcome: one of :data:`OUTCOME_ACKED`, :data:`OUTCOME_RETRIED`,
101        :data:`OUTCOME_REJECTED`, :data:`OUTCOME_DEAD_LETTERED` and
102        :data:`OUTCOME_PARKED` — the same words the other libraries put on a
103        span
104    :param reason: why it was set aside, when it was
105    :param delay: how long until the next attempt, when there is one
106    """
107
108    outcome: str
109    reason: str | None = None
110    delay: timedelta | None = None
111
112    @property
113    def dead_lettered(self) -> bool:
114        """Whether the message went to the dead-letter queue.
115
116        True for an outright rejection as well as for an exhausted one: both
117        end up in the same queue, and the difference between the two words is
118        *who decided*, which :attr:`outcome` records and this does not.
119
120        False for :data:`OUTCOME_PARKED`, and that is the whole point of the
121        parked queue: a message nobody could read is a different problem with a
122        different answer from one that failed five times, and whoever drains
123        the dead letters should not have to sort the two apart by hand.
124        """
125        return self.outcome in (OUTCOME_REJECTED, OUTCOME_DEAD_LETTERED)
126
127    @property
128    def parked(self) -> bool:
129        """Whether the message went to the parked queue."""
130        return self.outcome == OUTCOME_PARKED
131
132    def __str__(self) -> str:
133        return self.outcome

What the consumer did with a delivery, once it had decided.

Reported before it is carried out rather than after, because one of them takes time: a consumer-side retry sleeps out the backoff before the message is republished, and a listener told afterwards would be describing the delivery minutes later, on a span that had been open the whole while.

Parameters
  • outcome: one of OUTCOME_ACKED, OUTCOME_RETRIED, OUTCOME_REJECTED, OUTCOME_DEAD_LETTERED and OUTCOME_PARKED — the same words the other libraries put on a span
  • reason: why it was set aside, when it was
  • delay: how long until the next attempt, when there is one
Settlement( outcome: str, reason: str | None = None, delay: datetime.timedelta | None = None)
outcome: str
reason: str | None
delay: datetime.timedelta | None
dead_lettered: bool
112    @property
113    def dead_lettered(self) -> bool:
114        """Whether the message went to the dead-letter queue.
115
116        True for an outright rejection as well as for an exhausted one: both
117        end up in the same queue, and the difference between the two words is
118        *who decided*, which :attr:`outcome` records and this does not.
119
120        False for :data:`OUTCOME_PARKED`, and that is the whole point of the
121        parked queue: a message nobody could read is a different problem with a
122        different answer from one that failed five times, and whoever drains
123        the dead letters should not have to sort the two apart by hand.
124        """
125        return self.outcome in (OUTCOME_REJECTED, OUTCOME_DEAD_LETTERED)

Whether the message went to the dead-letter queue.

True for an outright rejection as well as for an exhausted one: both end up in the same queue, and the difference between the two words is who decided, which outcome records and this does not.

False for OUTCOME_PARKED, and that is the whole point of the parked queue: a message nobody could read is a different problem with a different answer from one that failed five times, and whoever drains the dead letters should not have to sort the two apart by hand.

parked: bool
127    @property
128    def parked(self) -> bool:
129        """Whether the message went to the parked queue."""
130        return self.outcome == OUTCOME_PARKED

Whether the message went to the parked queue.

SettlementListener = collections.abc.Callable[[Settlement], None]
class TextCodec:
184class TextCodec:
185    """Reads and writes text.
186
187    For messages that really are text — a line of a log, a command somebody
188    typed — rather than a structure that happens to be readable. Prefer
189    :class:`JsonCodec` for anything with fields.
190    """
191
192    @property
193    def content_type(self) -> str:
194        return TEXT_CONTENT_TYPE
195
196    def encode(self, payload: Any) -> bytes:
197        if isinstance(payload, str):
198            return payload.encode("utf-8")
199        if isinstance(payload, bytes):
200            return payload
201        raise TypeError(
202            f"acemq: TextCodec cannot encode a {type(payload).__name__}; "
203            "it takes a string or bytes"
204        )
205
206    def decode(self, body: bytes, content_type: str | None = None) -> Any:
207        try:
208            return body.decode("utf-8")
209        except UnicodeDecodeError as failure:
210            raise FatalError(f"acemq: this message is not UTF-8 text: {failure}") from failure
211
212    def can_decode(self, content_type: str | None) -> bool:
213        """Accepts ``text/*`` and nothing else.
214
215        Unlike :class:`JsonCodec` it does not answer for a message with no
216        content type, because there is no reason to think an untyped message is
217        text — and a codec that decodes anything into a string never fails, so
218        it would quietly hand a handler the printable rendering of a body it
219        should have refused.
220        """
221        return content_type is not None and content_type.lower().startswith("text/")

Reads and writes text.

For messages that really are text — a line of a log, a command somebody typed — rather than a structure that happens to be readable. Prefer JsonCodec for anything with fields.

content_type: str
192    @property
193    def content_type(self) -> str:
194        return TEXT_CONTENT_TYPE
def encode(self, payload: Any) -> bytes:
196    def encode(self, payload: Any) -> bytes:
197        if isinstance(payload, str):
198            return payload.encode("utf-8")
199        if isinstance(payload, bytes):
200            return payload
201        raise TypeError(
202            f"acemq: TextCodec cannot encode a {type(payload).__name__}; "
203            "it takes a string or bytes"
204        )
def decode(self, body: bytes, content_type: str | None = None) -> Any:
206    def decode(self, body: bytes, content_type: str | None = None) -> Any:
207        try:
208            return body.decode("utf-8")
209        except UnicodeDecodeError as failure:
210            raise FatalError(f"acemq: this message is not UTF-8 text: {failure}") from failure
def can_decode(self, content_type: str | None) -> bool:
212    def can_decode(self, content_type: str | None) -> bool:
213        """Accepts ``text/*`` and nothing else.
214
215        Unlike :class:`JsonCodec` it does not answer for a message with no
216        content type, because there is no reason to think an untyped message is
217        text — and a codec that decodes anything into a string never fails, so
218        it would quietly hand a handler the printable rendering of a body it
219        should have refused.
220        """
221        return content_type is not None and content_type.lower().startswith("text/")

Accepts text/* and nothing else.

Unlike JsonCodec it does not answer for a message with no content type, because there is no reason to think an untyped message is text — and a codec that decodes anything into a string never fails, so it would quietly hand a handler the printable rendering of a body it should have refused.

@dataclass
class Topology:
188@dataclass
189class Topology:
190    """A description of what a service needs a broker to have.
191
192    Every method that adds something returns the topology, so a description
193    reads as one expression. The order things are added in does not matter:
194    :meth:`apply` declares exchanges first, then queues, then the bindings
195    between them, because that is the order a broker will accept them in.
196    """
197
198    _exchanges: list[_NamedExchange] = field(default_factory=list)
199    _queues: list[_NamedQueue] = field(default_factory=list)
200    _bindings: list[Binding] = field(default_factory=list)
201
202    def exchange(
203        self,
204        name: str,
205        kind: str = "topic",
206        *,
207        durable: bool = True,
208        auto_delete: bool = False,
209        args: Mapping[str, Any] | None = None,
210    ) -> Topology:
211        """Adds an exchange.
212
213        :param name: what to call it
214        :param kind: ``direct``, ``topic``, ``fanout`` or ``headers``
215        :param durable: survives a broker restart
216        :param auto_delete: goes away when its last binding does
217        :param args: broker-specific arguments
218        :returns: this topology
219        """
220        if not name:
221            raise ValueError("acemq: an exchange needs a name")
222        if not kind:
223            raise ValueError(f"acemq: exchange {name!r} needs a kind (direct, topic, fanout)")
224        self._exchanges.append(
225            _NamedExchange(
226                name,
227                ExchangeSpec(
228                    kind=kind,
229                    durable=durable,
230                    auto_delete=auto_delete,
231                    args=dict(args or {}),
232                ),
233            )
234        )
235        return self
236
237    def queue(
238        self,
239        name: str,
240        *,
241        durable: bool = True,
242        auto_delete: bool = False,
243        exclusive: bool = False,
244        quorum: bool | None = None,
245        dead_letter: bool = False,
246        retry: RetryPolicy | None = None,
247        args: Mapping[str, Any] | None = None,
248    ) -> Topology:
249        """Adds a queue, and the queues that catch what it cannot handle.
250
251        A durable queue is a **quorum** queue unless it is asked to be something
252        else. That is what Java has always declared and what the other libraries
253        declare now, and it has to be the same everywhere: the queue type is one
254        more argument the broker compares, so a Java service and a Python
255        service that disagree about ``orders`` cannot both consume it. Pass
256        ``quorum=False`` for a classic queue, which is declared by leaving
257        ``x-queue-type`` off entirely — the spelling every AceMQ library uses,
258        and therefore the only one a broker finds equivalent to theirs.
259
260        ``exclusive``, ``auto_delete`` and ``durable=False`` all mean a queue
261        that belongs to one connection, and RabbitMQ refuses a quorum queue that
262        is any of those. Such a queue is declared classic without being asked,
263        because the alternative is a declaration the broker rejects with a
264        message that never mentions the word quorum. Asking for both — a queue
265        that is ``quorum=True`` and exclusive — is refused here instead, where
266        the contradiction is written down.
267
268        A queue whose ``args`` already name a kind keeps it: that is how
269        :func:`acemq_amqp.patterns.stream` declares a stream, and a caller who
270        wrote ``x-queue-type`` down meant it. Writing it down *and* passing
271        ``quorum`` is two answers to one question and is refused.
272
273        ``dead_letter`` declares ``{name}.dlq`` alongside it and points the
274        broker at it, so a message this queue rejects or lets expire lands
275        somewhere an operator can find by name rather than disappearing. It
276        declares ``{name}.parked`` too, because a consumer on this queue sends
277        anything it cannot decode there — and a library that parks messages into
278        a queue nobody declared has only moved the disappearance somewhere else.
279
280        Neither dead-letters itself: a dead-letter queue that dead-letters is a
281        loop, and a loop is how a poison message becomes an outage. Both are
282        reached through :data:`DEAD_LETTER_EXCHANGE`, bound on their own names,
283        which is what Java has always done and is therefore what a broker shared
284        with a Java service already has. A service that wants its own
285        dead-letter exchange instead should leave ``dead_letter`` alone and pass
286        the arguments in ``args``.
287
288        ``retry`` declares the rung queues that policy's long waits need: one
289        ``{name}.retry.{delay}`` per distinct delay at or above the policy's
290        threshold, each declared with :func:`rung_args` — the delay as
291        ``x-message-ttl``, :data:`RETRY_EXCHANGE` as the dead-letter target, and
292        this queue as the routing key. The binding that brings an expired
293        message home, ``{name}`` to :data:`RETRY_EXCHANGE` on ``{name}``, is
294        made here and not left to a caller to remember: without it every rung
295        expires into an exchange that routes nowhere, and a retry that is
296        dropped by the broker looks exactly like one that is still waiting.
297
298        It takes the policy rather than a list of delays on purpose. The rungs a
299        consumer will publish to are derived from the policy it is running, so
300        anything else here would be a second copy of the same list, free to drift
301        from the first — and the way that drift shows up is a retry published to
302        a queue nobody declared, at the moment the service is already failing.
303        One value produces both, or the topology is not a description of what the
304        service needs.
305
306        :param name: what to call it
307        :param durable: survives a broker restart
308        :param auto_delete: goes away when its last consumer does
309        :param exclusive: usable only by the connection that declared it
310        :param quorum: replicated rather than classic. ``None``, the default,
311            means quorum for a durable queue and classic for one belonging to a
312            single connection
313        :param dead_letter: also declare and wire ``{name}.dlq``, and declare
314            ``{name}.parked``
315        :param retry: also declare the rung queues this policy's long waits use
316        :param args: broker-specific arguments
317        :returns: this topology
318        :raises ValueError: when the queue cannot be what it was asked to be
319        """
320        if not name:
321            raise ValueError("acemq: a queue needs a name")
322
323        arguments: dict[str, Any] = dict(args or {})
324        kind = _queue_type(
325            name,
326            arguments,
327            durable=durable,
328            auto_delete=auto_delete,
329            exclusive=exclusive,
330            quorum=quorum,
331        )
332        if kind is not None:
333            arguments[QUEUE_TYPE_ARG] = kind
334        if dead_letter:
335            # Refused rather than resolved, because either answer would be a
336            # guess about which of two conflicting instructions was meant.
337            conflicting = sorted(
338                key
339                for key in arguments
340                if key in (DEAD_LETTER_EXCHANGE_ARG, DEAD_LETTER_ROUTING_KEY_ARG)
341            )
342            if conflicting:
343                raise ValueError(
344                    f"acemq: queue {name!r} asks for dead_letter=True and also sets "
345                    + ", ".join(conflicting)
346                    + "; pick one"
347                )
348            arguments[DEAD_LETTER_EXCHANGE_ARG] = DEAD_LETTER_EXCHANGE
349            arguments[DEAD_LETTER_ROUTING_KEY_ARG] = naming.dead_letter_queue(name)
350
351        self._queues.append(
352            _NamedQueue(
353                name,
354                QueueSpec(
355                    durable=durable,
356                    auto_delete=auto_delete,
357                    exclusive=exclusive,
358                    args=arguments,
359                ),
360            )
361        )
362        if dead_letter:
363            self._managed_exchange(DEAD_LETTER_EXCHANGE)
364            # Classic, deliberately, whatever the queue in front of them is.
365            # Java declares both QueueType.CLASSIC and these are shared names:
366            # a dead-letter queue nobody consumes gains nothing from being
367            # replicated, and gaining it here would cost a Java service its
368            # declaration of the same queue.
369            for target in (naming.dead_letter_queue(name), naming.parked_queue(name)):
370                self._queues.append(_NamedQueue(target, QueueSpec(durable=durable)))
371                self._bindings.append(
372                    Binding(queue=target, exchange=DEAD_LETTER_EXCHANGE, routing_key=target)
373                )
374        if retry is not None:
375            rungs = retry.broker_rungs()
376            if rungs:
377                self._managed_exchange(RETRY_EXCHANGE)
378                # Classic too, and for the same reason as ``.dlq``: a rung is
379                # declared by every service consuming the source queue, Java
380                # declares it QueueType.CLASSIC, and :func:`rung_args` is the
381                # whole argument table — three keys, no ``x-queue-type``.
382                for rung in rungs:
383                    self._queues.append(
384                        _NamedQueue(
385                            naming.retry_queue(name, rung),
386                            QueueSpec(durable=durable, args=rung_args(name, rung)),
387                        )
388                    )
389                # One binding brings every expired message back to the queue it
390                # came from. Made here rather than offered as something to add,
391                # because a missing binding loses every retry silently: the
392                # broker drops what a direct exchange cannot route and says
393                # nothing, so the queue looks quiet rather than broken.
394                self._bindings.append(
395                    Binding(queue=name, exchange=RETRY_EXCHANGE, routing_key=name)
396                )
397        return self
398
399    def _managed_exchange(self, name: str) -> None:
400        """Declares one of this library's own exchanges, once.
401
402        Several queues in one topology each need :data:`RETRY_EXCHANGE` and
403        :data:`DEAD_LETTER_EXCHANGE`, and declaring either twice is what
404        :meth:`validate` refuses. A caller who has already named the same
405        exchange keeps their own spec: theirs is the deliberate one.
406        """
407        if any(entry.name == name for entry in self._exchanges):
408            return
409        self._exchanges.append(_NamedExchange(name, _MANAGED_EXCHANGE))
410
411    def binding(self, queue: str, exchange: str, routing_key: str = "") -> Topology:
412        """Routes messages matching a key from an exchange to a queue.
413
414        :param queue: where messages end up, which this topology must declare
415        :param exchange: where they come from, which this topology must declare
416        :param routing_key: what to match, empty for a fanout
417        :returns: this topology
418        """
419        self._bindings.append(Binding(queue=queue, exchange=exchange, routing_key=routing_key))
420        return self
421
422    @property
423    def queues(self) -> list[str]:
424        """The queue names this topology declares, in declaration order."""
425        return [entry.name for entry in self._queues]
426
427    @property
428    def exchanges(self) -> list[str]:
429        """The exchange names this topology declares, in declaration order."""
430        return [entry.name for entry in self._exchanges]
431
432    @property
433    def bindings(self) -> list[Binding]:
434        """The bindings this topology declares."""
435        return list(self._bindings)
436
437    def validate(self) -> None:
438        """Reports what is wrong with the description itself.
439
440        A binding naming a queue the topology does not declare is the mistake
441        worth catching here. The broker would accept it if the queue happened to
442        exist already, and the service would then depend on something nothing
443        declares — which works until the day it is deployed somewhere new.
444
445        :raises ValueError: when the topology cannot be right
446        """
447        seen_queues: set[str] = set()
448        for queue in self._queues:
449            if queue.name in seen_queues:
450                raise ValueError(f"acemq: the topology declares queue {queue.name!r} twice")
451            seen_queues.add(queue.name)
452
453        seen_exchanges: set[str] = set()
454        for exchange in self._exchanges:
455            if exchange.name in seen_exchanges:
456                raise ValueError(
457                    f"acemq: the topology declares exchange {exchange.name!r} twice"
458                )
459            seen_exchanges.add(exchange.name)
460
461        for binding in self._bindings:
462            if binding.queue not in seen_queues:
463                raise ValueError(
464                    f"acemq: binding {binding} names queue {binding.queue!r}, "
465                    "which this topology does not declare"
466                )
467            if not binding.exchange:
468                raise ValueError(
469                    f"acemq: binding {binding} names the default exchange, "
470                    "which cannot be bound to"
471                )
472            if binding.exchange not in seen_exchanges:
473                raise ValueError(
474                    f"acemq: binding {binding} names exchange {binding.exchange!r}, "
475                    "which this topology does not declare"
476                )
477
478    def plan(self) -> list[PlanAction]:
479        """What :meth:`apply` would do, without doing it.
480
481        Deliberately not a difference against the live broker: AMQP offers no
482        way to enumerate what is there without the management API, and a plan
483        that quietly guessed would be worse than one that is honest about being
484        a statement of intent.
485
486        :returns: one action per thing that would be declared
487        :raises ValueError: when the topology cannot be right
488        """
489        self.validate()
490        actions = [
491            PlanAction("exchange", entry.name, _describe(entry.spec))
492            for entry in self._exchanges
493        ]
494        actions += [
495            PlanAction("queue", entry.name, _describe(entry.spec)) for entry in self._queues
496        ]
497        actions += [
498            PlanAction(
499                "binding", binding.queue, f"from {binding.exchange} on {binding.routing_key}"
500            )
501            for binding in self._bindings
502        ]
503        return actions
504
505    async def apply(self, transport: Transport) -> None:
506        """Declares everything, in the order a broker needs.
507
508        It stops at the first failure. A queue that already exists with
509        different settings is refused by the broker with ``PRECONDITION_FAILED``,
510        and that refusal is passed on rather than swallowed: it means this
511        service and the broker disagree about what the queue is, and carrying on
512        would leave the service using a queue that is not the one it asked for.
513
514        :param transport: where to declare it
515        :raises ValueError: when the topology cannot be right
516        """
517        self.validate()
518        for exchange in self._exchanges:
519            await transport.declare_exchange(exchange.name, exchange.spec)
520        for queue in self._queues:
521            await transport.declare_queue(queue.name, queue.spec)
522        for binding in self._bindings:
523            await transport.bind(binding.queue, binding.exchange, binding.routing_key)
524
525    def __str__(self) -> str:
526        try:
527            actions = self.plan()
528        except ValueError as failure:
529            return f"Topology(invalid: {failure})"
530        header = (
531            f"Topology: {len(self._exchanges)} exchanges, "
532            f"{len(self._queues)} queues, {len(self._bindings)} bindings"
533        )
534        return "\n".join([header, *(f"  {action}" for action in actions)])

A description of what a service needs a broker to have.

Every method that adds something returns the topology, so a description reads as one expression. The order things are added in does not matter: apply() declares exchanges first, then queues, then the bindings between them, because that is the order a broker will accept them in.

Topology( _exchanges: list[acemq_amqp.topology._NamedExchange] = <factory>, _queues: list[acemq_amqp.topology._NamedQueue] = <factory>, _bindings: list[acemq_amqp.topology.Binding] = <factory>)
def exchange( self, name: str, kind: str = 'topic', *, durable: bool = True, auto_delete: bool = False, args: Mapping[str, typing.Any] | None = None) -> Topology:
202    def exchange(
203        self,
204        name: str,
205        kind: str = "topic",
206        *,
207        durable: bool = True,
208        auto_delete: bool = False,
209        args: Mapping[str, Any] | None = None,
210    ) -> Topology:
211        """Adds an exchange.
212
213        :param name: what to call it
214        :param kind: ``direct``, ``topic``, ``fanout`` or ``headers``
215        :param durable: survives a broker restart
216        :param auto_delete: goes away when its last binding does
217        :param args: broker-specific arguments
218        :returns: this topology
219        """
220        if not name:
221            raise ValueError("acemq: an exchange needs a name")
222        if not kind:
223            raise ValueError(f"acemq: exchange {name!r} needs a kind (direct, topic, fanout)")
224        self._exchanges.append(
225            _NamedExchange(
226                name,
227                ExchangeSpec(
228                    kind=kind,
229                    durable=durable,
230                    auto_delete=auto_delete,
231                    args=dict(args or {}),
232                ),
233            )
234        )
235        return self

Adds an exchange.

Parameters
  • name: what to call it
  • kind: direct, topic, fanout or headers
  • durable: survives a broker restart
  • auto_delete: goes away when its last binding does
  • args: broker-specific arguments :returns: this topology
def queue( self, name: str, *, durable: bool = True, auto_delete: bool = False, exclusive: bool = False, quorum: bool | None = None, dead_letter: bool = False, retry: RetryPolicy | None = None, args: Mapping[str, typing.Any] | None = None) -> Topology:
237    def queue(
238        self,
239        name: str,
240        *,
241        durable: bool = True,
242        auto_delete: bool = False,
243        exclusive: bool = False,
244        quorum: bool | None = None,
245        dead_letter: bool = False,
246        retry: RetryPolicy | None = None,
247        args: Mapping[str, Any] | None = None,
248    ) -> Topology:
249        """Adds a queue, and the queues that catch what it cannot handle.
250
251        A durable queue is a **quorum** queue unless it is asked to be something
252        else. That is what Java has always declared and what the other libraries
253        declare now, and it has to be the same everywhere: the queue type is one
254        more argument the broker compares, so a Java service and a Python
255        service that disagree about ``orders`` cannot both consume it. Pass
256        ``quorum=False`` for a classic queue, which is declared by leaving
257        ``x-queue-type`` off entirely — the spelling every AceMQ library uses,
258        and therefore the only one a broker finds equivalent to theirs.
259
260        ``exclusive``, ``auto_delete`` and ``durable=False`` all mean a queue
261        that belongs to one connection, and RabbitMQ refuses a quorum queue that
262        is any of those. Such a queue is declared classic without being asked,
263        because the alternative is a declaration the broker rejects with a
264        message that never mentions the word quorum. Asking for both — a queue
265        that is ``quorum=True`` and exclusive — is refused here instead, where
266        the contradiction is written down.
267
268        A queue whose ``args`` already name a kind keeps it: that is how
269        :func:`acemq_amqp.patterns.stream` declares a stream, and a caller who
270        wrote ``x-queue-type`` down meant it. Writing it down *and* passing
271        ``quorum`` is two answers to one question and is refused.
272
273        ``dead_letter`` declares ``{name}.dlq`` alongside it and points the
274        broker at it, so a message this queue rejects or lets expire lands
275        somewhere an operator can find by name rather than disappearing. It
276        declares ``{name}.parked`` too, because a consumer on this queue sends
277        anything it cannot decode there — and a library that parks messages into
278        a queue nobody declared has only moved the disappearance somewhere else.
279
280        Neither dead-letters itself: a dead-letter queue that dead-letters is a
281        loop, and a loop is how a poison message becomes an outage. Both are
282        reached through :data:`DEAD_LETTER_EXCHANGE`, bound on their own names,
283        which is what Java has always done and is therefore what a broker shared
284        with a Java service already has. A service that wants its own
285        dead-letter exchange instead should leave ``dead_letter`` alone and pass
286        the arguments in ``args``.
287
288        ``retry`` declares the rung queues that policy's long waits need: one
289        ``{name}.retry.{delay}`` per distinct delay at or above the policy's
290        threshold, each declared with :func:`rung_args` — the delay as
291        ``x-message-ttl``, :data:`RETRY_EXCHANGE` as the dead-letter target, and
292        this queue as the routing key. The binding that brings an expired
293        message home, ``{name}`` to :data:`RETRY_EXCHANGE` on ``{name}``, is
294        made here and not left to a caller to remember: without it every rung
295        expires into an exchange that routes nowhere, and a retry that is
296        dropped by the broker looks exactly like one that is still waiting.
297
298        It takes the policy rather than a list of delays on purpose. The rungs a
299        consumer will publish to are derived from the policy it is running, so
300        anything else here would be a second copy of the same list, free to drift
301        from the first — and the way that drift shows up is a retry published to
302        a queue nobody declared, at the moment the service is already failing.
303        One value produces both, or the topology is not a description of what the
304        service needs.
305
306        :param name: what to call it
307        :param durable: survives a broker restart
308        :param auto_delete: goes away when its last consumer does
309        :param exclusive: usable only by the connection that declared it
310        :param quorum: replicated rather than classic. ``None``, the default,
311            means quorum for a durable queue and classic for one belonging to a
312            single connection
313        :param dead_letter: also declare and wire ``{name}.dlq``, and declare
314            ``{name}.parked``
315        :param retry: also declare the rung queues this policy's long waits use
316        :param args: broker-specific arguments
317        :returns: this topology
318        :raises ValueError: when the queue cannot be what it was asked to be
319        """
320        if not name:
321            raise ValueError("acemq: a queue needs a name")
322
323        arguments: dict[str, Any] = dict(args or {})
324        kind = _queue_type(
325            name,
326            arguments,
327            durable=durable,
328            auto_delete=auto_delete,
329            exclusive=exclusive,
330            quorum=quorum,
331        )
332        if kind is not None:
333            arguments[QUEUE_TYPE_ARG] = kind
334        if dead_letter:
335            # Refused rather than resolved, because either answer would be a
336            # guess about which of two conflicting instructions was meant.
337            conflicting = sorted(
338                key
339                for key in arguments
340                if key in (DEAD_LETTER_EXCHANGE_ARG, DEAD_LETTER_ROUTING_KEY_ARG)
341            )
342            if conflicting:
343                raise ValueError(
344                    f"acemq: queue {name!r} asks for dead_letter=True and also sets "
345                    + ", ".join(conflicting)
346                    + "; pick one"
347                )
348            arguments[DEAD_LETTER_EXCHANGE_ARG] = DEAD_LETTER_EXCHANGE
349            arguments[DEAD_LETTER_ROUTING_KEY_ARG] = naming.dead_letter_queue(name)
350
351        self._queues.append(
352            _NamedQueue(
353                name,
354                QueueSpec(
355                    durable=durable,
356                    auto_delete=auto_delete,
357                    exclusive=exclusive,
358                    args=arguments,
359                ),
360            )
361        )
362        if dead_letter:
363            self._managed_exchange(DEAD_LETTER_EXCHANGE)
364            # Classic, deliberately, whatever the queue in front of them is.
365            # Java declares both QueueType.CLASSIC and these are shared names:
366            # a dead-letter queue nobody consumes gains nothing from being
367            # replicated, and gaining it here would cost a Java service its
368            # declaration of the same queue.
369            for target in (naming.dead_letter_queue(name), naming.parked_queue(name)):
370                self._queues.append(_NamedQueue(target, QueueSpec(durable=durable)))
371                self._bindings.append(
372                    Binding(queue=target, exchange=DEAD_LETTER_EXCHANGE, routing_key=target)
373                )
374        if retry is not None:
375            rungs = retry.broker_rungs()
376            if rungs:
377                self._managed_exchange(RETRY_EXCHANGE)
378                # Classic too, and for the same reason as ``.dlq``: a rung is
379                # declared by every service consuming the source queue, Java
380                # declares it QueueType.CLASSIC, and :func:`rung_args` is the
381                # whole argument table — three keys, no ``x-queue-type``.
382                for rung in rungs:
383                    self._queues.append(
384                        _NamedQueue(
385                            naming.retry_queue(name, rung),
386                            QueueSpec(durable=durable, args=rung_args(name, rung)),
387                        )
388                    )
389                # One binding brings every expired message back to the queue it
390                # came from. Made here rather than offered as something to add,
391                # because a missing binding loses every retry silently: the
392                # broker drops what a direct exchange cannot route and says
393                # nothing, so the queue looks quiet rather than broken.
394                self._bindings.append(
395                    Binding(queue=name, exchange=RETRY_EXCHANGE, routing_key=name)
396                )
397        return self

Adds a queue, and the queues that catch what it cannot handle.

A durable queue is a quorum queue unless it is asked to be something else. That is what Java has always declared and what the other libraries declare now, and it has to be the same everywhere: the queue type is one more argument the broker compares, so a Java service and a Python service that disagree about orders cannot both consume it. Pass quorum=False for a classic queue, which is declared by leaving x-queue-type off entirely — the spelling every AceMQ library uses, and therefore the only one a broker finds equivalent to theirs.

exclusive, auto_delete and durable=False all mean a queue that belongs to one connection, and RabbitMQ refuses a quorum queue that is any of those. Such a queue is declared classic without being asked, because the alternative is a declaration the broker rejects with a message that never mentions the word quorum. Asking for both — a queue that is quorum=True and exclusive — is refused here instead, where the contradiction is written down.

A queue whose args already name a kind keeps it: that is how acemq_amqp.patterns.stream() declares a stream, and a caller who wrote x-queue-type down meant it. Writing it down and passing quorum is two answers to one question and is refused.

dead_letter declares {name}.dlq alongside it and points the broker at it, so a message this queue rejects or lets expire lands somewhere an operator can find by name rather than disappearing. It declares {name}.parked too, because a consumer on this queue sends anything it cannot decode there — and a library that parks messages into a queue nobody declared has only moved the disappearance somewhere else.

Neither dead-letters itself: a dead-letter queue that dead-letters is a loop, and a loop is how a poison message becomes an outage. Both are reached through DEAD_LETTER_EXCHANGE, bound on their own names, which is what Java has always done and is therefore what a broker shared with a Java service already has. A service that wants its own dead-letter exchange instead should leave dead_letter alone and pass the arguments in args.

retry declares the rung queues that policy's long waits need: one {name}acemq_amqp.retry.{delay} per distinct delay at or above the policy's threshold, each declared with rung_args() — the delay as x-message-ttl, RETRY_EXCHANGE as the dead-letter target, and this queue as the routing key. The binding that brings an expired message home, {name} to RETRY_EXCHANGE on {name}, is made here and not left to a caller to remember: without it every rung expires into an exchange that routes nowhere, and a retry that is dropped by the broker looks exactly like one that is still waiting.

It takes the policy rather than a list of delays on purpose. The rungs a consumer will publish to are derived from the policy it is running, so anything else here would be a second copy of the same list, free to drift from the first — and the way that drift shows up is a retry published to a queue nobody declared, at the moment the service is already failing. One value produces both, or the topology is not a description of what the service needs.

Parameters
  • name: what to call it
  • durable: survives a broker restart
  • auto_delete: goes away when its last consumer does
  • exclusive: usable only by the connection that declared it
  • quorum: replicated rather than classic. None, the default, means quorum for a durable queue and classic for one belonging to a single connection
  • dead_letter: also declare and wire {name}.dlq, and declare {name}.parked
  • retry: also declare the rung queues this policy's long waits use
  • args: broker-specific arguments :returns: this topology
Raises
  • ValueError: when the queue cannot be what it was asked to be
def binding( self, queue: str, exchange: str, routing_key: str = '') -> Topology:
411    def binding(self, queue: str, exchange: str, routing_key: str = "") -> Topology:
412        """Routes messages matching a key from an exchange to a queue.
413
414        :param queue: where messages end up, which this topology must declare
415        :param exchange: where they come from, which this topology must declare
416        :param routing_key: what to match, empty for a fanout
417        :returns: this topology
418        """
419        self._bindings.append(Binding(queue=queue, exchange=exchange, routing_key=routing_key))
420        return self

Routes messages matching a key from an exchange to a queue.

Parameters
  • queue: where messages end up, which this topology must declare
  • exchange: where they come from, which this topology must declare
  • routing_key: what to match, empty for a fanout :returns: this topology
queues: list[str]
422    @property
423    def queues(self) -> list[str]:
424        """The queue names this topology declares, in declaration order."""
425        return [entry.name for entry in self._queues]

The queue names this topology declares, in declaration order.

exchanges: list[str]
427    @property
428    def exchanges(self) -> list[str]:
429        """The exchange names this topology declares, in declaration order."""
430        return [entry.name for entry in self._exchanges]

The exchange names this topology declares, in declaration order.

bindings: list[acemq_amqp.topology.Binding]
432    @property
433    def bindings(self) -> list[Binding]:
434        """The bindings this topology declares."""
435        return list(self._bindings)

The bindings this topology declares.

def validate(self) -> None:
437    def validate(self) -> None:
438        """Reports what is wrong with the description itself.
439
440        A binding naming a queue the topology does not declare is the mistake
441        worth catching here. The broker would accept it if the queue happened to
442        exist already, and the service would then depend on something nothing
443        declares — which works until the day it is deployed somewhere new.
444
445        :raises ValueError: when the topology cannot be right
446        """
447        seen_queues: set[str] = set()
448        for queue in self._queues:
449            if queue.name in seen_queues:
450                raise ValueError(f"acemq: the topology declares queue {queue.name!r} twice")
451            seen_queues.add(queue.name)
452
453        seen_exchanges: set[str] = set()
454        for exchange in self._exchanges:
455            if exchange.name in seen_exchanges:
456                raise ValueError(
457                    f"acemq: the topology declares exchange {exchange.name!r} twice"
458                )
459            seen_exchanges.add(exchange.name)
460
461        for binding in self._bindings:
462            if binding.queue not in seen_queues:
463                raise ValueError(
464                    f"acemq: binding {binding} names queue {binding.queue!r}, "
465                    "which this topology does not declare"
466                )
467            if not binding.exchange:
468                raise ValueError(
469                    f"acemq: binding {binding} names the default exchange, "
470                    "which cannot be bound to"
471                )
472            if binding.exchange not in seen_exchanges:
473                raise ValueError(
474                    f"acemq: binding {binding} names exchange {binding.exchange!r}, "
475                    "which this topology does not declare"
476                )

Reports what is wrong with the description itself.

A binding naming a queue the topology does not declare is the mistake worth catching here. The broker would accept it if the queue happened to exist already, and the service would then depend on something nothing declares — which works until the day it is deployed somewhere new.

Raises
  • ValueError: when the topology cannot be right
def plan(self) -> list[acemq_amqp.topology.PlanAction]:
478    def plan(self) -> list[PlanAction]:
479        """What :meth:`apply` would do, without doing it.
480
481        Deliberately not a difference against the live broker: AMQP offers no
482        way to enumerate what is there without the management API, and a plan
483        that quietly guessed would be worse than one that is honest about being
484        a statement of intent.
485
486        :returns: one action per thing that would be declared
487        :raises ValueError: when the topology cannot be right
488        """
489        self.validate()
490        actions = [
491            PlanAction("exchange", entry.name, _describe(entry.spec))
492            for entry in self._exchanges
493        ]
494        actions += [
495            PlanAction("queue", entry.name, _describe(entry.spec)) for entry in self._queues
496        ]
497        actions += [
498            PlanAction(
499                "binding", binding.queue, f"from {binding.exchange} on {binding.routing_key}"
500            )
501            for binding in self._bindings
502        ]
503        return actions

What apply() would do, without doing it.

Deliberately not a difference against the live broker: AMQP offers no way to enumerate what is there without the management API, and a plan that quietly guessed would be worse than one that is honest about being a statement of intent.

:returns: one action per thing that would be declared

Raises
  • ValueError: when the topology cannot be right
async def apply(self, transport: Transport) -> None:
505    async def apply(self, transport: Transport) -> None:
506        """Declares everything, in the order a broker needs.
507
508        It stops at the first failure. A queue that already exists with
509        different settings is refused by the broker with ``PRECONDITION_FAILED``,
510        and that refusal is passed on rather than swallowed: it means this
511        service and the broker disagree about what the queue is, and carrying on
512        would leave the service using a queue that is not the one it asked for.
513
514        :param transport: where to declare it
515        :raises ValueError: when the topology cannot be right
516        """
517        self.validate()
518        for exchange in self._exchanges:
519            await transport.declare_exchange(exchange.name, exchange.spec)
520        for queue in self._queues:
521            await transport.declare_queue(queue.name, queue.spec)
522        for binding in self._bindings:
523            await transport.bind(binding.queue, binding.exchange, binding.routing_key)

Declares everything, in the order a broker needs.

It stops at the first failure. A queue that already exists with different settings is refused by the broker with PRECONDITION_FAILED, and that refusal is passed on rather than swallowed: it means this service and the broker disagree about what the queue is, and carrying on would leave the service using a queue that is not the one it asked for.

Parameters
  • transport: where to declare it
Raises
  • ValueError: when the topology cannot be right
@runtime_checkable
class Transport(typing.Protocol):
194@runtime_checkable
195class Transport(Protocol):
196    """A connection to a broker.
197
198    Async because that is the shape a broker client has: every one of these is a
199    round trip, and a library that hid them behind blocking calls would make one
200    slow broker stall a whole process. The blocking API in
201    :mod:`acemq_amqp.sync` is built on top of this rather than beside it.
202    """
203
204    async def declare_queue(self, name: str, spec: QueueSpec) -> None:
205        """Creates a queue if it is not already there."""
206
207    async def declare_exchange(self, name: str, spec: ExchangeSpec) -> None:
208        """Creates an exchange if it is not already there."""
209
210    async def bind(self, queue: str, exchange: str, routing_key: str) -> None:
211        """Routes messages matching a key from an exchange to a queue."""
212
213    async def publish(
214        self, exchange: str, routing_key: str, message: Outbound
215    ) -> PublishResult:
216        """Sends one message and reports what the broker said about it."""
217
218    async def consume(
219        self,
220        queue: str,
221        spec: ConsumeSpec,
222        deliver: Callable[[Delivery], Awaitable[None]],
223    ) -> Subscription:
224        """Delivers messages to ``deliver`` until the subscription is closed."""
225
226    async def close(self) -> None:
227        """Releases the connection."""

A connection to a broker.

Async because that is the shape a broker client has: every one of these is a round trip, and a library that hid them behind blocking calls would make one slow broker stall a whole process. The blocking API in acemq_amqp.sync is built on top of this rather than beside it.

Transport(*args, **kwargs)
1771def _no_init_or_replace_init(self, *args, **kwargs):
1772    cls = type(self)
1773
1774    if cls._is_protocol:
1775        raise TypeError('Protocols cannot be instantiated')
1776
1777    # Already using a custom `__init__`. No need to calculate correct
1778    # `__init__` to call. This can lead to RecursionError. See bpo-45121.
1779    if cls.__init__ is not _no_init_or_replace_init:
1780        return
1781
1782    # Initially, `__init__` of a protocol subclass is set to `_no_init_or_replace_init`.
1783    # The first instantiation of the subclass will call `_no_init_or_replace_init` which
1784    # searches for a proper new `__init__` in the MRO. The new `__init__`
1785    # replaces the subclass' old `__init__` (ie `_no_init_or_replace_init`). Subsequent
1786    # instantiation of the protocol subclass will thus use the new
1787    # `__init__` and no longer call `_no_init_or_replace_init`.
1788    for base in cls.__mro__:
1789        init = base.__dict__.get('__init__', _no_init_or_replace_init)
1790        if init is not _no_init_or_replace_init:
1791            cls.__init__ = init
1792            break
1793    else:
1794        # should not happen
1795        cls.__init__ = object.__init__
1796
1797    cls.__init__(self, *args, **kwargs)
async def declare_queue(self, name: str, spec: QueueSpec) -> None:
204    async def declare_queue(self, name: str, spec: QueueSpec) -> None:
205        """Creates a queue if it is not already there."""

Creates a queue if it is not already there.

async def declare_exchange(self, name: str, spec: ExchangeSpec) -> None:
207    async def declare_exchange(self, name: str, spec: ExchangeSpec) -> None:
208        """Creates an exchange if it is not already there."""

Creates an exchange if it is not already there.

async def bind(self, queue: str, exchange: str, routing_key: str) -> None:
210    async def bind(self, queue: str, exchange: str, routing_key: str) -> None:
211        """Routes messages matching a key from an exchange to a queue."""

Routes messages matching a key from an exchange to a queue.

async def publish( self, exchange: str, routing_key: str, message: Outbound) -> PublishResult:
213    async def publish(
214        self, exchange: str, routing_key: str, message: Outbound
215    ) -> PublishResult:
216        """Sends one message and reports what the broker said about it."""

Sends one message and reports what the broker said about it.

async def consume( self, queue: str, spec: ConsumeSpec, deliver: Callable[[Delivery], Awaitable[None]]) -> acemq_amqp.transport.Subscription:
218    async def consume(
219        self,
220        queue: str,
221        spec: ConsumeSpec,
222        deliver: Callable[[Delivery], Awaitable[None]],
223    ) -> Subscription:
224        """Delivers messages to ``deliver`` until the subscription is closed."""

Delivers messages to deliver until the subscription is closed.

async def close(self) -> None:
226    async def close(self) -> None:
227        """Releases the connection."""

Releases the connection.

class Verification(enum.Enum):
262class Verification(Enum):
263    """What an encrypted connection checks about the broker on the other end.
264
265    Named rather than left as a boolean because a ``verify=False`` in a
266    configuration file says nothing about what is being given up. These say it.
267    """
268
269    #: Check the certificate chain and the name on it. The only setting to use
270    #: against a broker holding anything that matters.
271    CERTIFICATE = "certificate"
272
273    #: Check nothing. Encrypted, and to whoever answered.
274    #: :func:`without_verifying_the_broker` is the way to ask for this, and its
275    #: docstring says what it costs.
276    NOTHING_AT_ALL = "nothing-at-all"

What an encrypted connection checks about the broker on the other end.

Named rather than left as a boolean because a verify=False in a configuration file says nothing about what is being given up. These say it.

CERTIFICATE = <Verification.CERTIFICATE: 'certificate'>
NOTHING_AT_ALL = <Verification.NOTHING_AT_ALL: 'nothing-at-all'>
@dataclass(frozen=True, slots=True)
class Wait:
50@dataclass(frozen=True, slots=True)
51class Wait:
52    """How long before the next attempt, and where the message spends it.
53
54    Two answers rather than one because they cannot be worked out separately:
55    jitter applies only to a wait spent in the consumer, so a caller given a
56    delay alone could not tell whether it had already been moved — and a
57    jittered delay does not name a rung queue.
58
59    :param delay: how long the message waits
60    :param in_broker: whether it waits on a rung queue rather than here
61    """
62
63    delay: timedelta
64    in_broker: bool

How long before the next attempt, and where the message spends it.

Two answers rather than one because they cannot be worked out separately: jitter applies only to a wait spent in the consumer, so a caller given a delay alone could not tell whether it had already been moved — and a jittered delay does not name a rung queue.

Parameters
  • delay: how long the message waits
  • in_broker: whether it waits on a rung queue rather than here
Wait(delay: datetime.timedelta, in_broker: bool)
delay: datetime.timedelta
in_broker: bool
__version__ = '0.7.12'
def accept() -> Ack:
136def accept() -> Ack:
137    """Confirms the message. It will not be delivered again."""
138    return Ack(Action.ACCEPT)

Confirms the message. It will not be delivered again.

async def aggregate_health( *checks: HealthCheck, timeout: float = 5.0) -> HealthReport:
604async def aggregate_health(*checks: HealthCheck, timeout: float = 5.0) -> HealthReport:
605    """Runs every check at once and combines the answers.
606
607    The combined status is the worst of them: one thing being down makes the
608    whole report down, because a service that cannot reach its broker is not
609    ready however healthy the rest of it is.
610
611    At once rather than in turn, so a slow check does not add its latency to the
612    others, and under a deadline, so one that ignores the instruction above
613    still cannot hang the probe. A check that times out is reported as down with
614    the reason, which is the honest reading: something that will not answer in
615    five seconds is not something a request can depend on.
616
617    :param checks: what to ask
618    :param timeout: how long to give all of them together
619    :returns: the combined report
620    """
621    if not checks:
622        return HealthReport(HealthStatus.UP)
623
624    async def run(check: HealthCheck) -> tuple[str, HealthReport]:
625        try:
626            return check.name, await check.check()
627        except Exception as failure:
628            return check.name, HealthReport(
629                HealthStatus.DOWN, f"the check itself failed: {failure}"
630            )
631
632    tasks = [asyncio.ensure_future(run(check)) for check in checks]
633    try:
634        done, pending = await asyncio.wait(tasks, timeout=timeout)
635    except asyncio.CancelledError:
636        for task in tasks:
637            task.cancel()
638        raise
639
640    parts: dict[str, Any] = {}
641    worst = HealthStatus.UP
642    reasons: list[str] = []
643
644    for task in pending:
645        task.cancel()
646    for index, task in enumerate(tasks):
647        if task in done:
648            name, report = task.result()
649        else:
650            name = checks[index].name
651            report = HealthReport(HealthStatus.DOWN, f"did not answer within {timeout}s")
652        parts[name] = report
653        if report.status is HealthStatus.DOWN:
654            worst = HealthStatus.DOWN
655            reasons.append(name)
656        elif report.status is HealthStatus.DEGRADED:
657            if worst is not HealthStatus.DOWN:
658                worst = HealthStatus.DEGRADED
659            reasons.append(name)
660
661    return HealthReport(worst, ", ".join(reasons), parts=parts)

Runs every check at once and combines the answers.

The combined status is the worst of them: one thing being down makes the whole report down, because a service that cannot reach its broker is not ready however healthy the rest of it is.

At once rather than in turn, so a slow check does not add its latency to the others, and under a deadline, so one that ignores the instruction above still cannot hang the probe. A check that times out is reported as down with the reason, which is the honest reading: something that will not answer in five seconds is not something a request can depend on.

Parameters
  • checks: what to ask
  • timeout: how long to give all of them together :returns: the combined report
def codec_by_name(name: str) -> Codec:
319def codec_by_name(name: str) -> Codec:
320    """Builds the codec registered under a name.
321
322    :param name: the registered name
323    :returns: a new instance
324    :raises KeyError: when nothing is registered under that name
325    """
326    with _registry_lock:
327        build = _registry.get(name)
328    if build is None:
329        raise KeyError(
330            f"acemq: no codec named {name!r} is registered; known: {', '.join(codec_names())}"
331        )
332    return build()

Builds the codec registered under a name.

Parameters
  • name: the registered name :returns: a new instance
Raises
  • KeyError: when nothing is registered under that name
def codec_names() -> list[str]:
335def codec_names() -> list[str]:
336    """The registered codec names, sorted."""
337    with _registry_lock:
338        return sorted(_registry)

The registered codec names, sorted.

async def connect( url: str, *, codec: Codec | None = None, origin: str | None = None, retry: RetryPolicy | None = None, prefetch: int = 20, max_outstanding_publishes: int = 1000, confirm_timeout: datetime.timedelta = datetime.timedelta(seconds=10), security: Security | None = None, on_publish: Sequence[Callable[[PublishContext, Callable[[PublishContext], Awaitable[PublishResult]]], Awaitable[PublishResult]]] = (), on_consume: Sequence[Callable[[ConsumeContext, Callable[[ConsumeContext], Awaitable[Ack]]], Awaitable[Ack]]] = (), observer: Observer | None = None, **transport_options: Any) -> Connection:
2012async def connect(
2013    url: str,
2014    *,
2015    codec: Codec | None = None,
2016    origin: str | None = None,
2017    retry: RetryPolicy | None = None,
2018    prefetch: int = DEFAULT_PREFETCH,
2019    max_outstanding_publishes: int = DEFAULT_MAX_OUTSTANDING_PUBLISHES,
2020    confirm_timeout: timedelta = DEFAULT_CONFIRM_TIMEOUT,
2021    security: Security | None = None,
2022    on_publish: Sequence[PublishInterceptor] = (),
2023    on_consume: Sequence[ConsumeInterceptor] = (),
2024    observer: Observer | None = None,
2025    **transport_options: Any,
2026) -> Connection:
2027    """Opens a connection to a broker.
2028
2029    The URL's scheme picks the transport, and says whether the connection is
2030    encrypted: ``amqps://`` is, ``amqp://`` is not. Both need aio-pika, which is
2031    why it is imported here rather than by the package: a program that only
2032    reads AceMQ envelopes should not be made to install an AMQP client to import
2033    this library.
2034
2035    An ``amqps://`` URL is verified against the machine's trust store and will
2036    not speak anything older than TLS 1.2, with or without a ``security``. Pass
2037    one to name a different authority, to present a client certificate, or to
2038    supply the login separately from the URL — which is how a password stays out
2039    of a connection string that ends up in a log.
2040
2041    :param url: where the broker is
2042    :param codec: what publishers and consumers use unless they say otherwise
2043    :param origin: what to stamp on published messages
2044    :param retry: what consumers use unless they say otherwise
2045    :param prefetch: how many unacknowledged messages a consumer holds
2046    :param max_outstanding_publishes: how many publishes may be waiting for the
2047        broker at once, a thousand by default. See :class:`Connection`
2048    :param confirm_timeout: how long a publish waits for room before raising
2049    :param security: how to verify the broker and who to log in as. See
2050        :class:`~acemq_amqp.Security`
2051    :param on_publish: what to wrap every publish in, outermost first. See
2052        :mod:`acemq_amqp.interceptors`
2053    :param on_consume: what to wrap every handler in
2054    :param observer: where the numbers go. See :mod:`acemq_amqp.telemetry`
2055    :param transport_options: passed to the transport, which for RabbitMQ is
2056        :func:`aio_pika.connect_robust`
2057    :returns: the connection
2058    :raises SecurityError: when the security settings cannot be honoured, before
2059        anything is dialled
2060    """
2061    scheme = urlsplit(url).scheme
2062    if scheme not in ("amqp", "amqps"):
2063        raise AceMQError(
2064            f"acemq: no transport knows how to reach {scheme or url!r}; "
2065            "this library speaks amqp:// and amqps://"
2066        )
2067
2068    # Resolved here rather than in the transport, so that a missing certificate
2069    # or an unset password variable is an exception from connect() with the
2070    # setting named in it, rather than a handshake failure from somewhere in
2071    # aio-pika that says only that the socket closed.
2072    settings = security or Security()
2073    url = settings.applied_to(url)
2074    # The caller's own options win: transport_options is the escape hatch for
2075    # everything this library does not model, and a Security built from defaults
2076    # should not quietly overrule an ssl_context somebody passed by hand.
2077    transport_options = {**settings.transport_options(url), **transport_options}
2078
2079    try:
2080        from .rabbitmq import RabbitMQTransport
2081    except ImportError as missing:  # pragma: no cover - depends on how it was installed
2082        raise AceMQError(
2083            "acemq: reaching a broker needs aio-pika, which is an optional extra. "
2084            'Install it with pip install "acemq-amqp[rabbitmq]"'
2085        ) from missing
2086
2087    transport = await RabbitMQTransport.connect(url, **transport_options)
2088    return Connection(
2089        transport,
2090        codec=codec,
2091        origin=origin,
2092        retry=retry,
2093        prefetch=prefetch,
2094        max_outstanding_publishes=max_outstanding_publishes,
2095        confirm_timeout=confirm_timeout,
2096        on_publish=on_publish,
2097        on_consume=on_consume,
2098        observer=observer,
2099    )

Opens a connection to a broker.

The URL's scheme picks the transport, and says whether the connection is encrypted: amqps:// is, amqp:// is not. Both need aio-pika, which is why it is imported here rather than by the package: a program that only reads AceMQ envelopes should not be made to install an AMQP client to import this library.

An amqps:// URL is verified against the machine's trust store and will not speak anything older than TLS 1.2, with or without a security. Pass one to name a different authority, to present a client certificate, or to supply the login separately from the URL — which is how a password stays out of a connection string that ends up in a log.

Parameters
  • url: where the broker is
  • codec: what publishers and consumers use unless they say otherwise
  • origin: what to stamp on published messages
  • retry: what consumers use unless they say otherwise
  • prefetch: how many unacknowledged messages a consumer holds
  • max_outstanding_publishes: how many publishes may be waiting for the broker at once, a thousand by default. See Connection
  • confirm_timeout: how long a publish waits for room before raising
  • security: how to verify the broker and who to log in as. See ~acemq_amqp.Security
  • on_publish: what to wrap every publish in, outermost first. See acemq_amqp.interceptors
  • on_consume: what to wrap every handler in
  • observer: where the numbers go. See acemq_amqp.telemetry
  • transport_options: passed to the transport, which for RabbitMQ is aio_pika.connect_robust() :returns: the connection
Raises
  • SecurityError: when the security settings cannot be honoured, before anything is dialled
def consume_chain( interceptors: Sequence[Callable[[ConsumeContext, Callable[[ConsumeContext], Awaitable[Ack]]], Awaitable[Ack]]], handle: Callable[[ConsumeContext], Awaitable[Ack]]) -> Callable[[ConsumeContext], Awaitable[Ack]]:
248def consume_chain(
249    interceptors: Sequence[ConsumeInterceptor], handle: ConsumeNext
250) -> ConsumeNext:
251    """Folds interceptors around a handler, first registered outermost.
252
253    :param interceptors: what to wrap it in, in registration order
254    :param handle: the innermost work, which runs the handler
255    :returns: the whole thing as one callable, or ``handle`` when there is
256        nothing to wrap it in
257    """
258    composed = handle
259    for interceptor in reversed(interceptors):
260        composed = _consume_step(interceptor, composed)
261    return composed

Folds interceptors around a handler, first registered outermost.

Parameters
  • interceptors: what to wrap it in, in registration order
  • handle: the innermost work, which runs the handler :returns: the whole thing as one callable, or handle when there is nothing to wrap it in
def credentials_from_environment( username_variable: str = 'ACEMQ_USERNAME', password_variable: str = 'ACEMQ_PASSWORD') -> Callable[Credentials]:
189def credentials_from_environment(
190    username_variable: str = "ACEMQ_USERNAME",
191    password_variable: str = "ACEMQ_PASSWORD",
192) -> CredentialsSource:
193    """Reads two environment variables, each time it is asked.
194
195    :param username_variable: the variable holding the account name
196    :param password_variable: the variable holding the password
197    :returns: a source that reads them
198    :raises SecurityError: when either variable is unset, naming which one
199    """
200
201    def read() -> Credentials:
202        username = os.environ.get(username_variable)
203        if username is None:
204            raise SecurityError(
205                f"acemq: {username_variable} is not set, so there is no broker username"
206            )
207        password = os.environ.get(password_variable)
208        if password is None:
209            raise SecurityError(
210                f"acemq: {password_variable} is not set, so there is no broker password"
211            )
212        return Credentials(username, password)
213
214    return read

Reads two environment variables, each time it is asked.

Parameters
  • username_variable: the variable holding the account name
  • password_variable: the variable holding the password :returns: a source that reads them
Raises
  • SecurityError: when either variable is unset, naming which one
def credentials_from_file( path: str | os.PathLike[str], *, username: str = '') -> Callable[Credentials]:
217def credentials_from_file(path: FilePath, *, username: str = "") -> CredentialsSource:
218    """Reads a secret from a file, each time it is asked.
219
220    This is how a mounted Kubernetes secret and a Docker secret arrive, and
221    reading it per connection rather than once at import is what lets a rotated
222    file take effect without a restart.
223
224    The file holds the password alone when ``username`` is given, and
225    ``username:password`` when it is not. Trailing whitespace is trimmed,
226    because a file written by an editor almost always ends in a newline and a
227    password with a newline on the end fails in a way nobody enjoys diagnosing.
228
229    :param path: the file to read
230    :param username: the account name, when the file holds only the secret
231    :returns: a source that reads the file
232    :raises SecurityError: when the file is missing, empty, or holds no username
233        and none was given
234    """
235
236    def read() -> Credentials:
237        where = os.fspath(path)
238        try:
239            with open(path, encoding="utf-8") as handle:
240                contents = handle.read().rstrip(" \t\r\n")
241        except OSError as unreadable:
242            raise SecurityError(
243                f"acemq: cannot read the credentials file {where!r}: {unreadable}"
244            ) from unreadable
245
246        if not contents:
247            raise SecurityError(f"acemq: the credentials file {where!r} is empty")
248        if username:
249            return Credentials(username, contents)
250        account, separator, secret = contents.partition(":")
251        if not separator:
252            raise SecurityError(
253                f"acemq: the credentials file {where!r} holds no username, and none was "
254                "given; write it as username:password, or pass username= to "
255                "credentials_from_file"
256            )
257        return Credentials(account, secret)
258
259    return read

Reads a secret from a file, each time it is asked.

This is how a mounted Kubernetes secret and a Docker secret arrive, and reading it per connection rather than once at import is what lets a rotated file take effect without a restart.

The file holds the password alone when username is given, and username:password when it is not. Trailing whitespace is trimmed, because a file written by an editor almost always ends in a newline and a password with a newline on the end fails in a way nobody enjoys diagnosing.

Parameters
  • path: the file to read
  • username: the account name, when the file holds only the secret :returns: a source that reads the file
Raises
  • SecurityError: when the file is missing, empty, or holds no username and none was given
def dead_letter_queue(queue: str) -> str:
35def dead_letter_queue(queue: str) -> str:
36    """``orders.new`` becomes ``orders.new.dlq``."""
37    return queue + DEAD_LETTER_SUFFIX

orders.new becomes orders.new.dlq.

async def declare_where_failures_go( transport: Transport, queue: str, retry: RetryPolicy | None = None) -> None:
604async def declare_where_failures_go(
605    transport: Transport, queue: str, retry: RetryPolicy | None = None
606) -> None:
607    """Declares the queues a consumer of ``queue`` will need when it fails.
608
609    ``acemq.dlx``, ``{queue}.dlq``, ``{queue}.parked`` and the two bindings that
610    reach them; and, when ``retry`` has waits long enough to be held by the
611    broker, ``acemq.retry``, one rung queue per distinct wait, and the binding
612    that brings an expired message back to ``queue``. Exactly what
613    :meth:`Topology.queue` writes for the same arguments, and exactly what Java's
614    consumer declares when it starts.
615
616    Called by :meth:`acemq_amqp.Connection.consume` before it subscribes, which
617    is the point of it: everything here is somewhere a message goes when the
618    handler has given up, and a message republished to a queue that is not there
619    is discarded by the broker without a word. Applying a topology first is still
620    the right way to run a service — it is reviewable, and it declares the source
621    queue with the ``x-dead-letter-exchange`` this cannot — but a service that was
622    deployed without one should lose a message to a bug rather than to a missing
623    queue.
624
625    The source queue is deliberately not declared. It belongs to whoever set the
626    service up, who chose its type and its arguments, and guessing wrong is a
627    ``PRECONDITION_FAILED`` that stops the consumer starting at all.
628
629    Safe to call repeatedly, and safe to call after :meth:`Topology.apply` or
630    before it: every declaration here carries the same arguments the topology
631    would send, so the second one to arrive is a no-op rather than a refusal.
632
633    :param transport: where to declare it
634    :param queue: the queue being consumed
635    :param retry: the policy the consumer is running, whose long waits decide
636        which rungs exist. ``None``, or a policy that holds every wait in the
637        consumer, declares only the dead-letter half
638    """
639    await transport.declare_exchange(DEAD_LETTER_EXCHANGE, _MANAGED_EXCHANGE)
640    for target in (naming.dead_letter_queue(queue), naming.parked_queue(queue)):
641        # Classic, whatever the queue in front of them is, for the reason
642        # :meth:`Topology.queue` gives: these are shared names, and a Java
643        # service declaring the same queue declares it classic.
644        await transport.declare_queue(target, QueueSpec(durable=True))
645        await transport.bind(target, DEAD_LETTER_EXCHANGE, target)
646
647    # Keyed by name rather than by delay. Two delays can render to the same name,
648    # and a second queue by that name with a different time-to-live is not a
649    # second rung — it is the PRECONDITION_FAILED that stops this consumer
650    # starting. The first delay to reach a name owns it, which is what Java does.
651    rungs: dict[str, timedelta] = {}
652    for delay in retry.broker_rungs() if retry is not None else []:
653        rungs.setdefault(naming.retry_queue(queue, delay), delay)
654    if not rungs:
655        return
656
657    await transport.declare_exchange(RETRY_EXCHANGE, _MANAGED_EXCHANGE)
658    for name, delay in rungs.items():
659        await transport.declare_queue(
660            name, QueueSpec(durable=True, args=rung_args(queue, delay))
661        )
662    # One binding brings every expired message home. Without it a rung expires
663    # into an exchange that routes nowhere, and the broker drops what it cannot
664    # route in silence: the retry looks like one still waiting rather than one
665    # that is gone.
666    await transport.bind(queue, RETRY_EXCHANGE, queue)

Declares the queues a consumer of queue will need when it fails.

acemq.dlx, {queue}.dlq, {queue}.parked and the two bindings that reach them; and, when retry has waits long enough to be held by the broker, acemq.retry, one rung queue per distinct wait, and the binding that brings an expired message back to queue. Exactly what Topology.queue() writes for the same arguments, and exactly what Java's consumer declares when it starts.

Called by acemq_amqp.Connection.consume() before it subscribes, which is the point of it: everything here is somewhere a message goes when the handler has given up, and a message republished to a queue that is not there is discarded by the broker without a word. Applying a topology first is still the right way to run a service — it is reviewable, and it declares the source queue with the x-dead-letter-exchange this cannot — but a service that was deployed without one should lose a message to a bug rather than to a missing queue.

The source queue is deliberately not declared. It belongs to whoever set the service up, who chose its type and its arguments, and guessing wrong is a PRECONDITION_FAILED that stops the consumer starting at all.

Safe to call repeatedly, and safe to call after Topology.apply() or before it: every declaration here carries the same arguments the topology would send, so the second one to arrive is a no-op rather than a refusal.

Parameters
  • transport: where to declare it
  • queue: the queue being consumed
  • retry: the policy the consumer is running, whose long waits decide which rungs exist. None, or a policy that holds every wait in the consumer, declares only the dead-letter half
def exponential_retry( max_attempts: int, initial_delay: datetime.timedelta, max_delay: datetime.timedelta = datetime.timedelta(0)) -> RetryPolicy:
231def exponential_retry(
232    max_attempts: int,
233    initial_delay: timedelta,
234    max_delay: timedelta = ZERO,
235) -> RetryPolicy:
236    """Doubling delays with 20% jitter, which is the sane default.
237
238    :param max_attempts: total deliveries including the first
239    :param initial_delay: the wait before the second attempt
240    :param max_delay: the ceiling, or zero for none
241    """
242    return RetryPolicy(
243        max_attempts=max_attempts,
244        initial_delay=initial_delay,
245        multiplier=2.0,
246        max_delay=max_delay,
247        jitter_factor=0.2,
248    )

Doubling delays with 20% jitter, which is the sane default.

Parameters
  • max_attempts: total deliveries including the first
  • initial_delay: the wait before the second attempt
  • max_delay: the ceiling, or zero for none
def fixed_retry( max_attempts: int, delay: datetime.timedelta) -> RetryPolicy:
251def fixed_retry(max_attempts: int, delay: timedelta) -> RetryPolicy:
252    """The same wait every time, with no jitter."""
253    return RetryPolicy(max_attempts=max_attempts, initial_delay=delay, multiplier=1.0)

The same wait every time, with no jitter.

def in_progress( delay: datetime.timedelta = datetime.timedelta(seconds=5)) -> Ack:
179def in_progress(delay: timedelta = DEFAULT_IN_PROGRESS_DELAY) -> Ack:
180    """Puts the message back because somebody else is still working on it.
181
182    What :func:`~acemq_amqp.patterns.idempotent` answers when a redelivery finds
183    a claim that is live but not yet confirmed. Accepting it would lose the
184    message if that other handler then died; running it would do the work
185    twice. So it goes back on its own queue after ``delay``, with its attempt
186    **unchanged**: a claim held elsewhere says nothing about whether this
187    message can be handled, so it neither spends a retry nor can exhaust the
188    policy and be dead-lettered. When the claim is confirmed the next look is a
189    duplicate and is accepted; when its lease runs out, it is taken over.
190
191    The wait is spent in the consumer, holding the delivery, as a short retry's
192    is; keep it short beside the store's claim timeout.
193
194    :param delay: how long until the message is looked at again
195    """
196    return Ack(Action.IN_PROGRESS, delay=delay)

Puts the message back because somebody else is still working on it.

What ~acemq_amqp.patterns.idempotent() answers when a redelivery finds a claim that is live but not yet confirmed. Accepting it would lose the message if that other handler then died; running it would do the work twice. So it goes back on its own queue after delay, with its attempt unchanged: a claim held elsewhere says nothing about whether this message can be handled, so it neither spends a retry nor can exhaust the policy and be dead-lettered. When the claim is confirmed the next look is a duplicate and is accepted; when its lease runs out, it is taken over.

The wait is spent in the consumer, holding the delivery, as a short retry's is; keep it short beside the store's claim timeout.

Parameters
  • delay: how long until the message is looked at again
def is_development_certificate(certificate: bytes) -> bool:
 97def is_development_certificate(certificate: bytes) -> bool:
 98    """Whether a certificate carries :data:`DEVELOPMENT_MARKER`.
 99
100    Reads DER, and reads PEM by decoding it first — the marker is plain ASCII
101    inside a DER certificate and is nowhere to be seen in the Base64 that wraps
102    it, so a PEM file searched as it stands would come back clean every time. A
103    PEM holding several certificates is read in full, so a chain file is checked
104    entry by entry.
105
106    The marker is looked for in the encoded certificate as a whole rather than in
107    a parsed subject, which is deliberate and is what lets this live in the
108    dependency-free core: an X.509 parser is not in the standard library, and the
109    answer would be the same one. Any distinguished name carrying the marker —
110    subject or issuer, ``O=`` as Go and .NET write it or ``OU=`` as Java does —
111    puts these bytes in the encoding, and there is nothing else a certificate
112    could plausibly contain them for.
113
114    Erring towards refusing is the right direction for this particular question.
115    A certificate that somehow contains the marker without being a development
116    one is refused, which is an inconvenience; the failure in the other direction
117    is a production system trusting an authority whose private key is on GitHub.
118
119    :param certificate: the certificate, DER-encoded or PEM-encoded
120    :returns: whether it is marked as a development certificate
121    """
122    if _MARKER_BYTES_LOWER in certificate.lower():
123        return True
124    return any(_MARKER_BYTES_LOWER in block.lower() for block in _der_blocks_in(certificate))

Whether a certificate carries DEVELOPMENT_MARKER.

Reads DER, and reads PEM by decoding it first — the marker is plain ASCII inside a DER certificate and is nowhere to be seen in the Base64 that wraps it, so a PEM file searched as it stands would come back clean every time. A PEM holding several certificates is read in full, so a chain file is checked entry by entry.

The marker is looked for in the encoded certificate as a whole rather than in a parsed subject, which is deliberate and is what lets this live in the dependency-free core: an X.509 parser is not in the standard library, and the answer would be the same one. Any distinguished name carrying the marker — subject or issuer, O= as Go and .NET write it or OU= as Java does — puts these bytes in the encoding, and there is nothing else a certificate could plausibly contain them for.

Erring towards refusing is the right direction for this particular question. A certificate that somehow contains the marker without being a development one is refused, which is an inconvenience; the failure in the other direction is a production system trusting an authority whose private key is on GitHub.

Parameters
  • certificate: the certificate, DER-encoded or PEM-encoded :returns: whether it is marked as a development certificate
def no_retry() -> RetryPolicy:
226def no_retry() -> RetryPolicy:
227    """One delivery, no second chance."""
228    return RetryPolicy(max_attempts=1)

One delivery, no second chance.

def park(error: BaseException | None = None) -> Ack:
162def park(error: BaseException | None = None) -> Ack:
163    """Sets the message aside on ``{queue}.parked`` without trying again.
164
165    For when the message is not readable at all: a body in a format nothing
166    here understands, a version this service was never taught. The engine
167    already parks a body its codec cannot decode; this is how a handler that
168    worked out the same thing one layer further in says so.
169
170    :func:`reject` is the neighbouring word and the difference is worth
171    keeping. A rejected message is a *bad request* — understood, and refused —
172    and it belongs with the other dead letters. A parked one was never
173    understood, and mixing the two means whoever drains the dead-letter queue
174    has to sort them by hand to find the producer that is emitting rubbish.
175    """
176    return Ack(Action.PARK, error)

Sets the message aside on {queue}.parked without trying again.

For when the message is not readable at all: a body in a format nothing here understands, a version this service was never taught. The engine already parks a body its codec cannot decode; this is how a handler that worked out the same thing one layer further in says so.

reject() is the neighbouring word and the difference is worth keeping. A rejected message is a bad request — understood, and refused — and it belongs with the other dead letters. A parked one was never understood, and mixing the two means whoever drains the dead-letter queue has to sort them by hand to find the producer that is emitting rubbish.

def parked_queue(queue: str) -> str:
40def parked_queue(queue: str) -> str:
41    """``orders.new`` becomes ``orders.new.parked``."""
42    return queue + PARKED_SUFFIX

orders.new becomes orders.new.parked.

def prometheus_text(metrics: Metrics) -> str:
454def prometheus_text(metrics: Metrics) -> str:
455    """Renders a :class:`Metrics` in the Prometheus text format.
456
457    Written out rather than through a client library, so a service that only
458    wants a scrape endpoint needs nothing installed. The format is small and
459    stable: a TYPE line, then samples, with dots in the metric name — and in the
460    label names, where ``routing.key`` and ``message.type`` have the same
461    problem — replaced by underscores, because Prometheus allows them in
462    neither.
463
464    A real registry — exemplars, native histograms, a shared process collector —
465    is what :mod:`acemq_amqp.prometheus` is for, and it needs the extra.
466
467    :param metrics: what to render
468    :returns: the body of a ``/acemq-metrics`` response
469    """
470    lines: list[str] = []
471    for name, value in sorted(metrics.counts.items()):
472        metric, labels = _split_key(name)
473        base = _prometheus_name(metric)
474        lines += [f"# TYPE {base} counter", f"{base}{labels} {value}"]
475    for name, value in sorted(metrics.gauges.items()):
476        metric, labels = _split_key(name)
477        base = _prometheus_name(metric)
478        lines += [f"# TYPE {base} gauge", f"{base}{labels} {value}"]
479    for name, summary in sorted(metrics.durations.items()):
480        metric, labels = _split_key(name)
481        base = _prometheus_name(metric)
482        lines += [
483            f"# TYPE {base} summary",
484            f"{base}_count{labels} {summary.count}",
485            f"{base}_sum{labels} {summary.total}",
486        ]
487    return "\n".join(lines) + ("\n" if lines else "")

Renders a Metrics in the Prometheus text format.

Written out rather than through a client library, so a service that only wants a scrape endpoint needs nothing installed. The format is small and stable: a TYPE line, then samples, with dots in the metric name — and in the label names, where routing.key and message.type have the same problem — replaced by underscores, because Prometheus allows them in neither.

A real registry — exemplars, native histograms, a shared process collector — is what acemq_amqp.prometheus is for, and it needs the extra.

Parameters
  • metrics: what to render :returns: the body of a /acemq-metrics response
def publish_chain( interceptors: Sequence[Callable[[PublishContext, Callable[[PublishContext], Awaitable[PublishResult]]], Awaitable[PublishResult]]], publish: Callable[[PublishContext], Awaitable[PublishResult]]) -> Callable[[PublishContext], Awaitable[PublishResult]]:
228def publish_chain(
229    interceptors: Sequence[PublishInterceptor], publish: PublishNext
230) -> PublishNext:
231    """Folds interceptors around a publish, first registered outermost.
232
233    Built backwards because that is what nesting is: the last interceptor wraps
234    the publish, the one before wraps that, and the first ends up on the
235    outside, seeing the message before anybody and its result after everybody.
236
237    :param interceptors: what to wrap it in, in registration order
238    :param publish: the innermost work, which actually sends the message
239    :returns: the whole thing as one callable, or ``publish`` when there is
240        nothing to wrap it in
241    """
242    composed = publish
243    for interceptor in reversed(interceptors):
244        composed = _publish_step(interceptor, composed)
245    return composed

Folds interceptors around a publish, first registered outermost.

Built backwards because that is what nesting is: the last interceptor wraps the publish, the one before wraps that, and the first ends up on the outside, seeing the message before anybody and its result after everybody.

Parameters
  • interceptors: what to wrap it in, in registration order
  • publish: the innermost work, which actually sends the message :returns: the whole thing as one callable, or publish when there is nothing to wrap it in
def register_codec(name: str, build: Callable[[], Codec]) -> None:
305def register_codec(name: str, build: Callable[[], Codec]) -> None:
306    """Makes a codec available by name.
307
308    So that configuration can name a format without the code that reads the
309    configuration importing every format it might name. Registering the same
310    name twice replaces the first, which is what lets a test override a default.
311
312    :param name: what configuration will call it, such as ``"json"``
313    :param build: makes an instance
314    """
315    with _registry_lock:
316        _registry[name] = build

Makes a codec available by name.

So that configuration can name a format without the code that reads the configuration importing every format it might name. Registering the same name twice replaces the first, which is what lets a test override a default.

Parameters
  • name: what configuration will call it, such as "json"
  • build: makes an instance
def reject(error: BaseException | None = None) -> Ack:
152def reject(error: BaseException | None = None) -> Ack:
153    """Dead-letters the message without trying again.
154
155    For when the message itself is the problem — a field that cannot be missing
156    is missing, a reference points at nothing — rather than when the world is
157    temporarily unhelpful.
158    """
159    return Ack(Action.REJECT, error)

Dead-letters the message without trying again.

For when the message itself is the problem — a field that cannot be missing is missing, a reference points at nothing — rather than when the world is temporarily unhelpful.

def retry(error: BaseException | None = None) -> Ack:
141def retry(error: BaseException | None = None) -> Ack:
142    """Returns the message to be tried again.
143
144    The retry policy decides whether there is another attempt left; when there
145    is not, the message is dead-lettered with ``error`` as the reason. A
146    :class:`Fatal` error skips the remaining attempts, because they would all
147    fail the same way.
148    """
149    return Ack(Action.RETRY, error)

Returns the message to be tried again.

The retry policy decides whether there is another attempt left; when there is not, the message is dead-lettered with error as the reason. A Fatal error skips the remaining attempts, because they would all fail the same way.

def retry_queue(queue: str, delay: datetime.timedelta) -> str:
45def retry_queue(queue: str, delay: timedelta) -> str:
46    """``orders.new`` and 30s become ``orders.new.retry.30s``.
47
48    The delay is in the name because a delay queue is per-delay: its
49    ``x-message-ttl`` is fixed at declaration, so a policy with four different
50    waits needs four queues, and an operator should be able to tell which is
51    which without reading its arguments.
52    """
53    return f"{queue}.retry.{_short(delay)}"

orders.new and 30s become orders.new.retry.30s.

The delay is in the name because a delay queue is per-delay: its x-message-ttl is fixed at declaration, so a policy with four different waits needs four queues, and an operator should be able to tell which is which without reading its arguments.

def rung_args(source: str, delay: datetime.timedelta) -> dict[str, typing.Any]:
120def rung_args(source: str, delay: timedelta) -> dict[str, Any]:
121    """The arguments a ``{source}.retry.{delay}`` queue must be declared with.
122
123    Exactly three keys, and the same three in every AceMQ library: the delay as
124    ``x-message-ttl``, :data:`RETRY_EXCHANGE` as the dead-letter target, and the
125    source queue as the routing key it is sent under. Nothing consumes a rung;
126    the time-to-live is the only thing that ever takes a message out of one, and
127    the binding made by :meth:`Topology.queue` is what brings it home.
128
129    The TTL is on the queue rather than on each message, which is the whole
130    reason a policy needs several queues instead of one. RabbitMQ expires
131    messages only from the head of a queue, so a single queue holding
132    per-message TTLs releases nothing while a message with a long one sits at
133    the front: a thirty-second wait queued behind a ten-minute wait becomes a
134    ten-minute wait. The delays that come out would bear no relation to the ones
135    that went in, and the bug would only appear under the load that puts two
136    different waits on the queue at once.
137
138    :param source: the queue being consumed, which is where an expired message
139        goes back to
140    :param delay: how long the rung holds a message
141    :returns: the argument table, which is a contract and is pinned by a test
142    """
143    return {
144        MESSAGE_TTL_ARG: int(delay.total_seconds() * 1000),
145        DEAD_LETTER_EXCHANGE_ARG: RETRY_EXCHANGE,
146        DEAD_LETTER_ROUTING_KEY_ARG: source,
147    }

The arguments a {source}acemq_amqp.retry.{delay} queue must be declared with.

Exactly three keys, and the same three in every AceMQ library: the delay as x-message-ttl, RETRY_EXCHANGE as the dead-letter target, and the source queue as the routing key it is sent under. Nothing consumes a rung; the time-to-live is the only thing that ever takes a message out of one, and the binding made by Topology.queue() is what brings it home.

The TTL is on the queue rather than on each message, which is the whole reason a policy needs several queues instead of one. RabbitMQ expires messages only from the head of a queue, so a single queue holding per-message TTLs releases nothing while a message with a long one sits at the front: a thirty-second wait queued behind a ten-minute wait becomes a ten-minute wait. The delays that come out would bear no relation to the ones that went in, and the bug would only appear under the load that puts two different waits on the queue at once.

Parameters
  • source: the queue being consumed, which is where an expired message goes back to
  • delay: how long the rung holds a message :returns: the argument table, which is a contract and is pinned by a test
def without_verifying_the_broker( *, client_certificate: 'FilePath | None' = None, client_key: 'FilePath | None' = None, client_key_password: str | None = None, credentials: Credentials | Callable[Credentials] | None = None) -> Security:
648def without_verifying_the_broker(
649    *,
650    client_certificate: FilePath | None = None,
651    client_key: FilePath | None = None,
652    client_key_password: str | None = None,
653    credentials: Credentials | CredentialsSource | None = None,
654) -> Security:
655    """Encrypts the connection and checks nothing about who is on the other end.
656
657    **This gives up most of what TLS is for, and it is worth being exact about
658    which part.** The traffic cannot be read by somebody watching the network.
659    Nothing whatsoever stops that somebody from *being* the broker: any
660    certificate is accepted, from any issuer, for any name, so an attacker who
661    can answer on the address in your URL receives every message you publish and
662    every password you log in with, over a connection that looks encrypted in
663    every log and every metric.
664
665    It exists because there is one situation where the alternative is worse: a
666    broker with a self-signed certificate on a laptop or in a test fixture,
667    where the choice is otherwise between this and turning encryption off
668    entirely. It is deliberately a function with a long name rather than a
669    ``verify=False``, because it should be impossible to end up here by pasting
670    a keyword argument and difficult to leave in a file nobody rereads.
671
672    **It is wrong in production, always.** Against a real broker, name the
673    authority instead — ``Security(certificate_authority="ca.crt")`` — which is
674    one more line and gives back everything this gives up. Self-signed
675    certificates are fine there too: put the broker's own certificate in the
676    file and it becomes the one authority you trust.
677
678    :param client_certificate: a PEM certificate to present
679    :param client_key: the PEM private key for it
680    :param client_key_password: the passphrase on that key, when it has one
681    :param credentials: the broker login, or a source asked for it
682    :returns: the configuration
683    """
684    return Security(
685        client_certificate=client_certificate,
686        client_key=client_key,
687        client_key_password=client_key_password,
688        credentials=credentials,
689        verification=Verification.NOTHING_AT_ALL,
690    )

Encrypts the connection and checks nothing about who is on the other end.

This gives up most of what TLS is for, and it is worth being exact about which part. The traffic cannot be read by somebody watching the network. Nothing whatsoever stops that somebody from being the broker: any certificate is accepted, from any issuer, for any name, so an attacker who can answer on the address in your URL receives every message you publish and every password you log in with, over a connection that looks encrypted in every log and every metric.

It exists because there is one situation where the alternative is worse: a broker with a self-signed certificate on a laptop or in a test fixture, where the choice is otherwise between this and turning encryption off entirely. It is deliberately a function with a long name rather than a verify=False, because it should be impossible to end up here by pasting a keyword argument and difficult to leave in a file nobody rereads.

It is wrong in production, always. Against a real broker, name the authority instead — Security(certificate_authority="ca.crt") — which is one more line and gives back everything this gives up. Self-signed certificates are fine there too: put the broker's own certificate in the file and it becomes the one authority you trust.

Parameters
  • client_certificate: a PEM certificate to present
  • client_key: the PEM private key for it
  • client_key_password: the passphrase on that key, when it has one
  • credentials: the broker login, or a source asked for it :returns: the configuration