Serialization

Nobody should have to think about this for the common case, so the common case needs no configuration:

mq.publisher("orders", "order.placed", Order.class).send(order);   // JSON
mq.consume("orders.new", Order.class, m -> handle(m.payload()));   // decoded

Write one format, read all

A publisher writes one format. A consumer reads every format on the classpath. That asymmetry is deliberate and it is what makes changing format two ordinary releases instead of a flag day:

1. Deploy consumers that can read the new format. They still read the old one.
2. Deploy publishers that write it.

The bytes on a queue are a contract with services that are not being redeployed right now. A queue carrying two formats at once is a queue nobody can write a consumer against — so the place to choose is once, where the destination is named.

Choosing a format

mq.publisher("legacy", "order", Order.class).asXml();
mq.publisher("config", "changed", Config.class).asYaml();
mq.publisher("flags", "changed", Flags.class).as(Codecs.byName("toml"));
mq.publisher("files", "file.new", byte[].class).asBytes();
mq.publisher("logs", "line", String.class).asText();
mq.publisher("events", "order", Order.class).as(new AvroCodec(registry));

Each returns a new publisher; the original is untouched. A long-lived object that quietly changes what it writes is worse than one that does not.

Names for Codecs.byName: json, text, bytes, plus xml, yaml, toml when their modules are on the classpath. string is accepted as another name for text, which is what .NET and Ruby call the same codec; text stays the canonical name and is the only one Codecs.names() lists.

Add the module for anything beyond JSON:

Codecs are found by ServiceLoader, so adding the dependency is the whole installation. Asking for a format whose module is absent fails immediately, by name, rather than at the first message.

YAML or TOML

Both are for the same audience: a message a person reads and edits as well as a program. They are text, larger than JSON and slower to parse, so neither belongs on a high-volume path — they earn their place where somebody actually looks at the message.

Between them:

TOML's rigidity is the point. YAML has a dozen ways to write the same thing and a famous habit of turning NO into false; if a person edits the message and a machine acts on it, fewer ways to be wrong is worth more than expressiveness.

A TOML document is a table. The top level has to be an object — a list, a string or a number has no TOML representation, and the codec refuses rather than writing something back. That refusal exists because Jackson does not: given a list it writes = ['a', 'b'], a key-less assignment that its own parser then rejects. Deeply nested payloads read poorly in TOML too; where the message is a tree rather than a table, JSON is the honest answer.

Avro and Protobuf

Neither can be built without a schema, so neither has a no-argument shortcut — asAvro() would only be able to fail:

Codec avro = new AvroCodec(schemaRegistry, OrderPlaced.getClassSchema());
mq.publisher("events", "order.placed", OrderPlaced.class).as(avro);

Avro uses the Confluent wire framing — a zero byte, a four-byte big-endian schema id, then the body — so consumers written against other Confluent-compatible tooling can read it.

What each accepts on the read side

A codec writes one content type and reads several. The write side is a choice; the read side is whatever the tool that wrote the message happened to say, and refusing a label that means exactly the same bytes turns a readable message into a poison one.

application/vnd.google.protobuf is what Google's own tooling and most schema registries write, so a message labelled that way is ordinary rather than exotic.

text/json is a legacy alias that is never correct to write and is written all the same, by older .NET stacks and a good deal of PHP. Go, Python and Ruby accept it, so Java does too — a read set that differs by language means a message one library reads and the library beside it dead-letters. The write side is unchanged.

The two Avro framings are the exception, and they stay separate: a registered message begins with five bytes a fixed-schema codec would read as the start of the first field, which does not throw — it produces a record whose every value is wrong. Each accepts only its own framing.

Say what the bytes are

A fixed-schema AvroCodec reads the content type before it looks at the bytes. Told avro/binary, it decodes the body as it stands. Told application/vnd.acemq.avro, it refuses, because that is the other framing.

Only when nothing useful was said does it fall back to guessing from the first byte, and a leading zero is then read as Confluent framing. That guess is a last resort because it cannot be made accurate: an Avro body begins with a zero byte whenever its first field encodes to zero — an empty string, a 0, a false, the first branch of a union — all ordinary values. Publishing with a content type set is what removes the ambiguity, and every publisher in this library sets one.

The schema registry

Avro's bytes and Protobuf's carry no account of what they are. A reader has to already hold the schema the writer used, and what travels in the message is a small integer standing for it. Something has to remember which integer means which schema.

JdbcSchemaRegistry registry = new JdbcSchemaRegistry(dataSource);
registry.createSchemaIfAbsent();

Identifiers have to be stable forever. A message published today may be read next year by a consumer looking up the schema it was written with. A registry that hands out fresh identifiers on restart makes every message written before the restart unreadable, and does it silently — the bytes still parse as a schema, just not the right one.

JdbcSchemaRegistry caches both directions in memory and never invalidates, because neither answer can change: an identifier stands for one schema forever, and a schema keeps the identifier it was given. Registration is idempotent by content — the same schema text registered from eight replicas at once yields one row and one identifier.

It is not Confluent's registry and does not try to be: no compatibility checking, no versioning UI, no REST API. It remembers which integer means which schema, which is the part the wire format depends on. Where a Confluent registry is already running, point the codec at that instead.

createSchemaIfAbsent() is for development. In production the table belongs in whatever migration tool already owns the schema.

Schema resolution

Handed the writer's schema and the reader's, Avro reconciles the two: a field the writer added that the reader does not declare is skipped, and a field the writer omitted is filled in from the reader's default. Handed only the writer's, there is nothing to reconcile, and the record arrives in the shape it was written.

So there is one rule, and it is the same rule in all five AceMQ libraries:

Resolution happens when the library has a reader schema to resolve onto.

What differs between the languages is where a reader schema comes from, and therefore how often there is one. Nothing about the bytes differs.

This is not an inconsistency waiting to be flattened. A library that resolves and a library that does not are both right about the same bytes — they are answering different questions, because only one of them was told what the reader expects.

The case that bites is a field the writer removed that the reader declares with a default. With resolution, the field arrives carrying that default. Without it, the field is simply absent: a missing key, whatever the language calls one. A consumer written against the reader schema then reads a value that was never on the wire, or fails to read a field it is sure it declared, and which of those happens is decided entirely by whether a reader schema was in play.

The other direction is the one people expect to be dangerous and is not. A field the writer added that the reader does not declare is skipped under resolution and present without it, and either way the fields the reader does declare come back correct — the unknown field does not shift the ones after it.

Both cases are pinned, with the bytes, in acemq-amqp-test/src/test/resources/fixtures/avro-resolution-fixtures.json, which every AceMQ library carries a copy of. It records the decoded value under each behaviour, as resolved and writerShape, and which library lands on which.

Asking for resolution in Java

// Resolves: every message is reconciled with the schema this consumer holds.
Codec codec = AvroCodec.registered(registry, OrderPlaced.getClassSchema());

// Does not resolve: the record arrives in the shape the producer wrote it.
Codec asWritten = AvroCodec.registered(registry);

Reach for the first unless there is a reason not to. Being able to redeploy a producer without its consumers is the whole point of putting a schema id on the front of the message, and resolution is the half of that which happens on the read side.

A generated class can also be the decode target directly:

OrderPlaced order = AvroCodec.registered(registry)
        .decode(body, OrderPlaced.class);

The class carries its own schema, so that schema is the reader schema and the resolution is the same one described above — you simply do not have to name it twice. Both routes give the same record; use whichever reads better where you are. Naming the schema explicitly is still the only option when the reader schema is not a generated class.

(Through the 0.7 line this threw a ClassCastException: the resolution happened, but a registry codec always built a generic reader and the result could not be handed back as the class asked for. On those versions, name the schema.)

Your own format

public final class CsvCodec implements Codec {
    public byte[] encode(Object payload) { ... }
    public <T> T decode(byte[] body, Class<T> type) { ... }
    public String contentType() { return "text/csv"; }
    public boolean canDecode(String contentType) { ... }
}

mq.publisher("reports", "daily", Report.class).as(new CsvCodec());

Register it for the read side by publishing a CodecProvider through ServiceLoader, and consumers will accept it alongside the built-ins.