Schema evolution in distributed Async message-passing systems

Message schemas in distributed systems

In asynchronous message-passing systems, producers and consumers communicate through a broker rather than connecting directly. A producer publishes a message to a specific topic, while a consumer subscribes to that topic and processes the messages it receives. This decoupling allows producers and consumers to operate and be deployed independently.

Throughout this article, we use a publish-subscribe model with topics as the running example.

This model is especially common in systems that involve physical devices. In the architecture used throughout this article, the producer is a robot operating in the field, while the consumer is a backend responsible for processing the received data.

For this communication to work correctly, both sides need to agree on the structure of the data being exchanged: which fields a message contains, what types those fields have, and which of them are required. We will refer to this contract as the message schema.

For example, assume that a robot sends a message with the following structure:\

{"battery": 80}

The robot needs to know how to construct such a message, while the backend needs to know how to interpret it correctly. Before it can be transmitted, however, the message needs to be serialized.

We encountered this problem while working on distributed communication between autonomous robots and the backend of Thoro.ai's fleet-management platform.

Serialization and deserialization

Serialization converts application-level data into a representation that can be transmitted as a message payload.

For example, the producer may represent a message internally as:

message = %{battery: 80}

and serialize it to JSON:

payload = JSON.encode!(message)

producing:

{"battery": 80}

The serialized representation becomes the message payload sent through the broker.

On the consumer side, the reverse process takes place. The received payload is deserialized back into a representation that the application can work with:

message = JSON.decode!(payload)
# %{"battery" => 80}

JSON is used here only as a simple example. Other serialization formats represent data differently and may handle schema evolution in different ways.

Serialization defines how data is represented for transmission, but it does not replace the shared contract. The producer and consumer still need to agree on the structure and meaning of the message.

As long as both sides use the same version of the message schema, this is straightforward. The interesting part begins when that schema needs to change.

Example: adding a field to a message schema

Assume that the current version of the message schema contains only a battery field:

{"battery": 80}

We now want to extend the schema with a new distance field:

{"battery": 80, "distance": 100}

An additive schema change like this is a relatively simple example of schema evolution, making it useful for understanding compatibility during a migration.

The change itself is straightforward. The deployment is not.

The producer and consumer are independent components and are not necessarily updated at the same time. During the rollout, some producers may still use the old schema while others already use the new one. The consumer therefore needs to be prepared to receive both variants.

During the migration, the backend must tolerate messages in which distance is missing, because older producers still use the previous schema:

def handle_message(%{"battery" => battery, "distance" => distance}) do
  process_message(battery, distance)
end

def handle_message(%{"battery" => battery}) do
  process_message(battery, nil)
end

This allows the new consumer to process both an old message:

{"battery": 80}

and a new one:

{"battery": 80,"distance": 100}

In the target v2 schema, distance is a required field for new producers. The consumer accepts its absence only to remain compatible with messages produced according to v1. This is an example of backward compatibility: the new consumer can still process messages produced according to the older schema.

The opposite direction is also important. If producers are upgraded first, an old consumer may start receiving the new distance field. If the old consumer tolerates unknown fields, the message may still be processed correctly. If it performs strict validation and accepts only the fields defined in the old schema, the same message may be rejected. A consumer that can correctly process messages produced according to a newer schema is forward compatible with that change.

These two directions of compatibility become especially important during rollout, when different producer and consumer versions coexist.

Backward vs forward compatibility

During the migration, four producer-consumer version combinations are possible:

The first and last cases are straightforward because both sides use the same version of the contract.
The two mixed-version cases are the important ones.

Producer v1 → Consumer v2 is the state we want to support first. The producer still sends the old message format, while the new consumer already understands both versions. This is the backward-compatible path.

Producer v2 → Consumer v1 is more risky. The new producer sends a field the old consumer does not know about. If the old consumer ignores unknown fields, the message may still be processed correctly. If it validates the payload strictly, the message may be rejected.

This is why, for this type of schema change, upgrading the consumer first is usually the safer rollout strategy. It avoids making the migration depend on the forward compatibility of the old consumer.

When to introduce a compatibility layer

For simple schema changes, keeping the consumer backward compatible is often the simplest solution. If the only difference between two schema versions is the addition of a new field, supporting both directly in the consumer may add very little complexity.
In systems with physical devices, however, the migration period can be much longer than a typical service rollout. Some producers may remain on older software versions for weeks, months, or years, turning backward compatibility into a long-lived requirement rather than a temporary deployment state.

As more legacy schemas accumulate, or the transformations between them become more complex, compatibility logic can significantly increase the complexity of the consumer itself. At that point, an alternative is to move this responsibility into a dedicated compatibility layer.

The compatibility layer translates messages using an older schema into the current canonical representation before they reach the consumer. This allows the consumer to operate on a single schema version and keeps legacy-specific logic outside the core business code.
The trade-off is additional infrastructure and operational complexity. A compatibility layer therefore makes sense when maintaining legacy support inside the consumer becomes more costly or difficult than maintaining the translation layer itself.
For short-lived migrations with small schema differences, backward compatibility in the consumer is usually sufficient. For long-lived legacy support or increasingly complex transformations, isolating compatibility logic can keep the consumer significantly easier to maintain.

Key takeaways for schema evolution

Schema evolution in distributed message-passing systems requires thinking about compatibility between independently deployed producers and consumers. Even a simple change, such as adding a new field, can create mixed-version states during rollout. For this type of migration, updating the consumer first is usually the safer approach.
If supporting legacy schemas starts to significantly complicate the consumer, moving compatibility logic into a dedicated compatibility layer can be a reasonable alternative.

To learn more about what we’ve been doing in the robotics field, on the example of Thoro.ai, see our Thoro.ai case study showcasing our contributions.

‍