Messaging Basics

In the realm of software development, the way applications communicate with each other is critical to achieving scalability, flexibility, and resilience. This is where messaging comes into play, acting as a powerful tool in a software architect’s toolkit. Let’s delve into messaging paradigms, patterns, and their value in modern software architectures.

Fundamentals

There are many reasons to choose Distributed systems or Microservices architectures, simplicity is not one of them, these architectures do however help to force some good design patterns such as loose coupling between bounded contexts, for more on bounded contexts please ready my Domain Driven Design Article. Messaging is, at its essence is a way of exchanging information asynchronously unlike traditional function calls where a component directly invokes another, messaging separates the sender from the receiver. This decoupling offers several significant benefits:

• Loose Coupling : Components don’t need intimate knowledge of each other’s implementation details, reducing dependencies and making applications more flexible and maintainable.
• Scalability : Senders and receivers can be scaled independently to handle varying loads, ensuring system responsiveness even during peak usage.
• Resilience : Messaging systems introduce fault tolerance by queuing messages and retrying if delivery initially fails. This helps ensure that critical information is eventually processed.

Components of messaging architectures

There are many terms that you might hear thrown around when talking about messaging :

• Message Brokers : Software that sits between senders and receivers, handling the complexities of message delivery, routing, and storage. Popular examples include Azure Service Bus, RabbitMQ, Apache Kafka, and ActiveMQ.
• Message Queues : While sometimes used interchangeably with message brokers, classic message queues like Amazon SQS or IBM MQ usually focus more on reliable queuing and less on the topic-based features.
• Queues : Used for point-to-point messaging. Messages are added to a queue by the sender and a single receiver processes them in the order they were received. Provides reliability and load balancing. A Queue should have a defined data schema.
• Topics : Used for publish-subscribe messaging. Messages are published to a topic, and multiple subscribers can consume them. This enables broadcast-style communication. A Topic should have a defined data schema.
• Subscription : A Child of a Topic essentially a Queue that a copy of all messages are placed on for each consumer to consume.
• Consumers : Sometimes called Subscribers or Receivers these are the components within the system that listen to Topics or Queues.
• Producers : Sometimes called Publishers or Senders these are the components within the system that create and send to Topics or Queues.
• Dead Letter : When a message is undeliverable (Often due to transient failures persisting across retries) the message can moved into the Dead Letter Queue (DLQ).

Types of messages

There are 3 main types of messages

• Command : ‘Do Something’ these are usually sent internally within your domain or system, an example in a Financial system could be to Instruct a Deposit. It is expected that a Command has a single consumer.
• Event : ‘Something has happened’ in the example above, once the deposit has been instructed we could event a Deposit has been Instructed. An event has an unknown number of consumers.
• Document : These are less common, Document messages encapsulate a self-contained unit of data in a structured format (like XML, JSON, or similar). This structure makes it easier for the receiving component to understand and process the information.

When sending a Command it should be known who is going to consume the message and what action they will take. When sending an Event it is not known who will consume the message or what actions they will then take. An example I like to use, is if I Was to ask one of my friends Can you please make me a Coffee that would be aligned to a Command, I have asked something of them, I would either expect them to either say no, or make me a coffee. On the other hand, if I was in a room with my friends and I said I'm thirsty this is aligned to an Event, I have updated them on my thirst status, maybe one of my friends would make me a coffee, maybe two would, maybe no-one would.

Messaging Patterns

There are two fundamental patterns that are commonly used:

1. Point-to-point (Aligned to Queues)
   • A single sender transmits messages to a queue.
   • A single receiver consumes messages from the queue.
   • It ensures that each message is processed only once.
   • Ideal for task distribution and load balancing.
   • Aligned to Commands.
2. Publish-Subscribe (Aligned to Topics)
   • Senders (publishers) broadcast messages to a topic.
   • Multiple interested receivers (subscribers) can listen to the topic and process the messages.
   • Best for event notifications and real-time data updates.
   • Aligned to Events.

Scalability

If all calls to your web server simply send a message which are then picked up by a farm of consumers to process, the response time from the web server is significantly faster which means that they can handle more clients and it appears more responsible to clients. Assuming your consumer farm has appropriate back pressure on its queue this will mean that you can process slower without having a real effect on clients.

Consuming Messages

Unlike most systems, Performing a Read Operation from a Messaging Broker is destructive, this is because messages are intended to be consumed then forgotten. The way to resiliently consume messages is to take a Peak Lock on the message, this will hide the message on the Topic or Queue While it is locked, if the consumer is able to process the message successfully the message can then be Acknowledged (ACK) or if not The locked can be Abandoned, removing the lock and un-hiding it for processing, or Dead-Lettered Moving it to the Dead Letter Sub-queue.

Reliability

Messaging Brokers generally have different levels of guarantees:

• At Most Once Delivery : The messaging system will attempt to deliver a message once, but there’s no guarantee of success. Messages may be lost if there’s a failure during transmission or before the receiver acknowledges receipt. This is the least reliable option however if Speed is your highest priority this is the option for you.
• At Least Once Delivery : The messaging system guarantees that a message will be delivered at least once. However, in case of failures, duplicate messages are possible. The receiver must be prepared to handle these duplicates, often using idempotency techniques (ensuring an operation performed multiple times has the same end result). If ensuring that every message is processed then this is the option for you.
• Exactly Once Delivery : The gold standard of messaging guarantees! The system ensures a message is delivered once and only once. This is the most complex to achieve, and typically involves advanced techniques like distributed transactions and deduplication on the receiving side. If you cannot achieve idempotency in your consumers and processing a message twice would be disastrous then this is what you require.

Conclusion

As modern software systems become increasingly complex and interconnected, messaging stands as an indispensable architectural paradigm. By embracing loose coupling, promoting scalability, and ensuring resilience, messaging empowers developers to build distributed applications that are flexible, responsive, and able to thrive in demanding environments.