The outbox pattern emerges as a powerful solution for ensuring reliable event delivery, guaranteeing events reach their destination even in the face of failures. This introductory tech article delves into the outbox pattern’s high-level structure, exploring how it empowers us to confidently update entities and trigger downstream actions with the peace of mind that events will be published, no matter what.
When a user makes a call to an API to update their address typically this will result in an update to the data store, in event driven systems there would also be an expectation of an event being sent so that all downstream consumers are notified of the change and can act accordingly.
On the happy path (given there are no unforeseen issues) this is fine, but when working in distributed systems or cloud systems, every piece of infrastructure must be unavailable. In the above example there are two ways in which this will adversely affect the application:
- If the changes are persisted to the database first but the messaging bus is unavailable, the User’s address will be updated, but the event will not be sent.
- If the changes are placed on the messaging bus first but the database is unavailable, downstream systems will be notified of a change that did not take place.
This is the problem that the Outbox Pattern aims to solve.
Outbox Pattern
Simply put, the outbox pattern is storing messages inside of the same data store as the entity and then publishing it is a second operation.
- In the event that the database is unavailable nothing is saved or sent.
- In the event of the messaging bus being unavailable, the publisher will continue attempting to send the message until it is successful.
Benefits
- Events are ensured to be sent.
- The ability to not have to wait for the message to be sent to return a successful message to the end users (resulting in faster response times for the API)
Extra Complexities
- Requires a separate publisher
Summary
In summary the outbox pattern is a must when working in event driven systems as missing changes can cause issues downstream. I have also seen the use of this pattern save a significant amount of time (hundreds of separate) from API calls as there is no requirement to wait for the message to be sent before returning a successful status to the caller.