Choosing Event Streaming or Messaging for your Architecture

Want to understand what to consider when choosing Event Streaming and Messaging for your Architecture?

They both are heavily used when it comes to Architecture, they are fantastic ways to manage the interactions, communication and behaviour between your services or systems and often there is not a one size fits all approach you should take.

Just because you have taken one of these approaches in a previous solution does not mean that it the right approach for your next, hopefully this post will be able to give you a high level overview on what you should consider and help you to mange your Architecture decision.

Before you start

About this post:

  • 10 – 20 min average reading time
  • Suitable for beginner through to advanced

What you will gain reading this post:

  • An understanding of the differences and use cases for choosing the different approaches.
  • Understand the cloud technologies available and which to choose.

What you can do to help support:

  • Like, comment and share this article
  • Follow this blog to receive notifications of new postings

Now, let’s get started.

Okay, what terminology do I need to understand before progressing?

What is an Event?

  • An event encapsulates a change in state (what has happened)
  • It should be lightweight and carry only enough information required about the change in state
  • An event is distributed to notify any interested parties
  • Events are distributed through channels such as streaming and messaging

An example, when a consumer borrows a DVD, the DVD state changes from “for rent” to “rented”.

What is a Message?

  • A message encapsulates the intention / action (what has to happen)
  • It is not lightweight containing a payload of all data required for processing
  • A message can be formatted and adhere to a contract to be suitable for the interested parties
  • A message is distributed through channels such as messaging

An example, when a customer places a reservation on a DVD but requests it be collected from another location, the booking system receives a message to reserve the stock and trigger a transfer for the DVD to another location.

Isn’t Events and Messages interchangeable?

A message is often interchanged with the term Event, however it is intended for different purposes even though there are many commonalities between them.

Sample Command and Event Flow Diagram
Sample Command and Event Flow Diagram

Referring to the CQRS pattern, checkout the Command message (message) and Event message (event) which demonstrates the differences between them.

What is Event Sourcing?

  • Event sourcing is a way to atomically update state and publish events.
  • Event sourcing is about a business object being persisted by storing a sequence of change in state events, that when replayed will show the entire state of the business object, rather than saving only the current state.

An example is a Payment Service.

  • It stores a Payment by persisting its change in state events – Created, Paid, Cancelled.
  • Each event would contain sufficient data to reconstruct the Payment’s state.

What is a Stream?

  • stream consists of immutable data, only inserting new events, whereas existing events cannot be changed.
  • Streams are persistent, durable and fault tolerant.

What is Streaming?

  • Streaming of data is the constant flow of events where each event should contain enough information to reflect the change in state.
  • It allows for the processing of data to occur in real-time (data in motion) and is different from the traditional approach for the processing of static data to occur (data at rest) at a later point in time, known as batch processing
  • Streaming data is unbounded, meaning it has no real beginning and no real end
  • Each event is processed as it occurs and is managed accordingly

An example of this can be found in the stock market.

  • When a stock price change, a new event is created containing the time and day, the stock identifier, and its new trade price which in this example is the change in state
  • Given there are thousands of stocks, and thousands of trades happening every second, this results in a constant stream of data

What is Publisher (pub)?

  • An application that publishes a message to a topic
  • It does not know the destination of the message

What is Subscriber (sub)?

  • An application that registers itself to a topic to receive the messages
  • It does not know the source of the message

What is a Queue?

  • A Queue is a mechanism for system to system communication that is asynchronous in behaviour
  • Queues store messages until they are processed and deleted.
  • Each message is processed only once and for a single consumer.
  • Queues can be used to decouple heavyweight processing, to control the flow of an influx or batch data, and to support erratic workloads.
  • This is typically used in Serverless and Microservices architectures.

What is Topic?

  • Is a channel that maintains a list of subscribers to relay messages to that are received from publishers

Great, now the terminology is clear, let’s explore these in greater detail

Why use Event Streaming?

Event streaming is really powerful when you have events that you want to be able to process and perform analysis in real-time allowing your systems to immediately take action.

Use Case
Processing
– Payments
– Stocks
Detection
– Fraud
– Anomaly
Maintenance
– Predictive
Analytics
– IoT
Event Streaming Use Case Table
  • Event streaming is used for real-time processing and analysis of changes in state in the data through events
  • It can persist events supporting the rebuilding of the state through replaying events in the order they were received
  • It allows multiple consumers to receive each event
Event streaming pattern
Event Streaming pattern

What Cloud services are available?

Depending on the cloud provider you are using the options are quite extensible and depending on your use case the suitable technology choice may differ.

Below is a simplistic decision flow to help you determine which technology may be more suitable depending on your use case, for more complex scenarios it would be worth asking yourself more questions to determine if the particular technology choice will be suitable.

Event Streaming technology options
Event Streaming technology options

Why use Messaging?

Messaging is powerful when it comes to decoupling your systems and providing a highly available and durable solutions, apart from a few use cases, messaging is more of a way to manage your system interactions and control the ingestion of your data.

Pub / Sub

Use Case
Applications
– Stateful
Workflows
– Asynchronous
Workloads
– Balancing
Systems
– Stocks
Notifications
– Email
– SMS
Delivery
– Multiple consumers
– Ordered
Messaging Pub / Sub Use Case Table
  • Publish-subscribe is commonly referred to as pub-sub
  • Pub-sub moves data from producers to consumers
  • It allows multiple consumers to receive each message in a topic
  • Publish-subscribe ensures that each consumer receives messages in a topic in the exact order in which they were received by the messaging system
Message Pub / Sub (Push / Pull) pattern
Message Pub / Sub (Push / Pull) pattern

Queueing

Use Case
Workflows
– Asynchronous
Workloads
– Balancing
Systems
– Pay Check
– Task Lists
– Work Queues
Delivery
– One consumer
– Ordering not important
Message Queueing Use Case Table
  • Message queueing ensures that for exactly one consumer each message is delivered and processed
  • It does not ensure that messages are delivered or processed in order
  • However, each message is removed from the queue once it has been delivered but it does requires consumer acknowledgement
Message Queue (push / pull) pattern

What Cloud services are available?

Once again, depending on the cloud provider you are using the options are quite extensible and depending on your use case the suitable technology choice may differ.

Below is a simplistic decision flow to help you determine which technology may be more suitable depending on your use case, for more complex scenarios it would be worth asking yourself more questions to determine if the particular technology choice will be suitable.

Messaging Pub / Sub & Queueing technology options
Messaging Pub / Sub & Queueing technology options

Conclusion

  • Do you understand your use case?
  • Do you understand how this may evolve?
  • What does the roadmap for your product look like?

This will be the driver for the technology choice that you should use.

If you require…

  • Processing and analysis of your data in real-time
  • If you have a constant flow of data changing in state that you may want to persist, replay or use to recreate your state

Then event streaming is the path to explore.

However, if you require…

  • To have decoupled, highly durable and highly available Architecture
  • The execution to be asynchronous in behaviour
  • If you don’t require real-time processing of your data, or if you are issuing commands

Then messaging is the path to explore.

Does that mean that you can’t combine these approaches?

Not at all, however event streaming is a much more complex setup and therefore my recommendation is if your use case won’t evolve and require real-time processing and analysis of your data, then the complexity that event streaming adds to your solution most likely is not worth it.

Did this help?

  • Like, comment and share this article
  • Follow this blog to receive notifications of new postings
  • View previous postings
Share your support
Spread the love

Author: Robert Leggett

I am a passionate, pragmatic and focused leader for people and technology. I love to converse in Spanish, play video games and spend time with my family.

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.