Communication Protocols & Message Brokers for Microservices

In this article, I am going to discuss the messaging protocols and message brokers for microservices. Microservice architecture is one of the trends and most talked-about software architecture which addresses the most reported issues in monolithic applications. It is useful to develop large complex systems with loosely coupled modules. Therefore, each microservice is a mini-application with its distinct architecture and logic. So, It’s easier to develop, understand, and maintain individual services.

Key Note:- How microservices can be communicated with each other once we have broken down the single application into small modules? 🤔

So, microservices can be communicated via both messaging and REST APIs. In the realm of microservices, it’s common to employ messaging protocols to facilitate asynchronous communication among microservices. We can further classify the above two communication protocols as below.

Synchronous VS. Asynchronous Communication

Communication Protocols

1. REST

Synchronous communication type and caller waits for a response from the server. The most commonly used architectural style for request and response communication.

Drawbacks of using REST in Microservices

1. Multiple round trips (latency) — The client often needs to execute multiple trips to the server to fetch all the data the client requires.
2. Blocking — The client is blocked and is waiting for a server response. It Can affect performance if the application thread is processing other concurrent requests.
3. Tight coupling — The client and server need to know about each other.

2. Messaging Protocol

This protocol is widely used in a microservices architecture and it follows the asynchronous protocol. Service sends a message without waiting for a response and one or more end services process the message asynchronously. So asynchronous communication is usually managed through a message broker I will discuss more information about message brokers from the next section.

Advantages of using Messaging in Microservices

1. Loose coupling — The message producer does not need to know about the consumer(s).
2. Multiple subscribers — Using a publisher/subscriber (pub/sub) model, multiple consumers can subscribe to receive events.
3. Resiliency or failure isolation — If the consumer fails, the producer can still send messages. The messages will be picked up when the consumer recovers from failure.
4. Non-blocking — The producers and consumers can send and process messages at their own pace.

What is a Message Broker? 🤔

Okay, Now let’s talk about message brokers.

Message broker is an application that supports to exchange of messages between applications, systems, and services in an asynchronous manner. Message brokers enable efficient message queuing, routing, and delivery.

Except for the above primary functionalities, brokers provide scalability, reliability, transparency, and extendable security features as well. It helps to prevent the loss of essential data and allows systems to continue operating even with intermittent connectivity or lag on networks.

Nowadays message brokers are a popular way of implementing message queues due to the fact that they provide a lot of functionality out of the box and support to integrate applications and automate message transmission.

Components of Message Broker? 📃

• Producer:-

Responsible for transmitting messages & also known as publishers in the publish/subscribe model.

• Consumer:-

Receives messages waiting in the message broker & also known as subscribers in the publish-subscribe pattern.

• Queue or Topic:-

Folders in a system and message brokers use them to store messages. Queue is best suited for point-to-point communication with a single receiver, while a Topic is ideal for broadcasting messages to multiple interested parties.

Common Challenges of Message Brokers😮

• Message Consumption and Rollback Handling

After consuming a message from the queue, the message is no longer available in the queue. If a consumer fails to process the message, the message is lost and may need a rollback.

• Tight coupling with the messaging infrastructure

Since using a particular vendor/messaging infrastructure you may face some difficulties when switching to another vendor/messaging infrastructure later.

• Operational complexity

Message queues do not come out of the box. Need to create, configure, and monitor.

Messaging Patterns

1. PUB/SUB Pattern

• The publisher sends a message to a channel(Topic) on a message broker.
• One or more subscribers subscribe to the channel and receive messages from the channel in an asynchronous manner.
• This pattern is useful when a microservice needs to broadcast information to a significant number of consumers.

Advantages of Pub/Sub Pattern

1. Broadcast Communication and Dynamic Subscriptions
2. Decouple publishers and subscribers

• Publishers and subscribers can be managed independently.
• Messages can be managed even if one or more subscribers are offline.

3. Increase scalability and improve the responsiveness of the publisher

• Can easily scale to accommodate a large number of subscribers and publishers.
• The messaging infrastructure is responsible for ensuring messages are delivered to interested subscribers.

2. Queue Based Pattern / Point-to-point(P2P)

• In this pattern, one or many consumers can access that queue. However, any message in that queue can be received by only one consumer.

Note:- If you need to configure two or more consumers for guaranteed message delivery( if a consumer crashes, the remaining one can continue the processing), we have to make sure that the message is consumed only by a single consumer. In that case, the broker can be configured to dispatch the messages among the consumers in a round-robin manner. So that the workload is load-balanced across consumers.

• Queue acts as a buffer, storing the message until it is retrieved by the receiver.
• This pattern is useful to ensure that a message is received by only one consumer application.

Advantages Queue-Based Pattern

1. Integration:- common messaging infrastructure
2. Guaranteed Delivery:- Messages remain in the queue until they are consumed.
3. Scalability & availability: The number of queues and the number of services can be scaled to meet demand, Delays arising in the producer or consumer won’t have an immediate or direct impact on the services.
4. Message Order:- messages are typically processed in the order they are received

When choosing a Message Broker…🙄

Currently, many message brokers are available and you have to choose the correct one according to your business requirements. So you can consider the below factors as a help to select the right messaging infrastructure for async communication.

• Scalability — Number of messages sent per second or the ability to scale automatically when there is a load surge on the message broker
• Data persistency — Ability to recover messages in case of reboot/failure
• Consumer capability — whether the broker can manage one-to-one and/or one-to-many consumers
• Monitoring — whether monitoring capabilities are available
• Push and pull queue — Ability to handle push and pull delivery by message queues
• Security — Proper authentication and authorization for messaging queues and topics
• Automatic failover — Ability to connect automatically to a failover broker when one broker fails without impacting publisher/consumer

Popular Message Brokers

• Apache Kafka
• Apache ActiveMQ
• RabbitMQ
• Amazon SQS

Message Broker Comparison

Here, I’ve compared the most popular brokers according to their features and functionalities. As I mentioned earlier, selecting a one-message broker totally depends on the business requirements.

As an example, if you need to integrate your services with a message broker that can be downloaded free, configured in local servers, and has a management interface to manage the publishers and queues, you have to select one among Apache ActiveMQ or RabbitMQ.

Message Broker Comparison

• Even though Kafka doesn’t have an inbuilt management console, you can configure a third-party application such as Confluent to manage and monitor Apache Kafka. Apache Kafka has a very good storage system and it provides data persistency and stores streams of records.
• If you move with the AWS SQS, no need to maintain a message broker server or infrastructure on your side. AWS will guarantee for message brokers and you can raise any issue related to infrastructure or anything related to message broker and AWS will take care of them. Further, AWS will automatically handle the load of messages and cater to them accordingly by scaling up.

That’s it and hope you’ve got a clear understanding of microservice communication and message brokers. See you in the next article. 😀