Pub-Sub: The Digital Megaphone

1. Beyond 1-to-1 Messaging

In a standard Message Queue (Point-to-Point), one message is processed by one consumer. This is great for work distribution (e.g., “Resize this image”).

But what if multiple services need to know about an event?

  • User Signs Up:
    • Email Service needs to send a welcome email.
    • Analytics Service needs to log the event.
    • Fraud Service needs to check the IP.

If we use a standard queue, only one of them will get the message. We need Pub-Sub.

2. The Pub-Sub Model

Publish-Subscribe decouples the sender (Publisher) from the receivers (Subscribers). Think of it like a Radio Station: The DJ (Publisher) broadcasts, and anyone tuned in (Subscribers) hears it.

Core Components

  1. Publisher: Sends a message to a Topic (not a specific queue).
  2. Topic (Exchange): The “Post Office” that decides where to route the message.
  3. Subscription: Services “subscribe” to the topic.
  4. Fanout: The broker copies the message to ALL subscribers.

3. Exchange Types Deep Dive

Not all Pub-Sub is the same. In AMQP (RabbitMQ), the “Topic” is actually called an Exchange. There are 4 main types:

Exchange Type Routing Logic Use Case
Direct Exact Match (routing_key == binding_key) Unicast routing (e.g., error.log → ErrorQueue).
Fanout Broadcasts to ALL queues. Ignores key. “Mass Notification” (e.g., New User Signup → Email, Analytics, Fraud).
Topic Pattern Match with Wildcards (*, #). Multicast routing (e.g., payment.us.* vs payment.#).
Headers Matches message headers (metadata) instead of key. Complex routing where the routing key string isn’t enough.

[!TIP] Performance: Fanout is the fastest (O(1) complexity) because it blindly copies messages. Topic exchanges are slower (O(N) complexity) because they must parse the string pattern against the routing table.

4. Interactive Demo: Topic Exchange & Wildcards

Visualize how messages are routed using Routing Keys.

  • Scenario: A Payment System emitting events like payment.us.success or payment.eu.error.
  • Goal: Route specific events to specific services using Wildcards (* matches one word, # matches many).
  • Action: Click the buttons to publish different events and watch who receives them.
Publisher (Payment Gateway)
Tap to emit event
🔀
TOPIC
Sub: # (All)
📜
Audit Log
0
Sub: payment.us.*
🇺🇸
US Region
0
Sub: *.error
🚨
PagerDuty
0

5. Why is this powerful?

1. Loose Coupling (Plug & Play)

If we need to add a new Recommendation Service next month, we don’t change the Payment Gateway code. We just add a new subscriber. The Publisher doesn’t know (or care) who is listening.

2. Parallel Processing

All subscribers receive the message simultaneously (or near simultaneously).

  • Email sent: 200ms
  • Analytics logged: 50ms
  • Fraud check: 100ms Total Time: Max(200, 50, 100) = 200ms (Parallel), instead of 350ms (Sequential).

3. Fanout Cost Analysis

Be careful with Fanout. If you have 1 event and 100 subscribers, the broker must create 100 copies.

  • CPU Cost: Low (it’s just a pointer copy).
  • Network Cost: High (100x bandwidth).
  • Storage Cost: High (if durable queues are used).

6. Filtering Strategies

Where should the filtering happen?

A. Broker-Side Filtering (Efficient)

The broker (Exchange) decides which queue gets the message.

  • Example: RabbitMQ Topic Exchange.
  • Logic: If no one binds to sys.debug, the message is discarded immediately at the broker.
  • Pros: Saves network bandwidth and consumer CPU.
  • Cons: Broker works harder (CPU usage for pattern matching).

B. Consumer-Side Filtering (Wasteful)

The consumer receives everything and filters it in code.

  • Example: All services listen to a “Firehose”.
  • Logic: if (msg.type != 'error') return;
  • Pros: Dumb broker (fast).
  • Cons: Massive waste of bandwidth. The consumer wakes up for nothing.

[!TIP] Always prefer Broker-Side Filtering for high-volume systems. Only send data to services that actually need it.

7. Understanding Wildcards

Sometimes you don’t want all messages. You want to subscribe to a subset.

Wildcards (RabbitMQ Example)

  • Topic format: service.region.status
  • Wildcard * (Star): Matches exactly one word.
    • payment.us.* matches payment.us.success but NOT payment.us.db.error.
  • Wildcard # (Hash): Matches zero or more words.
    • payment.# matches payment.error, payment.us.success, and payment.debug.level.1.

8. Delivery Semantics

When doing Pub-Sub, you must decide your guarantee level:

Semantics Description Pros Cons
At-Most-Once Fire and Forget. Message might be lost. Fastest. No state tracking. Data loss possible.
At-Least-Once Retries until ack received. No data loss. Duplicates (Requires Idempotency).
Exactly-Once Hardest to achieve. Transactional. Perfect consistency. High latency, complex.

9. Summary

  • Use Queues for 1-to-1 work distribution.
  • Use Pub-Sub for 1-to-Many notifications.
  • Enables Event-Driven Architecture where services react to state changes.
  • Use Topic Routing to filter noise at the source to save bandwidth.