Message Queues

TL;DR

Message queue: Async communication. Producer sends, consumer processes. Benefits: Decoupling, buffering, retry. Kafka: High throughput. RabbitMQ: Flexible routing.

Core Concepts

Why Message Queues?

Without queue: User uploads → Server resizes (5s) → User waits 😢
With queue: User uploads → Queue job (50ms) → User sees ✅
            Background worker processes later

Messaging Patterns

Point-to-Point: Each message consumed once (background jobs) Pub/Sub: Each message delivered to all subscribers (events)

Kafka vs RabbitMQ

Tool	Throughput	Use Case
Kafka	Millions/sec	Event streaming, replay
RabbitMQ	100K/sec	Complex routing, jobs
AWS SQS	Unlimited	Managed, serverless

Delivery Guarantees

Guarantee	Meaning	Implementation
At-most-once	May lose	No ack
At-least-once	May duplicate	Retry until ack
Exactly-once	No loss/dupes	Distributed tx

At-least-once + idempotency = effectively exactly-once

def process_message(msg):
    if redis.exists(f"processed:{msg.id}"):
        return  # Skip duplicate
    send_email(msg.email, msg.content)
    redis.set(f"processed:{msg.id}", 1, ex=86400)

Quick Reference

• Kafka: Event streaming, high throughput, replay
• RabbitMQ: Complex routing, traditional MQ
• At-least-once + idempotent consumer = reliable processing

---