Decoupling document automation with RabbitMQ

Transport companies still drown in CMR consignment notes — the paperwork that follows every international shipment. Automating them sounds simple until you hit reality: documents arrive in unpredictable bursts, some need OCR, some need validation against an ERP, and none of them can be silently lost. Here’s how I used RabbitMQ to make that pipeline boring, in the best possible way.

The problem with doing it inline

The naive version processes each document the moment it arrives: receive, OCR, validate, store — all in one request. It works in the demo and falls apart in production:

• A burst of 300 documents at 8am overwhelms the OCR step.
• One slow external validation call blocks everything behind it.
• If the process crashes mid-way, the document is gone.

The work is variable and failure-prone, but the intake must always be instant and reliable. That mismatch is exactly what a queue is for.

Decoupling intake from processing

The fix is to split the system in two with a broker in the middle:

1. Producer — accepts a document, persists the raw payload, and publishes a message. It returns in milliseconds.
2. RabbitMQ — durably holds the work until a consumer is ready.
3. Consumers — pull messages at their own pace, do the heavy lifting (OCR, validation, ERP sync), and acknowledge only when done.

// Publish once the raw document is safely stored.
err := ch.PublishWithContext(ctx,
    "cmr",            // exchange
    "cmr.received",   // routing key
    false, false,
    amqp.Publishing{
        DeliveryMode: amqp.Persistent, // survive a broker restart
        ContentType:  "application/json",
        MessageId:    doc.ID,
        Body:         payload,
    },
)

Now an 8am spike just makes the queue longer for a few minutes. The intake never degrades, and I can scale consumers horizontally to drain it faster.

Ordering, acknowledgements and idempotency

Three rules keep the pipeline correct under load:

• Manual acks. A consumer acknowledges a message after the work succeeds. Crash before that, and RabbitMQ redelivers it.
• Prefetch limits. QoS(prefetch=N) stops a single consumer from grabbing the whole queue and choking.
• Idempotency keys. Every message carries the document id; processing is idempotent, so a redelivery can’t create duplicates.

When things fail

Failure is a first-class path, not an afterthought:

• Transient errors are retried with backoff via a delayed re-queue.
• Messages that fail repeatedly land in a dead-letter queue for inspection instead of poisoning the main flow.
• The DLQ is monitored, so a bad document becomes an alert — never a silent loss.

Takeaways

The queue didn’t just add throughput; it changed the system’s shape. Intake and processing now fail independently, scale independently, and recover on their own. The mental model became: accept fast, persist always, process when ready, and never drop a message.

A message broker is one of those investments that feels like overkill on day one and indispensable by week two.