Dead Letter Queue for Jitterbit Message Queue

Overview

This page describes how to implement a Dead Letter Queue (DLQ) design pattern using the Jitterbit Message Queue Service and the Jitterbit MQ connector.

A DLQ pattern routes messages that cannot be processed successfully through a series of retry queues with increasing delays before placing them in a final DLQ for manual inspection.

NACK and requeue limitations

A common integration pattern is to use the NACK activity with the Requeue Messages After NACK option to return a message to the queue when processing fails. For example, an order invoicing integration may query an ERP system repeatedly until an order reaches the expected status.

This approach has two significant drawbacks when used with Quorum queues:

• Messages that cannot be processed immediately are returned to the head of the queue and retried continuously, consuming processing resources.
• Quorum queues enforce a delivery limit of 20 attempts. After a message has been negatively acknowledged 20 times, it is permanently removed from the queue without an error message.

The DLQ pattern uses the NACK activity with the Reject Messages After NACK option instead, and implements explicit routing to a set of retry queues with escalating delays, as described in the following sections.

Pattern overview

The DLQ pattern uses multiple queues and a set of Studio operations:

1. Messages are processed from a main queue.
2. On failure, the message is rejected (not requeued) and sent to the first retry queue with an incremented retryCount value in the payload.
3. A scheduled operation reads from each retry queue after its delay interval and forwards messages back to the main queue for reprocessing.
4. After a configurable maximum number of attempts, messages are sent to the DLQ for manual inspection.

Queue setup

Create the following five queues in the Management Console Message queues page. All queues should use the Quorum type for durability.

Message payload requirements

To track failed processing attempts, add a retryCount field (an integer starting at 0) to your message payload. On each failed processing attempt, configure the main processor operation to perform the following steps:

1. Read retryCount from the messageBody field in the Get activity response schema.

2. Increment the value using a script or transformation.

3. Write the updated value back via the messageBody field in the Send activity request schema when routing to the retry queue.

{
"orderId": "ORD-12345",
"retryCount": 0
}

Workflow design

Main processor operation

Configure this operation to read from orders.main and attempt to process each message.

1. Use the Get activity to retrieve a message from orders.main.

2. Attempt to process the message (for example, query the ERP system for the expected order status).

3. Use operation actions to configure the following actions:

   • On Success: Run an operation that uses the Acknowledge activity to acknowledge the message.

   • On Fail: Run an operation that performs the following steps:

     1. Read retryCount from the message payload and increment it by 1.

     2. Use the Send activity to send the updated message to the appropriate retry queue:

        retryCount value	Destination queue
        1	orders.retry.1m
        2	orders.retry.5m
        3 or 4	orders.retry.30m
        Greater than 4	orders.dlq

     3. Use the NACK activity with Reject Messages After NACK to remove the message from orders.main.

Retry forwarder operations

Create one forwarder operation for each retry queue. Configure each operation to read messages from its retry queue, send them back to orders.main for reprocessing, and run on a schedule that matches the intended delay:

Configure each operation to follow these steps:

1. Use the Get activity to retrieve a message from the retry queue.

2. Use the Send activity to send the message to orders.main, preserving the retryCount value in the payload.

3. Use the Acknowledge activity to acknowledge the message from the retry queue.

DLQ inspector operation (optional)

Create an operation that reads from orders.dlq to alert on or log messages that have exhausted all retry attempts. This operation can send email notifications, write to a log, or trigger a manual review workflow.