Retries, Failures, Duplicates, and Dead-Lettering in Service Bus
Service Bus gives you at-least-once delivery, so your design has to assume that messages can fail, be retried, and arrive more than once. Four mechanisms work together:
- Client retries for transient errors when talking to Service Bus
- Message redelivery (peek-lock and delivery count) when processing fails
- Dead-lettering for messages that can't be processed
- Duplicate handling, through broker-side detection and idempotent consumers
1. Two kinds of retry
Client-side retries cover transient failures when sending or receiving (network blips, throttling, server busy). The Azure SDK retries automatically, and you can tune it:
var client = new ServiceBusClient(
"mynamespace.servicebus.windows.net",
new DefaultAzureCredential(),
new ServiceBusClientOptions
{
RetryOptions = new ServiceBusRetryOptions
{
Mode = ServiceBusRetryMode.Exponential,
MaxRetries = 5,
Delay = TimeSpan.FromSeconds(1),
MaxDelay = TimeSpan.FromSeconds(30)
}
});
Message redelivery covers failures in your processing logic. This is controlled by peek-lock settlement, covered next.
2. Peek-lock settlement and redelivery
In peek-lock mode, a received message is locked (default lock duration 30 seconds, max 5 minutes). You then settle it:
| Action | Effect |
|---|---|
| Complete | Message is removed. Success. |
| Abandon | Lock is released; the message becomes available again and delivery count increments. |
| Defer | Message stays in the queue but is only retrievable by its sequence number. Useful for out-of-order processing. |
| Dead-letter | Message moves to the dead-letter queue immediately. |
| (do nothing / crash / lock expires) | Message becomes available again and delivery count increments. |
Use ServiceBusProcessor, which handles concurrency, lock renewal, and settlement for you:
var processor = client.CreateProcessor("orders", new ServiceBusProcessorOptions
{
MaxConcurrentCalls = 10,
AutoCompleteMessages = false, // settle explicitly
MaxAutoLockRenewalDuration = TimeSpan.FromMinutes(10),
PrefetchCount = 20
});
processor.ProcessMessageAsync += async args =>
{
try
{
await HandleAsync(args.Message, args.CancellationToken);
await args.CompleteMessageAsync(args.Message);
}
catch (TransientException)
{
// Retry: becomes visible again, delivery count + 1
await args.AbandonMessageAsync(args.Message);
}
catch (PermanentException ex)
{
// Retrying won't help: dead-letter right away
await args.DeadLetterMessageAsync(args.Message,
deadLetterReason: "ValidationFailed",
deadLetterErrorDescription: ex.Message);
}
};
processor.ProcessErrorAsync += args =>
{
logger.LogError(args.Exception,
"Service Bus error from {Source} on {Entity}", args.ErrorSource, args.EntityPath);
return Task.CompletedTask;
};
await processor.StartProcessingAsync();
Key point: classify failures. Transient failures (timeouts, 503s from a dependency, deadlocks) should be abandoned or retried. Permanent failures (malformed payload, validation errors, missing required data) should be dead-lettered immediately, since retrying them just burns delivery attempts.
Lock duration and long processing
If processing takes longer than the lock duration, the lock expires, the message is redelivered to another consumer, and your original Complete call fails with MessageLockLost. Avoid this by:
- Setting
MaxAutoLockRenewalDurationso the processor renews the lock automatically - Setting an appropriate lock duration on the entity
- Moving very long work into a durable workflow and keeping message handlers short
Retry timing: Service Bus has no built-in backoff on redelivery
An abandoned message is available immediately, which can cause a tight retry loop that hammers a failing dependency. Common patterns:
- In-process retry first: retry transient calls with exponential backoff (Polly or
Microsoft.Extensions.Http.Resilience) within the handler before abandoning. - Scheduled re-enqueue: complete the original and send a copy scheduled for later, which gives you true delayed retries:
var retry = new ServiceBusMessage(args.Message)
{
ScheduledEnqueueTime = DateTimeOffset.UtcNow.AddSeconds(Math.Pow(2, attempt) * 10)
};
retry.ApplicationProperties["attempt"] = attempt + 1;
await sender.SendMessageAsync(retry);
await args.CompleteMessageAsync(args.Message);
If you do this, do the send and complete in a transaction so you don't lose or duplicate the message (see below), and track the attempt count yourself.
- Circuit breaker: if a dependency is down, stop receiving for a while instead of cycling messages through delivery attempts.
3. Dead-letter queues (DLQ)
Every queue and every topic subscription has a dead-letter sub-queue, addressed as <entity>/$deadletterqueue. Messages end up there when:
MaxDeliveryCountis exceeded (default 10). This is the "poison message" safety net.- TTL expires and dead-lettering on expiration is enabled (
DeadLetteringOnMessageExpiration). - Your code dead-letters explicitly with a reason and description.
- A subscription filter evaluation fails (when configured to dead-letter on filter evaluation exceptions).
- Session-related issues, in some cases.
The DLQ is not automatically processed and has no TTL of its own, so messages stay until you deal with them. It also counts against the entity's size quota, so an ignored DLQ can eventually fill the queue.
Reading dead-lettered messages:
var dlqReceiver = client.CreateReceiver("orders", new ServiceBusReceiverOptions
{
SubQueue = SubQueue.DeadLetter
});
var messages = await dlqReceiver.ReceiveMessagesAsync(maxMessages: 20, TimeSpan.FromSeconds(5));
foreach (var m in messages)
{
Console.WriteLine($"{m.MessageId}: {m.DeadLetterReason} - {m.DeadLetterErrorDescription}");
}
For system dead-lettering, the reason is set for you (for example MaxDeliveryCountExceeded or TTLExpiredException). For your own dead-lettering, always set deadLetterReason and deadLetterErrorDescription so triage is easy.
What to do with dead-lettered messages
- Alert when the DLQ message count is above zero (Azure Monitor metric
DeadletteredMessages, filtered by entity). Unmonitored DLQs are where messages go to be forgotten. - Inspect them (Service Bus Explorer in the portal or in Visual Studio, or code like the above) and read the reason.
- Fix the cause: a bug, a bad payload, or a downstream outage that has since recovered.
- Resubmit after the fix: receive from the DLQ, send a clone to the original entity, then complete the DLQ copy. The Azure portal's Service Bus Explorer also supports resending dead-lettered messages.
- Or discard/archive messages that are genuinely invalid, recording where they went.
await foreach (var dead in dlqReceiver.ReceiveMessagesAsync())
{
var clone = new ServiceBusMessage(dead); // copies body and properties
await sender.SendMessageAsync(clone);
await dlqReceiver.CompleteMessageAsync(dead);
}
Automate DLQ handling with an Azure Function triggered from the DLQ path (orders/$deadletterqueue) if you want logging, notification, or auto-resubmission with a retry cap.
Tuning MaxDeliveryCount
- The default of 10 is often too high for failures that will never succeed and too low for slow-recovering dependencies.
- Pair it with how long a failure typically lasts. If a dependency outage can last 10 minutes and messages retry instantly, 10 attempts can be exhausted in seconds, sending healthy messages to the DLQ. This is another reason to add backoff.
4. Duplicate messages
Duplicates arrive from two directions:
Producer-side duplicates. A sender times out after the broker accepted the message and retries, so the broker gets two copies.
Consumer-side duplicates. A message is delivered more than once because the lock expired, the consumer crashed after processing but before completing, or you abandoned it. This is inherent in at-least-once delivery.
Broker-side duplicate detection (producer duplicates)
Enable it when creating the queue or topic and set a detection window (default 10 minutes, max 7 days). Service Bus then discards any message whose MessageId matches one already received within the window.
await adminClient.CreateQueueAsync(new CreateQueueOptions("orders")
{
RequiresDuplicateDetection = true,
DuplicateDetectionHistoryTimeWindow = TimeSpan.FromMinutes(30)
});
// Set a deterministic MessageId derived from the business key
var message = new ServiceBusMessage(body) { MessageId = $"order-{order.Id}" };
Important details:
- It only works if you set a stable, meaningful
MessageId. Random GUIDs defeat it. - It can only be enabled at entity creation time. You can't turn it on later.
- It only covers duplicates sent within the window. It does not help with redelivery to consumers.
- It isn't supported in the same way on partitioned or some Premium configurations. Check the current docs for your tier.
Idempotent consumers (the real solution)
Because at-least-once delivery can't be fully eliminated, make processing idempotent, so handling a message twice has the same effect as handling it once.
Common approaches:
- Natural idempotency: use upserts (
MERGE,INSERT ... ON CONFLICT DO NOTHING) and "set status = X" rather than "increment by 1." - Processed-message table (inbox pattern): record
MessageId(or a business key) in the same database transaction as the business change. If the insert fails on a unique key, the message was already handled, so skip it and complete it.
// In one DB transaction:
// INSERT INTO ProcessedMessages (MessageId) VALUES (@id) -- unique constraint
// ... apply business change ...
// If the insert violates the constraint: already processed, so just complete the message.
- Idempotency keys on downstream calls: pass a stable key to payment providers and similar APIs so they dedupe too.
- Check delivery count (
message.DeliveryCount > 1) as a hint that a message may have been processed before, then verify against state.
5. Atomic operations with transactions
A common failure mode: you send a follow-up message and then crash before completing the original, producing duplicates or lost work. Service Bus transactions group operations on the same connection so they succeed or fail together:
var client = new ServiceBusClient(fqdn, credential,
new ServiceBusClientOptions { EnableCrossEntityTransactions = true });
var receiver = client.CreateReceiver("orders");
var sender = client.CreateSender("payments");
var message = await receiver.ReceiveMessageAsync();
using (var scope = new TransactionScope(TransactionScopeAsyncFlowOption.Enabled))
{
await sender.SendMessageAsync(new ServiceBusMessage($"charge-{message.MessageId}"));
await receiver.CompleteMessageAsync(message);
scope.Complete();
}
Transactions span only Service Bus operations. They do not include your database. For database plus message consistency, use the outbox pattern: write the outgoing message to an outbox table in the same DB transaction as your business change, and have a separate relay publish it to Service Bus (which, combined with consumer idempotency, gives effectively-once results).
6. Ordering and sessions
If order matters, use sessions. They give FIFO per SessionId and exclusive processing by one consumer at a time. Be aware that a failing message in a session can block the rest of that session until it's dead-lettered or completed, so set a sensible MaxDeliveryCount and handle poison messages.
7. Azure Functions behavior
With the Service Bus trigger:
- The host calls Complete if your function succeeds and Abandon if it throws (when
autoCompleteMessagesis true, the default). - Delivery count and DLQ rules are the same as above, so repeated exceptions end in the DLQ after
MaxDeliveryCount. - Tune concurrency in
host.json:
{
"extensions": {
"serviceBus": {
"maxConcurrentCalls": 16,
"maxAutoLockRenewalDuration": "00:10:00",
"autoCompleteMessages": true
}
}
}
- Use
ServiceBusMessageActionswithautoCompleteMessages: falseif you want explicit control (dead-letter with a reason, defer, and so on). - Setting
MaxDeliveryCounttoo high combined with a flaky dependency can multiply load. Use Functions' built-in retry policies only for non-Service-Bus triggers; for Service Bus, rely on the broker's redelivery and delivery count.
8. Monitoring checklist
Alert on:
- Dead-lettered message count > 0
- Active message count / queue depth growing steadily (consumers falling behind or failing)
- Oldest message age rising
- Throttled requests and server errors on the namespace
- Lock-lost and processing exceptions in Application Insights
Log MessageId, DeliveryCount, CorrelationId, and the entity name in every handler so a single message can be traced across retries.
