arthrop0d

Brief articles on varied topics.

Amazon SQS Visibility Timeouts and Receipt Handles

amazon web servicesdistributed systems

Amazon SQS visibility timeouts temporarily hide a message after a consumer receives it. The consumer gets an opaque receipt handle for that delivery and must use that handle to delete the message after successful processing.

This gives SQS a way to support work that can recover from worker failures without making consumers process the same message concurrently by default. It also means SQS provides at-least-once delivery: a message can be delivered more than once, so processing must tolerate duplicates.

The problem visibility timeouts solve

An SQS message is not normally removed when a worker reads it. If receiving removed it immediately, a worker crash after receiving but before finishing would lose the work permanently.

But leaving every received message visible creates the opposite problem. Several workers could receive the same message at once and perform the same operation. SQS therefore separates receiving from deleting:

  1. A worker receives the message.
  2. SQS hides it from ordinary receives for a period of time.
  3. The worker performs the work.
  4. If the work succeeds, the worker deletes the message.
  5. If the worker fails or takes too long, the message becomes visible again.

That hidden period is the visibility timeout. It is not a lock that permanently reserves the message, and it is not a deadline by which SQS deletes it. It is a temporary period during which SQS normally prevents other receives from returning that message.

A visibility timeout protects in-progress work from immediate competition; deletion confirms that the work is finished.

What a receive returns

A call to ReceiveMessage returns the message body and attributes, along with a receipt handle. A receipt handle is an opaque token representing that particular receive operation. It is not the message ID and should not be constructed, parsed, or treated as a permanent identifier.

A worker should keep the handle with the message while it processes the message. When processing succeeds, it calls SQS DeleteMessage with that handle. This deletion is the practical equivalent of an acknowledgement: it tells SQS that the message no longer needs to be delivered.

For example, a Python worker using the AWS SDK might follow this shape:

response = sqs.receive_message(
    QueueUrl=queue_url,
    MaxNumberOfMessages=1,
    VisibilityTimeout=60,
)

for message in response.get("Messages", []):
    handle = message["ReceiptHandle"]
    process(message["Body"])
    sqs.delete_message(QueueUrl=queue_url, ReceiptHandle=handle)

The delete should happen after the side effect succeeds. Deleting first avoids duplicates only by risking lost work if process then fails.

Receipt handles are tied to deliveries, not just to the underlying message. If a message becomes visible and is received again, SQS returns a new receipt handle. The worker processing that later delivery should use the latest handle it received. A handle from an earlier delivery should not be retained as a reusable acknowledgement token.

When processing takes too long

Suppose the visibility timeout is 60 seconds, but processing takes 90 seconds. At approximately 60 seconds, the message becomes visible again. Another worker can receive it while the first worker is still running. Both workers may then perform the same operation.

This is the usual source of duplicate processing in an SQS incident: a log may show a message being received twice, or a second attempt may appear while the first attempt has not logged completion. The original worker did not necessarily lose the message; its temporary visibility simply expired.

A worker that knows it needs more time can call ChangeMessageVisibility with the current receipt handle to extend the timeout. Some consumers do this periodically as a heartbeat. The extension is still temporary, so the worker must handle failures between extensions and between finishing the work and deleting the message.

The timeout should be longer than normal processing time, but no timeout eliminates every duplicate. A worker can crash immediately after completing a database update but before deleting the SQS message. When the timeout expires, SQS has no reliable knowledge that the update completed, so it can deliver the message again. Network failures can create the same uncertainty: the delete might have succeeded even though the worker never received its response, or the worker might retry an operation whose outcome it cannot determine.

Why this is at-least-once delivery

At-least-once delivery means SQS attempts to make a message available one or more times. It does not promise that a successful processing attempt and deletion happen as one indivisible operation. The work and the deletion are separate operations with a failure window between them.

Consumers should therefore make processing idempotent, meaning repeating the same request produces the same final result rather than applying the effect twice. Common approaches include storing an event or message ID with a database result, using a conditional database write, or making an external operation accept an idempotency key.

The design rule is simple: set a visibility timeout that covers ordinary work, extend it for unusually long work, and assume a message can still be delivered again. Treat the receipt handle as the credential for deleting that specific delivery, and delete only after the processing outcome is safely committed.

← All articles