A retry is a second delivery

Retry logic reads as a scheduling problem and is written as one. Pick a base, pick a factor, pick a cap, add jitter so the herd does not return together. All four of those decisions are about time, all four are easy to test, and none of them is the one that costs money.

The one that costs money is that a retry is a second delivery. If the first attempt charged the card and then timed out on the way back, the failure your worker saw is a network failure and the charge is real. The retry charges it again. Your schedule was perfect and your customer was billed twice.

At-least-once is the only thing on offer

There is no at-most-once delivery that is also reliable, and there is no exactly-once delivery over a network that can drop the acknowledgement. What there is, is at-least-once delivery plus a way for the receiver to recognise a repeat. That second half is yours to build, and no queue can build it for you, because only your code knows what “the same job” means.

The recogniser is an idempotency key: a value that is stable across every attempt of one job and different for every other job. Stable is the part people get wrong. A key generated inside the handler is generated again on the retry, and it is a different key, and the retry sails through.

Where the key has to come from

The key has to be created when the job is enqueued, stored with the job, and sent unchanged on every attempt.

That is the whole rule, and Bidewell enforces the awkward end of it: a job that has no way to accept a key is refused at enqueue rather than at the second attempt. Refusing it at enqueue is a worse developer experience by a few minutes and a better outcome by however much the double charge was.

Downstream, the key has to reach something that can reject a duplicate. In practice that is one of three things, in descending order of how much you will enjoy them:

A unique constraint in your own database, on the key. The insert fails, you catch the violation, you return the original result. This is the cheapest correct answer and it is available to everybody.

A provider’s own idempotency header. Most payment and messaging APIs have one. Pass your key straight through; do not invent a second one.

A dedupe table with a time window, when the downstream has neither. This one is a compromise: the window has to be longer than the whole retry schedule or a late retry slips past it. Ours runs to a minute and thirty-two seconds worst case, which is the figure to size the window against, and it is on the limits page for that reason rather than being something you have to derive.

The jobs that should not retry at all

Not every job wants a schedule. A webhook the caller will resend on its own does not need one from you as well. An import a human kicked off should fail loudly at the first attempt, because the human is still at their desk and can look at it. A job that writes to a system with no key and no constraint is safer running once and being dead-lettered than running nine times and being wrong eight of them.

For those, the correct policy has no schedule in it at all. It runs once, and if it fails it goes to the dead letter with its error, and somebody decides. That is not a degraded configuration, it is the right one, and it is worth being able to say so plainly rather than setting the attempt count to one and hoping the intent survives the next code review.