Task recovery does not make email delivery exactly-once

Mastra's background-task recovery and an email provider's delivery state are separate state machines. A worker can lose ownership after a process crash while the destination has already accepted its request. A lease coordinates workers that share task storage; it cannot roll back an effect committed by another system.

In @mastra/core@1.72.0, a running task records an owner and lease expiry (ownerId and leaseExpiresAt). The owner renews that lease while it works. If the process disappears, another worker can reclaim the task after the lease expires. The matching storage adapter must persist and enforce those fields; the core package cannot provide that boundary by itself.

The difficult interval is not the time before the request. It is the gap after a destination accepts an operation but before Mastra saves the tool result and marks the task complete. If the worker dies there, storage says the task is unfinished while the destination may already have acted. Recovery has to make progress from the state it can observe, not from a transaction it never shared.

That leaves two different questions:

• Task ownership: which worker may execute and complete this task now?

• External effect: has the destination already applied this logical operation?

A lease answers the first question. It does not answer the second. A recovered task can therefore be working as designed and still issue a request that causes a duplicate effect.

Two state machines, one failure window {#failure-window}

The experiment makes the boundary visible by using separate databases: one local libSQL file for Mastra's task state and one SQLite file for a synthetic HTTP receiver. The worker can commit the task result only after the tool returns; the receiver can commit its effect before that happens. Neither database can atomically commit or undo the other's write.

This is a schematic, not a screen capture or a claim that every provider behaves this way. It describes the boundary S4 deliberately tested. If the receiver instead recognizes a stable idempotency key, it can accept B's repeated HTTP request as a duplicate without applying the logical effect a second time. The request still happened; the destination's behavior changed.

The question was narrow: after interruption, does a second worker respect the current owner, and what happens if the external action was already accepted? The fixed protocol defines ten scenarios, each run five times. The experiment used two separate Node worker processes sharing one local libSQL file. Each worker had its own in-process pubsub, so coordination between workers came from persisted state, startup recovery and lease expiry—not a shared in-memory message bus.

Tasks were enqueued directly through mastra.backgroundTaskManager.enqueue() and run by a static executor registered with registerStaticExecutor(). The tool, send_receipt, posted to a local HTTP receiver and then waited 2,500 ms before returning. That delay preserved the interval in which the receiver could have acted but the task result had not yet been saved. The receiver used its own SQLite database. It recorded every HTTP request separately from each accepted effect. In keyed scenarios, a unique idempotency key and a transaction made the first request for that key effective; later requests with the same key were recorded as duplicates.

No actual email was sent. The receiver was a controlled stand-in, not a provider integration. No model was called and no paid inference occurred. This tests task and destination state transitions; it does not test an agent's judgment or a real email service's delivery rules.

The tested pair was @mastra/core@1.72.0 with @mastra/libsql@1.24.0. A compatibility control paired the same core with the older @mastra/libsql@1.23.3, whose declared peer range allowed installation without warning. The task lease was 4,000 ms, the heartbeat interval was approximately 1,333 ms, maxRetries was 1, and the task timeout was 60 seconds. The host was an Apple M5 with 24 GB RAM, macOS Darwin 27.0 and Node v26.0.0.

The run protocol was fixed before the five-repetition pass. All 50 runs were classified as observed: each had the runtime and receiver evidence needed for classification. A Rust build ran on the same machine during part of the experiment. Takeover times are therefore measurements from this loaded local machine, not timing guarantees or estimates of a production failure rate.

The table reports requests received and effects accepted per run; every scenario had five runs. An effect counts only when the synthetic receiver recorded it. A and B are separate worker processes.

The current core-and-adapter pair respected a live lease in S2 and recovered ownership after interruption in S3–S6. The exact takeover window varied by scenario: with a 4,000 ms lease, S4's five measured takeovers were 3,979–4,007 ms after interruption; graceful release in S7 reduced that to 281–285 ms. These are observed elapsed times, not the configured lease duration expressed as a service-level guarantee.

The outcome counts also separate facts that are easy to collapse into a single “retry” number. S4 and S5 each made two requests per run; their effect counts differed because only S5's destination enforced the key. In all recovered runs, retry_count stayed at zero, including runs with two tool executions. That field did not tell us how many times the tool called the receiver.

The storage adapter is part of the guarantee {#adapter}

The older-adapter scenarios are not merely a comparison of two version numbers. They test whether the storage layer actually participates in the ownership protocol. In this experiment, core 1.72.0 installed alongside @mastra/libsql@1.23.3 without a peer warning, but the task rows did not persist an owner or lease. B could therefore run while A was still alive.

P1 enabled the destination key. It recorded two requests and one effect; B ran while A was active, and the stored result came from A. P2 used the keyed stale-owner setup: B ran at startup, two requests reached the receiver, one effect was accepted, and the stored result came from B. The key prevented a second effect in those tests, but it did not restore the missing task-ownership state.

P3 removed the key and did not kill either worker. B still ran while A was alive; the receiver recorded two requests and two effects. Duplicate external work therefore did not require a process crash when the adapter failed to enforce ownership. In the tested pair, by contrast, S2's live owner kept B from executing.

The practical check is compatibility across the boundary: verify that the selected adapter persists the fields and conditional ownership updates required by the core version. An install that completes quietly is not proof that the adapter implements the protocol. Check the task row and observed worker executions, rather than inferring lease enforcement from the core package version alone.

The explainer video for this test follows run S4-1. A clip and two frames show the same boundary: A's request is accepted, A is killed, B starts about four seconds later and the provider counts a second email. The clip's camera tracking is a visual treatment; the event log it follows is the recorded one.

The frames above come from S4-1. The figure below opens a different repetition, S4-2, with exact times instead of an invented example: it is recorded as runs/S4-2/record.json, and the plotted t values are milliseconds relative to that run's start. All five S4 runs produced two requests and two effects.

Inspect the sanitized event extract behind the timeline. The extract omits local process and owner identifiers; event names, relative times, worker labels, acceptance results and effect IDs are copied from the recorded run.

At t=711 ms, A sent the first request without an idempotency key. At t=721 ms, the recorded response was accepted as effect 1. The orchestrator sent SIGKILL at t=743 ms—22 ms after that response—and A exited three milliseconds later. The receiver's accepted effect was already outside the task store. Killing A could not retract it.

B began its attempt at t=4,731 ms, 3,988 ms after the interruption. B sent its request at t=5,033 ms; at t=5,055 ms, the receiver accepted effect 2 because no key was present. B finished at t=7,556 ms, and its result was stored. The task's final retryCount was still 0. In this run, recovery succeeded: B completed the task. The duplicate was also real in the receiver's recorded state.

This trace does not replace the five-run result or imply a universal timing. It demonstrates the precise order of events in one repetition. The aggregate result for S4 comes from all five recorded runs; the visual timeline makes one of them inspectable.

S4 and S5 deliberately place the crash at the same boundary. In S4, the receiver accepted A's request and A was killed before the task result was saved. B recovered the task and sent the operation again. With no destination key, the receiver accepted effect 2. That happened in all five repetitions.

In S5, both attempts sent the same stable operationId as Idempotency-Key. The receiver still saw two HTTP requests in every run, because the key does not prevent a worker from making another call. It accepted only one effect and marked B's request as a duplicate in all five runs. The experiment therefore supports a narrower claim than “the task ran once”: the destination's implemented deduplication reduced two requests to one accepted effect.

The scope of that observation matters. The test receiver used a unique key and an atomic insert; it was designed to honor the key. Real providers define their own key scope, retention period, conflict handling and response behavior. A key that expires before a retry, changes between attempts, or is ignored by the destination does not establish the same result. The logical operation ID must remain stable across attempts, and its semantics must match the provider's contract.

S6 tested another independent property. A was paused after acceptance, B was allowed to recover and finish, and then A resumed. With a stable key, the receiver recorded two requests but one effect; the stored task result remained B's in every repetition. Idempotency limited the external effect. Persisted ownership prevented the superseded worker from replacing the completed result. Neither fact substitutes for the other.

Treat task recovery, outbound calls and destination effects as separate observables. The experiment's retry_count stayed at zero while the receiver saw two requests and, in S4, two effects. A counter that describes one internal retry path is not a substitute for counting task attempts or external outcomes.

For a real email integration, keep a stable operation ID for the logical message and pass it as the provider's idempotency key on every attempt. Confirm the provider's documented key scope and retention instead of assuming that a header has universal semantics. A retry should not create a new logical operation ID simply because a different worker owns the task.

Validate the failure window directly: first verify that a second worker does not run while a compatible adapter records a live owner; then interrupt after the provider accepts but before the task result is persisted. Observe, separately, the task attempts, HTTP requests, provider responses and delivered effects. Also inspect the persisted owner and final result after a stale worker resumes. These checks correspond to different invariants; passing one does not prove the others.

The source reviewed for this experiment also distinguishes recovery policy from key behavior: stale-task redispatch requires maxRetries > 0; with maxRetries = 0, the task is marked failed rather than redispatched. The 50 runs all used maxRetries = 1, so that zero-retry behavior is a source-level finding, not an outcome measured in this run set.

The fixed protocol describes the system, destination behavior, scenarios, evidence collected and classification rules. The machine-readable aggregate contains the five-run results for all ten scenarios. It does not include the raw per-run databases and logs; the linked S4-2 event extract is a small, sanitized view of one recorded run.

The runner, worker, receiver, analyzer and package manifests are available as runner, worker, receiver, analyzer, current-adapter manifest and older-adapter manifest. The .mjs.txt and .py.txt suffixes serve code as downloadable text; rename them to .mjs and .py in the directory layout described by the protocol.

Use Node v26.0.0 and record the versions actually resolved when repeating the run. The manifests pin direct dependencies, not a transitive lockfile. The runner deletes and recreates the matching runs/<scenario>-<rep> directory, so run it in a disposable working copy rather than beside evidence you need to retain. The analyzer produces a summary from the run records; keep the request count distinct from accepted effects when checking its output.

The lease answers which worker may own and complete a task when the matching adapter persists the ownership boundary. Destination idempotency answers whether a destination that honors a stable key has already applied the logical operation. In this controlled test, the lease coordinated task execution; the receiver's key reduced accepted effects while leaving request counts at two. Calling task recovery itself “exactly-once delivery” would combine two guarantees the experiment kept separate.

The evidence covers one local libSQL adapter, one synthetic receiver, one Apple M5 machine, Node v26.0.0 and five repetitions per scenario. It is enough to show consistent behavior in these runs, not to estimate a production failure rate. No real email, provider, CRM, model, durable-agent recovery, multiple hosts or other storage adapters were tested. The receiver's idempotency behavior was implemented specifically for the experiment; other destinations may have different semantics.

The useful result is a design boundary, not a universal delivery claim: recover task ownership with a compatible adapter, and separately make the external operation safe to repeat using semantics the destination actually supports. Then test and observe both boundaries. The Mastra release and adapter change in PR #24841 describe the implementation context; the attached protocol and measurements show what this particular test did.