Mastra background-task recovery experiment
Date: 2026-10-12

Question
Can a recovered background task avoid duplicate external effects after a worker is interrupted? Keep task ownership, destination requests, and accepted destination effects as separate observations.

Setup
- Node v26.0.0 on Apple M5, 24 GB RAM, macOS Darwin 27.0.
- Two Mastra worker processes share a local libSQL file.
- A separate local HTTP receiver stores requests and accepted effects in its own SQLite file.
- Current pair: @mastra/core 1.72.0 + @mastra/libsql 1.24.0.
- Adapter comparison: @mastra/core 1.72.0 + @mastra/libsql 1.23.3.
- Lease: 4,000 ms; heartbeat: about 1,333 ms; task timeout: 60,000 ms; maxRetries: 1.
- The receiver uses a unique idempotency key and a transaction in keyed scenarios. It is a synthetic destination, not an email provider.
- Ten scenarios, five repetitions each; all 50 runs were observed. A Rust build ran on the machine during part of the experiment, so measured times are not service guarantees.

Scenarios
S1: one worker, no interruption.
S2: start B while A runs a 9-second task.
S3: SIGKILL A before the receiver request.
S4: SIGKILL A after the receiver accepts the request, without an idempotency key.
S5: same boundary as S4, with Idempotency-Key equal to the stable operationId.
S6: freeze A after receiver acceptance, let B finish, then resume A.
S7: SIGTERM A after task start.
P1: with the older adapter and a key, start B while A is running.
P2: with the older adapter and a key, freeze A after acceptance, let B finish, then resume A.
P3: with the older adapter and no key, start B while A is running; no crash.

Run in a disposable copy
Preserve the asset directory layout shown by the download links. Rename run.mjs.txt, worker.mjs.txt, and receiver.mjs.txt to .mjs; rename analyze.py.txt to analyze.py.

  npm install --prefix experiment
  npm install --prefix experiment-old-adapter
  node experiment/run.mjs all 5
  python3 analyze.py

To run only one scenario, replace "all" with its ID, for example `node experiment/run.mjs S4 5`. The runner removes and recreates each matching `runs/<scenario>-<rep>` directory. It writes detailed JSONL events, task-store and receiver databases, process logs, and record.json files under runs/. The analyzer writes aggregate results/summary.json and results/summary.md.

Interpretation
A request is an HTTP call received by the synthetic destination. An effect is a unique effect accepted by that destination. A second request may be classified as a duplicate and not create a second effect. These results do not establish exactly-once delivery for any real provider. The published results.json contains aggregate results, not per-run databases or event logs.