Why AI Agents Double Charge: Crash-Safe Tool Execution Under the Hood

Imagine your engineering team just built a Subscription Renewal Agent. The workflow is straightforward: charge the customer's credit card, then grant the entitlement on their account. One morning, customer support reports that a user was billed twice for a single renewal cycle. Naturally, the team's first suspicion is model hallucination. But when you inspect the production logs, you discover the agent executed exactly what its framework instructed: when encountering a timeout, retry the step. The model did not hallucinate; the flaw sat squarely in the retry boundary.

Incidents like this reveal a deceptive trap in distributed agent design. Even if your payment provider supports idempotency keys, duplicate charges can still happen. When a retry occurs one layer above the component generating the key, the provider perceives that retry as an entirely new transaction. The standard engineering advice on the internet suggests, "Just pass an idempotency key to every write tool." While true in spirit, that advice is woefully incomplete. Who derives the key? Where is it persisted? What happens when a crash strikes during the window between side effect execution and completion recording? Without clear answers to these questions, idempotency keys offer false comfort.

The core premise of this article is simple: Idempotency is not a property of a single function call; it is a property of the entire execution path. Rather than discussing abstract theory, we will build a minimal agent runtime, deliberately terminate the process at critical lifecycle boundaries, count duplicate side effects in a database ledger, and climb a six-tier Guard Ladder from Naive Resume up to Durable Execution, Atomic Commits, and UNKNOWN state reconciliation. Every tier exposes why the previous solution breaks under failure. By the end, you will walk away with an architecture decision flowchart and an end-to-end production checklist. This article builds upon my previous deep dives into Exactly-Once Delivery and Distributed Locks, so having those foundations will help tie the concepts together.

Table of Contents

• Step 1: The Incident Anatomy
• Step 2: Intent vs Attempt, The Mental Model
• Step 3: Anatomy of a Tool Call and the Four Crash Windows
• Step 4: Building the Crash Injection Lab
• Step 5: The Guard Ladder
  • Level 0: Naive Resume
  • Level 1: Hash of Args as Key
  • Level 2: Stable Key from Durable State
  • Level 3: Durable Execution
  • Level 4: Atomic Effect and Completion Record
  • Level 5: UNKNOWN State and Reconciliation
• Step 6: Results, What the Matrix Tells Us
• Step 7: How Production Systems Do It
• Recap and Production Checklist

Step 1: The Incident Anatomy

Let us trace the failure timeline step by step. The agent selects the payment tool, the tool wrapper sends a payment creation request to Stripe, and Stripe debits 250 USD from the card. However, during the return transit of the response, a network drop occurs. The wrapper never receives acknowledgment of success and raises a timeout error back to the agent runtime. The framework's default retry policy takes over and re-runs the entire step containing both the charge and the entitlement grant. Consequently, a second charge reaches Stripe, followed by the grant.

Notice the critical distinction: from Stripe's vantage point, that second request was not a retry; it was an entirely distinct charge. In many systems, idempotency keys are generated dynamically inside the tool wrapper on every invocation. When retries happen one layer above that wrapper, the wrapper generates a fresh UUID. The idempotency layer functioned as implemented, but it answered the wrong architectural question. A real-world incident postmortem detailing this exact vulnerability is documented in this engineering analysis (opens in a new tab), where both payments and downstream entitlement grants were duplicated due to a retry boundary placed above the key generation scope.

Step 2: Intent vs Attempt, The Mental Model

Consider an intuitive restaurant analogy. A waiter takes your table's dinner order and carries it to the kitchen. The kitchen begins preparing the meal, but on the return trip, the waiter's intercom cuts out. Not knowing whether the kitchen received the ticket, the waiter submits the order again, resulting in two identical dishes arriving at your table. The classic remedy is writing a unique order token on the ticket so the kitchen can identify that it already processed that exact order. That token serves as the idempotency key. But an essential question remains: what happens if the waiter misplaces their notepad entirely? Can they still recreate that identical token? The remainder of this post addresses that exact challenge.

This analogy underscores the need to decouple two core concepts: Intent and Attempt. When an LLM decides to trigger a tool, that decision represents an Intent (for example, "refund 250 USD to customer cus_42"). When network drops, timeouts, or process crashes happen, every subsequent re-execution represents an Attempt. The single duty of an idempotency mechanism is collapsing multiple Attempts into exactly one Intent. Any key instantiated fresh on each attempt represents an attempt-level identifier, making deduplication across attempts impossible.

I discussed network delivery semantics in depth in my previous post, so here is the essential conclusion: across unreliable networks, only two honest delivery guarantees exist: at-most-once and at-least-once. "Exactly-once" delivery is not a fundamental transport primitive; it is achieved purely by combining at-least-once delivery with idempotent execution semantics. This principle holds true even within sophisticated durable workflow systems such as Temporal. While a workflow completes once and activity results are logged once, individual activities execute with at-least-once guarantees. Committing a side effect and recording its completion are two separate physical operations, and duplicate mutations are born in the fracture between them.

Step 3: Anatomy of a Tool Call and the Four Crash Windows

Inside an agent runtime loop, every tool invocation traverses five sequential milestones: the LLM selects a tool, the runtime logs that intent, the tool performs the mutation against an external service, the runtime records completion, and finally the execution result is passed back into the LLM context. Between these five milestones lie four distinct failure intervals, designated as W1 through W4.

Window W3 represents the most dangerous failure mode in distributed computing. The external provider has committed the mutation, yet local durable storage holds no record of that completion. When the agent process restarts after a crash, it finds a recorded intent lacking a completion marker, leaving it incapable of determining whether the remote operation succeeded. We designate this state of uncertainty as the UNKNOWN state. A naive resume strategy invariably assumes that an unrecorded completion implies the effect never took place, directly triggering duplicate executions. In an empirical evaluation performed by the Durable Agent Runtime (opens in a new tab) research project across 60 crash points, 34 crashes occurred after side effect execution; naive recovery re-executed the action in 52.9% of those instances, producing 44 unwanted duplicate mutations.

This vulnerability is magnified because retries in modern agent frameworks span multiple stacked layers. The underlying HTTP client retries on socket disconnects, the tool wrapper retries on HTTP 5xx codes, the agent framework retries failed workflow steps, and the LLM itself evaluates error feedback and re-emits tool calls. Each layer acts sensibly in isolation, but their combination creates multiplicative amplification. If each of the four layers retries three times, a single intent can produce up to 81 outgoing calls (3 x 3 x 3 x 3). This highlights why key placement is vital: if the idempotency key is generated inside the innermost layer while retries happen at the outermost boundary, every high-level attempt behaves as an entirely new intent.

Step 4: Building the Crash Injection Lab

To move beyond abstract concepts, we construct an end-to-end crash simulation harness. The lab architecture consists of four components: a scripted LLM emulator (guaranteeing deterministic tool selection), three target tools (charge, send_email, and grant_entitlement), and a Side-Effect Ledger Database where every real-world mutation inserts an immutable record. A Crash Injector terminates the runtime process via SIGKILL at an exact window, after which recovery is triggered. Finally, an Oracle compares the resulting ledger against a crash-free baseline, detecting duplicate or omitted actions.

The crash injector implementation is deliberately compact: it inspects an environment variable and terminates the running process immediately, bypassing cleanup routines and finally blocks, faithfully reproducing a sudden hardware failure or container eviction.

// crash.ts
export type CrashWindow = "W1" | "W2" | "W3" | "W4";

export function crashPoint(window: CrashWindow) {
  if (process.env.CRASH_AT === window) {
    console.error(`[lab] simulated crash at ${window}`);
    process.kill(process.pid, "SIGKILL"); // no cleanup, no finally blocks, like a real crash
  }
}

// runtime.ts (the four windows are exactly the four gaps in the tool call lifecycle)
async function runStep(run: Run, call: ToolCall) {
  crashPoint("W1");                                       // nothing happened yet
  await appendEvent(run.id, "TOOL_INVOKED", call);        // intent is durable now
  crashPoint("W2");                                       // intent logged, no effect yet
  const result = await tools[call.tool](call.args, call); // effect reaches the provider
  crashPoint("W3");                                       // effect done, no record
  await appendEvent(run.id, "TOOL_COMMITTED", { callId: call.id, result });
  crashPoint("W4");                                       // recorded, LLM not told yet
  return result;
}

The side-effect ledger and oracle require a single table and one verification query. Each real effect (such as charging a card or dispatching an email) inserts an immutable row into this ledger. The oracle verifies that every tool was executed exactly once per run ID.

CREATE TABLE side_effects (
  id         BIGSERIAL PRIMARY KEY,
  run_id     TEXT NOT NULL,
  tool       TEXT NOT NULL,
  detail     JSONB NOT NULL,
  created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);

-- Oracle: expected result is exactly one row per tool
SELECT tool, count(*) AS times
FROM side_effects
WHERE run_id = $1
GROUP BY tool;

We evaluate every combination of Guard Level and Crash Window deterministically without sampling. A compact shell script executes the complete matrix: the primary invocation terminates via SIGKILL, the recovery runner resumes execution without faults, and the Oracle appends test outcomes to a CSV file.

for level in 0 1 2 3 4 5; do
  for w in W1 W2 W3 W4; do
    pnpm lab:reset
    LEVEL=$level CRASH_AT=$w pnpm lab:run || true   # first run dies by SIGKILL
    LEVEL=$level pnpm lab:run                        # resume run, no crash
    pnpm lab:oracle --level $level --window $w >> results/summary.csv
  done
done

The evaluation measures four specific metrics: Duplicate Side Effect Count, Lost Side Effect Count, Recovery Latency, and Redundant LLM Invocations. With a fixed seed, the entire test suite runs via a single command, producing reproducible outcomes.

Step 5: The Guard Ladder

Now comes the core engineering implementation. We construct six consecutive Guard Levels. Each level addresses specific vulnerabilities while carrying its own boundary failure mode. The methodology is systematic: at Level N, we demonstrate where the preceding level fails, seal that gap, and uncover where the new level itself breaks down.

Level 0: Naive Resume

Level 0 offers no durability guarantees. Following a crash, the runtime ignores past logs and re-executes the plan from scratch. This represents the default state of most initial prototypes.

async function resumeNaive(run: Run) {
  // no event log, no key: start the whole plan again
  return runAgent(run.input);
}

Crashes in W1 and W2 pass cleanly because no external side effect was dispatched. But when crashes hit W3 or W4, the side effect has already committed, and resuming from scratch executes it a second time. This is where duplicate side effects originate.

Level 1: Hash of Args as Key

The intuitive next step is hashing the tool call arguments to construct an Idempotency Key. This approach works reliably as long as argument payloads remain identical. However, breakdown occurs when retries are handled by an LLM. On a retry, the model does not replay identical network bytes; it re-evaluates the prompt context alongside error messages and produces a newly reasoned tool call. In doing so, parameter names may shift, JSON key order changes, or units get reformatted.

import { createHash } from "node:crypto";

const sha = (v: unknown) =>
  createHash("sha256").update(JSON.stringify(v)).digest("hex");

// the first attempt
const first = { customer: "cus_42", amount: 25000, currency: "usd" };

// the retry: same intent, but the model re-reasoned and reworded the args
const second = { currency: "usd", amount: 250, customer: "cus_42", unit: "dollars" };

console.log(sha(first) === sha(second)); // false, so two keys, so two charges

Level 2: Stable Key from Durable State

The architectural remedy follows a core rule: The idempotency key must never derive from model output; it must derive from the runtime's durable execution state. When an LLM produces a tool call block, that block carries a unique identifier. When an agent orchestrator replays a turn, it preserves that block identifier. By combining runId, step, and the tool call ID, we produce a stable key. However, there is a strict prerequisite: the intent must be persisted to disk before passing W1. If persistence fails prior to W1, a crash will force a new LLM call, generating a fresh block ID and yielding a new key.

import { createHash } from "node:crypto";

export function idemKey(runId: string, step: number, toolCallId: string) {
  return createHash("sha256").update(`${runId}:${step}:${toolCallId}`).digest("hex");
}

// canonical JSON: sorted keys, so field order never matters
function canonical(v: unknown): string {
  if (Array.isArray(v)) return `[${v.map(canonical).join(",")}]`;
  if (v && typeof v === "object") {
    const o = v as Record<string, unknown>;
    return `{${Object.keys(o).sort().map((k) => `${JSON.stringify(k)}:${canonical(o[k])}`).join(",")}}`;
  }
  return JSON.stringify(v);
}

// the fingerprint is NOT the key. It only detects "same key, different args"
export const fingerprint = (args: unknown) =>
  createHash("sha256").update(canonical(args)).digest("hex");

If the external provider natively honors idempotency keys, we forward this stable key with the request. Following a crash in W3, recovery submits the identical key, prompting the provider to return the cached response.

const intent = await stripe.paymentIntents.create(
  {
    amount: 25000,
    currency: "usd",
    customer: "cus_42",
    payment_method: pm,
    confirm: true,
    off_session: true,
    metadata: { idem_key: key }, // makes later lookup possible (Level 5)
  },
  { idempotencyKey: key },       // the provider dedupes on this
);

Level 2 exposes a glaring vulnerability: it functions only when the external provider natively supports and enforces idempotency keys. If an upstream service ignores keys, a crash in W3 leaves local storage empty, and the subsequent recovery request executes a duplicate mutation.

Level 3: Durable Execution

To solve this, we ascend to durable orchestration frameworks such as Temporal. In a durable execution system, every workflow step is written to an immutable Event History. When a worker process dies, another worker reconstructs runtime state by replaying that history. Completed activities read their cached results directly from history rather than executing again. This design necessitates deterministic workflow code: invoking non-deterministic calls like Date.now() or random generators inside workflow functions causes history replay mismatches. Developers must rely on framework primitives such as workflow.now() that return reproducible values during replay.

from datetime import timedelta
from temporalio import activity, workflow
from temporalio.common import RetryPolicy

# 'payments' and 'entitlements' are your own client wrappers (imports omitted)

@activity.defn
async def charge_customer(customer_id: str, amount_cents: int) -> str:
    info = activity.info()
    key = f"{info.workflow_id}:{info.activity_id}"   # stable across retries
    payment = await payments.charge(customer_id, amount_cents, idempotency_key=key)
    return payment.id

@activity.defn
async def grant_entitlement(customer_id: str, payment_id: str) -> None:
    info = activity.info()
    key = f"{info.workflow_id}:{info.activity_id}"
    await entitlements.grant(customer_id, payment_id, idempotency_key=key)

@workflow.defn
class RenewalWorkflow:
    @workflow.run
    async def run(self, customer_id: str, amount_cents: int) -> str:
        payment_id = await workflow.execute_activity(
            charge_customer,
            args=[customer_id, amount_cents],
            start_to_close_timeout=timedelta(seconds=30),
            retry_policy=RetryPolicy(maximum_attempts=5),
        )
        await workflow.execute_activity(
            grant_entitlement,
            args=[customer_id, payment_id],
            start_to_close_timeout=timedelta(seconds=15),
        )
        return payment_id

This reveals the most critical lesson of Level 3: Temporal executes activities with at-least-once semantics. If an activity completes its external side effect but crashes before acknowledging completion to the Temporal cluster, the engine re-runs the activity. This is the exact equivalent of window W3 under a different name. Temporal's documentation explicitly emphasizes that activities executing external side effects must remain idempotent, recommending composite keys built from Workflow ID and Activity ID.

The same consideration applies to Sagas and compensating actions. The pattern of "if Step 3 fails, roll back Step 2 and Step 1" sounds clean, but compensation is itself an external side effect. Issuing a refund twice drains capital twice. Compensations must adhere to the exact same idempotency key derivation discipline. Durable execution organizes complexity; it does not eliminate distributed systems realities.

Level 4: Atomic Effect and Completion Record

We arrive at an architectural foundation: If the side effect and its completion record commit within the same database transaction, the crash window between them ceases to exist. Window W3 disappears entirely. This pattern is achievable whenever the mutation occurs within your own relational database, such as provisioning account entitlements, granting platform credits, or updating an order state.

export async function grantEntitlementAtomic(
  runId: string, step: number, customerId: string, plan: string,
) {
  const key = `${runId}:${step}:grant`;
  const client = await pool.connect();
  try {
    await client.query("BEGIN");

    const ins = await client.query(
      `INSERT INTO entitlements (idem_key, customer_id, plan)
       VALUES ($1, $2, $3)
       ON CONFLICT (idem_key) DO NOTHING
       RETURNING idem_key`,
      [key, customerId, plan],
    );

    await client.query(
      `INSERT INTO run_events (run_id, seq, type, payload)
       VALUES ($1, $2, 'STEP_COMPLETED', $3)
       ON CONFLICT (run_id, seq) DO NOTHING`,
      [runId, step, JSON.stringify({ key, applied: ins.rowCount === 1 })],
    );

    await client.query("COMMIT"); // both rows, or neither
  } catch (e) {
    await client.query("ROLLBACK");
    throw e;
  } finally {
    client.release();
  }
}

This approach offers an additional architectural benefit. Because every STEP_COMPLETED event is recorded into an append-only event log, the agent runtime does not need custom state serialization: replaying the log reconstructs runtime state deterministically. This concept directly mirrors the Transactional Outbox and Inbox patterns explored in my earlier writing. Where Outbox binds message publishing to local state changes within one transaction, Level 4 binds tool execution to completion logging within an atomic boundary.

Level 5: UNKNOWN State and Reconciliation

The final level confronts reality: when an external API lacks native idempotency keys and the mutation occurs beyond your local database, the uncertainty of W3 cannot be engineered away. Architecture must account for it explicitly. We introduce a formal operation state machine: PENDING, INVOKED, COMMITTED, UNKNOWN, and FAILED. The governing rule is strict: an agent must never treat a timeout as proof that an effect did not occur. It transitions the operation to UNKNOWN, then performs reconciliation against the remote service.

CREATE TABLE tool_operations (
  idem_key         TEXT PRIMARY KEY,
  run_id           TEXT NOT NULL,
  tool             TEXT NOT NULL,
  args_fingerprint TEXT NOT NULL,
  state            TEXT NOT NULL
    CHECK (state IN ('PENDING','INVOKED','COMMITTED','UNKNOWN','FAILED')),
  result           JSONB,
  updated_at       TIMESTAMPTZ NOT NULL DEFAULT now()
);

async function setState(key: string, state: string, result?: unknown) {
  await pool.query(
    `UPDATE tool_operations
     SET state = $2, result = COALESCE($3, result), updated_at = now()
     WHERE idem_key = $1`,
    [key, state, result === undefined ? null : JSON.stringify(result)],
  );
}

export async function executeOnce(
  call: ToolCall,
  effect: (key: string) => Promise<unknown>,
  lookup?: (key: string) => Promise<unknown | null>, // ask the provider "did this happen?"
) {
  const key = idemKey(call.runId, call.step, call.toolCallId);
  const fp = fingerprint(call.args);

  // atomic claim: only one attempt wins the INSERT
  const claim = await pool.query(
    `INSERT INTO tool_operations (idem_key, run_id, tool, args_fingerprint, state)
     VALUES ($1, $2, $3, $4, 'INVOKED')
     ON CONFLICT (idem_key) DO NOTHING
     RETURNING idem_key`,
    [key, call.runId, call.tool, fp],
  );

  if (claim.rowCount === 0) {
    const { rows } = await pool.query(
      `SELECT state, result, args_fingerprint FROM tool_operations WHERE idem_key = $1`,
      [key],
    );
    const op = rows[0];
    if (op.args_fingerprint !== fp) throw new Error("IDEMPOTENCY_KEY_REUSED_WITH_DIFFERENT_ARGS");
    if (op.state === "COMMITTED") return op.result;           // replay the recorded result
    if (op.state === "FAILED") throw new Error("OPERATION_PREVIOUSLY_FAILED");

    // INVOKED or UNKNOWN: an earlier attempt may have reached the provider
    if (!lookup) {
      await setState(key, "UNKNOWN");
      throw new NeedsHumanApproval(call, key);                // no way to know, so ask a human
    }
    const found = await lookup(key);
    if (found) {
      await setState(key, "COMMITTED", found);
      return found;
    }
    // provider has no record: fall through and run the effect once more
  }

  try {
    const result = await effect(key);
    await setState(key, "COMMITTED", result);
    return result;
  } catch (err) {
    // 4xx validation error before any effect: FAILED. Anything ambiguous: UNKNOWN
    await setState(key, isDefinitelyNotSent(err) ? "FAILED" : "UNKNOWN");
    throw err;
  }
}

For non-idempotent operations where automated reconciliation is impossible (such as irreversible production deployments), human-in-the-loop approval is the final safety net. However, human approval must bind cryptographically to the exact tool call and parameter fingerprint, preventing approval hijacking where one action is authorized but another executed. Furthermore, classify tools by retry safety: safe, idempotent, and unsafe. Return structured retry metadata including retryable and retry_after_seconds to stop agents from applying blanket retry rules to non-retryable 4xx client errors.

Step 6: Results, What the Matrix Tells Us

Spanning six Guard Levels across four Crash Windows produces a 24-cell evaluation grid. The heatmap below summarizes the architectural guarantees across each level: where duplicates occur, where they are eliminated, and where safety depends on upstream service compliance.

The evaluation matrix highlights three architectural realities. First, windows W1 and W2 present zero duplicate risk across all levels because crashes occur before side effect transmission. Second, window W3 is the primary battleground: Level 2 and Level 3 remain conditional because they rely on external providers honoring keys. Third, achieving zero duplicate mutations requires one of two paths: eliminating the window entirely via local atomic transactions (Level 4), or embracing uncertainty through state reconciliation (Level 5).

Each level introduces concrete infrastructure and engineering tradeoffs.

Regarding performance overhead, Temporal Cloud documentation reports an average scheduling overhead of approximately 50 ms for standard activity tasks. Factor this into your latency budget when evaluating Level 3 durable execution.

Step 7: How Production Systems Do It

Examining industry implementations, Stripe established the standard for API idempotency keys. Stripe retains idempotency keys for 24 hours, and Brandur Leach's reference implementation (opens in a new tab) introduces a recovery_point mechanism. Multi-step workflows do not re-run from the beginning following a crash; they resume from the latest verified progress marker. This closely mirrors our Level 5 state machine with finer granularity.

In the durable execution landscape, The New Stack reported that Temporal exceeds 3,000 enterprise customers, with benchmarks proving that terminating agent workers mid-step introduces zero data loss or duplicate charges. "Durable by default" is becoming a baseline requirement for production AI agent procurement. Model Context Protocol (MCP) systems face identical constraints: in Stateless Streamable HTTP transports, broken connections drop in-flight requests, prompting clients to retransmit. Making every MCP write tool idempotent is a direct operational necessity. Similarly, agent framework checkpointing is often misunderstood: while checkpoints enable fast resume after crashes, they are recorded after step completion. A crash between side effect mutation and checkpoint persistence will re-execute the action, meaning checkpointing alone does not prevent duplicate mutations.

Observability deserves emphasis. Industry telemetry guides show production tool calls fail between 3% and 15% of the time, and localized failures cascade silently across agent workflows. When an incident occurs, you need forensic certainty. Every tool execution trace must capture three fields: the derived idempotency key, the operational state, and whether the attempt represented an initial execution or a replayed cached result. Without these markers, validating that an operation executed "exactly once" is impossible.

Recap and Production Checklist

To distill this entire analysis into a single principle: Breaking the link between retries and duplicate side effects requires deriving idempotency keys from durable runtime state rather than model output, and anchoring those keys at the exact layer where retries take place. Window W3 can either be eliminated via local atomic commits (Level 4) or acknowledged and resolved via state reconciliation (Level 5), but it cannot be ignored. "Exactly-once" remains the product of at-least-once transport and idempotent side effects.

Selecting the right level depends on one architectural question: where does the physical side effect occur?

Before launching write-capable agents into production, audit your architecture against this checklist:

• Does every write tool accept a caller-provided idempotency key?
• Is the key derived from durable runtime state (such as runId, step, and tool call ID) rather than argument hashes?
• Is the intent logged durably before the external request is dispatched?
• Are the retry trigger and the idempotency key located at the exact same architectural layer?
• How does the system handle a crash in window W3? Has this been tested under simulated failure?
• When external APIs do not deduplicate requests, does the runtime support an UNKNOWN state with reconciliation?
• Are non-idempotent operations protected by human-in-the-loop approvals bound to exact parameter fingerprints?
• Does key retention TTL comfortably exceed your longest possible agent execution duration?
• Are compensating actions (such as refunds or rollbacks) designed with the same idempotency guarantees?
• Do distributed traces record the idempotency key, operational state, and replay-versus-fresh execution status for every tool step?

Further reading and references:

• Why 'Exactly Once' Delivery Is Mostly a Lie and Distributed Locks Under the Hood
• Stripe: Designing robust and predictable APIs with idempotency (opens in a new tab)
• Brandur Leach: Implementing Stripe-like Idempotency Keys in Postgres (opens in a new tab)
• Temporal: Error handling in the Python SDK (opens in a new tab)
• Durable Agent Runtime: Open source crash-resume evaluation (opens in a new tab)
• Lab repository: agent-crash-lab