Recovery after restart

How to restore agent context after downtime or restart

When an AI agent restarts ??whether from a crash, deployment update, or scheduled maintenance ??it loses in-memory context. This guide explains how to query on-chain state and reconstruct operational context.

Recovery sequence

On startup, the agent should execute these checks in order:

  1. Verify PDA existence ??Confirm the AICWallet PDA exists at the expected address. If not, the wallet was never issued or is on a different network.
  2. Check wallet status ??Read the AICWallet account to verify is_active = true and the wallet hasn't been deactivated.
  3. Check will status ??Read the AIWill account. If updated_by_ai = false, the will hasn't been activated yet ??the agent should call create_will.
  4. Evaluate liveness ??Compare last_heartbeat + death_timeout against current time. If close to death, send heartbeat immediately.
  5. Check balances ??Query both the agent pubkey (for transaction fees) and the PDA (for operational funds). Alert if either is critically low.

On-chain queries

Data neededAccount to readKey fields
Wallet exists & activeAICWallet PDAis_active, issuer
Will statusAIWill PDAupdated_by_ai, beneficiaries
Liveness deadlineAIWill PDAlast_heartbeat, death_timeout
Transaction fee balanceAgent pubkey accountSOL balance (lamports)
Operational balanceAICWallet PDASOL balance (lamports)

Example recovery check (Python)

Python
from solana.rpc.api import Client
from solders.pubkey import Pubkey

rpc = Client("https://api.devnet.solana.com")

# 1. Check if wallet PDA exists
wallet_info = rpc.get_account_info(wallet_pda)
if wallet_info.value is None:
    raise Exception("Wallet PDA not found - check network or address")

# 2. Parse wallet data and verify active
wallet_data = parse_aicw_wallet(wallet_info.value.data)
if not wallet_data.is_active:
    raise Exception("Wallet is deactivated")

# 3. Check will status
will_info = rpc.get_account_info(will_pda)
will_data = parse_ai_will(will_info.value.data)
if not will_data.updated_by_ai:
    print("Will not activated - should call create_will")

# 4. Check liveness
import time
current_time = int(time.time())
time_until_death = (will_data.last_heartbeat + will_data.death_timeout) - current_time
if time_until_death < 86400:  # Less than 1 day
    print(f"URGENT: Only {time_until_death}s until death - send heartbeat!")

# 5. Check balances
agent_balance = rpc.get_balance(agent_pubkey).value
pda_balance = rpc.get_balance(wallet_pda).value
print(f"Agent fee balance: {agent_balance / 1e9} SOL")
print(f"PDA operational balance: {pda_balance / 1e9} SOL")

Using MCP server for recovery

If using Track B (MCP server), the aicw_check_survival tool performs all these checks automatically and returns a structured status report. Simply ask your agent: "Check my AICW wallet status"

Common recovery scenarios

ScenarioDetectionAction
First startup after issuanceupdated_by_ai = falseCall create_will to activate
Near death timeoutTime until death < 1 daySend heartbeat immediately
Low fee balanceAgent pubkey < 0.01 SOLRequest funding or alert operator
Wallet already deadCurrent time > last_heartbeat + death_timeoutCannot recover ??will execution is pending

Best practices

  • Run recovery checks on every agent startup, not just after crashes
  • Log recovery results for debugging and audit purposes
  • Set up alerting for low balances and approaching death timeouts
  • Test recovery flow on devnet before relying on it in production