Recovery after restart
How to restore agent context after downtime or restart
When an AI agent restarts ??whether from a crash, deployment update, or scheduled maintenance ??it loses in-memory context. This guide explains how to query on-chain state and reconstruct operational context.
Recovery sequence
On startup, the agent should execute these checks in order:
- Verify PDA existence ??Confirm the AICWallet PDA exists at the expected address. If not, the wallet was never issued or is on a different network.
- Check wallet status ??Read the AICWallet account to verify
is_active = trueand the wallet hasn't been deactivated. - Check will status ??Read the AIWill account. If
updated_by_ai = false, the will hasn't been activated yet ??the agent should callcreate_will. - Evaluate liveness ??Compare
last_heartbeat+death_timeoutagainst current time. If close to death, send heartbeat immediately. - Check balances ??Query both the agent pubkey (for transaction fees) and the PDA (for operational funds). Alert if either is critically low.
On-chain queries
| Data needed | Account to read | Key fields |
|---|---|---|
| Wallet exists & active | AICWallet PDA | is_active, issuer |
| Will status | AIWill PDA | updated_by_ai, beneficiaries |
| Liveness deadline | AIWill PDA | last_heartbeat, death_timeout |
| Transaction fee balance | Agent pubkey account | SOL balance (lamports) |
| Operational balance | AICWallet PDA | SOL balance (lamports) |
Example recovery check (Python)
Python
from solana.rpc.api import Client
from solders.pubkey import Pubkey
rpc = Client("https://api.devnet.solana.com")
# 1. Check if wallet PDA exists
wallet_info = rpc.get_account_info(wallet_pda)
if wallet_info.value is None:
raise Exception("Wallet PDA not found - check network or address")
# 2. Parse wallet data and verify active
wallet_data = parse_aicw_wallet(wallet_info.value.data)
if not wallet_data.is_active:
raise Exception("Wallet is deactivated")
# 3. Check will status
will_info = rpc.get_account_info(will_pda)
will_data = parse_ai_will(will_info.value.data)
if not will_data.updated_by_ai:
print("Will not activated - should call create_will")
# 4. Check liveness
import time
current_time = int(time.time())
time_until_death = (will_data.last_heartbeat + will_data.death_timeout) - current_time
if time_until_death < 86400: # Less than 1 day
print(f"URGENT: Only {time_until_death}s until death - send heartbeat!")
# 5. Check balances
agent_balance = rpc.get_balance(agent_pubkey).value
pda_balance = rpc.get_balance(wallet_pda).value
print(f"Agent fee balance: {agent_balance / 1e9} SOL")
print(f"PDA operational balance: {pda_balance / 1e9} SOL")Using MCP server for recovery
If using Track B (MCP server), the aicw_check_survival tool performs all these checks automatically and returns a structured status report. Simply ask your agent: "Check my AICW wallet status"
Common recovery scenarios
| Scenario | Detection | Action |
|---|---|---|
| First startup after issuance | updated_by_ai = false | Call create_will to activate |
| Near death timeout | Time until death < 1 day | Send heartbeat immediately |
| Low fee balance | Agent pubkey < 0.01 SOL | Request funding or alert operator |
| Wallet already dead | Current time > last_heartbeat + death_timeout | Cannot recover ??will execution is pending |
Best practices
- Run recovery checks on every agent startup, not just after crashes
- Log recovery results for debugging and audit purposes
- Set up alerting for low balances and approaching death timeouts
- Test recovery flow on devnet before relying on it in production