How to configure watchdog for auto-restart? (Uptime management)

Understanding Watchdog Mechanisms in Blockchain Node Operations

1. A watchdog is a dedicated monitoring process that observes the health and responsiveness of blockchain node software such as Geth, Erigon, or Solana-validator.

2. It continuously checks for liveness signals including HTTP API availability, RPC endpoint responsiveness, and internal heartbeat logs.

3. When a node fails to respond within predefined thresholds—like missing three consecutive /health checks—the watchdog triggers recovery actions.

4. This mechanism prevents silent failures where a node appears running but no longer participates in consensus or relays transactions.

5. In decentralized infrastructure, uptime directly impacts validator rewards, mempool propagation speed, and RPC service SLAs for dApp developers.

Core Configuration Files and Parameters

1. Systemd-based watchdogs rely on Restart=always, RestartSec=10, and StartLimitIntervalSec=60 in the .service file.

2. The WatchdogSec=30 directive enables systemd’s built-in watchdog timer, requiring the binary to emit WD_NOTIFY=1 at regular intervals.

3. For non-systemd environments, custom scripts use curl -f http://localhost:8545/health --max-time 5 to validate RPC liveness before initiating systemctl restart.

4. Environment variables like ETH_RPC_URL and VALIDATOR_KEY_PATH must persist across restarts via ExecStartPre directives or external config mounts.

5. Log rotation policies must be enforced so watchdog-triggered restarts do not fill disk space with unrotated debug traces from failed sync attempts.

Integration with Consensus Layer Health Signals

1. Modern validators require cross-layer verification: the execution client must report synced status while the consensus client confirms attestation participation.

2. A robust watchdog queries both endpoints—e.g., GET /eth/v1/node/syncing on Lighthouse and POST /admin/peers on Besu—and treats mismatched states as critical.

3. Beacon chain finality stalls are detected by comparing current finalized epoch against the latest known value stored in Redis or SQLite.

4. If slashing protection databases become unreachable, the watchdog halts restart loops and escalates to PagerDuty instead of risking double-signing.

5. Peer count decay below 25 over 90 seconds triggers emergency peer refresh via hardcoded bootnodes before full process termination.

Security Constraints in Auto-Restart Workflows

1. Restart privileges are restricted to a dedicated system user with no shell access and minimal capabilities via Capabilities=CAP_SYS_ADMIN+ep.

2. Private keys remain mounted read-only from encrypted volumes; watchdog processes never hold decryption keys or memory-mapped keyfiles.

3. Each restart increments a monotonic counter stored in /run/watchdog/restart_count, which locks further restarts if exceeding five in one hour.

4. All restart events write immutable entries to journald with _SYSTEMD_UNIT=validator.service and _TRANSPORT=journal, enabling correlation with on-chain slash detection alerts.

5. TLS certificate expiration checks run pre-restart using openssl x509 -in /etc/ssl/certs/rpc.crt -checkend 86400 to avoid revocation-induced downtime.

Frequently Asked Questions

Q: Can watchdog restarts cause nonce misalignment in transaction broadcasting?A: No. Nonce management resides outside the node process in external signers like Fireblocks or local ledger wallets. The node only reads pending nonce values via eth_getTransactionCount; restarts do not reset or overwrite them.

Q: Does automatic restart interfere with Ethereum’s fork detection logic?A: Not when configured correctly. Fork-aware watchdogs parse the response from eth_chainId and eth_getBlockByNumber before restart. If chain ID mismatches persist across three polls, the process halts and emits FATAL_CHAIN_MISMATCH instead of restarting.

Q: How does the watchdog handle database corruption during fast sync?A: It detects leveldb corruption signatures in stderr output—such as “Corruption: checksum mismatch”—and triggers a safe rollback to last verified snapshot rather than restarting the same broken state.

Q: Is it safe to enable watchdog on a node running inside a Docker container?A: Yes, provided the container uses --init, mounts /dev/kmsg, and runs with --restart=unless-stopped disabled to avoid conflict with host-level systemd supervision.