How to automate mining rig reboots when it goes offline?

Monitoring System Integration

1. Deploy a lightweight agent on the mining rig’s host OS that continuously reports hash rate, GPU temperature, and pool connection status to a central server.

2. Configure thresholds for critical metrics—such as zero accepted shares for 90 seconds or GPU utilization dropping below 5% for over two minutes.

3. Use Prometheus with custom exporters to scrape rig health data every 15 seconds, feeding into Grafana dashboards for real-time visibility.

4. Integrate SNMP traps or Syslog forwarding to capture kernel-level failures like PCIe link drops or driver crashes that may not surface in miner logs.

5. Assign unique identifiers to each rig using MAC address hashing or serial number tagging to avoid misidentification during mass alerts.

Remote Reboot Triggers

1. Set up an HTTP webhook endpoint on a separate VPS that receives POST payloads from the monitoring system when offline conditions are confirmed.

2. Authenticate incoming requests using HMAC signatures derived from shared secrets to prevent spoofed reboot commands.

3. Route validated triggers to a Python script that executes IPMI or Wake-on-LAN commands based on hardware support—Supermicro boards respond to ipmitool chassis power cycle, while consumer rigs rely on WoL magic packets.

4. Enforce a cooldown window of 180 seconds after each reboot attempt to avoid cascading restart loops caused by persistent firmware hangs.

5. Log all trigger events—including timestamp, source IP, rig ID, and reason code—to a local SQLite database with daily rotation.

Firmware-Level Recovery

1. Flash rigs with BIOS versions that enable “Restore on AC Power Loss” set to “Power On”, ensuring automatic boot after brief outages.

2. Patch NVIDIA drivers with custom init scripts that reload nvidia-smi and restart the miner process if GPU memory errors exceed three occurrences per minute.

3. Use watchdog timers via the Linux kernel’s softdog module, configured to trigger a hard reset if the miner process fails to write to /dev/watchdog every 60 seconds.

4. Embed a minimal BusyBox recovery partition on the SSD that boots independently if the main OS fails to mount or hangs at init.

5. Disable USB autosuspend for GPU risers by adding usbcore.autosuspend=-1 to kernel boot parameters to prevent enumeration failures.

Network Resilience Configuration

1. Assign static ARP entries on the upstream switch for each rig’s IP to prevent DHCP lease expiration from breaking remote access.

2. Run dnsmasq locally on the rig to cache DNS queries for mining pools, reducing dependency on external resolvers during partial network degradation.

3. Bind the miner binary to a specific interface using SO_BINDTODEVICE to avoid routing flaps when multi-homed rigs experience NIC failover.

4. Implement TCP keepalive settings in the miner’s configuration: tcp-keepalive = 60, to detect dead pool connections faster than default timeouts.

5. Use conntrack -D --orig-dst [rig-ip] in firewall scripts to flush stale NAT state entries that block SSH access post-reboot.

Common Questions

Q: Can I use Telegram bots to manually trigger reboots?Yes—configure a Telegram bot with a private group chat, then parse incoming messages via Webhook API. Validate sender IDs against a pre-approved list before executing systemctl reboot on the target host.

Q: Why does my rig come back online but show 0 MH/s after auto-reboot?This usually indicates the miner binary failed to start due to missing environment variables or incorrect CUDA_VISIBLE_DEVICES binding. Ensure your reboot script sources ~/.bashrc and sets GPU indices explicitly.

Q: Is it safe to run watchdog-triggered reboots on ASIC miners?No—most ASIC firmware lacks watchdog support and may brick if forced into uncontrolled power cycles. Stick to vendor-provided REST APIs like Bitmain’s BMMiner web interface for controlled resets.

Q: How do I test reboot automation without risking live mining?Create a dummy rig VM with identical OS, GPU drivers, and miner version. Simulate offline states using iptables DROP rules on outbound pool ports, then verify alert-to-reboot latency stays under 120 seconds.