How to configure HiveOS to auto-restart on GPU crash? (Stability)

Understanding GPU Crash Behavior in HiveOS

1. GPU crashes in HiveOS environments typically manifest as sudden mining process termination, driver hang detection, or kernel-level NMI interrupts triggered by hardware watchdog timeouts.

2. The system logs capture these events in /var/log/syslog with patterns like 'NVRM: Xid (PCI:0000:xx:00.0): 79' for NVIDIA or 'amdgpu: GPU fault detected' for AMD devices.

3. HiveOS does not treat GPU crashes as fatal system failures by default; instead, it relies on user-defined recovery logic within the mining service layer.

4. Kernel-level GPU resets may occur automatically if nvidia-smi --gpu-reset is enabled in driver configuration, but this requires explicit activation and root privileges.

5. Persistent crashes often correlate with unstable overclocking profiles, insufficient PCIe power delivery, or thermal throttling exceeding 95°C sustained for over 60 seconds.

Core Configuration Files for Auto-Restart Logic

1. The primary control point resides in /hive-config/user-config.yaml, where the watchdog section defines crash detection intervals and restart thresholds.

2. Editing /etc/systemd/system/hive-miner.service allows customization of Restart=on-failure, RestartSec=15, and StartLimitIntervalSec=300 to prevent infinite loops during persistent failure.

3. The gpu-health-check.sh script located in /hive/sbin/ executes every 45 seconds and triggers hive-restart when nvidia-smi -q -d MEMORY | grep 'Used' | awk '{print $3}' returns zero or invalid values three consecutive times.

4. Custom hooks can be injected via /hive-config/hooks/post-gpu-fail.sh, enabling actions such as fan speed override, core clock rollback, or temporary pool switching before restart.

5. HiveOS firmware version 0.6-218 introduced GPU_WATCHDOG_ENABLED=true in /etc/default/hiveos, which activates low-level PCI bus polling every 8 seconds to detect device disappearance.

Hardware-Level Recovery Mechanisms

1. Motherboard BIOS settings must enable PCIe ASPM L1 Substates and disable Fast Boot to ensure proper GPU re-enumeration after reset.

2. Power supply units rated below 80 PLUS Gold certification frequently fail to sustain transient 300W+ load spikes during GPU recovery sequences, causing secondary system reboots.

3. For multi-GPU rigs, the PCIe slot assignment order in BIOS affects reset isolation—slots connected to separate CPU PCIe root complexes allow independent recovery.

4. NVIDIA datacenter GPUs require NVreg_RegistryDwords='EnableMSI=0' in /etc/modprobe.d/nvidia.conf to prevent MSI interrupt storms during rapid restart cycles.

5. AMD RX 7000 series cards need amdgpu.gpu_recovery=1 added to kernel boot parameters in /etc/default/grub to activate hardware-assisted recovery paths.

Verification and Diagnostic Procedures

1. Simulate a controlled crash using echo 1 > /sys/bus/pci/devices/0000:xx:00.0/remove followed by echo 1 > /sys/bus/pci/rescan to validate restart timing and log capture accuracy.

2. Monitor journalctl -u hive-miner.service -n 100 --no-pager for entries containing 'GPU health check failed' and subsequent 'Restarting miner service' messages.

3. Confirm successful hardware-level recovery by checking lspci -vv -s 0000:xx:00.0 | grep 'LnkSta:' output shows stable Speed 16GT/s and Width x16 after restart.

4. Validate temperature stability post-recovery using hive-stats -f gpu_temp to ensure no thermal runaway occurs within first 120 seconds of resumed operation.

5. Cross-reference /var/log/hive/gpu-watchdog.log timestamps against dmesg | grep -i 'nvidia|amdgpu' to confirm alignment between software detection and kernel event logging.

Frequently Asked Questions

Q: Does HiveOS auto-restart functionality work with ASIC miners?A: No. Auto-restart logic is exclusively designed for GPU-based mining stacks. ASIC controllers operate independently and require external watchdog hardware or pool-side failover mechanisms.

Q: Can I configure different restart delays per GPU in a multi-GPU rig?A: Not natively. HiveOS applies uniform restart policies across all detected GPUs. Per-device timing requires custom systemd overrides tied to individual PCI addresses.

Q: Will auto-restart preserve my overclocking profile after recovery?A: Yes. HiveOS reloads the active /hive-config/oc-profile.yaml during each miner service restart, maintaining all voltage, memory, and core clock settings.

Q: Is there a risk of filesystem corruption during forced GPU reset?A: Minimal. HiveOS uses read-only root filesystem mounts and writes only to volatile /run and journal buffers during crash recovery sequences.