> For the complete documentation index, see [llms.txt](https://asus-isg-aidc.gitbook.io/guide/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://asus-isg-aidc.gitbook.io/guide/latest/cli/commandreference/hardwarehealth.md).

# Hardware Health

| Developer | Last modified |
| --------- | ------------- |
| AIDC Team | 2026/08/19    |

## Table of Contents

* [1. Overview](#overview)
* [2. Views](#views)
* [3. What Gets Checked](#what-gets-checked)
* [4. Common Flags](#common-flags)
* [5. Usage Examples](#usage-examples)

***

## 1. Overview <a href="#overview" id="overview"></a>

{% hint style="warning" %}
**⚠️ Prerequisite: `aidc-cli init` required first**

This command depends on settings saved by [`aidc-cli init`](/guide/latest/cli/commandreference/init.md). Before running it, make sure you have completed:

* [`aidc-cli init inventory`](/guide/latest/cli/commandreference/init.md#inventory) — render the node inventory from `aidc.csv`
* SSH access to the nodes — [`aidc-cli deploy install`](/guide/latest/cli/commandreference/deploy.md) sets this up during OS deployment; for nodes that already have an OS, run [`aidc-cli deploy access-key`](/guide/latest/cli/commandreference/deploy.md) or fill the `password` column in `aidc.csv`

For the full setup sequence, see the [Getting Started](/guide/latest/cli/gettingstarted.md) guide.
{% endhint %}

The `chkhw` command runs in-band hardware health checks and produces either a compact view or a normalized summary view.

```bash
aidc-cli chkhw <brief|summary> [flags]
```

***

## 2. Views <a href="#views" id="views"></a>

| Subcommand | Description                                      |
| ---------- | ------------------------------------------------ |
| `brief`    | Run compact in-band hardware health checks       |
| `summary`  | Run a normalized in-band hardware health summary |

***

## 3. What Gets Checked <a href="#what-gets-checked" id="what-gets-checked"></a>

Both views run the same set of checks per host; `brief` returns just a pass/fail map, while `summary` includes the full detail behind each result.

| Check                    | Description                                                                                                                  |
| ------------------------ | ---------------------------------------------------------------------------------------------------------------------------- |
| `memory_edac`            | ECC correctable/uncorrectable memory error counters (Linux EDAC)                                                             |
| `cpu_error_state`        | CPU thermal-throttle counts and machine-check (MCE) events                                                                   |
| `storage_health`         | NVMe SMART critical warnings/media errors and RAID (`mdstat`) degradation                                                    |
| `network_error_counters` | RX/TX error counters per network interface                                                                                   |
| `pcie_aer`               | PCIe Advanced Error Reporting (AER) events in the kernel log                                                                 |
| `sensor_visibility`      | Presence of hardware monitoring (`hwmon`) sensors                                                                            |
| `bmc_event_log`          | BMC System Event Log (SEL) entries; flags Critical/Non-recoverable events                                                    |
| `gpu_xid`                | NVIDIA GPU Xid errors in the kernel log                                                                                      |
| `gpu_nvlink`             | NVLink link status, with root-cause hints on inactive links; also checks the Grace↔Blackwell C2C interconnect on GB200/GH200 |

{% hint style="info" %}
`bmc_event_log` reports `UNKNOWN` when the BMC's SEL is unavailable. `gpu_nvlink` reports `UNKNOWN` on hosts without an NVIDIA GPU; when links are inactive, it also reports the NVIDIA Fabric Manager service state and any `vfio-pci` GPU passthrough bindings as a likely cause. On GB200/GH200 superchips, `gpu_nvlink` additionally checks the Grace↔Blackwell chip-to-chip (C2C) interconnect — platforms that don't support C2C report `NOT_SUPPORTED` rather than failing.
{% endhint %}

***

## 4. Common Flags <a href="#common-flags" id="common-flags"></a>

Both `chkhw` views support the following flag:

| Flag       | Short | Required | Description                                                                                 |
| ---------- | :---: | :------: | ------------------------------------------------------------------------------------------- |
| `--stdout` |   —   |     ❌    | Output the raw hardware health JSON to stdout instead of only saving the generated artifact |

***

## 5. Usage Examples <a href="#usage-examples" id="usage-examples"></a>

```bash
# Run the compact hardware health view
aidc-cli chkhw brief

# Run the normalized summary view
aidc-cli chkhw summary

# Output the summary view to stdout as JSON
aidc-cli chkhw summary --stdout

# Run the summary view for selected nodes only
aidc-cli chkhw summary -l node01,node02
```

**See also:** [DCGM GPU Diagnostics](/guide/latest/cli/commandreference/dcgm.md), [Hardware Sensor](/guide/latest/cli/commandreference/hardwaresensor.md).