> For the complete documentation index, see [llms.txt](https://asus-isg-aidc.gitbook.io/guide/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://asus-isg-aidc.gitbook.io/guide/latest/cli/commandreference/dcgm.md).

# DCGM GPU Diagnostics

| Developer | Last modified |
| --------- | ------------- |
| AIDC Team | 2026/08/19    |

## Table of Contents

* [1. Overview](#overview)
* [2. Subcommands](#subcommands)
  * [2.1 `diag`](#dcgm-diag)
  * [2.2 `discovery`](#dcgm-discovery)
* [3. Quick Summary](#quick-summary)
* [4. Usage Examples](#usage-examples)
* [5. Report Output](#report-output)
  * [5.1 `diag` Output](#diag-output)
  * [5.2 `discovery` Output](#discovery-output)

***

## 1. Overview <a href="#overview" id="overview"></a>

{% hint style="warning" %}
**⚠️ Prerequisite: `aidc-cli init` required first**

This command depends on settings saved by [`aidc-cli init`](/guide/latest/cli/commandreference/init.md). Before running it, make sure you have completed:

* [`aidc-cli init inventory`](/guide/latest/cli/commandreference/init.md#inventory) — render the node inventory from `aidc.csv`
* SSH access to the nodes — [`aidc-cli deploy install`](/guide/latest/cli/commandreference/deploy.md) sets this up during OS deployment; for nodes that already have an OS, run [`aidc-cli deploy access-key`](/guide/latest/cli/commandreference/deploy.md) or fill the `password` column in `aidc.csv`

For the full setup sequence, see the [Getting Started](/guide/latest/cli/gettingstarted.md) guide.
{% endhint %}

The `dcgm` command runs NVIDIA Data Center GPU Manager (DCGM) diagnostics against the GPU nodes in your inventory. It exposes two read-only, one-shot operations: an active health diagnostic (`diag`) that exercises the GPUs at a chosen intensity, and a topology listing (`discovery`) that reports the GPUs, NvSwitches, ConnectX adapters, and CPUs visible to DCGM.

Both operations are read-only: they never change persistent GPU or system state. Results are aggregated across all targeted hosts into a single report.

```bash
aidc-cli dcgm <subcommand> [flags]
```

{% hint style="info" %}
Each target node must have the NVIDIA driver installed, the DCGM package present, and the `nvidia-dcgm` host-engine service running. Nodes without a GPU are reported as `NO_GPU_DETECTED` and do not fail an otherwise-clean fleet run. Nodes where DCGM is missing or its service is stopped are reported as `DCGM_UNAVAILABLE` with a remediation hint.
{% endhint %}

***

## 2. Subcommands <a href="#subcommands" id="subcommands"></a>

### 2.1 `diag` — Run DCGM Active Diagnostic <a href="#dcgm-diag" id="dcgm-diag"></a>

Runs the DCGM active diagnostic (`dcgmi diag`) at the selected run level and aggregates the per-GPU pass/warn/fail results for every targeted host.

```bash
aidc-cli dcgm diag [flags]
```

| Flag       | Short | Required | Default | Description                                                                                                  |
| ---------- | :---: | :------: | :-----: | ------------------------------------------------------------------------------------------------------------ |
| `--run`    |  `-r` |     ❌    |   `2`   | Diagnostic run level: `1` (quick), `2` (medium), `3` (long stress test). Only `1`, `2`, or `3` are accepted. |
| `--strict` |   —   |     ❌    | `false` | Treat `WARN` results as failures for exit-code purposes.                                                     |
| `--yes`    |  `-y` |     ❌    | `false` | Skip the run level 3 stress-test confirmation prompt (required for non-interactive runs).                    |
| `--stdout` |   —   |     ❌    | `false` | Print the raw JSON report to stdout in addition to writing the artifact file.                                |

{% hint style="warning" %}
Run level 3 executes an **extended GPU stress test under heavy load** that can run for a long time. `aidc-cli` asks for confirmation before starting it:

```
⚠️  run level 3 會對 GPU 執行長時間壓力測試,期間 GPU 進入高負載,請先確認節點沒有正在執行的生產工作負載。
⚠️  Run level 3 runs an extended GPU stress test under heavy load. Make sure no production workload is running on the target nodes.
繼續? / Continue? [y/N]:
```

Make sure no production workload is running on the target nodes before you proceed. In automation, pass `--yes` to skip the prompt.
{% endhint %}

***

### 2.2 `discovery` — List GPU/NvSwitch/CPU Topology <a href="#dcgm-discovery" id="dcgm-discovery"></a>

Lists the GPU, NvSwitch, ConnectX, and CPU topology that DCGM can see on each targeted host (`dcgmi discovery -l`). This is a fast, read-only inventory of the accelerator topology.

```bash
aidc-cli dcgm discovery [flags]
```

| Flag       | Short | Required | Default | Description                                                                   |
| ---------- | :---: | :------: | :-----: | ----------------------------------------------------------------------------- |
| `--stdout` |   —   |     ❌    | `false` | Print the raw JSON report to stdout in addition to writing the artifact file. |

***

## 3. Quick Summary <a href="#quick-summary" id="quick-summary"></a>

| Subcommand  | Description                                                  |
| ----------- | ------------------------------------------------------------ |
| `diag`      | Run the DCGM active GPU diagnostic at run level 1, 2, or 3.  |
| `discovery` | List the GPU/NvSwitch/ConnectX/CPU topology visible to DCGM. |

***

## 4. Usage Examples <a href="#usage-examples" id="usage-examples"></a>

```bash
# Run the default medium-level diagnostic across all GPU nodes
aidc-cli dcgm diag

# Quick diagnostic on two specific nodes
aidc-cli dcgm diag --run 1 --limit-hosts gpu01,gpu02

# Long stress test, non-interactive (skips the confirmation prompt)
aidc-cli dcgm diag --run 3 --yes

# Treat warnings as failures and stream the JSON report to the terminal
aidc-cli dcgm diag --strict --stdout

# List the GPU/NvSwitch/CPU topology and print it to the terminal
aidc-cli dcgm discovery --stdout
```

***

## 5. Report Output <a href="#report-output" id="report-output"></a>

With `--stdout`, both subcommands print a JSON report aggregating every targeted host. The `overall_status` of each host is one of `PASS`, `WARN`, `FAIL`, `UNKNOWN`, `NO_GPU_DETECTED`, or `DCGM_UNAVAILABLE`.

### 5.1 `diag` Output <a href="#diag-output" id="diag-output"></a>

```json
{
  "task_id": "20260717-dcgm-diag",
  "task_description": "DCGM diagnostic",
  "current_time": "2026-07-17 14:30",
  "report": [
    {
      "hostname": "gpu01",
      "ip_address": "10.0.0.11",
      "gpu_count": 8,
      "driver_version": "560.35.03",
      "dcgm_version": "3.3.9",
      "overall_status": "PASS",
      "run_level": 2,
      "checks": {
        "deployment": { "status": "PASS", "gpus": [] },
        "integration": { "status": "PASS", "gpus": [] },
        "stress": { "status": "PASS", "gpus": [] }
      }
    },
    {
      "hostname": "gpu02",
      "ip_address": "10.0.0.12",
      "overall_status": "DCGM_UNAVAILABLE",
      "message": "nvidia-dcgm service is not running (systemctl is-active: inactive)",
      "remediation": "start and enable the DCGM host engine: 'systemctl enable --now nvidia-dcgm'"
    }
  ],
  "summary": {
    "total_hosts": 2,
    "total_gpus": 8,
    "status_counts": {
      "PASS": 1,
      "WARN": 0,
      "FAIL": 0,
      "UNKNOWN": 0,
      "NO_GPU_DETECTED": 0,
      "DCGM_UNAVAILABLE": 1
    },
    "attention_hosts": ["gpu02"]
  }
}
```

{% hint style="info" %}
`dcgm diag --stdout` also sets the process exit code so it can be used in automated pipelines:

* `0` — all clear (every host `PASS`, or `WARN` when `--strict` is not set)
* `1` — a real diagnostic failure (any host `FAIL`, or any `WARN` when `--strict` is set)
* `2` — inconclusive (no outright failure, but at least one host was `DCGM_UNAVAILABLE` or `UNKNOWN`)

Hosts reported as `NO_GPU_DETECTED` are ignored for exit-code purposes.
{% endhint %}

### 5.2 `discovery` Output <a href="#discovery-output" id="discovery-output"></a>

`discovery` reports the topology visible to DCGM on each host: the `gpus`, `nvswitches`, `connectx`, and `cpus` arrays. The example below is **abbreviated** — a real GPU node lists one entry per GPU/NvSwitch/CPU; only representative entries are shown here.

```json
{
  "task_id": "20260717-dcgm-discovery",
  "task_description": "DCGM diagnostic",
  "current_time": "2026-07-17 14:35",
  "report": [
    {
      "hostname": "gpu01",
      "ip_address": "10.0.0.11",
      "gpu_count": 8,
      "driver_version": "560.35.03",
      "dcgm_version": "3.3.9",
      "overall_status": "PASS",
      "gpus": [
        {
          "gpu_id": 0,
          "name": "NVIDIA H100 80GB HBM3",
          "pci_bus_id": "00000000:1B:00.0",
          "device_uuid": "GPU-1a2b3c4d-0001-0000-0000-000000000000"
        },
        {
          "gpu_id": 1,
          "name": "NVIDIA H100 80GB HBM3",
          "pci_bus_id": "00000000:43:00.0",
          "device_uuid": "GPU-1a2b3c4d-0002-0000-0000-000000000000"
        }
      ],
      "nvswitches": [
        { "switch_id": 0 },
        { "switch_id": 1 }
      ],
      "connectx": [],
      "cpus": [
        { "cpu_id": 0 }
      ]
    }
  ],
  "summary": {
    "total_hosts": 1,
    "total_gpus": 8,
    "status_counts": {
      "PASS": 1,
      "WARN": 0,
      "FAIL": 0,
      "UNKNOWN": 0,
      "NO_GPU_DETECTED": 0,
      "DCGM_UNAVAILABLE": 0
    },
    "attention_hosts": []
  }
}
```

{% hint style="info" %}
`discovery` is a read-only topology listing — it does not run any diagnostic tests, so hosts that reach DCGM report `overall_status: PASS`. Hosts without a GPU or without a running DCGM service still appear as `NO_GPU_DETECTED` / `DCGM_UNAVAILABLE`.
{% endhint %}

**See also:** [Hardware Health](/guide/latest/cli/commandreference/hardwarehealth.md), [Hardware Sensor](/guide/latest/cli/commandreference/hardwaresensor.md), [SAT Baselines and Validation](/guide/latest/cli/commandreference/sat.md).