> For the complete documentation index, see [llms.txt](https://asus-isg-aidc.gitbook.io/guide/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://asus-isg-aidc.gitbook.io/guide/latest/cli/troubleshooting.md).

# Troubleshooting

| Developer | Last modified |
| --------- | ------------- |
| AIDC Team | 2026/08/21    |

## 1. License Issues

### License is not active

**Symptom**: `License is not active. Please activate the application using aidc-cli license.`

**Solution**:

```bash
# Activate with your license key
aidc-cli license active -k "YOUR-LICENSE-KEY"

# Check license status and expiry
aidc-cli license show
```

### License expired

**Symptom**: Commands are unavailable even after activation.

**Solution**: Check the `Expiry Date` field in `aidc-cli license show`. Contact your AIDC administrator for a new license key.

***

## 2. Inventory & DHCP Issues

### Duplicate IPs detected

**Symptom**: `init inventory` fails with duplicate IP error.

**Solution**: Check your CSV database files for duplicate IP addresses. Each IP must be unique across all CSV entries.

### DHCP subnet mismatch

**Symptom**: `dhcp-validate` reports IPs outside the DHCP subnet.

**Solution**:

```bash
# Check DHCP settings
aidc-cli init dhcp-get

# Fix subnet/netmask to match your CSV IPs
aidc-cli init dhcp-set -s <subnet> -m <netmask> -r <gateway> -n <next-server> -g <range-start>,<range-end>
```

### Device limit exceeded

**Symptom**: `init inventory` fails with device limit error.

**Solution**: Your license limits the number of managed devices. Check with `aidc-cli license show` and verify your CSV doesn't exceed the `Device Limit`.

***

## 3. BMC Issues

### BMC unreachable

**Symptom**: BMC commands timeout or fail to connect.

**Solution**:

1. Verify BMC IP in CSV is correct and reachable: `ping <bmc_ip>`
2. Check BMC credentials: `aidc-cli init bmc-get`
3. Ensure management node can reach BMC network segment
4. Use `-l` flag to test on a single node first: `aidc-cli bmc bmc-ver -l node01`

### BMC password authentication failed

**Symptom**: Command fails with authentication error on BMC.

**Solution**:

```bash
# Reset BMC password
aidc-cli init bmc-set -p "NewPassword"

# If BMC is using factory default password, use init-pwd
aidc-cli bmc init-pwd
```

***

## 4. Deployment Issues

### PXE boot fails

**Symptom**: Nodes don't boot from PXE after `deploy install`.

**Solution**:

1. Verify DHCP is enabled: `aidc-cli deploy dhcp-enabled`
2. Check DHCP configuration: `aidc-cli init dhcp-get`
3. Validate network: `aidc-cli chknet pxe`
4. Ensure PXE boot order is set: `aidc-cli bmc pxe-boot`

### Image not found

**Symptom**: `deploy image-download` fails or no active image set.

**Solution**:

```bash
# Discover available images
aidc-cli init image-discovery

# List available images
aidc-cli init image-list

# Set active downloadable image
aidc-cli init image-active

# Download the image
aidc-cli deploy image-download
```

***

## 5. Driver Issues

### NVIDIA GPU driver — force reboot warning

**Symptom**: System reboots unexpectedly after GPU driver install.

**Expected Behavior**: `driver nv-gpu` explicitly warns that it will force a system reboot. Plan accordingly.

```bash
# Uninstall if needed
aidc-cli driver nv-gpu --uninstall -f <driver-file>
```

### Mellanox OFED — file flag required

**Symptom**: `Error: --file flag is required for installation`

**Solution**: Provide the driver package path (without `.tgz` extension):

```bash
aidc-cli driver mlnx-ofed -f MLNX_OFED_LINUX-24.04-0.6.6.0
```

### DOCA driver — package manager required

**Symptom**: `Error: --pkg-mgr flag is required`

**Solution**:

```bash
# For RHEL
aidc-cli driver doca-host-nic -m dnf -f <doca-file>

# For Ubuntu
aidc-cli driver doca-host-nic -m apt -f <doca-file>
```

***

## 6. Network Issues

### Network validation failures

**Symptom**: `chknet pxe` or `chknet bmc` reports unreachable nodes.

**Solution**:

1. Check physical connectivity (cables, switch ports)
2. Verify IP configuration in CSV
3. Ensure `init inventory` has been run after CSV changes
4. Check firewall rules: `aidc-cli security firewall-list-rule`

### InfiniBand validation — format option

**Symptom**: IB report is hard to read.

**Solution**: Specify report format:

```bash
# Group by host
aidc-cli chknet ib -f host

# Group by device
aidc-cli chknet ib -f device
```

***

## 7. Firewall & SELinux Issues

### Firewall rule not taking effect

**Solution**:

```bash
# Ensure firewall is enabled first
aidc-cli security firewall-enable

# Add rule with port
aidc-cli security firewall-add-rule -p 8080/tcp

# Or add rule with service name
aidc-cli security firewall-add-rule -s http,https

# Verify rules
aidc-cli security firewall-list-rule
```

### SELinux changes require reboot

**Expected Behavior**: Both `selinux-enable` and `selinux-disable` require a system reboot to take effect. The CLI does **not** auto-reboot; you must do it manually or use `aidc-cli bmc power-reset`.

***

## 8. Service Deployment Issues

### Slurm — missing compute node flag

**Symptom**: `aidc-cli slurm slurmd` fails.

**Solution**: The `-c` flag is required to specify compute nodes:

```bash
aidc-cli slurm slurmd -c node01,node02,node03
```

### NMX — missing password flags

**Symptom**: `aidc-cli nmx deploy` fails with missing flag error.

**Solution**: All three password flags are required:

```bash
aidc-cli nmx deploy \
  -W "kong-rw-password" \
  -E "kong-ro-password" \
  -V "vm-password"
```

### WEKA — BMC not configured

**Symptom**: `Error: BMC_USERNAME is not configured.`

**Solution**:

```bash
# Configure BMC credentials first
aidc-cli init bmc-set -p "your-bmc-password"

# Then deploy WEKA
aidc-cli weka server deploy -I weka-4.4.10.150-2.0.2.iso
```

***

## 9. General Debugging

### Check execution logs

AIDC-CLI logs each task execution with a remark (e.g., `bmc_power-on`). Check:

* Console output for immediate errors
* Log files for detailed execution output

### Run on a single node first

Always test on one node before running on the full cluster:

```bash
aidc-cli <command> <subcommand> -l node01
```

### Verify SSH connectivity

```bash
# Test SSH connectivity to a target node
ssh <node-hostname> hostname
```

### Common flag mistakes

| Mistake                                   | Correction                                                                                           |
| ----------------------------------------- | ---------------------------------------------------------------------------------------------------- |
| Missing required flag                     | Check `aidc-cli <cmd> <subcmd> --help` for required flags                                            |
| Using `--install` and `--remove` together | These are mutually exclusive (Package command)                                                       |
| Providing multiple exclusive flags        | Some commands only accept one flag at a time (e.g., `pkg-set --add` vs `--delete` vs `--remove-all`) |