Ascend NPU Hardware Monitoring

MindCluster NPU Exporter provides Ascend NPU metrics. Run one exporter on every monitored NPU node.

Run Sections 1-3 on each monitored NPU node. Run Section 4 on the host where RL-Insight Server is running.

1. Check the existing NPU service

Check first so an existing exporter can be reused, avoiding a duplicate installation and port conflict:

npu-smi info
command -v npu-exporter
pgrep -af npu-exporter
curl --noproxy '*' -fsS http://<NODE_IP>:8082/metrics | head

npu-smi info must display the NPU correctly before the exporter can collect through the driver. Continue according to the result:

/metrics returns successfully: skip installation and startup, then go directly to Section 4, “Register NPU monitoring endpoints on the RL-Insight Server host.”
A binary exists but /metrics is unreachable: skip installation and go to Section 3, “Start and verify NPU Exporter.” Check that another process is not using the selected port.
No binary exists: continue with Section 2.

2. Install NPU Exporter

The following script installs the MindCluster 26.0.0 binary layout. Download and extraction run as the current user; commands that write system directories use sudo:

(
set -euo pipefail

VERSION=26.0.0
ARCH="$(uname -m)"
case "${ARCH}" in
  aarch64|x86_64) ;;
  *) echo "Unsupported architecture: ${ARCH}"; exit 1 ;;
esac

PACKAGE="Ascend-mindxdl-npu-exporter_${VERSION}_linux-${ARCH}.zip"
WORK_DIR="$(mktemp -d)"
trap 'rm -rf "${WORK_DIR}"' EXIT

curl -fL \
  "https://gitcode.com/Ascend/mind-cluster/releases/download/v${VERSION}/${PACKAGE}" \
  -o "${WORK_DIR}/${PACKAGE}"
unzip -q "${WORK_DIR}/${PACKAGE}" -d "${WORK_DIR}/package"

sudo install -m 0500 "${WORK_DIR}/package/npu-exporter" /usr/local/bin/npu-exporter
sudo install -m 0400 "${WORK_DIR}/package/metricConfiguration.json" /usr/local/metricConfiguration.json
sudo install -m 0400 "${WORK_DIR}/package/pluginConfiguration.json" /usr/local/pluginConfiguration.json
sudo install -d -m 0750 /var/log/mindx-dl/npu-exporter
)

Running the binary directly does not require a systemd service file. See the official Ascend installation guide for non-root installation and version-specific details.

3. Start and verify NPU Exporter

<NODE_IP> is the local IP of the machine where NPU Exporter is being installed. Replace it, then start the binary directly:

Note: The following command requires the current user to have valid sudo credentials or to run it from a root shell. After nohup starts, it cannot accept a sudo password interactively.

nohup sudo env \
  LD_LIBRARY_PATH=/usr/local/Ascend/driver/lib64/driver:/usr/local/Ascend/driver/lib64/common \
  GOGC=50 GOMAXPROCS=2 GODEBUG=madvdontneed=1 \
  /usr/local/bin/npu-exporter \
  -ip=<NODE_IP> \
  -port=8082 \
  -updateTime=5 \
  -logFile=/var/log/mindx-dl/npu-exporter/npu-exporter.log &

/usr/local/Ascend/driver is the Ascend driver installation directory. NPU Exporter uses LD_LIBRARY_PATH to load driver libraries such as libdcmi.so from this directory. If the driver is installed elsewhere, replace these paths with the actual driver path.

nohup runs the process in the background, and Exporter logs go to the file specified by -logFile. Collection stops if the process exits; use Docker, Supervisor, or another process manager for production deployments.

Verify on the current NPU node:

curl --noproxy '*' -fsS http://<NODE_IP>:8082/metrics | head

4. Register NPU monitoring endpoints on the RL-Insight Server host

Run all commands in this section on the host where RL-Insight Server is running. Create npu_targets.yaml:

jobs:
  - job_name: npu-exporter
    targets:
      - target: "<NODE_01_IP>:8082"
        labels:
          node: node-01
      - target: "<NODE_02_IP>:8082"
        labels:
          node: node-02

Confirm that RL-Insight Server is running on the current host. Skip the first command if it is already running:

rl-insight server start --detach
rl-insight server targets add npu_targets.yaml

targets add only registers targets and reloads Prometheus; it does not manage NPU Exporter. After registration, view NPU utilization, memory, power, temperature, and network throughput in the RL-Insight Grafana dashboards.

RL-Insight NPU hardware metrics