Name card folders by GPU UUID, record serial in serial.txt

Folder identity switches from the physical serial (which neither
nvflash nor nvidia-smi exposes on the 170HX) to the auto-read GPU
UUID from nvidia-smi. The physical serial probe result (or N/A) is
saved to <folder>/serial.txt and report.json. No manual serial
prompt anymore; --serial only overrides the recorded serial.
Unidentified runs fall back to unidentified/<bdf>/. AGENTS.md
invariant and README updated to match.
This commit is contained in:
2026-10-15 00:17:13 +02:00
parent 70a6ff9b1a
commit ac53b83f70
4 changed files with 50 additions and 27 deletions
+1 -1
View File
@@ -14,7 +14,7 @@ This repository checks one NVIDIA CMP 170HX at a time on a fresh Ubuntu 26.04 x8
## Invariants ## Invariants
- A card folder is named from an automatically read **physical serial number**. Never request manual serial entry or substitute a GPU UUID or PCI address for a serial. When serial lookup fails, continue the run under `unidentified/` and report the missing serial. - A card folder is named from the automatically read **GPU UUID** (`nvidia-smi`). The physical serial (nvflash probe, else `N/A`) is recorded in `<folder>/serial.txt` and `report.json`. Never prompt for manual serial entry. When even the UUID is unavailable, continue the run under `unidentified/<bdf>/` and report it.
- Match every card-sensitive operation to the detected PCI address. A host may have another NVIDIA GPU even though only one CMP 170HX is being checked. - Match every card-sensitive operation to the detected PCI address. A host may have another NVIDIA GPU even though only one CMP 170HX is being checked.
- Measure capacitor-related lane width before unlocking. The width result is independent of cmpunlocker: Gen1 x16 passes the width check, while x4 fails it. Do not claim the capacitor is physically verified; the width is an observed link result on the current host. - Measure capacitor-related lane width before unlocking. The width result is independent of cmpunlocker: Gen1 x16 passes the width check, while x4 fails it. Do not claim the capacitor is physically verified; the width is an observed link result on the current host.
- Keep the final PCIe acceptance check after unlock. Preserve a red `[ERROR]` message and failing exit for any final link other than Gen2 x16. Gen1 x16 passes the earlier width check but fails the final card check. - Keep the final PCIe acceptance check after unlock. Preserve a red `[ERROR]` message and failing exit for any final link other than Gen2 x16. Gen1 x16 passes the earlier width check but fails the final card check.
+8 -7
View File
@@ -2,7 +2,7 @@
Repo: https://git.qbuissondebon.eu.org/qbuissondebon/cmp170hx_check (SSH push: `git@git.ssh.qbuissondebon.eu.org:qbuissondebon/cmp170hx_check.git`) Repo: https://git.qbuissondebon.eu.org/qbuissondebon/cmp170hx_check (SSH push: `git@git.ssh.qbuissondebon.eu.org:qbuissondebon/cmp170hx_check.git`)
Single-card checker for **NVIDIA CMP 170HX** cards (PCI IDs `10de:20c2` / `10de:2082`) — run it **one card at a time**: it waits for the card, verifies the PCIe capacitor mod, records PCIe bus health, backs up VBIOS, runs the cmpunlocker verification, offers the **opt-in** cmpunlocker unlock install (then a cold reboot and a re-run for the final PCIe check), and optionally stresses VRAM. Each card gets its own folder + JSON report named by serial, so the folders accumulate as you go through your batch (e.g. 23 cards). Single-card checker for **NVIDIA CMP 170HX** cards (PCI IDs `10de:20c2` / `10de:2082`) — run it **one card at a time**: it waits for the card, verifies the PCIe capacitor mod, records PCIe bus health, backs up VBIOS, runs the cmpunlocker verification, offers the **opt-in** cmpunlocker unlock install (then a cold reboot and a re-run for the final PCIe check), and optionally stresses VRAM. Each card gets its own folder + JSON report named by GPU UUID, so the folders accumulate as you go through your batch (e.g. 23 cards).
Reference harness (auto-cloned into `harness/` by preflight): Reference harness (auto-cloned into `harness/` by preflight):
@@ -84,13 +84,13 @@ python3 check_cards.py --help # all flags
Flow of a normal run: Flow of a normal run:
1. Waits for a CMP 170HX on the bus — prompts you to insert/reseat the card, Enter triggers a PCI rescan (`q` quits) 1. Waits for a CMP 170HX on the bus — prompts you to insert/reseat the card, Enter triggers a PCI rescan (`q` quits)
2. Serial auto-read from nvflash UUID; falls back to a manual label prompt 2. Folder `cards/<GPU-UUID>/` from the auto-read GPU UUID (`nvidia-smi`); physical serial probed from nvflash and saved to `serial.txt` (`N/A` when the card exposes none) — never prompts
3. VBIOS backup → `cards/<serial>/bios.rom` (nvflash, sysfs ROM fallback) 3. VBIOS backup → `cards/<GPU-UUID>/bios.rom` (nvflash, sysfs ROM fallback)
4. PCIe checks: link speed/width, AER counters, full config-space dump 4. PCIe checks: link speed/width, AER counters, full config-space dump
5. Link retrain test + full bus rescan test (device must survive both) 5. Link retrain test + full bus rescan test (device must survive both)
6. Kernel log scrape (`dmesg`/`journalctl`) for this BDF: Xid / AER / NVRM lines 6. Kernel log scrape (`dmesg`/`journalctl`) for this BDF: Xid / AER / NVRM lines
7. `nvidia-smi` snapshot + `harness/cmpunlocker/verify.sh` (read-only) 7. `nvidia-smi` snapshot + `harness/cmpunlocker/verify.sh` (read-only)
8. **Opt-in unlock**: `cmpunlocker unlock (install)` — skipped with a note when memory is already unlocked, otherwise asks `Run cmpunlocker install.sh now? (y/N)` (default **n**; `--unlock` runs it without asking). On yes: `git pull --ff-only` on the harness clone, then `install.sh` runs from `harness/cmpunlocker/` with **live terminal output** (full log → `cards/<serial>/install_unlocker.log`; the kernel-module build takes 5–15 min) 8. **Opt-in unlock**: `cmpunlocker unlock (install)` — skipped with a note when memory is already unlocked, otherwise asks `Run cmpunlocker install.sh now? (y/N)` (default **n**; `--unlock` runs it without asking). On yes: `git pull --ff-only` on the harness clone, then `install.sh` runs from `harness/cmpunlocker/` with **live terminal output** (full log → `cards/<GPU-UUID>/install_unlocker.log`; the kernel-module build takes 5–15 min)
9. **After a successful install**: the VRAM stress is skipped — **cold reboot the host** (`sudo shutdown -h now`, then power it back on) and **re-run the same command** for the final PCIe check, which must report **Gen2 x16** 9. **After a successful install**: the VRAM stress is skipped — **cold reboot the host** (`sudo shutdown -h now`, then power it back on) and **re-run the same command** for the final PCIe check, which must report **Gen2 x16**
10. **Last step, optional**: `Run 5-min Vulkan VRAM stress test now? (y/N)` — refused by default, `--vulkan always|never` to override 10. **Last step, optional**: `Run 5-min Vulkan VRAM stress test now? (y/N)` — refused by default, `--vulkan always|never` to override
11. Result printed + `report.json` written into the card's folder 11. Result printed + `report.json` written into the card's folder
@@ -111,7 +111,8 @@ Exit codes: `0` PASS/WARN · `1` FAIL · `2` no card found · `3` preflight bloc
``` ```
cards/ cards/
└── <serial>/ └── <GPU-UUID>/
├── serial.txt # physical serial, or N/A
├── bios.rom # VBIOS backup (+ sha256 in report) ├── bios.rom # VBIOS backup (+ sha256 in report)
├── report.json # full result: link, AER before/after, retrain, ├── report.json # full result: link, AER before/after, retrain,
│ # rescan, dmesg, verify, vulkan, PASS/WARN/FAIL │ # rescan, dmesg, verify, vulkan, PASS/WARN/FAIL
@@ -131,7 +132,7 @@ cards/
| `--preflight` | setup only: sudo precondition + package install + harness clone, then exit | | `--preflight` | setup only: sudo precondition + package install + harness clone, then exit |
| `--demo` | run the whole pipeline against fake hardware (self-test) | | `--demo` | run the whole pipeline against fake hardware (self-test) |
| `--nvflash PATH` | explicit nvflash binary | | `--nvflash PATH` | explicit nvflash binary |
| `--serial XYZ` | override serial (skips nvflash auto-read) | | `--serial XYZ` | override physical serial recorded in `serial.txt` (folder stays UUID-named) |
| `--vulkan ask\|always\|never` | VRAM test policy (default: ask, last step) | | `--vulkan ask\|always\|never` | VRAM test policy (default: ask, last step) |
| `--unlock` | run the cmpunlocker `install.sh` step without prompting (default: ask y/N) | | `--unlock` | run the cmpunlocker `install.sh` step without prompting (default: ask y/N) |
| `--skip-bios` | skip VBIOS backup | | `--skip-bios` | skip VBIOS backup |
@@ -142,5 +143,5 @@ cards/
This tool **reads** card state by default. The one step that changes the card is **opt-in**: This tool **reads** card state by default. The one step that changes the card is **opt-in**:
- **Unlock install (opt-in, explicit confirmation)** — runs only if you answer `y` to `Run cmpunlocker install.sh now? (y/N)` (default `n`) or pass `--unlock`. It executes `install.sh` **from the harness clone** (`harness/cmpunlocker/`, refreshed first with `git pull --ff-only`, no extra installer flags), needs root (sudo credentials are cached first), streams its output live, and saves the full log to `cards/<serial>/install_unlocker.log`. It is skipped entirely when memory is already unlocked, when the harness is not cloned, and in `--demo` mode. A successful install replaces kernel modules — the step then skips VRAM stress and asks you to cold reboot and re-run. - **Unlock install (opt-in, explicit confirmation)** — runs only if you answer `y` to `Run cmpunlocker install.sh now? (y/N)` (default `n`) or pass `--unlock`. It executes `install.sh` **from the harness clone** (`harness/cmpunlocker/`, refreshed first with `git pull --ff-only`, no extra installer flags), needs root (sudo credentials are cached first), streams its output live, and saves the full log to `cards/<GPU-UUID>/install_unlocker.log`. It is skipped entirely when memory is already unlocked, when the harness is not cloned, and in `--demo` mode. A successful install replaces kernel modules — the step then skips VRAM stress and asks you to cold reboot and re-run.
- **Everything else remains read-only**: `verify.sh` runs without installing anything, no modules are patched and no firmware is flashed (nvflash is only ever used with `--save`). The only other writes are local files (reports, logs), PCI link retrain/rescan sysfs triggers, and package installation during `--preflight` (explicitly gated behind the sudo precondition). - **Everything else remains read-only**: `verify.sh` runs without installing anything, no modules are patched and no firmware is flashed (nvflash is only ever used with `--save`). The only other writes are local files (reports, logs), PCI link retrain/rescan sysfs triggers, and package installation during `--preflight` (explicitly gated behind the sudo precondition).
+25 -13
View File
@@ -1,13 +1,13 @@
#!/usr/bin/env python3 #!/usr/bin/env python3
"""CMP 170HX single-card checker - ONE card per run, no queue. """CMP 170HX single-card checker - ONE card per run, no queue.
Per run: wait for card -> auto serial -> VBIOS backup -> PCIe link/AER/config Per run: wait for card -> auto UUID (folder) + serial probe -> VBIOS backup
checks -> retrain + rescan tests -> dmesg scrape -> cmpunlocker verify.sh -> PCIe link/AER/config checks -> retrain + rescan tests -> dmesg scrape
(read-only) -> opt-in cmpunlocker unlock (install.sh, y/N or --unlock; a -> cmpunlocker verify.sh (read-only) -> opt-in cmpunlocker unlock (install.sh,
successful install skips VRAM stress and asks for a cold reboot, then you y/N or --unlock; a successful install skips VRAM stress and asks for a cold
re-run this script for the final Gen2 x16 PCIe check) reboot, then you re-run this script for the final Gen2 x16 PCIe check)
-> optional 5-min Vulkan VRAM stress (last step, ask yes/no) -> optional 5-min Vulkan VRAM stress (last step, ask yes/no)
-> cards/<serial>/report.json. -> cards/<GPU-UUID>/report.json (+ serial.txt with the physical serial or N/A).
Scoring: FAIL on no-device/link-down/lost-after-rescan, and on a final Scoring: FAIL on no-device/link-down/lost-after-rescan, and on a final
post-unlock link other than Gen2 x16; WARN on correctable AER/Xid/width< x16/ post-unlock link other than Gen2 x16; WARN on correctable AER/Xid/width< x16/
@@ -121,12 +121,19 @@ def process_card(args):
ok(f"device found at {bdf} ({devid})") ok(f"device found at {bdf} ({devid})")
nf = X.nvflash_bin(args) nf = X.nvflash_bin(args)
serial, serial_src = X.get_serial(nf, args) uuid = X.get_gpu_uuid()
folder = X.CARDS_DIR / sanitize(serial) if uuid:
folder = X.CARDS_DIR / sanitize(uuid)
else:
warn("GPU UUID unavailable - filing run under unidentified/")
folder = X.UNIDENTIFIED_DIR / sanitize(bdf)
folder.mkdir(parents=True, exist_ok=True) folder.mkdir(parents=True, exist_ok=True)
serial, serial_src = X.get_serial(nf, args)
(folder / "serial.txt").write_text((serial or "N/A") + "\n")
info(f"folder: {folder}") info(f"folder: {folder}")
rep = { rep = {
"uuid": uuid, "folder": str(folder),
"serial": serial, "serial_source": serial_src, "started_at": now_iso(), "serial": serial, "serial_source": serial_src, "started_at": now_iso(),
"bdf": bdf, "device_id": devid, "host": X.run(["uname", "-n"])["out"].strip(), "bdf": bdf, "device_id": devid, "host": X.run(["uname", "-n"])["out"].strip(),
"kernel": X.run(["uname", "-r"])["out"].strip(), "kernel": X.run(["uname", "-r"])["out"].strip(),
@@ -220,12 +227,13 @@ def process_card(args):
(folder / "report.json").write_text(json.dumps(rep, indent=2)) (folder / "report.json").write_text(json.dumps(rep, indent=2))
print() print()
label = rep.get("uuid") or folder.name
if result == "PASS": if result == "PASS":
ok(f"{serial}: PASS") ok(f"{label}: PASS")
elif result == "WARN": elif result == "WARN":
warn(f"{serial}: WARN") warn(f"{label}: WARN")
else: else:
err(f"{serial}: FAIL") err(f"{label}: FAIL")
for f in fails: for f in fails:
err(" FAIL: " + f) err(" FAIL: " + f)
for w in warns: for w in warns:
@@ -237,7 +245,7 @@ def process_card(args):
# ---------------------------------------------------------------- demo # ---------------------------------------------------------------- demo
def demo(args): def demo(args):
"""Validate report pipeline without hardware: writes cards/DEMO-*.""" """Validate report pipeline without hardware: writes cards/GPU-DEMO-UUID-0001."""
step("Demo mode (no hardware)") step("Demo mode (no hardware)")
args.vulkan = "never" args.vulkan = "never"
args.serial = "DEMO-SERIAL-001" args.serial = "DEMO-SERIAL-001"
@@ -245,12 +253,14 @@ def demo(args):
orig_wait = globals()["wait_for_gpu"] orig_wait = globals()["wait_for_gpu"]
orig_detect, orig_check = X.detect_gpus, X.check_link orig_detect, orig_check = X.detect_gpus, X.check_link
orig_uuid = X.get_gpu_uuid
orig_aer, orig_dm, orig_cfg = X.aer_snapshot, X.dmesg_capture, X.dump_config orig_aer, orig_dm, orig_cfg = X.aer_snapshot, X.dmesg_capture, X.dump_config
orig_retrain, orig_rescan = X.retrain_test, X.rescan_test orig_retrain, orig_rescan = X.retrain_test, X.rescan_test
orig_smi, orig_verify = X.nvidia_smi_info, X.run_verify_unlocker orig_smi, orig_verify = X.nvidia_smi_info, X.run_verify_unlocker
orig_update, orig_installer = X.update_cmpunlocker, X.run_installer orig_update, orig_installer = X.update_cmpunlocker, X.run_installer
globals()["wait_for_gpu"] = lambda: "0000:0a:00.0" globals()["wait_for_gpu"] = lambda: "0000:0a:00.0"
X.detect_gpus = lambda: ["0000:0a:00.0"] X.detect_gpus = lambda: ["0000:0a:00.0"]
X.get_gpu_uuid = lambda: "GPU-DEMO-UUID-0001"
X.check_link = lambda b: ({"lnksta_speed": "5GT/s", "lnksta_width": "x4", "lnksta_gen": 2, X.check_link = lambda b: ({"lnksta_speed": "5GT/s", "lnksta_width": "x4", "lnksta_gen": 2,
"lnkcap_speed": "8GT/s", "lnkcap_width": "x16", "lnkcap_speed": "8GT/s", "lnkcap_width": "x16",
"lnksta_raw": "LnkSta: Speed 5GT/s, Width x4, TrErr-", "lnksta_raw": "LnkSta: Speed 5GT/s, Width x4, TrErr-",
@@ -274,6 +284,7 @@ def demo(args):
finally: finally:
globals()["wait_for_gpu"] = orig_wait globals()["wait_for_gpu"] = orig_wait
X.detect_gpus, X.check_link = orig_detect, orig_check X.detect_gpus, X.check_link = orig_detect, orig_check
X.get_gpu_uuid = orig_uuid
X.aer_snapshot, X.dmesg_capture, X.dump_config = orig_aer, orig_dm, orig_cfg X.aer_snapshot, X.dmesg_capture, X.dump_config = orig_aer, orig_dm, orig_cfg
X.retrain_test, X.rescan_test = orig_retrain, orig_rescan X.retrain_test, X.rescan_test = orig_retrain, orig_rescan
X.nvidia_smi_info, X.run_verify_unlocker = orig_smi, orig_verify X.nvidia_smi_info, X.run_verify_unlocker = orig_smi, orig_verify
@@ -289,7 +300,8 @@ def main():
help="tool check + package install (requires sudo) + harness clone, then exit") help="tool check + package install (requires sudo) + harness clone, then exit")
ap.add_argument("--demo", action="store_true", help="run pipeline with fake hardware (self-test)") ap.add_argument("--demo", action="store_true", help="run pipeline with fake hardware (self-test)")
ap.add_argument("--nvflash", help="path to nvflash binary") ap.add_argument("--nvflash", help="path to nvflash binary")
ap.add_argument("--serial", help="override serial (skips nvflash auto-read)") ap.add_argument("--serial", help="override physical serial recorded in serial.txt "
"(folder stays UUID-named)")
ap.add_argument("--skip-bios", action="store_true", help="skip VBIOS backup") ap.add_argument("--skip-bios", action="store_true", help="skip VBIOS backup")
ap.add_argument("--vulkan", choices=("ask", "always", "never"), default="ask", ap.add_argument("--vulkan", choices=("ask", "always", "never"), default="ask",
help="5-min Vulkan VRAM test policy (default ask, runs as last step)") help="5-min Vulkan VRAM test policy (default ask, runs as last step)")
+16 -6
View File
@@ -14,6 +14,7 @@ from pathlib import Path
SCRIPT_DIR = Path(__file__).resolve().parent SCRIPT_DIR = Path(__file__).resolve().parent
HARNESS_DIR = SCRIPT_DIR / "harness" HARNESS_DIR = SCRIPT_DIR / "harness"
CARDS_DIR = SCRIPT_DIR / "cards" CARDS_DIR = SCRIPT_DIR / "cards"
UNIDENTIFIED_DIR = SCRIPT_DIR / "unidentified"
REPOS = { REPOS = {
"170th-Street": "https://github.com/amoghmunikote/170th-Street.git", "170th-Street": "https://github.com/amoghmunikote/170th-Street.git",
@@ -633,12 +634,7 @@ def get_serial(nf, args):
warn("nvflash present but exposes no serial/UUID for this card") warn("nvflash present but exposes no serial/UUID for this card")
else: else:
warn("nvflash unavailable - cannot auto-read serial") warn("nvflash unavailable - cannot auto-read serial")
serial = "" return "N/A", "unavailable"
while not serial:
serial = ask("Enter serial number from the card label (folder name)")
if not serial:
warn("serial is required for the folder name")
return serial, "manual-label"
def save_bios(nf, out_path, args): def save_bios(nf, out_path, args):
@@ -692,6 +688,20 @@ def nvidia_smi_info():
return {"available": True, "gpus": rows} return {"available": True, "gpus": rows}
def get_gpu_uuid():
"""First GPU UUID from nvidia-smi (e.g. GPU-xxxx...), or None."""
if not which("nvidia-smi"):
return None
q = run(["nvidia-smi", "--query-gpu=uuid", "--format=csv,noheader"], timeout=30)
if q["rc"] != 0:
return None
for line in q["out"].splitlines():
line = line.strip()
if line.startswith("GPU-"):
return line
return None
def run_verify_unlocker(): def run_verify_unlocker():
if not VERIFY_SH.exists(): if not VERIFY_SH.exists():
return {"ran": False, "note": "harness/cmpunlocker not cloned"} return {"ran": False, "note": "harness/cmpunlocker not cloned"}