Name card folders by GPU UUID, record serial in serial.txt
Folder identity switches from the physical serial (which neither nvflash nor nvidia-smi exposes on the 170HX) to the auto-read GPU UUID from nvidia-smi. The physical serial probe result (or N/A) is saved to <folder>/serial.txt and report.json. No manual serial prompt anymore; --serial only overrides the recorded serial. Unidentified runs fall back to unidentified/<bdf>/. AGENTS.md invariant and README updated to match.
This commit is contained in:
@@ -14,7 +14,7 @@ This repository checks one NVIDIA CMP 170HX at a time on a fresh Ubuntu 26.04 x8
|
||||
|
||||
## Invariants
|
||||
|
||||
- A card folder is named from an automatically read **physical serial number**. Never request manual serial entry or substitute a GPU UUID or PCI address for a serial. When serial lookup fails, continue the run under `unidentified/` and report the missing serial.
|
||||
- A card folder is named from the automatically read **GPU UUID** (`nvidia-smi`). The physical serial (nvflash probe, else `N/A`) is recorded in `<folder>/serial.txt` and `report.json`. Never prompt for manual serial entry. When even the UUID is unavailable, continue the run under `unidentified/<bdf>/` and report it.
|
||||
- Match every card-sensitive operation to the detected PCI address. A host may have another NVIDIA GPU even though only one CMP 170HX is being checked.
|
||||
- Measure capacitor-related lane width before unlocking. The width result is independent of cmpunlocker: Gen1 x16 passes the width check, while x4 fails it. Do not claim the capacitor is physically verified; the width is an observed link result on the current host.
|
||||
- Keep the final PCIe acceptance check after unlock. Preserve a red `[ERROR]` message and failing exit for any final link other than Gen2 x16. Gen1 x16 passes the earlier width check but fails the final card check.
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
Repo: https://git.qbuissondebon.eu.org/qbuissondebon/cmp170hx_check (SSH push: `git@git.ssh.qbuissondebon.eu.org:qbuissondebon/cmp170hx_check.git`)
|
||||
|
||||
Single-card checker for **NVIDIA CMP 170HX** cards (PCI IDs `10de:20c2` / `10de:2082`) — run it **one card at a time**: it waits for the card, verifies the PCIe capacitor mod, records PCIe bus health, backs up VBIOS, runs the cmpunlocker verification, offers the **opt-in** cmpunlocker unlock install (then a cold reboot and a re-run for the final PCIe check), and optionally stresses VRAM. Each card gets its own folder + JSON report named by serial, so the folders accumulate as you go through your batch (e.g. 23 cards).
|
||||
Single-card checker for **NVIDIA CMP 170HX** cards (PCI IDs `10de:20c2` / `10de:2082`) — run it **one card at a time**: it waits for the card, verifies the PCIe capacitor mod, records PCIe bus health, backs up VBIOS, runs the cmpunlocker verification, offers the **opt-in** cmpunlocker unlock install (then a cold reboot and a re-run for the final PCIe check), and optionally stresses VRAM. Each card gets its own folder + JSON report named by GPU UUID, so the folders accumulate as you go through your batch (e.g. 23 cards).
|
||||
|
||||
Reference harness (auto-cloned into `harness/` by preflight):
|
||||
|
||||
@@ -84,13 +84,13 @@ python3 check_cards.py --help # all flags
|
||||
Flow of a normal run:
|
||||
|
||||
1. Waits for a CMP 170HX on the bus — prompts you to insert/reseat the card, Enter triggers a PCI rescan (`q` quits)
|
||||
2. Serial auto-read from nvflash UUID; falls back to a manual label prompt
|
||||
3. VBIOS backup → `cards/<serial>/bios.rom` (nvflash, sysfs ROM fallback)
|
||||
2. Folder `cards/<GPU-UUID>/` from the auto-read GPU UUID (`nvidia-smi`); physical serial probed from nvflash and saved to `serial.txt` (`N/A` when the card exposes none) — never prompts
|
||||
3. VBIOS backup → `cards/<GPU-UUID>/bios.rom` (nvflash, sysfs ROM fallback)
|
||||
4. PCIe checks: link speed/width, AER counters, full config-space dump
|
||||
5. Link retrain test + full bus rescan test (device must survive both)
|
||||
6. Kernel log scrape (`dmesg`/`journalctl`) for this BDF: Xid / AER / NVRM lines
|
||||
7. `nvidia-smi` snapshot + `harness/cmpunlocker/verify.sh` (read-only)
|
||||
8. **Opt-in unlock**: `cmpunlocker unlock (install)` — skipped with a note when memory is already unlocked, otherwise asks `Run cmpunlocker install.sh now? (y/N)` (default **n**; `--unlock` runs it without asking). On yes: `git pull --ff-only` on the harness clone, then `install.sh` runs from `harness/cmpunlocker/` with **live terminal output** (full log → `cards/<serial>/install_unlocker.log`; the kernel-module build takes 5–15 min)
|
||||
8. **Opt-in unlock**: `cmpunlocker unlock (install)` — skipped with a note when memory is already unlocked, otherwise asks `Run cmpunlocker install.sh now? (y/N)` (default **n**; `--unlock` runs it without asking). On yes: `git pull --ff-only` on the harness clone, then `install.sh` runs from `harness/cmpunlocker/` with **live terminal output** (full log → `cards/<GPU-UUID>/install_unlocker.log`; the kernel-module build takes 5–15 min)
|
||||
9. **After a successful install**: the VRAM stress is skipped — **cold reboot the host** (`sudo shutdown -h now`, then power it back on) and **re-run the same command** for the final PCIe check, which must report **Gen2 x16**
|
||||
10. **Last step, optional**: `Run 5-min Vulkan VRAM stress test now? (y/N)` — refused by default, `--vulkan always|never` to override
|
||||
11. Result printed + `report.json` written into the card's folder
|
||||
@@ -111,7 +111,8 @@ Exit codes: `0` PASS/WARN · `1` FAIL · `2` no card found · `3` preflight bloc
|
||||
|
||||
```
|
||||
cards/
|
||||
└── <serial>/
|
||||
└── <GPU-UUID>/
|
||||
├── serial.txt # physical serial, or N/A
|
||||
├── bios.rom # VBIOS backup (+ sha256 in report)
|
||||
├── report.json # full result: link, AER before/after, retrain,
|
||||
│ # rescan, dmesg, verify, vulkan, PASS/WARN/FAIL
|
||||
@@ -131,7 +132,7 @@ cards/
|
||||
| `--preflight` | setup only: sudo precondition + package install + harness clone, then exit |
|
||||
| `--demo` | run the whole pipeline against fake hardware (self-test) |
|
||||
| `--nvflash PATH` | explicit nvflash binary |
|
||||
| `--serial XYZ` | override serial (skips nvflash auto-read) |
|
||||
| `--serial XYZ` | override physical serial recorded in `serial.txt` (folder stays UUID-named) |
|
||||
| `--vulkan ask\|always\|never` | VRAM test policy (default: ask, last step) |
|
||||
| `--unlock` | run the cmpunlocker `install.sh` step without prompting (default: ask y/N) |
|
||||
| `--skip-bios` | skip VBIOS backup |
|
||||
@@ -142,5 +143,5 @@ cards/
|
||||
|
||||
This tool **reads** card state by default. The one step that changes the card is **opt-in**:
|
||||
|
||||
- **Unlock install (opt-in, explicit confirmation)** — runs only if you answer `y` to `Run cmpunlocker install.sh now? (y/N)` (default `n`) or pass `--unlock`. It executes `install.sh` **from the harness clone** (`harness/cmpunlocker/`, refreshed first with `git pull --ff-only`, no extra installer flags), needs root (sudo credentials are cached first), streams its output live, and saves the full log to `cards/<serial>/install_unlocker.log`. It is skipped entirely when memory is already unlocked, when the harness is not cloned, and in `--demo` mode. A successful install replaces kernel modules — the step then skips VRAM stress and asks you to cold reboot and re-run.
|
||||
- **Unlock install (opt-in, explicit confirmation)** — runs only if you answer `y` to `Run cmpunlocker install.sh now? (y/N)` (default `n`) or pass `--unlock`. It executes `install.sh` **from the harness clone** (`harness/cmpunlocker/`, refreshed first with `git pull --ff-only`, no extra installer flags), needs root (sudo credentials are cached first), streams its output live, and saves the full log to `cards/<GPU-UUID>/install_unlocker.log`. It is skipped entirely when memory is already unlocked, when the harness is not cloned, and in `--demo` mode. A successful install replaces kernel modules — the step then skips VRAM stress and asks you to cold reboot and re-run.
|
||||
- **Everything else remains read-only**: `verify.sh` runs without installing anything, no modules are patched and no firmware is flashed (nvflash is only ever used with `--save`). The only other writes are local files (reports, logs), PCI link retrain/rescan sysfs triggers, and package installation during `--preflight` (explicitly gated behind the sudo precondition).
|
||||
|
||||
+25
-13
@@ -1,13 +1,13 @@
|
||||
#!/usr/bin/env python3
|
||||
"""CMP 170HX single-card checker - ONE card per run, no queue.
|
||||
|
||||
Per run: wait for card -> auto serial -> VBIOS backup -> PCIe link/AER/config
|
||||
checks -> retrain + rescan tests -> dmesg scrape -> cmpunlocker verify.sh
|
||||
(read-only) -> opt-in cmpunlocker unlock (install.sh, y/N or --unlock; a
|
||||
successful install skips VRAM stress and asks for a cold reboot, then you
|
||||
re-run this script for the final Gen2 x16 PCIe check)
|
||||
Per run: wait for card -> auto UUID (folder) + serial probe -> VBIOS backup
|
||||
-> PCIe link/AER/config checks -> retrain + rescan tests -> dmesg scrape
|
||||
-> cmpunlocker verify.sh (read-only) -> opt-in cmpunlocker unlock (install.sh,
|
||||
y/N or --unlock; a successful install skips VRAM stress and asks for a cold
|
||||
reboot, then you re-run this script for the final Gen2 x16 PCIe check)
|
||||
-> optional 5-min Vulkan VRAM stress (last step, ask yes/no)
|
||||
-> cards/<serial>/report.json.
|
||||
-> cards/<GPU-UUID>/report.json (+ serial.txt with the physical serial or N/A).
|
||||
|
||||
Scoring: FAIL on no-device/link-down/lost-after-rescan, and on a final
|
||||
post-unlock link other than Gen2 x16; WARN on correctable AER/Xid/width< x16/
|
||||
@@ -121,12 +121,19 @@ def process_card(args):
|
||||
ok(f"device found at {bdf} ({devid})")
|
||||
|
||||
nf = X.nvflash_bin(args)
|
||||
serial, serial_src = X.get_serial(nf, args)
|
||||
folder = X.CARDS_DIR / sanitize(serial)
|
||||
uuid = X.get_gpu_uuid()
|
||||
if uuid:
|
||||
folder = X.CARDS_DIR / sanitize(uuid)
|
||||
else:
|
||||
warn("GPU UUID unavailable - filing run under unidentified/")
|
||||
folder = X.UNIDENTIFIED_DIR / sanitize(bdf)
|
||||
folder.mkdir(parents=True, exist_ok=True)
|
||||
serial, serial_src = X.get_serial(nf, args)
|
||||
(folder / "serial.txt").write_text((serial or "N/A") + "\n")
|
||||
info(f"folder: {folder}")
|
||||
|
||||
rep = {
|
||||
"uuid": uuid, "folder": str(folder),
|
||||
"serial": serial, "serial_source": serial_src, "started_at": now_iso(),
|
||||
"bdf": bdf, "device_id": devid, "host": X.run(["uname", "-n"])["out"].strip(),
|
||||
"kernel": X.run(["uname", "-r"])["out"].strip(),
|
||||
@@ -220,12 +227,13 @@ def process_card(args):
|
||||
(folder / "report.json").write_text(json.dumps(rep, indent=2))
|
||||
|
||||
print()
|
||||
label = rep.get("uuid") or folder.name
|
||||
if result == "PASS":
|
||||
ok(f"{serial}: PASS")
|
||||
ok(f"{label}: PASS")
|
||||
elif result == "WARN":
|
||||
warn(f"{serial}: WARN")
|
||||
warn(f"{label}: WARN")
|
||||
else:
|
||||
err(f"{serial}: FAIL")
|
||||
err(f"{label}: FAIL")
|
||||
for f in fails:
|
||||
err(" FAIL: " + f)
|
||||
for w in warns:
|
||||
@@ -237,7 +245,7 @@ def process_card(args):
|
||||
# ---------------------------------------------------------------- demo
|
||||
|
||||
def demo(args):
|
||||
"""Validate report pipeline without hardware: writes cards/DEMO-*."""
|
||||
"""Validate report pipeline without hardware: writes cards/GPU-DEMO-UUID-0001."""
|
||||
step("Demo mode (no hardware)")
|
||||
args.vulkan = "never"
|
||||
args.serial = "DEMO-SERIAL-001"
|
||||
@@ -245,12 +253,14 @@ def demo(args):
|
||||
|
||||
orig_wait = globals()["wait_for_gpu"]
|
||||
orig_detect, orig_check = X.detect_gpus, X.check_link
|
||||
orig_uuid = X.get_gpu_uuid
|
||||
orig_aer, orig_dm, orig_cfg = X.aer_snapshot, X.dmesg_capture, X.dump_config
|
||||
orig_retrain, orig_rescan = X.retrain_test, X.rescan_test
|
||||
orig_smi, orig_verify = X.nvidia_smi_info, X.run_verify_unlocker
|
||||
orig_update, orig_installer = X.update_cmpunlocker, X.run_installer
|
||||
globals()["wait_for_gpu"] = lambda: "0000:0a:00.0"
|
||||
X.detect_gpus = lambda: ["0000:0a:00.0"]
|
||||
X.get_gpu_uuid = lambda: "GPU-DEMO-UUID-0001"
|
||||
X.check_link = lambda b: ({"lnksta_speed": "5GT/s", "lnksta_width": "x4", "lnksta_gen": 2,
|
||||
"lnkcap_speed": "8GT/s", "lnkcap_width": "x16",
|
||||
"lnksta_raw": "LnkSta: Speed 5GT/s, Width x4, TrErr-",
|
||||
@@ -274,6 +284,7 @@ def demo(args):
|
||||
finally:
|
||||
globals()["wait_for_gpu"] = orig_wait
|
||||
X.detect_gpus, X.check_link = orig_detect, orig_check
|
||||
X.get_gpu_uuid = orig_uuid
|
||||
X.aer_snapshot, X.dmesg_capture, X.dump_config = orig_aer, orig_dm, orig_cfg
|
||||
X.retrain_test, X.rescan_test = orig_retrain, orig_rescan
|
||||
X.nvidia_smi_info, X.run_verify_unlocker = orig_smi, orig_verify
|
||||
@@ -289,7 +300,8 @@ def main():
|
||||
help="tool check + package install (requires sudo) + harness clone, then exit")
|
||||
ap.add_argument("--demo", action="store_true", help="run pipeline with fake hardware (self-test)")
|
||||
ap.add_argument("--nvflash", help="path to nvflash binary")
|
||||
ap.add_argument("--serial", help="override serial (skips nvflash auto-read)")
|
||||
ap.add_argument("--serial", help="override physical serial recorded in serial.txt "
|
||||
"(folder stays UUID-named)")
|
||||
ap.add_argument("--skip-bios", action="store_true", help="skip VBIOS backup")
|
||||
ap.add_argument("--vulkan", choices=("ask", "always", "never"), default="ask",
|
||||
help="5-min Vulkan VRAM test policy (default ask, runs as last step)")
|
||||
|
||||
@@ -14,6 +14,7 @@ from pathlib import Path
|
||||
SCRIPT_DIR = Path(__file__).resolve().parent
|
||||
HARNESS_DIR = SCRIPT_DIR / "harness"
|
||||
CARDS_DIR = SCRIPT_DIR / "cards"
|
||||
UNIDENTIFIED_DIR = SCRIPT_DIR / "unidentified"
|
||||
|
||||
REPOS = {
|
||||
"170th-Street": "https://github.com/amoghmunikote/170th-Street.git",
|
||||
@@ -633,12 +634,7 @@ def get_serial(nf, args):
|
||||
warn("nvflash present but exposes no serial/UUID for this card")
|
||||
else:
|
||||
warn("nvflash unavailable - cannot auto-read serial")
|
||||
serial = ""
|
||||
while not serial:
|
||||
serial = ask("Enter serial number from the card label (folder name)")
|
||||
if not serial:
|
||||
warn("serial is required for the folder name")
|
||||
return serial, "manual-label"
|
||||
return "N/A", "unavailable"
|
||||
|
||||
|
||||
def save_bios(nf, out_path, args):
|
||||
@@ -692,6 +688,20 @@ def nvidia_smi_info():
|
||||
return {"available": True, "gpus": rows}
|
||||
|
||||
|
||||
def get_gpu_uuid():
|
||||
"""First GPU UUID from nvidia-smi (e.g. GPU-xxxx...), or None."""
|
||||
if not which("nvidia-smi"):
|
||||
return None
|
||||
q = run(["nvidia-smi", "--query-gpu=uuid", "--format=csv,noheader"], timeout=30)
|
||||
if q["rc"] != 0:
|
||||
return None
|
||||
for line in q["out"].splitlines():
|
||||
line = line.strip()
|
||||
if line.startswith("GPU-"):
|
||||
return line
|
||||
return None
|
||||
|
||||
|
||||
def run_verify_unlocker():
|
||||
if not VERIFY_SH.exists():
|
||||
return {"ran": False, "note": "harness/cmpunlocker not cloned"}
|
||||
|
||||
Reference in New Issue
Block a user