diff --git a/AGENTS.md b/AGENTS.md index 71efbd9..2b043fe 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -14,7 +14,7 @@ This repository checks one NVIDIA CMP 170HX at a time on a fresh Ubuntu 26.04 x8 ## Invariants -- A card folder is named from an automatically read **physical serial number**. Never request manual serial entry or substitute a GPU UUID or PCI address for a serial. When serial lookup fails, continue the run under `unidentified/` and report the missing serial. +- A card folder is named from the automatically read **GPU UUID** (`nvidia-smi`). The physical serial (nvflash probe, else `N/A`) is recorded in `/serial.txt` and `report.json`. Never prompt for manual serial entry. When even the UUID is unavailable, continue the run under `unidentified//` and report it. - Match every card-sensitive operation to the detected PCI address. A host may have another NVIDIA GPU even though only one CMP 170HX is being checked. - Measure capacitor-related lane width before unlocking. The width result is independent of cmpunlocker: Gen1 x16 passes the width check, while x4 fails it. Do not claim the capacitor is physically verified; the width is an observed link result on the current host. - Keep the final PCIe acceptance check after unlock. Preserve a red `[ERROR]` message and failing exit for any final link other than Gen2 x16. Gen1 x16 passes the earlier width check but fails the final card check. diff --git a/README.md b/README.md index d1106eb..335df37 100755 --- a/README.md +++ b/README.md @@ -2,7 +2,7 @@ Repo: https://git.qbuissondebon.eu.org/qbuissondebon/cmp170hx_check (SSH push: `git@git.ssh.qbuissondebon.eu.org:qbuissondebon/cmp170hx_check.git`) -Single-card checker for **NVIDIA CMP 170HX** cards (PCI IDs `10de:20c2` / `10de:2082`) — run it **one card at a time**: it waits for the card, verifies the PCIe capacitor mod, records PCIe bus health, backs up VBIOS, runs the cmpunlocker verification, offers the **opt-in** cmpunlocker unlock install (then a cold reboot and a re-run for the final PCIe check), and optionally stresses VRAM. Each card gets its own folder + JSON report named by serial, so the folders accumulate as you go through your batch (e.g. 23 cards). +Single-card checker for **NVIDIA CMP 170HX** cards (PCI IDs `10de:20c2` / `10de:2082`) — run it **one card at a time**: it waits for the card, verifies the PCIe capacitor mod, records PCIe bus health, backs up VBIOS, runs the cmpunlocker verification, offers the **opt-in** cmpunlocker unlock install (then a cold reboot and a re-run for the final PCIe check), and optionally stresses VRAM. Each card gets its own folder + JSON report named by GPU UUID, so the folders accumulate as you go through your batch (e.g. 23 cards). Reference harness (auto-cloned into `harness/` by preflight): @@ -84,13 +84,13 @@ python3 check_cards.py --help # all flags Flow of a normal run: 1. Waits for a CMP 170HX on the bus — prompts you to insert/reseat the card, Enter triggers a PCI rescan (`q` quits) -2. Serial auto-read from nvflash UUID; falls back to a manual label prompt -3. VBIOS backup → `cards//bios.rom` (nvflash, sysfs ROM fallback) +2. Folder `cards//` from the auto-read GPU UUID (`nvidia-smi`); physical serial probed from nvflash and saved to `serial.txt` (`N/A` when the card exposes none) — never prompts +3. VBIOS backup → `cards//bios.rom` (nvflash, sysfs ROM fallback) 4. PCIe checks: link speed/width, AER counters, full config-space dump 5. Link retrain test + full bus rescan test (device must survive both) 6. Kernel log scrape (`dmesg`/`journalctl`) for this BDF: Xid / AER / NVRM lines 7. `nvidia-smi` snapshot + `harness/cmpunlocker/verify.sh` (read-only) -8. **Opt-in unlock**: `cmpunlocker unlock (install)` — skipped with a note when memory is already unlocked, otherwise asks `Run cmpunlocker install.sh now? (y/N)` (default **n**; `--unlock` runs it without asking). On yes: `git pull --ff-only` on the harness clone, then `install.sh` runs from `harness/cmpunlocker/` with **live terminal output** (full log → `cards//install_unlocker.log`; the kernel-module build takes 5–15 min) +8. **Opt-in unlock**: `cmpunlocker unlock (install)` — skipped with a note when memory is already unlocked, otherwise asks `Run cmpunlocker install.sh now? (y/N)` (default **n**; `--unlock` runs it without asking). On yes: `git pull --ff-only` on the harness clone, then `install.sh` runs from `harness/cmpunlocker/` with **live terminal output** (full log → `cards//install_unlocker.log`; the kernel-module build takes 5–15 min) 9. **After a successful install**: the VRAM stress is skipped — **cold reboot the host** (`sudo shutdown -h now`, then power it back on) and **re-run the same command** for the final PCIe check, which must report **Gen2 x16** 10. **Last step, optional**: `Run 5-min Vulkan VRAM stress test now? (y/N)` — refused by default, `--vulkan always|never` to override 11. Result printed + `report.json` written into the card's folder @@ -111,7 +111,8 @@ Exit codes: `0` PASS/WARN · `1` FAIL · `2` no card found · `3` preflight bloc ``` cards/ -└── / +└── / + ├── serial.txt # physical serial, or N/A ├── bios.rom # VBIOS backup (+ sha256 in report) ├── report.json # full result: link, AER before/after, retrain, │ # rescan, dmesg, verify, vulkan, PASS/WARN/FAIL @@ -131,7 +132,7 @@ cards/ | `--preflight` | setup only: sudo precondition + package install + harness clone, then exit | | `--demo` | run the whole pipeline against fake hardware (self-test) | | `--nvflash PATH` | explicit nvflash binary | -| `--serial XYZ` | override serial (skips nvflash auto-read) | +| `--serial XYZ` | override physical serial recorded in `serial.txt` (folder stays UUID-named) | | `--vulkan ask\|always\|never` | VRAM test policy (default: ask, last step) | | `--unlock` | run the cmpunlocker `install.sh` step without prompting (default: ask y/N) | | `--skip-bios` | skip VBIOS backup | @@ -142,5 +143,5 @@ cards/ This tool **reads** card state by default. The one step that changes the card is **opt-in**: -- **Unlock install (opt-in, explicit confirmation)** — runs only if you answer `y` to `Run cmpunlocker install.sh now? (y/N)` (default `n`) or pass `--unlock`. It executes `install.sh` **from the harness clone** (`harness/cmpunlocker/`, refreshed first with `git pull --ff-only`, no extra installer flags), needs root (sudo credentials are cached first), streams its output live, and saves the full log to `cards//install_unlocker.log`. It is skipped entirely when memory is already unlocked, when the harness is not cloned, and in `--demo` mode. A successful install replaces kernel modules — the step then skips VRAM stress and asks you to cold reboot and re-run. +- **Unlock install (opt-in, explicit confirmation)** — runs only if you answer `y` to `Run cmpunlocker install.sh now? (y/N)` (default `n`) or pass `--unlock`. It executes `install.sh` **from the harness clone** (`harness/cmpunlocker/`, refreshed first with `git pull --ff-only`, no extra installer flags), needs root (sudo credentials are cached first), streams its output live, and saves the full log to `cards//install_unlocker.log`. It is skipped entirely when memory is already unlocked, when the harness is not cloned, and in `--demo` mode. A successful install replaces kernel modules — the step then skips VRAM stress and asks you to cold reboot and re-run. - **Everything else remains read-only**: `verify.sh` runs without installing anything, no modules are patched and no firmware is flashed (nvflash is only ever used with `--save`). The only other writes are local files (reports, logs), PCI link retrain/rescan sysfs triggers, and package installation during `--preflight` (explicitly gated behind the sudo precondition). diff --git a/check_cards.py b/check_cards.py index 6b92164..a9451fc 100755 --- a/check_cards.py +++ b/check_cards.py @@ -1,13 +1,13 @@ #!/usr/bin/env python3 """CMP 170HX single-card checker - ONE card per run, no queue. -Per run: wait for card -> auto serial -> VBIOS backup -> PCIe link/AER/config -checks -> retrain + rescan tests -> dmesg scrape -> cmpunlocker verify.sh -(read-only) -> opt-in cmpunlocker unlock (install.sh, y/N or --unlock; a -successful install skips VRAM stress and asks for a cold reboot, then you -re-run this script for the final Gen2 x16 PCIe check) +Per run: wait for card -> auto UUID (folder) + serial probe -> VBIOS backup +-> PCIe link/AER/config checks -> retrain + rescan tests -> dmesg scrape +-> cmpunlocker verify.sh (read-only) -> opt-in cmpunlocker unlock (install.sh, +y/N or --unlock; a successful install skips VRAM stress and asks for a cold +reboot, then you re-run this script for the final Gen2 x16 PCIe check) -> optional 5-min Vulkan VRAM stress (last step, ask yes/no) --> cards//report.json. +-> cards//report.json (+ serial.txt with the physical serial or N/A). Scoring: FAIL on no-device/link-down/lost-after-rescan, and on a final post-unlock link other than Gen2 x16; WARN on correctable AER/Xid/width< x16/ @@ -121,12 +121,19 @@ def process_card(args): ok(f"device found at {bdf} ({devid})") nf = X.nvflash_bin(args) - serial, serial_src = X.get_serial(nf, args) - folder = X.CARDS_DIR / sanitize(serial) + uuid = X.get_gpu_uuid() + if uuid: + folder = X.CARDS_DIR / sanitize(uuid) + else: + warn("GPU UUID unavailable - filing run under unidentified/") + folder = X.UNIDENTIFIED_DIR / sanitize(bdf) folder.mkdir(parents=True, exist_ok=True) + serial, serial_src = X.get_serial(nf, args) + (folder / "serial.txt").write_text((serial or "N/A") + "\n") info(f"folder: {folder}") rep = { + "uuid": uuid, "folder": str(folder), "serial": serial, "serial_source": serial_src, "started_at": now_iso(), "bdf": bdf, "device_id": devid, "host": X.run(["uname", "-n"])["out"].strip(), "kernel": X.run(["uname", "-r"])["out"].strip(), @@ -220,12 +227,13 @@ def process_card(args): (folder / "report.json").write_text(json.dumps(rep, indent=2)) print() + label = rep.get("uuid") or folder.name if result == "PASS": - ok(f"{serial}: PASS") + ok(f"{label}: PASS") elif result == "WARN": - warn(f"{serial}: WARN") + warn(f"{label}: WARN") else: - err(f"{serial}: FAIL") + err(f"{label}: FAIL") for f in fails: err(" FAIL: " + f) for w in warns: @@ -237,7 +245,7 @@ def process_card(args): # ---------------------------------------------------------------- demo def demo(args): - """Validate report pipeline without hardware: writes cards/DEMO-*.""" + """Validate report pipeline without hardware: writes cards/GPU-DEMO-UUID-0001.""" step("Demo mode (no hardware)") args.vulkan = "never" args.serial = "DEMO-SERIAL-001" @@ -245,12 +253,14 @@ def demo(args): orig_wait = globals()["wait_for_gpu"] orig_detect, orig_check = X.detect_gpus, X.check_link + orig_uuid = X.get_gpu_uuid orig_aer, orig_dm, orig_cfg = X.aer_snapshot, X.dmesg_capture, X.dump_config orig_retrain, orig_rescan = X.retrain_test, X.rescan_test orig_smi, orig_verify = X.nvidia_smi_info, X.run_verify_unlocker orig_update, orig_installer = X.update_cmpunlocker, X.run_installer globals()["wait_for_gpu"] = lambda: "0000:0a:00.0" X.detect_gpus = lambda: ["0000:0a:00.0"] + X.get_gpu_uuid = lambda: "GPU-DEMO-UUID-0001" X.check_link = lambda b: ({"lnksta_speed": "5GT/s", "lnksta_width": "x4", "lnksta_gen": 2, "lnkcap_speed": "8GT/s", "lnkcap_width": "x16", "lnksta_raw": "LnkSta: Speed 5GT/s, Width x4, TrErr-", @@ -274,6 +284,7 @@ def demo(args): finally: globals()["wait_for_gpu"] = orig_wait X.detect_gpus, X.check_link = orig_detect, orig_check + X.get_gpu_uuid = orig_uuid X.aer_snapshot, X.dmesg_capture, X.dump_config = orig_aer, orig_dm, orig_cfg X.retrain_test, X.rescan_test = orig_retrain, orig_rescan X.nvidia_smi_info, X.run_verify_unlocker = orig_smi, orig_verify @@ -289,7 +300,8 @@ def main(): help="tool check + package install (requires sudo) + harness clone, then exit") ap.add_argument("--demo", action="store_true", help="run pipeline with fake hardware (self-test)") ap.add_argument("--nvflash", help="path to nvflash binary") - ap.add_argument("--serial", help="override serial (skips nvflash auto-read)") + ap.add_argument("--serial", help="override physical serial recorded in serial.txt " + "(folder stays UUID-named)") ap.add_argument("--skip-bios", action="store_true", help="skip VBIOS backup") ap.add_argument("--vulkan", choices=("ask", "always", "never"), default="ask", help="5-min Vulkan VRAM test policy (default ask, runs as last step)") diff --git a/cxlib.py b/cxlib.py index 9066cef..45fd872 100755 --- a/cxlib.py +++ b/cxlib.py @@ -14,6 +14,7 @@ from pathlib import Path SCRIPT_DIR = Path(__file__).resolve().parent HARNESS_DIR = SCRIPT_DIR / "harness" CARDS_DIR = SCRIPT_DIR / "cards" +UNIDENTIFIED_DIR = SCRIPT_DIR / "unidentified" REPOS = { "170th-Street": "https://github.com/amoghmunikote/170th-Street.git", @@ -633,12 +634,7 @@ def get_serial(nf, args): warn("nvflash present but exposes no serial/UUID for this card") else: warn("nvflash unavailable - cannot auto-read serial") - serial = "" - while not serial: - serial = ask("Enter serial number from the card label (folder name)") - if not serial: - warn("serial is required for the folder name") - return serial, "manual-label" + return "N/A", "unavailable" def save_bios(nf, out_path, args): @@ -692,6 +688,20 @@ def nvidia_smi_info(): return {"available": True, "gpus": rows} +def get_gpu_uuid(): + """First GPU UUID from nvidia-smi (e.g. GPU-xxxx...), or None.""" + if not which("nvidia-smi"): + return None + q = run(["nvidia-smi", "--query-gpu=uuid", "--format=csv,noheader"], timeout=30) + if q["rc"] != 0: + return None + for line in q["out"].splitlines(): + line = line.strip() + if line.startswith("GPU-"): + return line + return None + + def run_verify_unlocker(): if not VERIFY_SH.exists(): return {"ran": False, "note": "harness/cmpunlocker not cloned"}