diff --git a/AGENTS.md b/AGENTS.md index 2b043fe..12e998b 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -18,9 +18,9 @@ This repository checks one NVIDIA CMP 170HX at a time on a fresh Ubuntu 26.04 x8 - Match every card-sensitive operation to the detected PCI address. A host may have another NVIDIA GPU even though only one CMP 170HX is being checked. - Measure capacitor-related lane width before unlocking. The width result is independent of cmpunlocker: Gen1 x16 passes the width check, while x4 fails it. Do not claim the capacitor is physically verified; the width is an observed link result on the current host. - Keep the final PCIe acceptance check after unlock. Preserve a red `[ERROR]` message and failing exit for any final link other than Gen2 x16. Gen1 x16 passes the earlier width check but fails the final card check. -- Preserve cold-boot state by card identity. GPU UUID may be used internally for this purpose, but not as a folder name. +- Preserve cold-boot state by card identity. The GPU UUID is the folder name and may be used internally for state tracking. - The cmpunlocker harness is cloned from upstream **main** into `harness/cmpunlocker/`, and `git pull --ff-only` runs before every install (a failed pull warns and continues with the existing checkout). Invoke `install.sh` only with flags the checked-out version actually accepts — currently **no flags**; never pass `--no-passthrough` to older checkouts without verifying them first. -- VBIOS backup is optional and read-only. Select the detected PCI address for nvflash and sysfs ROM operations. Never add a flashing step to the normal workflow. +- VBIOS backup is optional. Prefer nvflash `--save`, then the sysfs ROM fallback, and report the precise failure cause (driver-loaded vs missing rom node). A driver unload -> save -> reload retry is opt-in only (y/N prompt or `--bios-unload`). Never add a flashing step to the normal workflow. ## Working in this repository diff --git a/README.md b/README.md index 335df37..e94a061 100755 --- a/README.md +++ b/README.md @@ -85,7 +85,7 @@ Flow of a normal run: 1. Waits for a CMP 170HX on the bus — prompts you to insert/reseat the card, Enter triggers a PCI rescan (`q` quits) 2. Folder `cards//` from the auto-read GPU UUID (`nvidia-smi`); physical serial probed from nvflash and saved to `serial.txt` (`N/A` when the card exposes none) — never prompts -3. VBIOS backup → `cards//bios.rom` (nvflash, sysfs ROM fallback) +3. VBIOS backup → `cards//bios.rom` (nvflash, sysfs ROM fallback). On failure the script reports the precise cause (driver-loaded vs missing rom node) and offers an opt-in driver unload → save → reload retry (`Unload NVIDIA driver and retry nvflash --save now? (y/N)`, default **n**; `--bios-unload` runs it without asking) 4. PCIe checks: link speed/width, AER counters, full config-space dump 5. Link retrain test + full bus rescan test (device must survive both) 6. Kernel log scrape (`dmesg`/`journalctl`) for this BDF: Xid / AER / NVRM lines @@ -136,12 +136,14 @@ cards/ | `--vulkan ask\|always\|never` | VRAM test policy (default: ask, last step) | | `--unlock` | run the cmpunlocker `install.sh` step without prompting (default: ask y/N) | | `--skip-bios` | skip VBIOS backup | +| `--bios-unload` | unload NVIDIA driver to retry `nvflash --save` without prompting (console may freeze briefly, driver is reloaded after) | | `--no-install` | never install packages (skips the sudo precondition) | | `--no-clone` | don't clone the harness repos | ## Safety -This tool **reads** card state by default. The one step that changes the card is **opt-in**: +This tool **reads** card state by default. Two steps can change host state, both **opt-in**: - **Unlock install (opt-in, explicit confirmation)** — runs only if you answer `y` to `Run cmpunlocker install.sh now? (y/N)` (default `n`) or pass `--unlock`. It executes `install.sh` **from the harness clone** (`harness/cmpunlocker/`, refreshed first with `git pull --ff-only`, no extra installer flags), needs root (sudo credentials are cached first), streams its output live, and saves the full log to `cards//install_unlocker.log`. It is skipped entirely when memory is already unlocked, when the harness is not cloned, and in `--demo` mode. A successful install replaces kernel modules — the step then skips VRAM stress and asks you to cold reboot and re-run. +- **VBIOS unload-retry (opt-in, explicit confirmation)** — offered only after both backup methods fail with the driver loaded: unloads the NVIDIA modules, retries `nvflash --save`, then reloads the modules (local console may freeze briefly). Triggered by `y` or `--bios-unload`; skipped in `--demo` mode. - **Everything else remains read-only**: `verify.sh` runs without installing anything, no modules are patched and no firmware is flashed (nvflash is only ever used with `--save`). The only other writes are local files (reports, logs), PCI link retrain/rescan sysfs triggers, and package installation during `--preflight` (explicitly gated behind the sudo precondition). diff --git a/check_cards.py b/check_cards.py index a9451fc..741af23 100755 --- a/check_cards.py +++ b/check_cards.py @@ -2,6 +2,7 @@ """CMP 170HX single-card checker - ONE card per run, no queue. Per run: wait for card -> auto UUID (folder) + serial probe -> VBIOS backup +(--skip-bios, or --bios-unload to retry with the driver unloaded) -> PCIe link/AER/config checks -> retrain + rescan tests -> dmesg scrape -> cmpunlocker verify.sh (read-only) -> opt-in cmpunlocker unlock (install.sh, y/N or --unlock; a successful install skips VRAM stress and asks for a cold @@ -303,6 +304,9 @@ def main(): ap.add_argument("--serial", help="override physical serial recorded in serial.txt " "(folder stays UUID-named)") ap.add_argument("--skip-bios", action="store_true", help="skip VBIOS backup") + ap.add_argument("--bios-unload", action="store_true", + help="unload NVIDIA driver to retry nvflash --save without prompting " + "(local console may freeze briefly, driver is reloaded after)") ap.add_argument("--vulkan", choices=("ask", "always", "never"), default="ask", help="5-min Vulkan VRAM test policy (default ask, runs as last step)") ap.add_argument("--unlock", action="store_true", diff --git a/cxlib.py b/cxlib.py index 45fd872..6f685b1 100755 --- a/cxlib.py +++ b/cxlib.py @@ -23,6 +23,7 @@ REPOS = { VERIFY_SH = HARNESS_DIR / "cmpunlocker" / "verify.sh" INSTALL_SH = HARNESS_DIR / "cmpunlocker" / "install.sh" GPU_IDS = ("20c2", "2082") +NVIDIA_MODULES = ("nvidia_uvm", "nvidia_drm", "nvidia_modeset", "nvidia") VULKAN_TEST_SECONDS = 300 SPEED_TO_GEN = {"2.5GT/s": 1, "5GT/s": 2, "8GT/s": 3, "16GT/s": 4, "32GT/s": 5, "64GT/s": 6} @@ -642,25 +643,25 @@ def save_bios(nf, out_path, args): if getattr(args, "skip_bios", False): result["note"] = "bios backup skipped (--skip-bios)" return result + notes = [] if nf: - r = as_root([nf, "--save", str(out_path)], timeout=90) - if r["rc"] == 0 and out_path.exists() and out_path.stat().st_size > 0: + ok_save, note = try_nvflash_save(nf, out_path) + if ok_save: data = out_path.read_bytes() - if data.strip(b"\x00") == b"": - result["note"] = "nvflash wrote an all-zero image" - else: - result.update(method="nvflash", path=str(out_path), size=len(data), - sha256=hashlib.sha256(data).hexdigest(), ok=True) - return result - else: - result["note"] = f"nvflash --save failed rc={r['rc']}: {(r['err'] or r['out'])[-200:]}" + result.update(method="nvflash", path=str(out_path), size=len(data), + sha256=hashlib.sha256(data).hexdigest(), ok=True) + return result + notes.append(note) else: - result["note"] = "nvflash unavailable" + notes.append("nvflash unavailable") for bdf in detect_gpus(): rom = sysfs(bdf) / "rom" + if not rom.exists(): + notes.append(f"no sysfs rom node for {bdf} (ROM BAR not exposed - fallback unavailable)") + continue if as_root(["sh", "-c", f"echo 1 > {rom}"], timeout=15)["rc"] != 0: - result["note"] += " | sysfs rom enable failed" + notes.append(f"sysfs rom enable denied for {bdf}") continue data = read_file(rom, binary=True) as_root(["sh", "-c", f"echo 0 > {rom}"], timeout=15) @@ -668,12 +669,102 @@ def save_bios(nf, out_path, args): out_path.write_bytes(data) result.update(method="sysfs-rom", path=str(out_path), size=len(data), sha256=hashlib.sha256(data).hexdigest(), ok=True) - result["note"] = (result["note"] + " | saved via sysfs rom").strip(" |") return result - result["note"] += " | sysfs rom empty/all-zero" + notes.append(f"sysfs rom empty/all-zero for {bdf}") + if any("no sysfs rom node" in n for n in notes): + notes.append(f"host lockdown={lockdown_state()} secureboot={secure_boot_state()}") + result["note"] = " | ".join(notes) + + # Opt-in last resort: unload the driver, retry nvflash --save, always reload. + if nf and any("driver loaded" in n for n in notes): + if getattr(args, "bios_unload", False) or ( + sys.stdin.isatty() + and ask("Unload NVIDIA driver and retry nvflash --save now? (y/N)", + "n").lower().startswith("y")): + result["note"] += " | " + retry_save_unloaded(nf, out_path, result) + else: + result["note"] += (" | skipped driver-unload retry (use --bios-unload; " + "local console may freeze while the driver is out)") return result +def try_nvflash_save(nf, out_path): + """One nvflash --save attempt. Returns (ok, note).""" + try: + if out_path.exists(): + out_path.unlink() + except OSError: + pass + r = as_root([nf, "--save", str(out_path)], timeout=90) + text = r["out"] + r["err"] + if r["rc"] == 0 and out_path.exists() and out_path.stat().st_size > 0: + data = out_path.read_bytes() + if data.strip(b"\x00") == b"": + return False, "nvflash wrote an all-zero image" + return True, "" + if "unload" in text.lower() and "kernel driver" in text.lower(): + return False, (f"nvflash --save refused: NVIDIA driver loaded (rc={r['rc']}) - " + "unload it first or use --bios-unload") + return False, f"nvflash --save failed rc={r['rc']}: {text[-200:].strip()}" + + +def retry_save_unloaded(nf, out_path, result): + """Unload NVIDIA modules, retry nvflash --save, always reload. Returns note.""" + warn("unloading NVIDIA driver - local display may freeze until reload") + _, unload_log = unload_nvidia_modules() + try: + ok_save, note = try_nvflash_save(nf, out_path) + if ok_save: + data = out_path.read_bytes() + result.update(method="nvflash-unload", path=str(out_path), size=len(data), + sha256=hashlib.sha256(data).hexdigest(), ok=True) + return "saved via nvflash after driver unload" + return f"nvflash --save still failed after unload: {note}" + finally: + ok_reload, reload_log = load_nvidia_modules() + for line in unload_log + reload_log: + info(line) + if ok_reload and which("nvidia-smi"): + ok("NVIDIA driver reloaded") + else: + err("NVIDIA driver reload incomplete - reboot the host") + + +def unload_nvidia_modules(): + """Best-effort rmmod of the NVIDIA stack. Returns (all_ok, log_lines).""" + log, all_ok = [], True + for mod in NVIDIA_MODULES: + r = as_root(["rmmod", mod], timeout=60) + log.append(f"rmmod {mod}: rc={r['rc']} {(r['err'] or r['out']).strip()[:160]}") + if r["rc"] != 0: + all_ok = False + return all_ok, log + + +def load_nvidia_modules(): + """Reload the NVIDIA stack in reverse order. Returns (all_ok, log_lines).""" + log, all_ok = [], True + for mod in reversed(NVIDIA_MODULES): + r = as_root(["modprobe", mod], timeout=120) + log.append(f"modprobe {mod}: rc={r['rc']} {(r['err'] or r['out']).strip()[:160]}") + if r["rc"] != 0: + all_ok = False + return all_ok, log + + +def lockdown_state(): + data = read_file("/sys/kernel/security/lockdown") + return data.strip() if data else "lockdown file unreadable" + + +def secure_boot_state(): + if not which("mokutil"): + return "mokutil not installed" + r = run(["mokutil", "--sb-state"], timeout=15) + lines = (r["out"] or r["err"]).strip().splitlines() + return lines[0].strip() if lines else "mokutil gave no output" + + # ---------------------------------------------------------------- nvidia / verify def nvidia_smi_info():