VBIOS backup: precise failure diagnosis plus opt-in unload retry

save_bios now distinguishes nvflash driver-loaded refusal from a
missing sysfs rom node (ROM BAR not exposed) vs enable denial,
and appends host lockdown/Secure Boot state to the note. New
opt-in last resort: unload NVIDIA modules, retry nvflash --save,
always reload (y/N prompt default n, or --bios-unload). Docs
(flow, flags, safety, AGENTS.md invariant) updated to match.

AGENTS.md

@@ -18,9 +18,9 @@ This repository checks one NVIDIA CMP 170HX at a time on a fresh Ubuntu 26.04 x8
 - Match every card-sensitive operation to the detected PCI address. A host may have another NVIDIA GPU even though only one CMP 170HX is being checked.
 - Measure capacitor-related lane width before unlocking. The width result is independent of cmpunlocker: Gen1 x16 passes the width check, while x4 fails it. Do not claim the capacitor is physically verified; the width is an observed link result on the current host.
 - Keep the final PCIe acceptance check after unlock. Preserve a red `[ERROR]` message and failing exit for any final link other than Gen2 x16. Gen1 x16 passes the earlier width check but fails the final card check.
-- Preserve cold-boot state by card identity. GPU UUID may be used internally for this purpose, but not as a folder name.
+- Preserve cold-boot state by card identity. The GPU UUID is the folder name and may be used internally for state tracking.
 - The cmpunlocker harness is cloned from upstream **main** into `harness/cmpunlocker/`, and `git pull --ff-only` runs before every install (a failed pull warns and continues with the existing checkout). Invoke `install.sh` only with flags the checked-out version actually accepts — currently **no flags**; never pass `--no-passthrough` to older checkouts without verifying them first.
-- VBIOS backup is optional and read-only. Select the detected PCI address for nvflash and sysfs ROM operations. Never add a flashing step to the normal workflow.
+- VBIOS backup is optional. Prefer nvflash `--save`, then the sysfs ROM fallback, and report the precise failure cause (driver-loaded vs missing rom node). A driver unload -> save -> reload retry is opt-in only (y/N prompt or `--bios-unload`). Never add a flashing step to the normal workflow.
 
 ## Working in this repository
 

README.md

@@ -85,7 +85,7 @@ Flow of a normal run:
 
 1. Waits for a CMP 170HX on the bus — prompts you to insert/reseat the card, Enter triggers a PCI rescan (`q` quits)
 2. Folder `cards/<GPU-UUID>/` from the auto-read GPU UUID (`nvidia-smi`); physical serial probed from nvflash and saved to `serial.txt` (`N/A` when the card exposes none) — never prompts
-3. VBIOS backup → `cards/<GPU-UUID>/bios.rom` (nvflash, sysfs ROM fallback)
+3. VBIOS backup → `cards/<GPU-UUID>/bios.rom` (nvflash, sysfs ROM fallback). On failure the script reports the precise cause (driver-loaded vs missing rom node) and offers an opt-in driver unload → save → reload retry (`Unload NVIDIA driver and retry nvflash --save now? (y/N)`, default **n**; `--bios-unload` runs it without asking)
 4. PCIe checks: link speed/width, AER counters, full config-space dump
 5. Link retrain test + full bus rescan test (device must survive both)
 6. Kernel log scrape (`dmesg`/`journalctl`) for this BDF: Xid / AER / NVRM lines
@@ -136,12 +136,14 @@ cards/
 | `--vulkan ask\|always\|never` | VRAM test policy (default: ask, last step) |
 | `--unlock` | run the cmpunlocker `install.sh` step without prompting (default: ask y/N) |
 | `--skip-bios` | skip VBIOS backup |
+| `--bios-unload` | unload NVIDIA driver to retry `nvflash --save` without prompting (console may freeze briefly, driver is reloaded after) |
 | `--no-install` | never install packages (skips the sudo precondition) |
 | `--no-clone` | don't clone the harness repos |
 
 ## Safety
 
-This tool **reads** card state by default. The one step that changes the card is **opt-in**:
+This tool **reads** card state by default. Two steps can change host state, both **opt-in**:
 
 - **Unlock install (opt-in, explicit confirmation)** — runs only if you answer `y` to `Run cmpunlocker install.sh now? (y/N)` (default `n`) or pass `--unlock`. It executes `install.sh` **from the harness clone** (`harness/cmpunlocker/`, refreshed first with `git pull --ff-only`, no extra installer flags), needs root (sudo credentials are cached first), streams its output live, and saves the full log to `cards/<GPU-UUID>/install_unlocker.log`. It is skipped entirely when memory is already unlocked, when the harness is not cloned, and in `--demo` mode. A successful install replaces kernel modules — the step then skips VRAM stress and asks you to cold reboot and re-run.
+- **VBIOS unload-retry (opt-in, explicit confirmation)** — offered only after both backup methods fail with the driver loaded: unloads the NVIDIA modules, retries `nvflash --save`, then reloads the modules (local console may freeze briefly). Triggered by `y` or `--bios-unload`; skipped in `--demo` mode.
 - **Everything else remains read-only**: `verify.sh` runs without installing anything, no modules are patched and no firmware is flashed (nvflash is only ever used with `--save`). The only other writes are local files (reports, logs), PCI link retrain/rescan sysfs triggers, and package installation during `--preflight` (explicitly gated behind the sudo precondition).

check_cards.py

@@ -2,6 +2,7 @@
 """CMP 170HX single-card checker - ONE card per run, no queue.
 
 Per run: wait for card -> auto UUID (folder) + serial probe -> VBIOS backup
+(--skip-bios, or --bios-unload to retry with the driver unloaded)
 -> PCIe link/AER/config checks -> retrain + rescan tests -> dmesg scrape
 -> cmpunlocker verify.sh (read-only) -> opt-in cmpunlocker unlock (install.sh,
 y/N or --unlock; a successful install skips VRAM stress and asks for a cold
@@ -303,6 +304,9 @@ def main():
     ap.add_argument("--serial", help="override physical serial recorded in serial.txt "
                     "(folder stays UUID-named)")
     ap.add_argument("--skip-bios", action="store_true", help="skip VBIOS backup")
+    ap.add_argument("--bios-unload", action="store_true",
+                    help="unload NVIDIA driver to retry nvflash --save without prompting "
+                    "(local console may freeze briefly, driver is reloaded after)")
     ap.add_argument("--vulkan", choices=("ask", "always", "never"), default="ask",
                     help="5-min Vulkan VRAM test policy (default ask, runs as last step)")
     ap.add_argument("--unlock", action="store_true",

cxlib.py

@@ -23,6 +23,7 @@ REPOS = {
 VERIFY_SH = HARNESS_DIR / "cmpunlocker" / "verify.sh"
 INSTALL_SH = HARNESS_DIR / "cmpunlocker" / "install.sh"
 GPU_IDS = ("20c2", "2082")
+NVIDIA_MODULES = ("nvidia_uvm", "nvidia_drm", "nvidia_modeset", "nvidia")
 VULKAN_TEST_SECONDS = 300
 
 SPEED_TO_GEN = {"2.5GT/s": 1, "5GT/s": 2, "8GT/s": 3, "16GT/s": 4, "32GT/s": 5, "64GT/s": 6}
@@ -642,25 +643,25 @@ def save_bios(nf, out_path, args):
     if getattr(args, "skip_bios", False):
         result["note"] = "bios backup skipped (--skip-bios)"
         return result
+    notes = []
     if nf:
-        r = as_root([nf, "--save", str(out_path)], timeout=90)
-        if r["rc"] == 0 and out_path.exists() and out_path.stat().st_size > 0:
+        ok_save, note = try_nvflash_save(nf, out_path)
+        if ok_save:
             data = out_path.read_bytes()
-            if data.strip(b"\x00") == b"":
-                result["note"] = "nvflash wrote an all-zero image"
-            else:
             result.update(method="nvflash", path=str(out_path), size=len(data),
                           sha256=hashlib.sha256(data).hexdigest(), ok=True)
             return result
+        notes.append(note)
     else:
-            result["note"] = f"nvflash --save failed rc={r['rc']}: {(r['err'] or r['out'])[-200:]}"
-    else:
-        result["note"] = "nvflash unavailable"
+        notes.append("nvflash unavailable")
 
     for bdf in detect_gpus():
         rom = sysfs(bdf) / "rom"
+        if not rom.exists():
+            notes.append(f"no sysfs rom node for {bdf} (ROM BAR not exposed - fallback unavailable)")
+            continue
         if as_root(["sh", "-c", f"echo 1 > {rom}"], timeout=15)["rc"] != 0:
-            result["note"] += " | sysfs rom enable failed"
+            notes.append(f"sysfs rom enable denied for {bdf}")
             continue
         data = read_file(rom, binary=True)
         as_root(["sh", "-c", f"echo 0 > {rom}"], timeout=15)
@@ -668,12 +669,102 @@ def save_bios(nf, out_path, args):
             out_path.write_bytes(data)
             result.update(method="sysfs-rom", path=str(out_path), size=len(data),
                           sha256=hashlib.sha256(data).hexdigest(), ok=True)
-            result["note"] = (result["note"] + " | saved via sysfs rom").strip(" |")
             return result
-        result["note"] += " | sysfs rom empty/all-zero"
+        notes.append(f"sysfs rom empty/all-zero for {bdf}")
+    if any("no sysfs rom node" in n for n in notes):
+        notes.append(f"host lockdown={lockdown_state()} secureboot={secure_boot_state()}")
+    result["note"] = " | ".join(notes)
+
+    # Opt-in last resort: unload the driver, retry nvflash --save, always reload.
+    if nf and any("driver loaded" in n for n in notes):
+        if getattr(args, "bios_unload", False) or (
+                sys.stdin.isatty()
+                and ask("Unload NVIDIA driver and retry nvflash --save now? (y/N)",
+                        "n").lower().startswith("y")):
+            result["note"] += " | " + retry_save_unloaded(nf, out_path, result)
+        else:
+            result["note"] += (" | skipped driver-unload retry (use --bios-unload; "
+                               "local console may freeze while the driver is out)")
     return result
 
 
+def try_nvflash_save(nf, out_path):
+    """One nvflash --save attempt. Returns (ok, note)."""
+    try:
+        if out_path.exists():
+            out_path.unlink()
+    except OSError:
+        pass
+    r = as_root([nf, "--save", str(out_path)], timeout=90)
+    text = r["out"] + r["err"]
+    if r["rc"] == 0 and out_path.exists() and out_path.stat().st_size > 0:
+        data = out_path.read_bytes()
+        if data.strip(b"\x00") == b"":
+            return False, "nvflash wrote an all-zero image"
+        return True, ""
+    if "unload" in text.lower() and "kernel driver" in text.lower():
+        return False, (f"nvflash --save refused: NVIDIA driver loaded (rc={r['rc']}) - "
+                       "unload it first or use --bios-unload")
+    return False, f"nvflash --save failed rc={r['rc']}: {text[-200:].strip()}"
+
+
+def retry_save_unloaded(nf, out_path, result):
+    """Unload NVIDIA modules, retry nvflash --save, always reload. Returns note."""
+    warn("unloading NVIDIA driver - local display may freeze until reload")
+    _, unload_log = unload_nvidia_modules()
+    try:
+        ok_save, note = try_nvflash_save(nf, out_path)
+        if ok_save:
+            data = out_path.read_bytes()
+            result.update(method="nvflash-unload", path=str(out_path), size=len(data),
+                          sha256=hashlib.sha256(data).hexdigest(), ok=True)
+            return "saved via nvflash after driver unload"
+        return f"nvflash --save still failed after unload: {note}"
+    finally:
+        ok_reload, reload_log = load_nvidia_modules()
+        for line in unload_log + reload_log:
+            info(line)
+        if ok_reload and which("nvidia-smi"):
+            ok("NVIDIA driver reloaded")
+        else:
+            err("NVIDIA driver reload incomplete - reboot the host")
+
+
+def unload_nvidia_modules():
+    """Best-effort rmmod of the NVIDIA stack. Returns (all_ok, log_lines)."""
+    log, all_ok = [], True
+    for mod in NVIDIA_MODULES:
+        r = as_root(["rmmod", mod], timeout=60)
+        log.append(f"rmmod {mod}: rc={r['rc']} {(r['err'] or r['out']).strip()[:160]}")
+        if r["rc"] != 0:
+            all_ok = False
+    return all_ok, log
+
+
+def load_nvidia_modules():
+    """Reload the NVIDIA stack in reverse order. Returns (all_ok, log_lines)."""
+    log, all_ok = [], True
+    for mod in reversed(NVIDIA_MODULES):
+        r = as_root(["modprobe", mod], timeout=120)
+        log.append(f"modprobe {mod}: rc={r['rc']} {(r['err'] or r['out']).strip()[:160]}")
+        if r["rc"] != 0:
+            all_ok = False
+    return all_ok, log
+
+
+def lockdown_state():
+    data = read_file("/sys/kernel/security/lockdown")
+    return data.strip() if data else "lockdown file unreadable"
+
+
+def secure_boot_state():
+    if not which("mokutil"):
+        return "mokutil not installed"
+    r = run(["mokutil", "--sb-state"], timeout=15)
+    lines = (r["out"] or r["err"]).strip().splitlines()
+    return lines[0].strip() if lines else "mokutil gave no output"
+
+
 # ---------------------------------------------------------------- nvidia / verify
 
 def nvidia_smi_info():