Add opt-in cmpunlocker unlock step with post-unlock Gen2 x16 acceptance
Insert a "cmpunlocker unlock (install)" step after unlock verification: y/N confirmation (default n) or --unlock runs install.sh from the harness clone (main branch, ff-only pull first) with live terminal output and full log to cards/<serial>/install_unlocker.log. On success the Vulkan stress is skipped until after the cold reboot. Score now enforces the post-unlock invariant: when verify reports unlocked-ok, any final link other than Gen2 x16 fails with a red [ERROR] and exit 1. README safety section and AGENTS.md harness cloning invariant updated to match.
This commit is contained in:
@@ -7,7 +7,7 @@ This repository checks one NVIDIA CMP 170HX at a time on a fresh Ubuntu 26.04 x8
|
|||||||
## Source and outputs
|
## Source and outputs
|
||||||
|
|
||||||
- `check_cards.py` owns the CLI, per-card flow, report scoring, and terminal result.
|
- `check_cards.py` owns the CLI, per-card flow, report scoring, and terminal result.
|
||||||
- `cxlib.py` owns host setup, pinned cmpunlocker download, card identity, subprocess logging, PCIe inspection, and optional VBIOS backup.
|
- `cxlib.py` owns host setup, the cmpunlocker harness clone/update and opt-in installer, card identity, subprocess logging, PCIe inspection, and optional VBIOS backup.
|
||||||
- `README.md` is the operator guide. Keep its flags, output paths, reboot steps, and exit codes aligned with the code.
|
- `README.md` is the operator guide. Keep its flags, output paths, reboot steps, and exit codes aligned with the code.
|
||||||
- `harness/`, `cards/`, and `unidentified/` contain generated source, logs, reports, or firmware backups. Do not commit their contents.
|
- `harness/`, `cards/`, and `unidentified/` contain generated source, logs, reports, or firmware backups. Do not commit their contents.
|
||||||
- The bundled `nvflash_5.867_linux/` contains the optional backup tool. Backups belong under `extras/`, not at repository root.
|
- The bundled `nvflash_5.867_linux/` contains the optional backup tool. Backups belong under `extras/`, not at repository root.
|
||||||
@@ -19,7 +19,7 @@ This repository checks one NVIDIA CMP 170HX at a time on a fresh Ubuntu 26.04 x8
|
|||||||
- Measure capacitor-related lane width before unlocking. The width result is independent of cmpunlocker: Gen1 x16 passes the width check, while x4 fails it. Do not claim the capacitor is physically verified; the width is an observed link result on the current host.
|
- Measure capacitor-related lane width before unlocking. The width result is independent of cmpunlocker: Gen1 x16 passes the width check, while x4 fails it. Do not claim the capacitor is physically verified; the width is an observed link result on the current host.
|
||||||
- Keep the final PCIe acceptance check after unlock. Preserve a red `[ERROR]` message and failing exit for any final link other than Gen2 x16. Gen1 x16 passes the earlier width check but fails the final card check.
|
- Keep the final PCIe acceptance check after unlock. Preserve a red `[ERROR]` message and failing exit for any final link other than Gen2 x16. Gen1 x16 passes the earlier width check but fails the final card check.
|
||||||
- Preserve cold-boot state by card identity. GPU UUID may be used internally for this purpose, but not as a folder name.
|
- Preserve cold-boot state by card identity. GPU UUID may be used internally for this purpose, but not as a folder name.
|
||||||
- The cmpunlocker archive is pinned by commit and SHA256 in `cxlib.py`. Check a pinned version's actual accepted installer flags before changing invocation. The currently pinned v0.3 installer does not accept `--no-passthrough`.
|
- The cmpunlocker harness is cloned from upstream **main** into `harness/cmpunlocker/`, and `git pull --ff-only` runs before every install (a failed pull warns and continues with the existing checkout). Invoke `install.sh` only with flags the checked-out version actually accepts — currently **no flags**; never pass `--no-passthrough` to older checkouts without verifying them first.
|
||||||
- VBIOS backup is optional and read-only. Select the detected PCI address for nvflash and sysfs ROM operations. Never add a flashing step to the normal workflow.
|
- VBIOS backup is optional and read-only. Select the detected PCI address for nvflash and sysfs ROM operations. Never add a flashing step to the normal workflow.
|
||||||
|
|
||||||
## Working in this repository
|
## Working in this repository
|
||||||
|
|||||||
@@ -1,16 +1,16 @@
|
|||||||
# cmp170hx_check
|
# cmp170hx_check
|
||||||
|
|
||||||
Single-card checker for **NVIDIA CMP 170HX** cards (PCI IDs `10de:20c2` / `10de:2082`) — run it **one card at a time**: it waits for the card, verifies the PCIe capacitor mod, records PCIe bus health, backs up VBIOS, runs the cmpunlocker verification, and optionally stresses VRAM. Each card gets its own folder + JSON report named by serial, so the folders accumulate as you go through your batch (e.g. 23 cards).
|
Single-card checker for **NVIDIA CMP 170HX** cards (PCI IDs `10de:20c2` / `10de:2082`) — run it **one card at a time**: it waits for the card, verifies the PCIe capacitor mod, records PCIe bus health, backs up VBIOS, runs the cmpunlocker verification, offers the **opt-in** cmpunlocker unlock install (then a cold reboot and a re-run for the final PCIe check), and optionally stresses VRAM. Each card gets its own folder + JSON report named by serial, so the folders accumulate as you go through your batch (e.g. 23 cards).
|
||||||
|
|
||||||
Reference harness (auto-cloned into `harness/` by preflight):
|
Reference harness (auto-cloned into `harness/` by preflight):
|
||||||
|
|
||||||
- [170th-Street](https://github.com/amoghmunikote/170th-Street) — community docs, incl. `modifications/pcie-capacitor-mod.md` (the pass criteria: `LnkSta Width x16` after the mod, `x4` when missing)
|
- [170th-Street](https://github.com/amoghmunikote/170th-Street) — community docs, incl. `modifications/pcie-capacitor-mod.md` (the pass criteria: `LnkSta Width x16` after the mod, `x4` when missing)
|
||||||
- [cmpunlocker](https://github.com/amoghmunikote/cmpunlocker) — `verify.sh` is executed **read-only** per card (nothing is installed or patched by this tool)
|
- [cmpunlocker](https://github.com/amoghmunikote/cmpunlocker) — `verify.sh` is executed **read-only** per card; `install.sh` is only run from the separate, opt-in unlock step (see [Safety](#safety))
|
||||||
|
|
||||||
## Requirements
|
## Requirements
|
||||||
|
|
||||||
- Linux x86-64 (tested target: **Ubuntu 26.04**), Python 3.8+
|
- Linux x86-64 (tested target: **Ubuntu 26.04**), Python 3.8+
|
||||||
- **sudo privileges** — precondition for `--preflight` package installation and for privileged checks (PCI rescan, link retrain, sysfs ROM, `dmesg`, `verify.sh`)
|
- **sudo privileges** — precondition for `--preflight` package installation, for privileged checks (PCI rescan, link retrain, sysfs ROM, `dmesg`, `verify.sh`), and for the opt-in unlock install
|
||||||
- `pciutils` (`lspci`, `setpci`), `git` — installed automatically by preflight
|
- `pciutils` (`lspci`, `setpci`), `git` — installed automatically by preflight
|
||||||
- NVIDIA driver with `nvidia-smi` (nvidia-open 610.xx+, see cmpunlocker) for unlock verification — preflight can install it for you (asks first)
|
- NVIDIA driver with `nvidia-smi` (nvidia-open 610.xx+, see cmpunlocker) for unlock verification — preflight can install it for you (asks first)
|
||||||
- `nvflash` — **bundled in this repo** (`nvflash_5.867_linux/x64/nvflash`), used directly by the script; no system install needed (see [nvflash](#nvflash-bundled-in-this-repo))
|
- `nvflash` — **bundled in this repo** (`nvflash_5.867_linux/x64/nvflash`), used directly by the script; no system install needed (see [nvflash](#nvflash-bundled-in-this-repo))
|
||||||
@@ -75,6 +75,7 @@ the files (keep the executable bit: `chmod +x`).
|
|||||||
python3 check_cards.py # check the connected card
|
python3 check_cards.py # check the connected card
|
||||||
python3 check_cards.py --preflight # setup: tools, packages, harness clone
|
python3 check_cards.py --preflight # setup: tools, packages, harness clone
|
||||||
python3 check_cards.py --demo # self-test with fake hardware (no GPU needed)
|
python3 check_cards.py --demo # self-test with fake hardware (no GPU needed)
|
||||||
|
python3 check_cards.py --unlock # check + run the cmpunlocker install without asking
|
||||||
python3 check_cards.py --help # all flags
|
python3 check_cards.py --help # all flags
|
||||||
```
|
```
|
||||||
|
|
||||||
@@ -87,8 +88,10 @@ Flow of a normal run:
|
|||||||
5. Link retrain test + full bus rescan test (device must survive both)
|
5. Link retrain test + full bus rescan test (device must survive both)
|
||||||
6. Kernel log scrape (`dmesg`/`journalctl`) for this BDF: Xid / AER / NVRM lines
|
6. Kernel log scrape (`dmesg`/`journalctl`) for this BDF: Xid / AER / NVRM lines
|
||||||
7. `nvidia-smi` snapshot + `harness/cmpunlocker/verify.sh` (read-only)
|
7. `nvidia-smi` snapshot + `harness/cmpunlocker/verify.sh` (read-only)
|
||||||
8. **Last step, optional**: `Run 5-min Vulkan VRAM stress test now? (y/N)` — refused by default, `--vulkan always|never` to override
|
8. **Opt-in unlock**: `cmpunlocker unlock (install)` — skipped with a note when memory is already unlocked, otherwise asks `Run cmpunlocker install.sh now? (y/N)` (default **n**; `--unlock` runs it without asking). On yes: `git pull --ff-only` on the harness clone, then `install.sh` runs from `harness/cmpunlocker/` with **live terminal output** (full log → `cards/<serial>/install_unlocker.log`; the kernel-module build takes 5–15 min)
|
||||||
9. Result printed + `report.json` written into the card's folder
|
9. **After a successful install**: the VRAM stress is skipped — **cold reboot the host** (`sudo shutdown -h now`, then power it back on) and **re-run the same command** for the final PCIe check, which must report **Gen2 x16**
|
||||||
|
10. **Last step, optional**: `Run 5-min Vulkan VRAM stress test now? (y/N)` — refused by default, `--vulkan always|never` to override
|
||||||
|
11. Result printed + `report.json` written into the card's folder
|
||||||
|
|
||||||
Remove the card, plug the next one, repeat — no queue, no indices, folders just pile up in `cards/`.
|
Remove the card, plug the next one, repeat — no queue, no indices, folders just pile up in `cards/`.
|
||||||
|
|
||||||
@@ -96,8 +99,8 @@ Remove the card, plug the next one, repeat — no queue, no indices, folders jus
|
|||||||
|
|
||||||
| Result | Criteria |
|
| Result | Criteria |
|
||||||
|--------|----------|
|
|--------|----------|
|
||||||
| **FAIL** | no device detected, link down/untrained, device lost after rescan, link failed retrain, active uncorrectable AER errors |
|
| **FAIL** | no device detected, link down/untrained, device lost after rescan, link failed retrain, active uncorrectable AER errors, **final post-unlock link other than Gen2 x16** (`[ERROR] final PCIe link after unlock is <speed> <width> - expected Gen2 x16`) |
|
||||||
| **WARN** | link width ≠ `x16` (capacitor mod missing/incomplete), Gen1 only (Gen2 unlock patch not active), correctable AER errors, kernel errors / Xid, VBIOS backup failed, cmpunlocker verify not `unlocked-ok`, errors during VRAM stress |
|
| **WARN** | link width ≠ `x16` (capacitor mod missing/incomplete, pre-unlock runs only), Gen1 only (Gen2 unlock patch not active, pre-unlock runs only), correctable AER errors, kernel errors / Xid, VBIOS backup failed, cmpunlocker verify not `unlocked-ok`, cmpunlocker install failed, errors during VRAM stress |
|
||||||
| **PASS** | everything clean |
|
| **PASS** | everything clean |
|
||||||
|
|
||||||
Exit codes: `0` PASS/WARN · `1` FAIL · `2` no card found · `3` preflight blocking failure.
|
Exit codes: `0` PASS/WARN · `1` FAIL · `2` no card found · `3` preflight blocking failure.
|
||||||
@@ -114,7 +117,9 @@ cards/
|
|||||||
├── lspci_after.txt # lspci -vvv (after retrain/rescan)
|
├── lspci_after.txt # lspci -vvv (after retrain/rescan)
|
||||||
├── config_space.hex # lspci -nnvvxxxx full hex dump
|
├── config_space.hex # lspci -nnvvxxxx full hex dump
|
||||||
├── dmesg.log # kernel lines related to this card
|
├── dmesg.log # kernel lines related to this card
|
||||||
└── verify_unlocker.log # cmpunlocker verify.sh output
|
├── verify_unlocker.log # cmpunlocker verify.sh output
|
||||||
|
└── install_unlocker.log # full install.sh output (only if the opt-in
|
||||||
|
# unlock step ran)
|
||||||
```
|
```
|
||||||
|
|
||||||
## Flags
|
## Flags
|
||||||
@@ -126,10 +131,14 @@ cards/
|
|||||||
| `--nvflash PATH` | explicit nvflash binary |
|
| `--nvflash PATH` | explicit nvflash binary |
|
||||||
| `--serial XYZ` | override serial (skips nvflash auto-read) |
|
| `--serial XYZ` | override serial (skips nvflash auto-read) |
|
||||||
| `--vulkan ask\|always\|never` | VRAM test policy (default: ask, last step) |
|
| `--vulkan ask\|always\|never` | VRAM test policy (default: ask, last step) |
|
||||||
|
| `--unlock` | run the cmpunlocker `install.sh` step without prompting (default: ask y/N) |
|
||||||
| `--skip-bios` | skip VBIOS backup |
|
| `--skip-bios` | skip VBIOS backup |
|
||||||
| `--no-install` | never install packages (skips the sudo precondition) |
|
| `--no-install` | never install packages (skips the sudo precondition) |
|
||||||
| `--no-clone` | don't clone the harness repos |
|
| `--no-clone` | don't clone the harness repos |
|
||||||
|
|
||||||
## Safety
|
## Safety
|
||||||
|
|
||||||
This tool only **reads** card state: `verify.sh` is run without installing anything, no modules are patched, no firmware is flashed. The only writes are local files, PCI link retrain/rescan sysfs triggers, package installation during `--preflight` (explicitly gated behind the sudo precondition), and your own reports.
|
This tool **reads** card state by default. The one step that changes the card is **opt-in**:
|
||||||
|
|
||||||
|
- **Unlock install (opt-in, explicit confirmation)** — runs only if you answer `y` to `Run cmpunlocker install.sh now? (y/N)` (default `n`) or pass `--unlock`. It executes `install.sh` **from the harness clone** (`harness/cmpunlocker/`, refreshed first with `git pull --ff-only`, no extra installer flags), needs root (sudo credentials are cached first), streams its output live, and saves the full log to `cards/<serial>/install_unlocker.log`. It is skipped entirely when memory is already unlocked, when the harness is not cloned, and in `--demo` mode. A successful install replaces kernel modules — the step then skips VRAM stress and asks you to cold reboot and re-run.
|
||||||
|
- **Everything else remains read-only**: `verify.sh` runs without installing anything, no modules are patched and no firmware is flashed (nvflash is only ever used with `--save`). The only other writes are local files (reports, logs), PCI link retrain/rescan sysfs triggers, and package installation during `--preflight` (explicitly gated behind the sudo precondition).
|
||||||
|
|||||||
+60
-6
@@ -3,11 +3,15 @@
|
|||||||
|
|
||||||
Per run: wait for card -> auto serial -> VBIOS backup -> PCIe link/AER/config
|
Per run: wait for card -> auto serial -> VBIOS backup -> PCIe link/AER/config
|
||||||
checks -> retrain + rescan tests -> dmesg scrape -> cmpunlocker verify.sh
|
checks -> retrain + rescan tests -> dmesg scrape -> cmpunlocker verify.sh
|
||||||
(read-only) -> optional 5-min Vulkan VRAM stress (last step, ask yes/no)
|
(read-only) -> opt-in cmpunlocker unlock (install.sh, y/N or --unlock; a
|
||||||
|
successful install skips VRAM stress and asks for a cold reboot, then you
|
||||||
|
re-run this script for the final Gen2 x16 PCIe check)
|
||||||
|
-> optional 5-min Vulkan VRAM stress (last step, ask yes/no)
|
||||||
-> cards/<serial>/report.json.
|
-> cards/<serial>/report.json.
|
||||||
|
|
||||||
Scoring (lenient): FAIL only on no-device/link-down/lost-after-rescan;
|
Scoring: FAIL on no-device/link-down/lost-after-rescan, and on a final
|
||||||
WARN on correctable AER/Xid/width< x16/verify issues.
|
post-unlock link other than Gen2 x16; WARN on correctable AER/Xid/width< x16/
|
||||||
|
verify issues/failed unlock install.
|
||||||
|
|
||||||
Exit codes: 0 = PASS/WARN, 1 = FAIL, 2 = no card, 3 = preflight blocking.
|
Exit codes: 0 = PASS/WARN, 1 = FAIL, 2 = no card, 3 = preflight blocking.
|
||||||
"""
|
"""
|
||||||
@@ -24,6 +28,8 @@ from cxlib import C, ask, detect_gpus, err, info, now_iso, ok, sanitize, step, w
|
|||||||
|
|
||||||
def score(rep):
|
def score(rep):
|
||||||
fails, warns = [], []
|
fails, warns = [], []
|
||||||
|
ver = rep.get("cmpunlocker_verify", {})
|
||||||
|
unlocked = ver.get("status") == "unlocked-ok"
|
||||||
if not rep.get("bdf"):
|
if not rep.get("bdf"):
|
||||||
fails.append("no CMP 170HX device detected on the bus")
|
fails.append("no CMP 170HX device detected on the bus")
|
||||||
link = rep.get("link", {})
|
link = rep.get("link", {})
|
||||||
@@ -33,10 +39,19 @@ def score(rep):
|
|||||||
if link.get("lnksta_width") in ("x0", "x1", None) or not link.get("lnksta_speed"):
|
if link.get("lnksta_width") in ("x0", "x1", None) or not link.get("lnksta_speed"):
|
||||||
fails.append(f"link down or untrained: {link.get('lnksta_raw') or 'no LnkSta'}")
|
fails.append(f"link down or untrained: {link.get('lnksta_raw') or 'no LnkSta'}")
|
||||||
width = link.get("lnksta_width")
|
width = link.get("lnksta_width")
|
||||||
|
gen = link.get("lnksta_gen")
|
||||||
|
if unlocked:
|
||||||
|
# Post-unlock final acceptance: only Gen2 x16 passes. The pre-unlock
|
||||||
|
# width/Gen1 warns are suppressed here - the error replaces them.
|
||||||
|
if width != "x16" or gen is None or gen < 2:
|
||||||
|
fails.append(f"[ERROR] final PCIe link after unlock is "
|
||||||
|
f"{link.get('lnksta_speed') or '?'} {width or '?'} "
|
||||||
|
f"- expected Gen2 x16")
|
||||||
|
else:
|
||||||
if width and width != "x16":
|
if width and width != "x16":
|
||||||
warns.append(f"link width {width} (capacitor mod expects x16 - mod likely "
|
warns.append(f"link width {width} (capacitor mod expects x16 - mod likely "
|
||||||
f"missing/incomplete on this card)")
|
f"missing/incomplete on this card)")
|
||||||
if link.get("lnksta_gen") == 1:
|
if gen == 1:
|
||||||
warns.append("link negotiated Gen1 only (cmpunlocker Gen2 patch/service not active?)")
|
warns.append("link negotiated Gen1 only (cmpunlocker Gen2 patch/service not active?)")
|
||||||
if rep.get("rescan", {}).get("ok") is False:
|
if rep.get("rescan", {}).get("ok") is False:
|
||||||
fails.append("device lost after bus rescan")
|
fails.append("device lost after bus rescan")
|
||||||
@@ -60,9 +75,11 @@ def score(rep):
|
|||||||
warns.append("kernel errors around VRAM stress window")
|
warns.append("kernel errors around VRAM stress window")
|
||||||
if not rep.get("bios", {}).get("ok"):
|
if not rep.get("bios", {}).get("ok"):
|
||||||
warns.append("VBIOS backup failed/incomplete")
|
warns.append("VBIOS backup failed/incomplete")
|
||||||
ver = rep.get("cmpunlocker_verify", {})
|
|
||||||
if ver.get("ran") and ver.get("status") not in ("unlocked-ok",):
|
if ver.get("ran") and ver.get("status") not in ("unlocked-ok",):
|
||||||
warns.append(f"cmpunlocker verify status: {ver.get('status')}")
|
warns.append(f"cmpunlocker verify status: {ver.get('status')}")
|
||||||
|
unlock = rep.get("unlock", {})
|
||||||
|
if unlock.get("attempted") and not unlock.get("ok"):
|
||||||
|
warns.append("cmpunlocker install failed (see install_unlocker.log)")
|
||||||
|
|
||||||
result = "FAIL" if fails else ("WARN" if warns else "PASS")
|
result = "FAIL" if fails else ("WARN" if warns else "PASS")
|
||||||
return result, fails, warns
|
return result, fails, warns
|
||||||
@@ -156,8 +173,38 @@ def process_card(args):
|
|||||||
info("verify: " + (f"rc={v.get('rc')} -> {v.get('status')}" if v.get("ran")
|
info("verify: " + (f"rc={v.get('rc')} -> {v.get('status')}" if v.get("ran")
|
||||||
else f"skipped - {v.get('note')}"))
|
else f"skipped - {v.get('note')}"))
|
||||||
|
|
||||||
|
step("cmpunlocker unlock (install)")
|
||||||
|
unlock_installed = False
|
||||||
|
if rep["cmpunlocker_verify"].get("status") == "unlocked-ok":
|
||||||
|
info("memory already unlocked - skipping install")
|
||||||
|
rep["unlock"] = {"attempted": False, "ok": False, "note": "already unlocked"}
|
||||||
|
elif getattr(args, "demo", False):
|
||||||
|
# --demo never prompts: force the unlock install path off.
|
||||||
|
rep["unlock"] = {"attempted": False, "ok": False, "note": "demo"}
|
||||||
|
info("unlock install skipped (demo mode)")
|
||||||
|
elif not X.INSTALL_SH.exists():
|
||||||
|
rep["unlock"] = {"attempted": False, "ok": False,
|
||||||
|
"note": "harness/cmpunlocker not cloned - run --preflight"}
|
||||||
|
warn("harness/cmpunlocker not cloned - run --preflight (unlock install skipped)")
|
||||||
|
elif args.unlock or ask("Run cmpunlocker install.sh now? (y/N)", "n").lower().startswith("y"):
|
||||||
|
X.update_cmpunlocker()
|
||||||
|
rep["unlock"] = X.run_installer(folder / "install_unlocker.log")
|
||||||
|
unlock_installed = bool(rep["unlock"].get("attempted") and rep["unlock"].get("ok"))
|
||||||
|
if unlock_installed:
|
||||||
|
# Kernel modules are replaced but the card is not unlocked until the
|
||||||
|
# cold reboot - VRAM stress against the half-applied state is unsafe.
|
||||||
|
rep["vulkan_test"] = {"requested": False, "ran": False,
|
||||||
|
"note": "skipped - unlock installed, reboot first"}
|
||||||
|
else:
|
||||||
|
warn("cmpunlocker install failed - continuing with the Vulkan step")
|
||||||
|
else:
|
||||||
|
rep["unlock"] = {"attempted": False, "ok": False, "note": "operator declined"}
|
||||||
|
info("unlock install skipped")
|
||||||
|
|
||||||
step("optional Vulkan VRAM test (last step)")
|
step("optional Vulkan VRAM test (last step)")
|
||||||
if args.vulkan == "never":
|
if unlock_installed:
|
||||||
|
info("vulkan test skipped - unlock installed, cold reboot first")
|
||||||
|
elif args.vulkan == "never":
|
||||||
rep["vulkan_test"] = {"requested": False, "ran": False, "note": "disabled via --vulkan never"}
|
rep["vulkan_test"] = {"requested": False, "ran": False, "note": "disabled via --vulkan never"}
|
||||||
elif args.vulkan == "always":
|
elif args.vulkan == "always":
|
||||||
rep["vulkan_test"] = X.run_vulkan_test()
|
rep["vulkan_test"] = X.run_vulkan_test()
|
||||||
@@ -201,6 +248,7 @@ def demo(args):
|
|||||||
orig_aer, orig_dm, orig_cfg = X.aer_snapshot, X.dmesg_capture, X.dump_config
|
orig_aer, orig_dm, orig_cfg = X.aer_snapshot, X.dmesg_capture, X.dump_config
|
||||||
orig_retrain, orig_rescan = X.retrain_test, X.rescan_test
|
orig_retrain, orig_rescan = X.retrain_test, X.rescan_test
|
||||||
orig_smi, orig_verify = X.nvidia_smi_info, X.run_verify_unlocker
|
orig_smi, orig_verify = X.nvidia_smi_info, X.run_verify_unlocker
|
||||||
|
orig_update, orig_installer = X.update_cmpunlocker, X.run_installer
|
||||||
globals()["wait_for_gpu"] = lambda: "0000:0a:00.0"
|
globals()["wait_for_gpu"] = lambda: "0000:0a:00.0"
|
||||||
X.detect_gpus = lambda: ["0000:0a:00.0"]
|
X.detect_gpus = lambda: ["0000:0a:00.0"]
|
||||||
X.check_link = lambda b: ({"lnksta_speed": "5GT/s", "lnksta_width": "x4", "lnksta_gen": 2,
|
X.check_link = lambda b: ({"lnksta_speed": "5GT/s", "lnksta_width": "x4", "lnksta_gen": 2,
|
||||||
@@ -218,6 +266,9 @@ def demo(args):
|
|||||||
X.rescan_test = lambda b: {"attempted": True, "ok": True, "note": "re-enumerated"}
|
X.rescan_test = lambda b: {"attempted": True, "ok": True, "note": "re-enumerated"}
|
||||||
X.nvidia_smi_info = lambda: {"available": False}
|
X.nvidia_smi_info = lambda: {"available": False}
|
||||||
X.run_verify_unlocker = lambda: {"ran": False, "note": "demo"}
|
X.run_verify_unlocker = lambda: {"ran": False, "note": "demo"}
|
||||||
|
X.update_cmpunlocker = lambda: {"ok": True, "note": "demo"}
|
||||||
|
X.run_installer = lambda log_path: {"attempted": False, "ok": False, "rc": None,
|
||||||
|
"note": "demo", "log": str(log_path)}
|
||||||
try:
|
try:
|
||||||
rep = process_card(args)
|
rep = process_card(args)
|
||||||
finally:
|
finally:
|
||||||
@@ -226,6 +277,7 @@ def demo(args):
|
|||||||
X.aer_snapshot, X.dmesg_capture, X.dump_config = orig_aer, orig_dm, orig_cfg
|
X.aer_snapshot, X.dmesg_capture, X.dump_config = orig_aer, orig_dm, orig_cfg
|
||||||
X.retrain_test, X.rescan_test = orig_retrain, orig_rescan
|
X.retrain_test, X.rescan_test = orig_retrain, orig_rescan
|
||||||
X.nvidia_smi_info, X.run_verify_unlocker = orig_smi, orig_verify
|
X.nvidia_smi_info, X.run_verify_unlocker = orig_smi, orig_verify
|
||||||
|
X.update_cmpunlocker, X.run_installer = orig_update, orig_installer
|
||||||
return rep
|
return rep
|
||||||
|
|
||||||
|
|
||||||
@@ -241,6 +293,8 @@ def main():
|
|||||||
ap.add_argument("--skip-bios", action="store_true", help="skip VBIOS backup")
|
ap.add_argument("--skip-bios", action="store_true", help="skip VBIOS backup")
|
||||||
ap.add_argument("--vulkan", choices=("ask", "always", "never"), default="ask",
|
ap.add_argument("--vulkan", choices=("ask", "always", "never"), default="ask",
|
||||||
help="5-min Vulkan VRAM test policy (default ask, runs as last step)")
|
help="5-min Vulkan VRAM test policy (default ask, runs as last step)")
|
||||||
|
ap.add_argument("--unlock", action="store_true",
|
||||||
|
help="run cmpunlocker install.sh without prompting (default: ask y/N)")
|
||||||
ap.add_argument("--no-install", action="store_true",
|
ap.add_argument("--no-install", action="store_true",
|
||||||
help="do not auto-install packages (skips the sudo precondition)")
|
help="do not auto-install packages (skips the sudo precondition)")
|
||||||
ap.add_argument("--no-clone", action="store_true", help="do not clone harness repos")
|
ap.add_argument("--no-clone", action="store_true", help="do not clone harness repos")
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
#!/usr/bin/env python3
|
#!/usr/bin/env python3
|
||||||
"""CMP 170HX checker - shared library (preflight, detection, PCIe checks, BIOS, verify, Vulkan)."""
|
"""CMP 170HX checker - shared library (preflight, detection, PCIe checks, BIOS, verify, unlock install, Vulkan)."""
|
||||||
import datetime
|
import datetime
|
||||||
import hashlib
|
import hashlib
|
||||||
import os
|
import os
|
||||||
@@ -7,6 +7,7 @@ import re
|
|||||||
import shutil
|
import shutil
|
||||||
import subprocess
|
import subprocess
|
||||||
import sys
|
import sys
|
||||||
|
import threading
|
||||||
import time
|
import time
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
|
|
||||||
@@ -19,6 +20,7 @@ REPOS = {
|
|||||||
"cmpunlocker": "https://github.com/amoghmunikote/cmpunlocker.git",
|
"cmpunlocker": "https://github.com/amoghmunikote/cmpunlocker.git",
|
||||||
}
|
}
|
||||||
VERIFY_SH = HARNESS_DIR / "cmpunlocker" / "verify.sh"
|
VERIFY_SH = HARNESS_DIR / "cmpunlocker" / "verify.sh"
|
||||||
|
INSTALL_SH = HARNESS_DIR / "cmpunlocker" / "install.sh"
|
||||||
GPU_IDS = ("20c2", "2082")
|
GPU_IDS = ("20c2", "2082")
|
||||||
VULKAN_TEST_SECONDS = 300
|
VULKAN_TEST_SECONDS = 300
|
||||||
|
|
||||||
@@ -76,9 +78,9 @@ def sanitize(name):
|
|||||||
|
|
||||||
# ---------------------------------------------------------------- subprocess
|
# ---------------------------------------------------------------- subprocess
|
||||||
|
|
||||||
def run(cmd, timeout=60, env=None):
|
def run(cmd, timeout=60, env=None, cwd=None):
|
||||||
try:
|
try:
|
||||||
p = subprocess.run(cmd, capture_output=True, text=True, timeout=timeout, env=env)
|
p = subprocess.run(cmd, capture_output=True, text=True, timeout=timeout, env=env, cwd=cwd)
|
||||||
return {"rc": p.returncode, "out": p.stdout, "err": p.stderr, "cmd": cmd}
|
return {"rc": p.returncode, "out": p.stdout, "err": p.stderr, "cmd": cmd}
|
||||||
except FileNotFoundError:
|
except FileNotFoundError:
|
||||||
return {"rc": 127, "out": "", "err": f"not found: {cmd[0]}", "cmd": cmd}
|
return {"rc": 127, "out": "", "err": f"not found: {cmd[0]}", "cmd": cmd}
|
||||||
@@ -87,11 +89,76 @@ def run(cmd, timeout=60, env=None):
|
|||||||
return {"rc": 124, "out": out, "err": "timeout", "cmd": cmd}
|
return {"rc": 124, "out": out, "err": "timeout", "cmd": cmd}
|
||||||
|
|
||||||
|
|
||||||
def as_root(cmd, timeout=60):
|
def run_live(cmd, cwd=None, timeout=600, log_path=None):
|
||||||
|
"""Run cmd while echoing every output line to the terminal as it arrives.
|
||||||
|
|
||||||
|
The complete output is also written to log_path (operator keeps the full log,
|
||||||
|
the terminal keeps the progress). stderr is merged into stdout, so out holds
|
||||||
|
everything. Returns the same {rc, out, err, cmd} shape as run(): rc 127 =
|
||||||
|
not found, rc 124 = timeout.
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
p = subprocess.Popen(cmd, cwd=cwd, stdout=subprocess.PIPE,
|
||||||
|
stderr=subprocess.STDOUT, text=True, bufsize=1)
|
||||||
|
except FileNotFoundError:
|
||||||
|
return {"rc": 127, "out": "", "err": f"not found: {cmd[0]}", "cmd": cmd}
|
||||||
|
except OSError as e:
|
||||||
|
return {"rc": 126, "out": "", "err": str(e), "cmd": cmd}
|
||||||
|
|
||||||
|
lines, done = [], threading.Event()
|
||||||
|
|
||||||
|
def _pump():
|
||||||
|
try:
|
||||||
|
for line in p.stdout or "":
|
||||||
|
lines.append(line)
|
||||||
|
print(line, end="", flush=True)
|
||||||
|
except (ValueError, OSError):
|
||||||
|
pass
|
||||||
|
done.set()
|
||||||
|
|
||||||
|
pump = threading.Thread(target=_pump, daemon=True)
|
||||||
|
pump.start()
|
||||||
|
|
||||||
|
timed_out = interrupted = False
|
||||||
|
try:
|
||||||
|
finished = done.wait(timeout)
|
||||||
|
except KeyboardInterrupt:
|
||||||
|
finished, interrupted = True, True
|
||||||
|
if not finished:
|
||||||
|
timed_out = True
|
||||||
|
if timed_out or interrupted:
|
||||||
|
try:
|
||||||
|
p.kill()
|
||||||
|
except OSError:
|
||||||
|
pass
|
||||||
|
pump.join(timeout=15)
|
||||||
|
rc = p.wait()
|
||||||
|
out = "".join(lines)
|
||||||
|
err = ""
|
||||||
|
if timed_out:
|
||||||
|
rc, err = 124, "timeout"
|
||||||
|
elif interrupted:
|
||||||
|
err = "interrupted"
|
||||||
|
|
||||||
|
if log_path:
|
||||||
|
path = Path(log_path)
|
||||||
|
try:
|
||||||
|
path.parent.mkdir(parents=True, exist_ok=True)
|
||||||
|
path.write_text(f"$ {' '.join(str(c) for c in cmd)}\n"
|
||||||
|
f"cwd: {cwd or os.getcwd()}\n"
|
||||||
|
f"{out}\nrc={rc}{f' ({err})' if err else ''}\n")
|
||||||
|
except OSError as e:
|
||||||
|
warn(f"could not write log {path}: {e}")
|
||||||
|
if interrupted:
|
||||||
|
raise KeyboardInterrupt
|
||||||
|
return {"rc": rc, "out": out, "err": err, "cmd": cmd}
|
||||||
|
|
||||||
|
|
||||||
|
def as_root(cmd, timeout=60, cwd=None):
|
||||||
if os.geteuid() == 0:
|
if os.geteuid() == 0:
|
||||||
return run(cmd, timeout=timeout)
|
return run(cmd, timeout=timeout, cwd=cwd)
|
||||||
if shutil.which("sudo"):
|
if shutil.which("sudo"):
|
||||||
r = run(["sudo", "-n"] + cmd, timeout=timeout)
|
r = run(["sudo", "-n"] + cmd, timeout=timeout, cwd=cwd)
|
||||||
if r["rc"] != 0 and "password is required" in (r["err"] + r["out"]).lower():
|
if r["rc"] != 0 and "password is required" in (r["err"] + r["out"]).lower():
|
||||||
r["err"] = "sudo requires a password (sudo -n failed)"
|
r["err"] = "sudo requires a password (sudo -n failed)"
|
||||||
return r
|
return r
|
||||||
@@ -644,6 +711,69 @@ def run_verify_unlocker():
|
|||||||
return {"ran": True, "rc": r["rc"], "status": status, "tail": lines[-30:]}
|
return {"ran": True, "rc": r["rc"], "status": status, "tail": lines[-30:]}
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------- unlock (opt-in install)
|
||||||
|
|
||||||
|
def update_cmpunlocker():
|
||||||
|
"""Bring harness/cmpunlocker up to date with upstream main before installing.
|
||||||
|
|
||||||
|
A failed `git pull --ff-only` (diverged checkout, offline host, ...) is not
|
||||||
|
fatal: we warn, keep the existing checkout, and let the install proceed.
|
||||||
|
"""
|
||||||
|
dest = HARNESS_DIR / "cmpunlocker"
|
||||||
|
if not (dest / ".git").exists():
|
||||||
|
info("harness/cmpunlocker missing - cloning ...")
|
||||||
|
if not ensure_repos():
|
||||||
|
return {"ok": False, "note": "harness/cmpunlocker not cloned and git clone failed"}
|
||||||
|
return {"ok": True, "note": "harness/cmpunlocker cloned"}
|
||||||
|
r = run(["git", "-C", str(dest), "pull", "--ff-only"], timeout=180)
|
||||||
|
if r["rc"] != 0:
|
||||||
|
msg = (r["err"] or r["out"]).strip().splitlines()
|
||||||
|
msg = msg[-1] if msg else f"rc={r['rc']}"
|
||||||
|
warn(f"git pull --ff-only failed: {msg} - continuing with the existing checkout")
|
||||||
|
return {"ok": False, "note": f"git pull --ff-only failed ({msg}) - using existing checkout"}
|
||||||
|
last = (r["out"].strip().splitlines() or ["already up to date"])[-1]
|
||||||
|
ok(f"cmpunlocker harness updated: {last}")
|
||||||
|
return {"ok": True, "note": f"updated: {last}"}
|
||||||
|
|
||||||
|
|
||||||
|
def run_installer(log_path):
|
||||||
|
"""Run install.sh from the harness clone (opt-in, root required).
|
||||||
|
|
||||||
|
Output streams live to the terminal and is fully saved to log_path.
|
||||||
|
Returns {"attempted", "ok", "rc", "note", "log"}; attempted=False when a
|
||||||
|
precondition (clone present, sudo usable) fails.
|
||||||
|
"""
|
||||||
|
log_path = Path(log_path)
|
||||||
|
result = {"attempted": False, "ok": False, "rc": None, "note": "", "log": str(log_path)}
|
||||||
|
if not INSTALL_SH.exists():
|
||||||
|
result["note"] = "harness/cmpunlocker not cloned"
|
||||||
|
warn(f"{result['note']} - run --preflight first")
|
||||||
|
return result
|
||||||
|
ok_sudo, why = sudo_precondition()
|
||||||
|
if not ok_sudo:
|
||||||
|
result["note"] = f"sudo precondition failed: {why}"
|
||||||
|
warn(result["note"])
|
||||||
|
return result
|
||||||
|
|
||||||
|
cmd = ["bash", str(INSTALL_SH)]
|
||||||
|
if os.geteuid() != 0:
|
||||||
|
cmd = ["sudo", "-n"] + cmd # credentials cached by sudo_precondition()
|
||||||
|
info(f"running install.sh in {INSTALL_SH.parent} - kernel module build takes 5-15 min, "
|
||||||
|
f"live output -> {log_path}")
|
||||||
|
r = run_live(cmd, cwd=str(INSTALL_SH.parent), timeout=1800, log_path=log_path)
|
||||||
|
result.update(attempted=True, rc=r["rc"])
|
||||||
|
if r["rc"] == 0:
|
||||||
|
result.update(ok=True, note="install.sh completed")
|
||||||
|
ok("cmpunlocker installed - cold reboot required: sudo shutdown -h now (then power on), "
|
||||||
|
"then re-run this script for the final PCIe check")
|
||||||
|
else:
|
||||||
|
tail = (r["err"] or r["out"]).strip().splitlines()
|
||||||
|
tail = tail[-1] if tail else ""
|
||||||
|
result["note"] = f"install.sh failed rc={r['rc']}" + (f": {tail}" if tail else "")
|
||||||
|
warn(f"{result['note']} (see {log_path})")
|
||||||
|
return result
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------- Vulkan test
|
# ---------------------------------------------------------------- Vulkan test
|
||||||
|
|
||||||
def vulkan_tool():
|
def vulkan_tool():
|
||||||
|
|||||||
Reference in New Issue
Block a user