Single-card flow; preflight sudo precondition + Ubuntu package install
This commit is contained in:
@@ -1,6 +1,6 @@
|
|||||||
# cmp170hx_check
|
# cmp170hx_check
|
||||||
|
|
||||||
Guided, one-card-at-a-time batch checker for **NVIDIA CMP 170HX** cards (PCI IDs `10de:20c2` / `10de:2082`) — verifies the PCIe capacitor mod, records PCIe bus health, backs up VBIOS, runs the cmpunlocker verification, and optionally stresses VRAM. Built for a 23-card run, with one folder + JSON report per card serial.
|
Single-card checker for **NVIDIA CMP 170HX** cards (PCI IDs `10de:20c2` / `10de:2082`) — run it **one card at a time**: it waits for the card, verifies the PCIe capacitor mod, records PCIe bus health, backs up VBIOS, runs the cmpunlocker verification, and optionally stresses VRAM. Each card gets its own folder + JSON report named by serial, so the folders accumulate as you go through your batch (e.g. 23 cards).
|
||||||
|
|
||||||
Reference harness (auto-cloned into `harness/` by preflight):
|
Reference harness (auto-cloned into `harness/` by preflight):
|
||||||
|
|
||||||
@@ -9,36 +9,51 @@ Reference harness (auto-cloned into `harness/` by preflight):
|
|||||||
|
|
||||||
## Requirements
|
## Requirements
|
||||||
|
|
||||||
- Linux x86-64, Python 3.8+
|
- Linux x86-64 (tested target: **Ubuntu 26.04**), Python 3.8+
|
||||||
- `pciutils` (`lspci`, `setpci`) — auto-installed if a package manager is present
|
- **sudo privileges** — precondition for `--preflight` package installation and for privileged checks (PCI rescan, link retrain, sysfs ROM, `dmesg`, `verify.sh`)
|
||||||
- Root (`sudo`) for: PCI bus rescan, link retrain, sysfs ROM read, `dmesg`, `verify.sh`
|
- `pciutils` (`lspci`, `setpci`), `git` — installed automatically by preflight
|
||||||
- NVIDIA driver with `nvidia-smi` (nvidia-open 610.xx+, see cmpunlocker) for unlock verification
|
- NVIDIA driver with `nvidia-smi` (nvidia-open 610.xx+, see cmpunlocker) for unlock verification — preflight can install it for you (asks first)
|
||||||
- `nvflash` (not packaged — download from NVIDIA or pass `--nvflash /path/to/nvflash`) for VBIOS backup and serial auto-read
|
- `nvflash` (not packaged — download from NVIDIA or pass `--nvflash /path/to/nvflash`) for VBIOS backup and serial auto-read
|
||||||
- Optional: `vkmark` or `vulkan-tools` for the 5-minute VRAM stress test
|
- Optional: `vkmark` or `vulkan-tools` for the 5-minute VRAM stress test
|
||||||
|
|
||||||
Missing tools are reported by preflight; packages are auto-installed where possible, otherwise instructions are printed.
|
## Preflight (setup step)
|
||||||
|
|
||||||
## Usage
|
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
python3 check_cards.py # full run, 23 cards, guided
|
python3 check_cards.py --preflight
|
||||||
python3 check_cards.py --preflight # tool check / install / harness clone only
|
```
|
||||||
|
|
||||||
|
On Ubuntu 26.04 this:
|
||||||
|
|
||||||
|
1. **Checks the sudo precondition** — runs `sudo -v` if needed so credentials are cached; if sudo is unavailable it prints the fix and **does not install anything** (exit code 3 if blocking tools are still missing)
|
||||||
|
2. Installs required packages via `apt-get update && apt-get install`: `git`, `pciutils`, `vulkan-tools`, `vkmark`
|
||||||
|
3. Offers to install the **NVIDIA driver** (`ubuntu-drivers install`, falling back to the `nvidia-driver` metapackage) when `nvidia-smi` is missing — reboot afterwards
|
||||||
|
4. Clones the two harness repos into `harness/`
|
||||||
|
5. Prints a tool status table; `nvflash` is reported with manual-install instructions (not packaged)
|
||||||
|
|
||||||
|
Skip installation with `--no-install` (no sudo needed at all).
|
||||||
|
|
||||||
|
## Usage (one card per run)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python3 check_cards.py # check the connected card
|
||||||
|
python3 check_cards.py --preflight # setup: tools, packages, harness clone
|
||||||
python3 check_cards.py --demo # self-test with fake hardware (no GPU needed)
|
python3 check_cards.py --demo # self-test with fake hardware (no GPU needed)
|
||||||
python3 check_cards.py --help # all flags
|
python3 check_cards.py --help # all flags
|
||||||
```
|
```
|
||||||
|
|
||||||
Normal run flow, repeated per card:
|
Flow of a normal run:
|
||||||
|
|
||||||
1. `Insert card N/23, then press Enter` (`s` = skip, `q` = quit)
|
1. Waits for a CMP 170HX on the bus — prompts you to insert/reseat the card, Enter triggers a PCI rescan (`q` quits)
|
||||||
2. GPU auto-detected on the bus (PCI rescan retry offered if absent)
|
2. Serial auto-read from nvflash UUID; falls back to a manual label prompt
|
||||||
3. Serial auto-read from nvflash UUID; falls back to a manual label prompt
|
3. VBIOS backup → `cards/<serial>/bios.rom` (nvflash, sysfs ROM fallback)
|
||||||
4. VBIOS backup → `cards/<serial>/bios.rom` (nvflash, sysfs ROM fallback)
|
4. PCIe checks: link speed/width, AER counters, full config-space dump
|
||||||
5. PCIe checks: link speed/width, AER counters, full config-space dump
|
5. Link retrain test + full bus rescan test (device must survive both)
|
||||||
6. Link retrain test + full bus rescan test (device must survive both)
|
6. Kernel log scrape (`dmesg`/`journalctl`) for this BDF: Xid / AER / NVRM lines
|
||||||
7. Kernel log scrape (`dmesg`/`journalctl`) for this BDF: Xid / AER / NVRM lines
|
7. `nvidia-smi` snapshot + `harness/cmpunlocker/verify.sh` (read-only)
|
||||||
8. `nvidia-smi` snapshot + `harness/cmpunlocker/verify.sh` (read-only)
|
8. **Last step, optional**: `Run 5-min Vulkan VRAM stress test now? (y/N)` — refused by default, `--vulkan always|never` to override
|
||||||
9. **Last step, optional**: `Run 5-min Vulkan VRAM stress test now? (y/N)` — refused by default, `--vulkan always|never` to override
|
9. Result printed + `report.json` written into the card's folder
|
||||||
10. Result + `report.json` written; progress saved to `progress.json` (resumable)
|
|
||||||
|
Remove the card, plug the next one, repeat — no queue, no indices, folders just pile up in `cards/`.
|
||||||
|
|
||||||
## Checks & scoring (lenient)
|
## Checks & scoring (lenient)
|
||||||
|
|
||||||
@@ -48,6 +63,8 @@ Normal run flow, repeated per card:
|
|||||||
| **WARN** | link width ≠ `x16` (capacitor mod missing/incomplete), Gen1 only (Gen2 unlock patch not active), correctable AER errors, kernel errors / Xid, VBIOS backup failed, cmpunlocker verify not `unlocked-ok`, errors during VRAM stress |
|
| **WARN** | link width ≠ `x16` (capacitor mod missing/incomplete), Gen1 only (Gen2 unlock patch not active), correctable AER errors, kernel errors / Xid, VBIOS backup failed, cmpunlocker verify not `unlocked-ok`, errors during VRAM stress |
|
||||||
| **PASS** | everything clean |
|
| **PASS** | everything clean |
|
||||||
|
|
||||||
|
Exit codes: `0` PASS/WARN · `1` FAIL · `2` no card found · `3` preflight blocking failure.
|
||||||
|
|
||||||
## Output
|
## Output
|
||||||
|
|
||||||
```
|
```
|
||||||
@@ -61,22 +78,21 @@ cards/
|
|||||||
├── config_space.hex # lspci -nnvvxxxx full hex dump
|
├── config_space.hex # lspci -nnvvxxxx full hex dump
|
||||||
├── dmesg.log # kernel lines related to this card
|
├── dmesg.log # kernel lines related to this card
|
||||||
└── verify_unlocker.log # cmpunlocker verify.sh output
|
└── verify_unlocker.log # cmpunlocker verify.sh output
|
||||||
progress.json # which card indices are done (resume support)
|
|
||||||
```
|
```
|
||||||
|
|
||||||
## Useful flags
|
## Flags
|
||||||
|
|
||||||
| Flag | Meaning |
|
| Flag | Meaning |
|
||||||
|------|---------|
|
|------|---------|
|
||||||
| `--cards N` / `--start N` | total cards (default 23) / first index |
|
| `--preflight` | setup only: sudo precondition + package install + harness clone, then exit |
|
||||||
|
| `--demo` | run the whole pipeline against fake hardware (self-test) |
|
||||||
| `--nvflash PATH` | explicit nvflash binary |
|
| `--nvflash PATH` | explicit nvflash binary |
|
||||||
| `--serial XYZ` | override serial for the next card |
|
| `--serial XYZ` | override serial (skips nvflash auto-read) |
|
||||||
| `--vulkan ask\|always\|never` | VRAM test policy (default: ask, last step) |
|
| `--vulkan ask\|always\|never` | VRAM test policy (default: ask, last step) |
|
||||||
| `--skip-bios` | skip VBIOS backup |
|
| `--skip-bios` | skip VBIOS backup |
|
||||||
| `--no-install` / `--no-clone` | don't auto-install packages / don't clone harness |
|
| `--no-install` | never install packages (skips the sudo precondition) |
|
||||||
| `--restart` | ignore `progress.json` and start over |
|
| `--no-clone` | don't clone the harness repos |
|
||||||
| `--no-pause` | don't wait for the insert prompt |
|
|
||||||
|
|
||||||
## Safety
|
## Safety
|
||||||
|
|
||||||
This tool only **reads** card state: `verify.sh` is run without installing anything, no modules are patched, no firmware is flashed. The only writes are local files, PCI link retrain/rescan sysfs triggers, and your own reports.
|
This tool only **reads** card state: `verify.sh` is run without installing anything, no modules are patched, no firmware is flashed. The only writes are local files, PCI link retrain/rescan sysfs triggers, package installation during `--preflight` (explicitly gated behind the sudo precondition), and your own reports.
|
||||||
|
|||||||
+65
-135
@@ -1,23 +1,23 @@
|
|||||||
#!/usr/bin/env python3
|
#!/usr/bin/env python3
|
||||||
"""CMP 170HX batch card checker - guided one-card-at-a-time workflow (23 cards).
|
"""CMP 170HX single-card checker - ONE card per run, no queue.
|
||||||
|
|
||||||
Per card: detect GPU -> auto serial -> VBIOS backup -> PCIe link/AER/config checks
|
Per run: wait for card -> auto serial -> VBIOS backup -> PCIe link/AER/config
|
||||||
-> retrain + rescan tests -> dmesg scrape -> cmpunlocker verify.sh (read-only)
|
checks -> retrain + rescan tests -> dmesg scrape -> cmpunlocker verify.sh
|
||||||
-> optional 5-min Vulkan VRAM stress (last step, ask yes/no) -> cards/<serial>/report.json.
|
(read-only) -> optional 5-min Vulkan VRAM stress (last step, ask yes/no)
|
||||||
|
-> cards/<serial>/report.json.
|
||||||
|
|
||||||
Scoring (lenient): FAIL only on no-device/link-down; WARN on correctable AER/Xid/
|
Scoring (lenient): FAIL only on no-device/link-down/lost-after-rescan;
|
||||||
width< x16/verify issues.
|
WARN on correctable AER/Xid/width< x16/verify issues.
|
||||||
|
|
||||||
|
Exit codes: 0 = PASS/WARN, 1 = FAIL, 2 = no card, 3 = preflight blocking.
|
||||||
"""
|
"""
|
||||||
import argparse
|
import argparse
|
||||||
import json
|
import json
|
||||||
import sys
|
import sys
|
||||||
import time
|
import time
|
||||||
from pathlib import Path
|
|
||||||
|
|
||||||
import cxlib as X
|
import cxlib as X
|
||||||
from cxlib import (C, ask, detect_gpus, err, info, now_iso, ok, sanitize, step, warn)
|
from cxlib import C, ask, detect_gpus, err, info, now_iso, ok, sanitize, step, warn
|
||||||
|
|
||||||
TOTAL_DEFAULT = X.DEFAULT_TOTAL
|
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------- scoring
|
# ---------------------------------------------------------------- scoring
|
||||||
@@ -56,7 +56,7 @@ def score(rep):
|
|||||||
if vu.get("requested") and vu.get("ran"):
|
if vu.get("requested") and vu.get("ran"):
|
||||||
if vu.get("tool_errors"):
|
if vu.get("tool_errors"):
|
||||||
warns.append("errors reported during Vulkan VRAM stress")
|
warns.append("errors reported during Vulkan VRAM stress")
|
||||||
if dm.get("errors") and vu.get("ran"):
|
if dm.get("errors"):
|
||||||
warns.append("kernel errors around VRAM stress window")
|
warns.append("kernel errors around VRAM stress window")
|
||||||
if not rep.get("bios", {}).get("ok"):
|
if not rep.get("bios", {}).get("ok"):
|
||||||
warns.append("VBIOS backup failed/incomplete")
|
warns.append("VBIOS backup failed/incomplete")
|
||||||
@@ -68,41 +68,39 @@ def score(rep):
|
|||||||
return result, fails, warns
|
return result, fails, warns
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------- card detect
|
||||||
|
|
||||||
|
def wait_for_gpu():
|
||||||
|
"""Wait for a single CMP 170HX (with rescan retries). Returns BDF or None."""
|
||||||
|
while True:
|
||||||
|
for _ in range(5):
|
||||||
|
bdfs = detect_gpus()
|
||||||
|
if bdfs:
|
||||||
|
if len(bdfs) > 1:
|
||||||
|
warn(f"multiple CMP 170HX devices present: {', '.join(bdfs)} - using {bdfs[0]}")
|
||||||
|
return bdfs[0]
|
||||||
|
time.sleep(1)
|
||||||
|
ans = ask("CMP 170HX not detected - insert/reseat the card, press Enter to rescan "
|
||||||
|
"(q=quit)", "")
|
||||||
|
if ans.lower().startswith("q"):
|
||||||
|
return None
|
||||||
|
if not X.rescan_bus():
|
||||||
|
warn("bus rescan not permitted (no root/sudo) - reseat the card physically")
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------- one card
|
# ---------------------------------------------------------------- one card
|
||||||
|
|
||||||
def find_gpu(timeout=45):
|
def process_card(args):
|
||||||
"""Wait for a CMP 170HX to appear, offering rescan retries."""
|
step("Card check")
|
||||||
deadline = time.time() + timeout
|
bdf = wait_for_gpu()
|
||||||
while time.time() < deadline:
|
|
||||||
bdfs = detect_gpus()
|
|
||||||
if len(bdfs) == 1:
|
|
||||||
return bdfs[0]
|
|
||||||
if len(bdfs) > 1:
|
|
||||||
warn(f"multiple CMP 170HX devices present: {', '.join(bdfs)}")
|
|
||||||
return bdfs[0]
|
|
||||||
time.sleep(2)
|
|
||||||
warn("no CMP 170HX detected yet")
|
|
||||||
if ask("trigger PCI bus rescan and retry? (y/N)", "n").lower().startswith("y"):
|
|
||||||
if X.rescan_bus():
|
|
||||||
ok("bus rescanned")
|
|
||||||
else:
|
|
||||||
err("rescan not permitted - reseat the card and press Enter")
|
|
||||||
input()
|
|
||||||
return find_gpu(timeout=30)
|
|
||||||
return None
|
|
||||||
|
|
||||||
|
|
||||||
def process_card(index, total, args):
|
|
||||||
header = f"Card {index}/{total}"
|
|
||||||
step(header)
|
|
||||||
bdf = find_gpu()
|
|
||||||
if not bdf:
|
if not bdf:
|
||||||
err("no device - skipping this card")
|
err("no card detected - nothing to do")
|
||||||
return None
|
return None
|
||||||
devid = ""
|
devid = ""
|
||||||
for line in X.run(["lspci", "-D", "-s", bdf, "-nn"])["out"].splitlines():
|
for line in X.run(["lspci", "-D", "-s", bdf, "-nn"])["out"].splitlines():
|
||||||
if "[10de:" in line:
|
if "[10de:" in line:
|
||||||
devid = line[line.index("[10de:") + 1:line.index("[10de:") + 10].rstrip("]")
|
i = line.index("[10de:")
|
||||||
|
devid = line[i + 1:i + 10].rstrip("]")
|
||||||
ok(f"device found at {bdf} ({devid})")
|
ok(f"device found at {bdf} ({devid})")
|
||||||
|
|
||||||
nf = X.nvflash_bin(args)
|
nf = X.nvflash_bin(args)
|
||||||
@@ -112,17 +110,17 @@ def process_card(index, total, args):
|
|||||||
info(f"folder: {folder}")
|
info(f"folder: {folder}")
|
||||||
|
|
||||||
rep = {
|
rep = {
|
||||||
"card_index": index, "total": total, "serial": serial, "serial_source": serial_src,
|
"serial": serial, "serial_source": serial_src, "started_at": now_iso(),
|
||||||
"started_at": now_iso(), "bdf": bdf, "device_id": devid, "host": X.run(["uname", "-n"])["out"].strip(),
|
"bdf": bdf, "device_id": devid, "host": X.run(["uname", "-n"])["out"].strip(),
|
||||||
"kernel": X.run(["uname", "-r"])["out"].strip(),
|
"kernel": X.run(["uname", "-r"])["out"].strip(),
|
||||||
}
|
}
|
||||||
|
|
||||||
step(f"[{index}/{total}] VBIOS backup")
|
step("VBIOS backup")
|
||||||
rep["bios"] = X.save_bios(nf, folder / "bios.rom", args)
|
rep["bios"] = X.save_bios(nf, folder / "bios.rom", args)
|
||||||
(ok if rep["bios"]["ok"] else warn)(f"bios: {rep['bios']['method'] or 'FAILED'} "
|
(ok if rep["bios"]["ok"] else warn)(f"bios: {rep['bios']['method'] or 'FAILED'} "
|
||||||
f"{rep['bios']['size']} bytes - {rep['bios']['note']}")
|
f"{rep['bios']['size']} bytes - {rep['bios']['note']}")
|
||||||
|
|
||||||
step(f"[{index}/{total}] PCIe link + config")
|
step("PCIe link + config")
|
||||||
link, lspci_txt = X.check_link(bdf)
|
link, lspci_txt = X.check_link(bdf)
|
||||||
rep["link"] = link
|
rep["link"] = link
|
||||||
(folder / "lspci.txt").write_text(lspci_txt)
|
(folder / "lspci.txt").write_text(lspci_txt)
|
||||||
@@ -130,7 +128,7 @@ def process_card(index, total, args):
|
|||||||
info(f"LnkSta: {link.get('lnksta_raw') or 'n/a'} (setpci gen={link.get('setpci_current_gen')})")
|
info(f"LnkSta: {link.get('lnksta_raw') or 'n/a'} (setpci gen={link.get('setpci_current_gen')})")
|
||||||
rep["aer_before"] = X.aer_snapshot(bdf)
|
rep["aer_before"] = X.aer_snapshot(bdf)
|
||||||
|
|
||||||
step(f"[{index}/{total}] retrain + rescan tests")
|
step("retrain + rescan tests")
|
||||||
rep["retrain"] = X.retrain_test(bdf)
|
rep["retrain"] = X.retrain_test(bdf)
|
||||||
info(f"retrain: ok={rep['retrain']['ok']} - {rep['retrain']['note']}")
|
info(f"retrain: ok={rep['retrain']['ok']} - {rep['retrain']['note']}")
|
||||||
rep["rescan"] = X.rescan_test(bdf)
|
rep["rescan"] = X.rescan_test(bdf)
|
||||||
@@ -141,24 +139,24 @@ def process_card(index, total, args):
|
|||||||
if lspci2:
|
if lspci2:
|
||||||
(folder / "lspci_after.txt").write_text(lspci2)
|
(folder / "lspci_after.txt").write_text(lspci2)
|
||||||
|
|
||||||
step(f"[{index}/{total}] kernel log scrape")
|
step("kernel log scrape")
|
||||||
dm = X.dmesg_capture(bdf)
|
dm = X.dmesg_capture(bdf)
|
||||||
rep["dmesg"] = dm
|
rep["dmesg"] = dm
|
||||||
(folder / "dmesg.log").write_text("\n".join(dm.get("related", [])))
|
(folder / "dmesg.log").write_text("\n".join(dm.get("related", [])))
|
||||||
info(f"{dm.get('note')} (source: {dm.get('source')})")
|
info(f"{dm.get('note')} (source: {dm.get('source')})")
|
||||||
if dm.get("errors"):
|
for e in dm.get("errors", [])[-5:]:
|
||||||
for e in dm["errors"][-5:]:
|
warn(e.strip()[:160])
|
||||||
warn(e.strip()[:160])
|
|
||||||
|
|
||||||
step(f"[{index}/{total}] nvidia-smi + cmpunlocker verify")
|
step("nvidia-smi + cmpunlocker verify")
|
||||||
rep["nvidia_smi"] = X.nvidia_smi_info()
|
rep["nvidia_smi"] = X.nvidia_smi_info()
|
||||||
rep["cmpunlocker_verify"] = X.run_verify_unlocker()
|
rep["cmpunlocker_verify"] = X.run_verify_unlocker()
|
||||||
v = rep["cmpunlocker_verify"]
|
v = rep["cmpunlocker_verify"]
|
||||||
(folder / "verify_unlocker.log").write_text(
|
(folder / "verify_unlocker.log").write_text(
|
||||||
"\n".join(v.get("tail", [])) or (v.get("note") or "not run"))
|
"\n".join(v.get("tail", [])) or (v.get("note") or "not run"))
|
||||||
info(f"verify: {'ran rc=' + str(v.get('rc')) + ' -> ' + str(v.get('status')) if v.get('ran') else 'skipped - ' + str(v.get('note'))}")
|
info("verify: " + (f"rc={v.get('rc')} -> {v.get('status')}" if v.get("ran")
|
||||||
|
else f"skipped - {v.get('note')}"))
|
||||||
|
|
||||||
step(f"[{index}/{total}] optional Vulkan VRAM test (last step)")
|
step("optional Vulkan VRAM test (last step)")
|
||||||
if args.vulkan == "never":
|
if args.vulkan == "never":
|
||||||
rep["vulkan_test"] = {"requested": False, "ran": False, "note": "disabled via --vulkan never"}
|
rep["vulkan_test"] = {"requested": False, "ran": False, "note": "disabled via --vulkan never"}
|
||||||
elif args.vulkan == "always":
|
elif args.vulkan == "always":
|
||||||
@@ -189,37 +187,6 @@ def process_card(index, total, args):
|
|||||||
return rep
|
return rep
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------- progress
|
|
||||||
|
|
||||||
def load_progress():
|
|
||||||
try:
|
|
||||||
return json.loads(X.PROGRESS_FILE.read_text())
|
|
||||||
except Exception:
|
|
||||||
return {"done": []}
|
|
||||||
|
|
||||||
|
|
||||||
def save_progress(prog):
|
|
||||||
X.PROGRESS_FILE.write_text(json.dumps(prog, indent=2))
|
|
||||||
|
|
||||||
|
|
||||||
def scoreboard(results):
|
|
||||||
step("Scoreboard")
|
|
||||||
counts = {"PASS": 0, "WARN": 0, "FAIL": 0}
|
|
||||||
for r in results:
|
|
||||||
if not r:
|
|
||||||
continue
|
|
||||||
counts[r["result"]] = counts.get(r["result"], 0) + 1
|
|
||||||
color = {"PASS": C.GREEN, "WARN": C.YELLOW, "FAIL": C.RED}[r["result"]]
|
|
||||||
print(f" {color}{r['result']:<4}{C.RESET} #{r['card_index']:>2} "
|
|
||||||
f"{r['serial']:<24} {r.get('bdf') or '-':<12} "
|
|
||||||
f"{r.get('link', {}).get('lnksta_width') or '?':<4} "
|
|
||||||
f"{r.get('link', {}).get('lnksta_speed') or '?'}")
|
|
||||||
skipped = [r for r in results if r is None]
|
|
||||||
print(f"\n PASS={counts.get('PASS', 0)} WARN={counts.get('WARN', 0)} "
|
|
||||||
f"FAIL={counts.get('FAIL', 0)} skipped={len(skipped)} "
|
|
||||||
f"total_seen={len(results)}/{TOTAL_DEFAULT}")
|
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------- demo
|
# ---------------------------------------------------------------- demo
|
||||||
|
|
||||||
def demo(args):
|
def demo(args):
|
||||||
@@ -228,16 +195,13 @@ def demo(args):
|
|||||||
args.vulkan = "never"
|
args.vulkan = "never"
|
||||||
args.serial = "DEMO-SERIAL-001"
|
args.serial = "DEMO-SERIAL-001"
|
||||||
args.skip_bios = True
|
args.skip_bios = True
|
||||||
real_find = globals()["find_gpu"]
|
|
||||||
globals()["find_gpu"] = lambda timeout=0: "0000:0a:00.0"
|
|
||||||
|
|
||||||
class FakeRun:
|
|
||||||
pass
|
|
||||||
|
|
||||||
|
orig_wait = globals()["wait_for_gpu"]
|
||||||
orig_detect, orig_check = X.detect_gpus, X.check_link
|
orig_detect, orig_check = X.detect_gpus, X.check_link
|
||||||
orig_aer, orig_dm, orig_cfg = X.aer_snapshot, X.dmesg_capture, X.dump_config
|
orig_aer, orig_dm, orig_cfg = X.aer_snapshot, X.dmesg_capture, X.dump_config
|
||||||
orig_retrain, orig_rescan = X.retrain_test, X.rescan_test
|
orig_retrain, orig_rescan = X.retrain_test, X.rescan_test
|
||||||
orig_smi, orig_verify = X.nvidia_smi_info, X.run_verify_unlocker
|
orig_smi, orig_verify = X.nvidia_smi_info, X.run_verify_unlocker
|
||||||
|
globals()["wait_for_gpu"] = lambda: "0000:0a:00.0"
|
||||||
X.detect_gpus = lambda: ["0000:0a:00.0"]
|
X.detect_gpus = lambda: ["0000:0a:00.0"]
|
||||||
X.check_link = lambda b: ({"lnksta_speed": "5GT/s", "lnksta_width": "x4", "lnksta_gen": 2,
|
X.check_link = lambda b: ({"lnksta_speed": "5GT/s", "lnksta_width": "x4", "lnksta_gen": 2,
|
||||||
"lnkcap_speed": "8GT/s", "lnkcap_width": "x16",
|
"lnkcap_speed": "8GT/s", "lnkcap_width": "x16",
|
||||||
@@ -255,9 +219,9 @@ def demo(args):
|
|||||||
X.nvidia_smi_info = lambda: {"available": False}
|
X.nvidia_smi_info = lambda: {"available": False}
|
||||||
X.run_verify_unlocker = lambda: {"ran": False, "note": "demo"}
|
X.run_verify_unlocker = lambda: {"ran": False, "note": "demo"}
|
||||||
try:
|
try:
|
||||||
rep = process_card(1, TOTAL_DEFAULT, args)
|
rep = process_card(args)
|
||||||
finally:
|
finally:
|
||||||
globals()["find_gpu"] = real_find
|
globals()["wait_for_gpu"] = orig_wait
|
||||||
X.detect_gpus, X.check_link = orig_detect, orig_check
|
X.detect_gpus, X.check_link = orig_detect, orig_check
|
||||||
X.aer_snapshot, X.dmesg_capture, X.dump_config = orig_aer, orig_dm, orig_cfg
|
X.aer_snapshot, X.dmesg_capture, X.dump_config = orig_aer, orig_dm, orig_cfg
|
||||||
X.retrain_test, X.rescan_test = orig_retrain, orig_rescan
|
X.retrain_test, X.rescan_test = orig_retrain, orig_rescan
|
||||||
@@ -268,66 +232,32 @@ def demo(args):
|
|||||||
# ---------------------------------------------------------------- main
|
# ---------------------------------------------------------------- main
|
||||||
|
|
||||||
def main():
|
def main():
|
||||||
global TOTAL_DEFAULT
|
ap = argparse.ArgumentParser(description="CMP 170HX single-card checker (one card per run)")
|
||||||
ap = argparse.ArgumentParser(description="CMP 170HX guided 23-card checker")
|
ap.add_argument("--preflight", action="store_true",
|
||||||
ap.add_argument("--preflight", action="store_true", help="only run tool check/install/clone")
|
help="tool check + package install (requires sudo) + harness clone, then exit")
|
||||||
ap.add_argument("--demo", action="store_true", help="run pipeline with fake hardware (self-test)")
|
ap.add_argument("--demo", action="store_true", help="run pipeline with fake hardware (self-test)")
|
||||||
ap.add_argument("--cards", type=int, default=TOTAL_DEFAULT, help="total cards (default 23)")
|
|
||||||
ap.add_argument("--start", type=int, default=1, help="first card index")
|
|
||||||
ap.add_argument("--nvflash", help="path to nvflash binary")
|
ap.add_argument("--nvflash", help="path to nvflash binary")
|
||||||
ap.add_argument("--serial", help="override serial for the next card (single-card run)")
|
ap.add_argument("--serial", help="override serial (skips nvflash auto-read)")
|
||||||
ap.add_argument("--skip-bios", action="store_true", help="skip VBIOS backup")
|
ap.add_argument("--skip-bios", action="store_true", help="skip VBIOS backup")
|
||||||
ap.add_argument("--vulkan", choices=("ask", "always", "never"), default="ask",
|
ap.add_argument("--vulkan", choices=("ask", "always", "never"), default="ask",
|
||||||
help="5-min Vulkan VRAM test policy (default ask, runs as last step)")
|
help="5-min Vulkan VRAM test policy (default ask, runs as last step)")
|
||||||
ap.add_argument("--no-install", action="store_true", help="do not auto-install missing packages")
|
ap.add_argument("--no-install", action="store_true",
|
||||||
|
help="do not auto-install packages (skips the sudo precondition)")
|
||||||
ap.add_argument("--no-clone", action="store_true", help="do not clone harness repos")
|
ap.add_argument("--no-clone", action="store_true", help="do not clone harness repos")
|
||||||
ap.add_argument("--restart", action="store_true", help="ignore previous progress.json")
|
|
||||||
ap.add_argument("--no-pause", action="store_true", help="do not wait for card insertion")
|
|
||||||
args = ap.parse_args()
|
args = ap.parse_args()
|
||||||
|
|
||||||
TOTAL_DEFAULT = args.cards
|
|
||||||
|
|
||||||
pf = X.preflight(auto_install=not args.no_install, do_clone=not args.no_clone)
|
pf = X.preflight(auto_install=not args.no_install, do_clone=not args.no_clone)
|
||||||
if args.preflight:
|
if args.preflight:
|
||||||
return 0
|
return 0 if pf.get("preflight_ok") else 3
|
||||||
if args.demo:
|
if args.demo:
|
||||||
demo(args)
|
demo(args)
|
||||||
return 0
|
return 0
|
||||||
|
|
||||||
X.CARDS_DIR.mkdir(exist_ok=True)
|
X.CARDS_DIR.mkdir(exist_ok=True)
|
||||||
prog = {"done": []} if args.restart else load_progress()
|
rep = process_card(args)
|
||||||
|
if not rep:
|
||||||
results = []
|
return 2
|
||||||
for i in range(args.start, args.cards + 1):
|
return 1 if rep.get("result") == "FAIL" else 0
|
||||||
if str(i) in prog.get("done", []):
|
|
||||||
info(f"card {i} already done (progress.json) - skipping")
|
|
||||||
results.append(None)
|
|
||||||
continue
|
|
||||||
if not args.no_pause:
|
|
||||||
ans = ask(f"Insert card {i}/{args.cards}, then press Enter "
|
|
||||||
f"(s=skip this card, q=quit)", "")
|
|
||||||
if ans.lower() == "q":
|
|
||||||
break
|
|
||||||
if ans.lower() == "s":
|
|
||||||
results.append(None)
|
|
||||||
prog.setdefault("done", []).append(str(i))
|
|
||||||
prog.setdefault("skipped", []).append(str(i))
|
|
||||||
save_progress(prog)
|
|
||||||
continue
|
|
||||||
rep = process_card(i, args.cards, args)
|
|
||||||
results.append(rep)
|
|
||||||
if rep:
|
|
||||||
prog.setdefault("done", []).append(str(i))
|
|
||||||
save_progress(prog)
|
|
||||||
else:
|
|
||||||
if not ask("retry this card? (y/N)", "n").lower().startswith("y"):
|
|
||||||
break
|
|
||||||
|
|
||||||
if results:
|
|
||||||
scoreboard(results)
|
|
||||||
done = len([r for r in results if r])
|
|
||||||
info(f"{done} card(s) processed. Output in {X.CARDS_DIR}")
|
|
||||||
return 0
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
if __name__ == "__main__":
|
||||||
@@ -335,5 +265,5 @@ if __name__ == "__main__":
|
|||||||
sys.exit(main())
|
sys.exit(main())
|
||||||
except KeyboardInterrupt:
|
except KeyboardInterrupt:
|
||||||
print()
|
print()
|
||||||
warn("interrupted - progress saved in progress.json")
|
warn("interrupted")
|
||||||
sys.exit(130)
|
sys.exit(130)
|
||||||
|
|||||||
@@ -13,7 +13,6 @@ from pathlib import Path
|
|||||||
SCRIPT_DIR = Path(__file__).resolve().parent
|
SCRIPT_DIR = Path(__file__).resolve().parent
|
||||||
HARNESS_DIR = SCRIPT_DIR / "harness"
|
HARNESS_DIR = SCRIPT_DIR / "harness"
|
||||||
CARDS_DIR = SCRIPT_DIR / "cards"
|
CARDS_DIR = SCRIPT_DIR / "cards"
|
||||||
PROGRESS_FILE = SCRIPT_DIR / "progress.json"
|
|
||||||
|
|
||||||
REPOS = {
|
REPOS = {
|
||||||
"170th-Street": "https://github.com/amoghmunikote/170th-Street.git",
|
"170th-Street": "https://github.com/amoghmunikote/170th-Street.git",
|
||||||
@@ -21,7 +20,6 @@ REPOS = {
|
|||||||
}
|
}
|
||||||
VERIFY_SH = HARNESS_DIR / "cmpunlocker" / "verify.sh"
|
VERIFY_SH = HARNESS_DIR / "cmpunlocker" / "verify.sh"
|
||||||
GPU_IDS = ("20c2", "2082")
|
GPU_IDS = ("20c2", "2082")
|
||||||
DEFAULT_TOTAL = 23
|
|
||||||
VULKAN_TEST_SECONDS = 300
|
VULKAN_TEST_SECONDS = 300
|
||||||
|
|
||||||
SPEED_TO_GEN = {"2.5GT/s": 1, "5GT/s": 2, "8GT/s": 3, "16GT/s": 4, "32GT/s": 5, "64GT/s": 6}
|
SPEED_TO_GEN = {"2.5GT/s": 1, "5GT/s": 2, "8GT/s": 3, "16GT/s": 4, "32GT/s": 5, "64GT/s": 6}
|
||||||
@@ -172,34 +170,120 @@ PKG_FOR = {"lspci": "pciutils", "setpci": "pciutils", "vulkaninfo": "vulkan-tool
|
|||||||
"vkcube": "vulkan-tools", "vkmark": "vkmark", "git": "git"}
|
"vkcube": "vulkan-tools", "vkmark": "vkmark", "git": "git"}
|
||||||
|
|
||||||
|
|
||||||
|
def os_release():
|
||||||
|
data = read_file("/etc/os-release") or ""
|
||||||
|
out = {}
|
||||||
|
for line in data.splitlines():
|
||||||
|
if "=" in line:
|
||||||
|
k, v = line.split("=", 1)
|
||||||
|
out[k] = v.strip().strip('"')
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
def sudo_precondition(interactive=True):
|
||||||
|
"""Precondition for package installation (--preflight): root or working sudo.
|
||||||
|
|
||||||
|
Tries passwordless sudo first; falls back to an interactive `sudo -v` so the
|
||||||
|
credentials get cached, after which plain sudo -n calls succeed.
|
||||||
|
"""
|
||||||
|
if os.geteuid() == 0:
|
||||||
|
return True, "running as root"
|
||||||
|
if not shutil.which("sudo"):
|
||||||
|
return False, "sudo is not installed"
|
||||||
|
r = run(["sudo", "-n", "true"], timeout=15)
|
||||||
|
if r["rc"] == 0:
|
||||||
|
return True, "sudo available (credentials cached)"
|
||||||
|
if interactive and sys.stdin.isatty():
|
||||||
|
info("sudo authentication required for package installation - enter your password")
|
||||||
|
try:
|
||||||
|
p = subprocess.run(["sudo", "-v"])
|
||||||
|
except KeyboardInterrupt:
|
||||||
|
return False, "interrupted"
|
||||||
|
if p.returncode == 0:
|
||||||
|
return True, "sudo authenticated just now"
|
||||||
|
return False, "sudo authentication failed"
|
||||||
|
return False, "sudo requires a password (non-interactive session)"
|
||||||
|
|
||||||
|
|
||||||
|
def install_nvidia_driver():
|
||||||
|
"""Ubuntu: prefer 'ubuntu-drivers install', else the nvidia-driver metapackage."""
|
||||||
|
if which("ubuntu-drivers"):
|
||||||
|
info("installing NVIDIA driver via 'ubuntu-drivers install'")
|
||||||
|
r = as_root(["ubuntu-drivers", "install"], timeout=900)
|
||||||
|
else:
|
||||||
|
pm = detect_pm()
|
||||||
|
if not pm:
|
||||||
|
return False, "no package manager available for driver install"
|
||||||
|
info("installing NVIDIA driver metapackage 'nvidia-driver'")
|
||||||
|
r = as_root(pm + ["nvidia-driver"], timeout=900)
|
||||||
|
return r["rc"] == 0, (r["out"][-2000:] + r["err"][-1000:])
|
||||||
|
|
||||||
|
|
||||||
def preflight(auto_install=True, do_clone=True):
|
def preflight(auto_install=True, do_clone=True):
|
||||||
step("Preflight")
|
step("Preflight")
|
||||||
report = {"tools": {}, "root": have_root()}
|
report = {"tools": {}, "root": have_root()}
|
||||||
critical_missing, optional_missing = [], []
|
critical_missing, optional_missing = [], []
|
||||||
|
|
||||||
|
rel = os_release()
|
||||||
|
report["os"] = {"id": rel.get("ID", ""), "version_id": rel.get("VERSION_ID", ""),
|
||||||
|
"pretty": rel.get("PRETTY_NAME", "unknown")}
|
||||||
|
info(f"OS: {report['os']['pretty']}")
|
||||||
|
|
||||||
for tool, (level, why) in TOOL_CHECKS.items():
|
for tool, (level, why) in TOOL_CHECKS.items():
|
||||||
path = which(tool)
|
path = which(tool)
|
||||||
report["tools"][tool] = {"found": bool(path), "path": path, "level": level, "why": why}
|
report["tools"][tool] = {"found": bool(path), "path": path, "level": level, "why": why}
|
||||||
if not path:
|
if not path:
|
||||||
(critical_missing if level.startswith("critical") else optional_missing).append(tool)
|
(critical_missing if level.startswith("critical") else optional_missing).append(tool)
|
||||||
|
|
||||||
|
want_nvidia = False
|
||||||
if auto_install:
|
if auto_install:
|
||||||
pkgs = sorted({PKG_FOR[t] for t in critical_missing + optional_missing if t in PKG_FOR})
|
pkgs = sorted({PKG_FOR[t] for t in critical_missing + optional_missing if t in PKG_FOR})
|
||||||
if pkgs:
|
if "nvidia-smi" in critical_missing and sys.stdin.isatty():
|
||||||
if detect_pm():
|
if ask("nvidia-smi missing - install the NVIDIA driver now? (y/N)", "n").lower().startswith("y"):
|
||||||
info(f"attempting to install: {' '.join(pkgs)}")
|
want_nvidia = True
|
||||||
_, log = install_packages(pkgs)
|
if pkgs or want_nvidia:
|
||||||
report["install_log"] = log
|
ok_sudo, why = sudo_precondition()
|
||||||
|
report["sudo"] = {"ok": ok_sudo, "why": why}
|
||||||
|
if not ok_sudo:
|
||||||
|
err(f"sudo PRECONDITION FAILED: {why}")
|
||||||
|
err("--preflight package installation requires sudo privileges. Fix one of:")
|
||||||
|
err(" - run 'sudo -v' first, then re-run this command")
|
||||||
|
err(" - run the script as root")
|
||||||
|
err(" - re-run with --no-install to skip package installation")
|
||||||
|
report["install_skipped"] = why
|
||||||
else:
|
else:
|
||||||
report["install_log"] = "no package manager - manual install required"
|
info(f"sudo precondition OK: {why}")
|
||||||
warn(f"no package manager found - install manually: {' '.join(pkgs)}")
|
pm = detect_pm()
|
||||||
for tool in list(critical_missing) + list(optional_missing):
|
if pkgs:
|
||||||
if which(tool):
|
if pm and pm[0] == "apt-get":
|
||||||
if tool in critical_missing:
|
info("apt-get update ...")
|
||||||
critical_missing.remove(tool)
|
as_root(["apt-get", "update"], timeout=300)
|
||||||
if tool in optional_missing:
|
if pm:
|
||||||
optional_missing.remove(tool)
|
info(f"installing: {' '.join(pkgs)}")
|
||||||
report["tools"][tool].update(found=True, path=which(tool))
|
_, log = install_packages(pkgs)
|
||||||
|
report["install_log"] = log
|
||||||
|
else:
|
||||||
|
report["install_log"] = "no package manager - manual install required"
|
||||||
|
warn(f"no package manager found - install manually: {' '.join(pkgs)}")
|
||||||
|
if want_nvidia:
|
||||||
|
if pm or which("ubuntu-drivers"):
|
||||||
|
ok_d, log_d = install_nvidia_driver()
|
||||||
|
report["nvidia_driver"] = {"ok": ok_d, "log": log_d}
|
||||||
|
if ok_d:
|
||||||
|
ok("NVIDIA driver installed - reboot, then re-run the check")
|
||||||
|
else:
|
||||||
|
warn("NVIDIA driver install failed (see report['nvidia_driver']['log'])")
|
||||||
|
else:
|
||||||
|
warn("no package manager - cannot install NVIDIA driver automatically")
|
||||||
|
for tool in list(critical_missing) + list(optional_missing):
|
||||||
|
if which(tool):
|
||||||
|
if tool in critical_missing:
|
||||||
|
critical_missing.remove(tool)
|
||||||
|
if tool in optional_missing:
|
||||||
|
optional_missing.remove(tool)
|
||||||
|
report["tools"][tool].update(found=True, path=which(tool))
|
||||||
|
else:
|
||||||
|
report["sudo"] = {"ok": True, "why": "nothing to install"}
|
||||||
|
|
||||||
if do_clone:
|
if do_clone:
|
||||||
ensure_repos()
|
ensure_repos()
|
||||||
@@ -216,16 +300,22 @@ def preflight(auto_install=True, do_clone=True):
|
|||||||
mark = f"{C.YELLOW}missing (optional){C.RESET}"
|
mark = f"{C.YELLOW}missing (optional){C.RESET}"
|
||||||
print(f" {t:<14} {mark} {C.DIM}{st['why']}{C.RESET}")
|
print(f" {t:<14} {mark} {C.DIM}{st['why']}{C.RESET}")
|
||||||
|
|
||||||
if "nvflash" in critical_missing:
|
if not which("nvflash"):
|
||||||
warn("nvflash missing - VBIOS backup will fall back to sysfs ROM, serial to manual entry")
|
warn("nvflash missing - VBIOS backup will fall back to sysfs ROM, serial to manual entry")
|
||||||
print(" Download Linux x64 nvflash from https://www.nvidia.com/en-us/drivers/nvflash/")
|
print(" Download Linux x64 nvflash from https://www.nvidia.com/en-us/drivers/nvflash/")
|
||||||
print(" and pass --nvflash /path/to/nvflash, or place it on PATH.")
|
print(" and pass --nvflash /path/to/nvflash, or place it on PATH.")
|
||||||
if "nvidia-smi" in critical_missing:
|
if not which("nvidia-smi"):
|
||||||
warn("nvidia-smi missing - cmpunlocker verify will be skipped (install nvidia-open driver)")
|
warn("nvidia-smi missing - cmpunlocker verify will be skipped (install nvidia-open driver)")
|
||||||
if not report["root"]:
|
if not report["root"]:
|
||||||
warn("no sudo and not root: privileged steps (rescan, retrain, rom, dmesg) will be skipped")
|
warn("no sudo and not root: privileged steps (rescan, retrain, rom, dmesg) will be skipped")
|
||||||
|
|
||||||
ok("preflight complete")
|
blocking = [t for t in ("git", "lspci", "setpci") if not which(t)]
|
||||||
|
report["blocking_missing"] = blocking
|
||||||
|
report["preflight_ok"] = not blocking
|
||||||
|
if blocking:
|
||||||
|
err("blocking tools still missing: " + ", ".join(blocking))
|
||||||
|
else:
|
||||||
|
ok("preflight complete")
|
||||||
return report
|
return report
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user