cmp170hx_check/README.md

# cmp170hx_check

Repo: https://git.qbuissondebon.eu.org/qbuissondebon/cmp170hx_check (SSH push: `git@git.ssh.qbuissondebon.eu.org:qbuissondebon/cmp170hx_check.git`)

Single-card checker for **NVIDIA CMP 170HX** cards (PCI IDs `10de:20c2` / `10de:2082`) — run it **one card at a time**: it waits for the card, verifies the PCIe capacitor mod, records PCIe bus health, backs up VBIOS, runs the cmpunlocker verification, offers the **opt-in** cmpunlocker unlock install (then a cold reboot and a re-run for the final PCIe check), and optionally stresses VRAM. Each card gets its own folder + JSON report named by GPU UUID, so the folders accumulate as you go through your batch (e.g. 23 cards).

Reference harness (auto-cloned into `harness/` by preflight):

- [170th-Street](https://github.com/amoghmunikote/170th-Street) — community docs, incl. `modifications/pcie-capacitor-mod.md` (the pass criteria: `LnkSta Width x16` after the mod, `x4` when missing)
- [cmpunlocker](https://github.com/amoghmunikote/cmpunlocker) — `verify.sh` is executed **read-only** per card; `install.sh` is only run from the separate, opt-in unlock step (see [Safety](#safety))

## Requirements

- Linux x86-64 (tested target: **Ubuntu 26.04**), Python 3.8+
- **sudo privileges** — precondition for `--preflight` package installation, for privileged checks (PCI rescan, link retrain, sysfs ROM, `dmesg`, `verify.sh`), and for the opt-in unlock install
- `pciutils` (`lspci`, `setpci`), `git` — installed automatically by preflight
- NVIDIA driver with `nvidia-smi` (nvidia-open, version list in cmpunlocker `driver/VERSION` — 610.xx / 615.xx) for unlock verification — preflight can install it for you (asks first)
- `nvflash` — **bundled in this repo** (`nvflash_5.867_linux/x64/nvflash`), used directly by the script; no system install needed (see [nvflash](#nvflash-bundled-in-this-repo))
- Optional: `vkmark` or `vulkan-tools` for the 5-minute VRAM stress test

## Preflight (setup step)

```bash
python3 check_cards.py --preflight
```

On Ubuntu 26.04 this:

1. **Checks the sudo precondition** — runs `sudo -v` if needed so credentials are cached; if sudo is unavailable it prints the fix and **does not install anything** (exit code 3 if blocking tools are still missing)
2. Installs required packages via `apt-get update && apt-get install`: `git`, `pciutils`, `vulkan-tools`, `vkmark`
3. Offers to install the **NVIDIA driver** (`ubuntu-drivers install`, falling back to the `nvidia-driver` metapackage) when `nvidia-smi` is missing — reboot afterwards
4. Clones the two harness repos into `harness/`
5. Prints a tool status table — the bundled `nvflash` is detected automatically and its path is shown

Skip installation with `--no-install` (no sudo needed at all).

## nvflash (bundled in this repo)

The Linux build of NVIDIA's `nvflash` 5.867 is committed to this repo at
`nvflash_5.867_linux/x64/nvflash` (plus `aarch64`/`ppc64`/`x86` variants) — the script
uses it **directly**, no system-wide installation required.

Binary resolution order:

1. `--nvflash /path/to/nvflash` (explicit override)
2. repo copy: `nvflash_*_linux/<arch>/nvflash`, then `./nvflash`
3. `nvflash` on `PATH` (optional system install)

Preflight reports `nvflash OK` and prints the resolved path.

```bash
python3 check_cards.py --preflight   # shows: nvflash OK / nvflash: <repo>/nvflash_5.867_linux/x64/nvflash
```

Notes:

- nvflash requires root: the script invokes it via `sudo` — cache credentials with `sudo -v` first, or run the script as root.
- Only `--save` (read-only backup) is used; if it fails while the NVIDIA driver is bound, the script automatically falls back to the sysfs ROM method.
- Do **not** use the AUR nvflash package (known broken), and ignore Windows `.exe` builds.

Optional — install it system-wide anyway:

```bash
sudo install -m 755 nvflash_5.867_linux/x64/nvflash /usr/local/bin/nvflash
nvflash --version
sudo nvflash --list        # must show the 170HX
```

Refreshing the bundled copy: download `nvflash_5.867_linux.zip` from
https://www.techpowerup.com/download/nvidia-nvflash/ , verify SHA256
`73ab51e121129059f1377b3bf21a8001f6649d64af2471455499c803a9c80301`, extract and replace
the files (keep the executable bit: `chmod +x`).

## Usage (one card per run)

```bash
python3 check_cards.py               # check the connected card
python3 check_cards.py --preflight   # setup: tools, packages, harness clone
python3 check_cards.py --demo        # self-test with fake hardware (no GPU needed)
python3 check_cards.py --unlock      # check + run the cmpunlocker install without asking
python3 check_cards.py --help        # all flags
```

Flow of a normal run:

1. Waits for a CMP 170HX on the bus — prompts you to insert/reseat the card, Enter triggers a PCI rescan (`q` quits)
2. Folder `cards/<GPU-UUID>/` from the auto-read GPU UUID (`nvidia-smi`); physical serial probed from nvflash and saved to `serial.txt` (`N/A` when the card exposes none) — never prompts
3. VBIOS backup → `cards/<GPU-UUID>/bios.rom` (nvflash, sysfs ROM fallback). On failure the script reports the precise cause (driver-loaded vs missing rom node) and offers an opt-in driver unload → save → reload retry (`Unload NVIDIA driver and retry nvflash --save now? (y/N)`, default **n**; `--bios-unload` runs it without asking)
4. PCIe checks: link speed/width, AER counters, full config-space dump
5. Link retrain test + full bus rescan test (device must survive both)
6. Kernel log scrape (`dmesg`/`journalctl`) for this BDF: Xid / AER / NVRM lines
7. `nvidia-smi` snapshot + `harness/cmpunlocker/verify.sh` (read-only)
8. **Opt-in unlock**: `cmpunlocker unlock (install)` — skipped with a note when memory is already unlocked, otherwise asks `Run cmpunlocker install.sh now? (y/N)` (default **n**; `--unlock` runs it without asking). On yes: `git pull --ff-only` on the harness clone, then `install.sh` runs from `harness/cmpunlocker/` with **live terminal output** (full log → `cards/<GPU-UUID>/install_unlocker.log`; the kernel-module build takes 5–15 min)
9. **After a successful install**: the VRAM stress is skipped — **cold reboot the host** (`sudo shutdown -h now`, then power it back on) and **re-run the same command** for the final PCIe check, which must report **Gen2 x16**
10. **Last step, optional**: `Run 5-min Vulkan VRAM stress test now? (y/N)` — refused by default, `--vulkan always|never` to override
11. Result printed + `report.json` written into the card's folder

Remove the card, plug the next one, repeat — no queue, no indices, folders just pile up in `cards/`.

## Checks & scoring (lenient)

| Result | Criteria |
|--------|----------|
| **FAIL** | no device detected, link down/untrained, device lost after rescan, link failed retrain, active uncorrectable AER errors, **final post-unlock link other than Gen2 x16** (`[ERROR] final PCIe link after unlock is <speed> <width> - expected Gen2 x16`) |
| **WARN** | link width ≠ `x16` (capacitor mod missing/incomplete, pre-unlock runs only), Gen1 only (Gen2 unlock patch not active, pre-unlock runs only), correctable AER errors, kernel errors / Xid, VBIOS backup failed, cmpunlocker verify not `unlocked-ok`, cmpunlocker install failed, errors during VRAM stress |
| **PASS** | everything clean |

Exit codes: `0` PASS/WARN · `1` FAIL · `2` no card found · `3` preflight blocking failure.

## Output

```
cards/
└── <GPU-UUID>/
    ├── serial.txt            # physical serial, or N/A
    ├── bios.rom              # VBIOS backup (+ sha256 in report)
    ├── report.json           # full result: link, AER before/after, retrain,
    │                          #   rescan, dmesg, verify, vulkan, PASS/WARN/FAIL
    ├── lspci.txt             # lspci -vvv (before)
    ├── lspci_after.txt       # lspci -vvv (after retrain/rescan)
    ├── config_space.hex      # lspci -nnvvxxxx full hex dump
    ├── dmesg.log             # kernel lines related to this card
    ├── verify_unlocker.log   # cmpunlocker verify.sh output
    └── install_unlocker.log  # full install.sh output (only if the opt-in
                              #   unlock step ran)
```

## Flags

| Flag | Meaning |
|------|---------|
| `--preflight` | setup only: sudo precondition + package install + harness clone, then exit |
| `--demo` | run the whole pipeline against fake hardware (self-test) |
| `--nvflash PATH` | explicit nvflash binary |
| `--serial XYZ` | override physical serial recorded in `serial.txt` (folder stays UUID-named) |
| `--vulkan ask\|always\|never` | VRAM test policy (default: ask, last step) |
| `--unlock` | run the cmpunlocker `install.sh` step without prompting (default: ask y/N) |
| `--skip-bios` | skip VBIOS backup |
| `--bios-unload` | unload NVIDIA driver to retry `nvflash --save` without prompting (console may freeze briefly, driver is reloaded after) |
| `--no-install` | never install packages (skips the sudo precondition) |
| `--no-clone` | don't clone the harness repos |

## Safety

This tool **reads** card state by default. Two steps can change host state, both **opt-in**:

- **Unlock install (opt-in, explicit confirmation)** — runs only if you answer `y` to `Run cmpunlocker install.sh now? (y/N)` (default `n`) or pass `--unlock`. It executes `install.sh` **from the harness clone** (`harness/cmpunlocker/`, refreshed first with `git pull --ff-only`, no extra installer flags), needs root (sudo credentials are cached first), streams its output live, and saves the full log to `cards/<GPU-UUID>/install_unlocker.log`. It is skipped entirely when memory is already unlocked, when the harness is not cloned, and in `--demo` mode. A successful install replaces kernel modules — the step then skips VRAM stress and asks you to cold reboot and re-run.
- **VBIOS unload-retry (opt-in, explicit confirmation)** — offered only after both backup methods fail with the driver loaded: unloads the NVIDIA modules, retries `nvflash --save`, then reloads the modules (local console may freeze briefly). Triggered by `y` or `--bios-unload`; skipped in `--demo` mode.
- **Everything else remains read-only**: `verify.sh` runs without installing anything, no modules are patched and no firmware is flashed (nvflash is only ever used with `--save`). The only other writes are local files (reports, logs), PCI link retrain/rescan sysfs triggers, and package installation during `--preflight` (explicitly gated behind the sudo precondition).