havoc-remaster/docs/HANDOFF.md

# Havoc Remaster - Session Handoff

Read this first. It is a self-contained bootstrap for continuing the Havoc reverse-engineering
and remaster work in a fresh session (including on Linux). It assumes no prior conversation
memory. Companion docs: `docs/PATCHES.md`, `docs/FORMATS.md`, `docs/METHODOLOGY.md`,
`docs/WEAPONS.md`.

Last updated at the end of the session that got the game booting and cracked GRAFIX.

---

## 1. What this project is

Reviving **Havoc** (Reality Bytes, Inc., 1995): a first-person, **software-rendered** 3D
vehicular-combat shooter for Win95/98. The binaries are **32-bit PE (i386), DirectDraw +
DirectSound, no Direct3D** (the 3D is a software rasterizer). It is NOT DOS/16-bit, so DOSBox is
the wrong tool.

**End goal:** port to **Godot** as a remaster with QoL (widescreen, higher-res, rebindable
input, fixed-timestep physics). Three planned variants:
1. Faithful 1:1 (original bugs preserved as an option, including FPS-coupled cockpit physics).
2. Remaster (bug fixes, widescreen, rebindable input, higher-res).
3. Roguelite (procedural worlds, extra tanks) as a separate mode.

**Two active work streams for the next sessions:**
- **A. Ghidra decompilation** of `HAVOC.EXE` to recover game logic (weapons, AI, physics, cheats,
  the FPS-coupling, level/entity loaders).
- **B. Asset cracking** of the remaining `WORLDS/` and `.FF` formats.

The user wants a **detailed blog post** of the whole RE process, so when you crack something,
write down the *discovery steps and dead ends*, not just the final spec. Keep notes in `docs/`.

---

## 2. Current state (what works / what is done)

- **The game boots and is fully playable on Windows 11** via dgVoodoo2 (DirectDraw wrapper,
  windowed). Loads levels, enemies shoot, etc. This was the big unlock this session.
- **NOCD + copy-protection fix:** `HAVOC_NOCD.EXE` is the patched original (28 patches). The
  final blocker was a copy-protection check in the data-file loader `0x42a480` that only accepted
  files loaded from the CD drive; a 14-byte patch makes it accept the local files. Full write-up
  and patch table in `docs/PATCHES.md`.
- **Formats cracked:** `.FF` archives, MUSIC.FF (17 songs + FL Studio pack), INTRFACE.FF bitmaps,
  and now **GRAFIX0N.DAT** (64x64 texture tilesets, `tools/grafix_extract.py`). See
  `docs/FORMATS.md`.
- **MAP layout partially decoded:** `MAP0101.MAP` is a 2-byte-per-cell grid, ~192x144 (384-byte
  rows, found via row autocorrelation). Cell semantics (which byte is tile vs height) still TBD.
- **FPS-coupling: root cause found.** It's not "no timestep at all" -- there's a real
  delta-time system (`compute_frame_delta_ticks_60hz_min1_clamp` @ `0x4092D0`, converts
  `timeGetTime()` to 60Hz ticks correctly), but it clamps its minimum return value to 1 tick.
  Above 60fps, less real time elapses between calls than one tick, so the clamp still credits
  a full tick -- simulation speed scales up in deterministic, linear proportion to render
  rate above that threshold (confirmed against the user's own description: "does not break
  ... deterministic ... behaves ... 'FASTER'"). Below/at 60fps the delta is accurate, which
  is exactly why dgVoodoo `FPSLimit=30` fixes it. Whether this differs between cockpit and
  minimal-HUD modes is still unconfirmed -- the functions found so far gate only on game
  state, not HUD mode. Full chase incl. two dead ends in `docs/METHODOLOGY.md` section 10;
  function-by-function status in `docs/FUNCTIONS.md`.

---

## 3. Environment (Windows -> Linux)

The repo lives on **navi** at `/Library/Development/devl/Havoc` (mounted on the Windows box as
`Z:\Development\devl\Havoc`). On Linux you work directly in that path.

Tooling used this session and how it maps to Linux:

| Purpose | Windows (this session) | Linux equivalent |
|---|---|---|
| Python | `C:/Python313/python.exe` | `python3` |
| Disassembly | `radare2` + `r2pipe` (pip) | same: `apt/pacman install radare2`, `pip install r2pipe` |
| Image output | Pillow (`pip install pillow`) | same |
| Decompiler | Ghidra 12.1.2 GUI (project at `tools/ghidra_project/`) | Ghidra 12.1.2 headless (see below) - now set up |
| Run the game | dgVoodoo2 on native Windows | Wine + dgVoodoo (test only; not needed for static work) |

**Windows-only tools that do NOT port (and are no longer needed):** the crash diagnosis used WER
minidumps from `%LOCALAPPDATA%\CrashDumps` parsed in pure Python, and a `ctypes` Win32 debugger
(`tools/dbg_path.py`). Those were for the boot crash, which is solved. Decompilation and asset
cracking are pure static analysis and run fine on Linux.

All the extraction tools (`ff_extract.py`, `grafix_extract.py`) are cross-platform Python
(they use `os.path`, avoid Windows-isms).

**Note:** the game itself only runs on Windows (or Wine). Linux sessions are for Ghidra +
asset extraction, which need only the files, not a running game.

**Ghidra is now installed on Linux (`Heimdall`, headless, no sudo needed):**
```sh
export JAVA_HOME=~/tools/jdk-21.0.11+10
export PATH=$JAVA_HOME/bin:$PATH
~/tools/ghidra_12.1.2_PUBLIC/support/analyzeHeadless \
  /Library/Development/devl/Havoc/tools/ghidra_project HAVOC \
  -process HAVOC_NOCD.EXE -noanalysis \
  -scriptPath /Library/Development/devl/Havoc/tools/ghidra_scripts \
  -postScript DecompileHavoc.java
```
`tools/ghidra_project/` is the **existing, real, previously-analyzed project** from an
earlier Windows/GUI session (gitignored, so invisible to `git status` - check the
filesystem). Do not create a new project. Full setup story (dead ends, the `NotOwnerException`
fix) is in `docs/METHODOLOGY.md` section 9. `tools/ghidra_scripts/DecompileHavoc.java` is a
working headless script that dumps decompiled C for a curated VA list to
`tools/ghidra_decompile.txt` - extend its `TARGET_VAS` array with new addresses as the
section 7 targets below get investigated. No GUI/display is available on this box; all
Ghidra work here is headless-only (scripting + batch decompile), not interactive
click-through.

**Resolved:** the project's binary was initially found to be a stale, superseded NOCD patch
iteration (MD5 mismatch vs. the current `HAVOC_NOCD.EXE`). Confirmed no manual annotation
existed to lose (0 of 987 functions user-named), so it was re-imported with `-overwrite` and
fully re-analyzed against the current binary. `PrintProgramMD5.java` (in `tools/
ghidra_scripts/`) confirms the project's `EXECUTABLE_MD5` now matches `md5sum
HAVOC_NOCD.EXE` exactly. `tools/ghidra_decompile.txt` has been regenerated against the
verified-current, working NOCD build. If `HAVOC_NOCD.EXE` gets repatched again, re-run the
`-overwrite` import (see `docs/METHODOLOGY.md` section 9 point 6) before trusting decompiled
output.

---

## 4. Repo, git, and constraints

- **Remote:** Forgejo, private repo `git.opensourcesolarpunk.com/pyr0ball/havoc-remaster`
  (personal `pyr0ball` account, not the CircuitForge org). Origin is already configured.
- **Pushing:** the user pushes on request. Token lives in a local `.env.mirror` file (outside the
  repo) as `FORGEJO_TOKEN` - **never print it, never store it in `.git/config`**. Push via
  `git -c http.extraheader="Authorization: token $TOKEN" push` for a one-time push.
- **Commits:** solo repo, history is on `main`. Commit style: conventional prefixes
  (`feat:`, `docs:`, `chore:`). Every commit message ends with the two trailer lines the harness
  provides (Co-Authored-By + Claude-Session).

**Hard constraints (do not violate):**
- **Never commit original game data or binaries** (copyright, Reality Bytes 1995):
  `*.EXE *.DLL *.FF *.DAT *.MAP *.INI *.cue *.bin`, `BIGFILE.DAT`, `WORLDS/`, `tmp_cd/`, the
  Ghidra install/project, and derived audio/art are all in `.gitignore`. When staging, use
  explicit paths (not `git add -A`) and sanity-check `git diff --cached --name-only`.
- **No emdashes in any output** (user style preference). Use plain hyphens.
- Document discovery process for the blog (see section 1).

---

## 5. The binary: layout and key addresses

`HAVOC_NOCD.EXE` = patched `HAVOC.EXE`, 485,376 bytes, 32-bit PE i386, image base `0x400000`,
loads at preferred base (so runtime VAs == static VAs).

**Section file-offset (FO) to virtual-address (VA):**

| Section | Raw FO | RVA | FO -> VA |
|---|---|---|---|
| `.text`  | 0x00400 | 0x01000 | `VA = FO + 0x400C00` |
| `.rdata` | 0x67000 | 0x79000 | `VA = FO + 0x412000` |
| `.data`  | 0x69400 | 0x7C000 | `VA = FO + 0x412C00` |
| `.idata` | 0x6EC00 | 0x82000 | `VA = FO + 0x413400` |

**Control-flow map (verified this session):**

| VA | What it is |
|---|---|
| `0x464657` | CRT entry point |
| `0x407C90` | WinMain. Allocs game object (vtable `0x4791A0`), calls `0x434040` (init); on success calls `vtable[7]` = `0x43AC40` (main loop) |
| `0x434040` | Main init: allocates ~30 subsystem objects, loads title/GAMIMAGE, palette fade. Calls `0x418600`. Returns AX=0 on success |
| `0x418600` | DirectDraw + archives + sound init. **GRAFIX/world files load during this** |
| `0x43AC40` | **MAIN GAME LOOP** (vtable[7]). State machine; state byte at `[this+0x18]`; loops while `0x10 <= state < 0xf0`. Per-frame calls `vtable[8]=0x436170`, `vtable[17]=0x43adc0` (msg pump), `vtable[9]=0x4361d0`. Exits when state = 0xf0. **No frame limiter here = FPS-coupled** |
| `0x4791A0` | Game object vtable (0x14-entry) |
| `0x42a480` | WORLDS\ data-file loader (the copy-protection function, now patched at FO 0x29a2b) |
| `0x439380` | Stream open method: `CreateFileA([this+4], ...)`, vtable `0x479af8` |
| `0x444db0` | GRAFIX0N.DAT loader (byte-swaps BE records; template for reading other WORLDS loaders) |
| `0x40C8E0` | Busy-wait frame limiter (~60/sec via timeGetTime), used in palette-fade / state transitions (`0x40CAE0`), NOT in the gameplay loop |
| `0x40b5b0` | Win32 WndProc dispatch (msg 2 WM_DESTROY -> `0x40b680` sets state=0xf0 + PostQuitMessage; focus msgs 7/0x1c handled) |
| `0x409FC0` | DirectSound init |
| `0x42D970` | show_error_dialog (patched to `RET` so failures do not popup) |
| `0x4824E4` | timeGetTime IAT; `[0x47e56c]` holds a 60Hz tick baseline (set once, not used as a per-frame delta) |

Reproduce any disassembly with r2:
```sh
python3 -c "import r2pipe; r=r2pipe.open('HAVOC_NOCD.EXE',['-e','bin.relocs.apply=true']); print(r.cmd('pd 60 @ 0x43ac40'))"
```

---

## 6. File formats: status

Cracked (see `docs/FORMATS.md` for specs and `tools/` for extractors):
- `.FF` "FlashFile" archive (`tools/ff_extract.py`).
- MUSIC.FF: 97 WAV segments + 17 PLST-sequenced songs + FL Studio pack.
- INTRFACE.FF: bitmap system (BitB/BitR headers, XPAL palettes, STR# string tables).
- **GRAFIX0N.DAT** (`tools/grafix_extract.py`): BE header (size@0x08, count@0x0c), 16-byte entry
  records (BE offset + id), per-graphic blocks (BE w/h/frames/id + 8-bit indexed pixels). 64x64.

TODO (the asset-cracking stream):
- **MAP<lvl>.MAP**: 2-byte-per-cell grid, ~192x144 confirmed. Next: disassemble its loader to get
  exact dims + which byte is tile vs height. Find the loader by searching for the code that opens
  "MAP" filenames (same pattern as GRAFIX via `0x42a480`).
- **STUF<lvl>.DAT**: entity placement. Contains ASCII `"Item"` records - very parseable. Start
  here for a quick win.
- **LAND<lvl>.DAT**: terrain geometry (hardest). BE fields; music-track id is a BE u16 at offset
  0x12 (per prior notes). Has 4-byte records around 0x20 (`17 70 00 c0` ...).
- **OBJECTS.FF**: 3D models (vehicles, enemies, pickups) - not yet touched.
- **World palette:** GRAFIX tiles currently render with a UI XPAL palette. The true per-world 3D
  palette is TBD - likely loaded during `0x418600` init or embedded per-world. Finding it makes
  all extracted art color-correct.

---

## 7. Ghidra decompilation stream (primary Linux task)

Load `HAVOC.EXE` (or `HAVOC_NOCD.EXE`) into Ghidra (x86 32-bit, image base 0x400000). Import the
key addresses from section 5 as labels to bootstrap. High-value targets, roughly in order:

1. **The FPS-coupling / render modes.** The user reports cockpit mode is FPS-coupled but the
   minimal HUD mode is not. Find the mode toggle and the two render/update paths. Determine
   whether minimal mode uses a frame limiter (Sleep/timeGetTime) or a fixed-timestep/delta update.
   That path is the template for the remaster's fixed-timestep, and possibly a cleaner in-binary
   fix than the dgVoodoo FPS cap. Start from the main loop `0x43AC40` and its per-frame methods
   `0x436170` / `0x4361d0`, and the busy-wait limiter `0x40C8E0`.
2. **Cheat code handler.** Codes `mmm` (max heatseeker), `vvv` (free vehicle), `sss` (shield),
   `hhh` (health), `aaa` (ammo). Find the keyboard-buffer matcher; it will reveal the full cheat
   list and the player/inventory structs.
3. **Weapon system.** 6 primary classes (Laser, Disc, Shockwave, Fireball, Plasma, VARG) infinite
   ammo; 14 secondary lines (see `docs/WEAPONS.md`, from string tables STR#7D02/7D03). Recover the
   weapon data tables, upgrade tiers, and firing logic.
4. **Player / vehicle physics** (movement constants, the FPS-coupled step) and **enemy AI**.
5. **Level + entity loaders** (MAP/LAND/STUF) - decompiling these also directly finishes the
   asset-cracking stream (get exact record layouts instead of guessing).
6. **World table**: 6 worlds stored in reverse (Overlord, Wasteland, Malterra, Tyrak, Fallout,
   Badlands), 3 levels + boss + bonus each. See `docs/FORMATS.md` and game-mechanics notes.

---

## 8. Methodology playbook (how the cracking was done)

- **Crack a format via its loader, not guesswork.** Find where the game opens the file
  (filenames are in `.data`; the loader `0x42a480` builds `WORLDS\<name>` paths), then read the
  loader's parse code. GRAFIX was solved by reading `0x444db0` (the 16-byte record stride and
  BE->LE byte-swap gave the record layout exactly).
- **These WORLDS files are big-endian** (Mac-origin data; the loaders byte-swap). Always try BE
  first for headers/records.
- **Find grid dimensions** by row autocorrelation (minimize mean abs diff between byte j and
  j+stride over candidate strides); then render grayscale at that width and eyeball for coherent
  structure. That is how MAP's 384-byte row (192x2) was found.
- **Verify visually.** Render extracted pixel data to PNG and build contact sheets; correct
  decodes look like real art. Pillow `Image.frombytes('P', (w,h), data); putpalette(...)`.
- **r2pipe pattern** for scanning: open with `bin.relocs.apply=true`, iterate `.text` FOs looking
  for opcode patterns (e.g. `ff 15` CALL [IAT], `a1`/`8b 05` mem reads) and convert FO->VA.

---

## 9. Tool inventory (`tools/`)

- `ff_extract.py` - `.FF` archive extractor (archives, songs, FL pack). `--help` for flags.
- `grafix_extract.py` - GRAFIX0N.DAT -> PNG (`-p <768-byte palette>`).
- `dbg_path.py` - Windows-only ctypes debugger (used for the boot crash; keep for reference).
- Many `r2*.py` / `find_*.py` / `analyze*.py` - one-off RE probes from the crash hunt and format
  work. Useful as r2pipe examples; not all are load-bearing.
- `run_probe.bat` / `run_and_log.bat` - Windows launch helpers (repro/testing on Windows only).

Palettes: extract with `python3 tools/ff_extract.py INTRFACE.FF -o tools/out_intrface`, palettes
land in `tools/out_intrface/raw/XPAL*.DAT` (768-byte RGB).

---

## 10. Immediate next steps (suggested)

1. (Asset, quick win) Crack `STUF<lvl>.DAT` entity records (ASCII `"Item"` markers).
2. (Asset) Disassemble the MAP loader to lock MAP dims + cell meaning; render a proper level map.
3. (Ghidra) Stand up the project, label the section-5 addresses, and chase the render-mode /
   FPS-coupling path (item 7.1) - it informs both the exe fix and the Godot physics.
4. (Remaster) Once STUF + MAP + a world palette are done, there is enough to scaffold the Godot
   project (empty today) and load a real level.

The FPS workaround (dgVoodoo `FPSLimit = 30`) is a stopgap; the user is tuning the value. A proper
fix is either a frame limiter patched into `0x43AC40` (the game imports `Sleep`) or, better,
adopting the minimal-mode timing path once Ghidra reveals it.