docs: add self-contained session handoff for continuing on Linux
docs/HANDOFF.md bootstraps a fresh session (no prior conversation memory): project goal + variants, current state (game boots, formats cracked), the Windows->Linux tooling map, repo/git constraints (copyright, no game data), the binary's section FO<->VA formulas and verified control-flow address map, format status, the Ghidra decompilation targets (FPS-coupling/render modes, cheats, weapons, AI, level loaders), the format-cracking methodology, and suggested next steps. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TN4Ytn3gdWRonNmHpisWQv
This commit is contained in:
parent
198977ed2b
commit
68cf36a3d0
1 changed files with 237 additions and 0 deletions
237
docs/HANDOFF.md
Normal file
237
docs/HANDOFF.md
Normal file
|
|
@ -0,0 +1,237 @@
|
|||
# Havoc Remaster - Session Handoff
|
||||
|
||||
Read this first. It is a self-contained bootstrap for continuing the Havoc reverse-engineering
|
||||
and remaster work in a fresh session (including on Linux). It assumes no prior conversation
|
||||
memory. Companion docs: `docs/PATCHES.md`, `docs/FORMATS.md`, `docs/METHODOLOGY.md`,
|
||||
`docs/WEAPONS.md`.
|
||||
|
||||
Last updated at the end of the session that got the game booting and cracked GRAFIX.
|
||||
|
||||
---
|
||||
|
||||
## 1. What this project is
|
||||
|
||||
Reviving **Havoc** (Reality Bytes, Inc., 1995): a first-person, **software-rendered** 3D
|
||||
vehicular-combat shooter for Win95/98. The binaries are **32-bit PE (i386), DirectDraw +
|
||||
DirectSound, no Direct3D** (the 3D is a software rasterizer). It is NOT DOS/16-bit, so DOSBox is
|
||||
the wrong tool.
|
||||
|
||||
**End goal:** port to **Godot** as a remaster with QoL (widescreen, higher-res, rebindable
|
||||
input, fixed-timestep physics). Three planned variants:
|
||||
1. Faithful 1:1 (original bugs preserved as an option, including FPS-coupled cockpit physics).
|
||||
2. Remaster (bug fixes, widescreen, rebindable input, higher-res).
|
||||
3. Roguelite (procedural worlds, extra tanks) as a separate mode.
|
||||
|
||||
**Two active work streams for the next sessions:**
|
||||
- **A. Ghidra decompilation** of `HAVOC.EXE` to recover game logic (weapons, AI, physics, cheats,
|
||||
the FPS-coupling, level/entity loaders).
|
||||
- **B. Asset cracking** of the remaining `WORLDS/` and `.FF` formats.
|
||||
|
||||
The user wants a **detailed blog post** of the whole RE process, so when you crack something,
|
||||
write down the *discovery steps and dead ends*, not just the final spec. Keep notes in `docs/`.
|
||||
|
||||
---
|
||||
|
||||
## 2. Current state (what works / what is done)
|
||||
|
||||
- **The game boots and is fully playable on Windows 11** via dgVoodoo2 (DirectDraw wrapper,
|
||||
windowed). Loads levels, enemies shoot, etc. This was the big unlock this session.
|
||||
- **NOCD + copy-protection fix:** `HAVOC_NOCD.EXE` is the patched original (28 patches). The
|
||||
final blocker was a copy-protection check in the data-file loader `0x42a480` that only accepted
|
||||
files loaded from the CD drive; a 14-byte patch makes it accept the local files. Full write-up
|
||||
and patch table in `docs/PATCHES.md`.
|
||||
- **Formats cracked:** `.FF` archives, MUSIC.FF (17 songs + FL Studio pack), INTRFACE.FF bitmaps,
|
||||
and now **GRAFIX0N.DAT** (64x64 texture tilesets, `tools/grafix_extract.py`). See
|
||||
`docs/FORMATS.md`.
|
||||
- **MAP layout partially decoded:** `MAP0101.MAP` is a 2-byte-per-cell grid, ~192x144 (384-byte
|
||||
rows, found via row autocorrelation). Cell semantics (which byte is tile vs height) still TBD.
|
||||
- **Known open gameplay issue:** movement/physics is **FPS-coupled** (one sim step per rendered
|
||||
frame, no fixed timestep). Workaround in place: dgVoodoo `FPSLimit` set to 30 (tunable). The
|
||||
user's memory says the in-game "minimal" HUD mode was NOT FPS-coupled while "cockpit" mode was,
|
||||
which suggests a limiter/delta path already exists in the binary for one mode. Finding that in
|
||||
Ghidra is a priority (see section 7).
|
||||
|
||||
---
|
||||
|
||||
## 3. Environment (Windows -> Linux)
|
||||
|
||||
The repo lives on **navi** at `/Library/Development/devl/Havoc` (mounted on the Windows box as
|
||||
`Z:\Development\devl\Havoc`). On Linux you work directly in that path.
|
||||
|
||||
Tooling used this session and how it maps to Linux:
|
||||
|
||||
| Purpose | Windows (this session) | Linux equivalent |
|
||||
|---|---|---|
|
||||
| Python | `C:/Python313/python.exe` | `python3` |
|
||||
| Disassembly | `radare2` + `r2pipe` (pip) | same: `apt/pacman install radare2`, `pip install r2pipe` |
|
||||
| Image output | Pillow (`pip install pillow`) | same |
|
||||
| Decompiler | (not yet used) | **Ghidra** (Java, cross-platform) - the main Linux task |
|
||||
| Run the game | dgVoodoo2 on native Windows | Wine + dgVoodoo (test only; not needed for static work) |
|
||||
|
||||
**Windows-only tools that do NOT port (and are no longer needed):** the crash diagnosis used WER
|
||||
minidumps from `%LOCALAPPDATA%\CrashDumps` parsed in pure Python, and a `ctypes` Win32 debugger
|
||||
(`tools/dbg_path.py`). Those were for the boot crash, which is solved. Decompilation and asset
|
||||
cracking are pure static analysis and run fine on Linux.
|
||||
|
||||
All the extraction tools (`ff_extract.py`, `grafix_extract.py`) are cross-platform Python
|
||||
(they use `os.path`, avoid Windows-isms).
|
||||
|
||||
**Note:** the game itself only runs on Windows (or Wine). Linux sessions are for Ghidra +
|
||||
asset extraction, which need only the files, not a running game.
|
||||
|
||||
---
|
||||
|
||||
## 4. Repo, git, and constraints
|
||||
|
||||
- **Remote:** Forgejo, private repo `git.opensourcesolarpunk.com/pyr0ball/havoc-remaster`
|
||||
(personal `pyr0ball` account, not the CircuitForge org). Origin is already configured.
|
||||
- **Pushing:** the user pushes on request. Token lives in a local `.env.mirror` file (outside the
|
||||
repo) as `FORGEJO_TOKEN` - **never print it, never store it in `.git/config`**. Push via
|
||||
`git -c http.extraheader="Authorization: token $TOKEN" push` for a one-time push.
|
||||
- **Commits:** solo repo, history is on `main`. Commit style: conventional prefixes
|
||||
(`feat:`, `docs:`, `chore:`). Every commit message ends with the two trailer lines the harness
|
||||
provides (Co-Authored-By + Claude-Session).
|
||||
|
||||
**Hard constraints (do not violate):**
|
||||
- **Never commit original game data or binaries** (copyright, Reality Bytes 1995):
|
||||
`*.EXE *.DLL *.FF *.DAT *.MAP *.INI *.cue *.bin`, `BIGFILE.DAT`, `WORLDS/`, `tmp_cd/`, the
|
||||
Ghidra install/project, and derived audio/art are all in `.gitignore`. When staging, use
|
||||
explicit paths (not `git add -A`) and sanity-check `git diff --cached --name-only`.
|
||||
- **No emdashes in any output** (user style preference). Use plain hyphens.
|
||||
- Document discovery process for the blog (see section 1).
|
||||
|
||||
---
|
||||
|
||||
## 5. The binary: layout and key addresses
|
||||
|
||||
`HAVOC_NOCD.EXE` = patched `HAVOC.EXE`, 485,376 bytes, 32-bit PE i386, image base `0x400000`,
|
||||
loads at preferred base (so runtime VAs == static VAs).
|
||||
|
||||
**Section file-offset (FO) to virtual-address (VA):**
|
||||
|
||||
| Section | Raw FO | RVA | FO -> VA |
|
||||
|---|---|---|---|
|
||||
| `.text` | 0x00400 | 0x01000 | `VA = FO + 0x400C00` |
|
||||
| `.rdata` | 0x67000 | 0x79000 | `VA = FO + 0x412000` |
|
||||
| `.data` | 0x69400 | 0x7C000 | `VA = FO + 0x412C00` |
|
||||
| `.idata` | 0x6EC00 | 0x82000 | `VA = FO + 0x413400` |
|
||||
|
||||
**Control-flow map (verified this session):**
|
||||
|
||||
| VA | What it is |
|
||||
|---|---|
|
||||
| `0x464657` | CRT entry point |
|
||||
| `0x407C90` | WinMain. Allocs game object (vtable `0x4791A0`), calls `0x434040` (init); on success calls `vtable[7]` = `0x43AC40` (main loop) |
|
||||
| `0x434040` | Main init: allocates ~30 subsystem objects, loads title/GAMIMAGE, palette fade. Calls `0x418600`. Returns AX=0 on success |
|
||||
| `0x418600` | DirectDraw + archives + sound init. **GRAFIX/world files load during this** |
|
||||
| `0x43AC40` | **MAIN GAME LOOP** (vtable[7]). State machine; state byte at `[this+0x18]`; loops while `0x10 <= state < 0xf0`. Per-frame calls `vtable[8]=0x436170`, `vtable[17]=0x43adc0` (msg pump), `vtable[9]=0x4361d0`. Exits when state = 0xf0. **No frame limiter here = FPS-coupled** |
|
||||
| `0x4791A0` | Game object vtable (0x14-entry) |
|
||||
| `0x42a480` | WORLDS\ data-file loader (the copy-protection function, now patched at FO 0x29a2b) |
|
||||
| `0x439380` | Stream open method: `CreateFileA([this+4], ...)`, vtable `0x479af8` |
|
||||
| `0x444db0` | GRAFIX0N.DAT loader (byte-swaps BE records; template for reading other WORLDS loaders) |
|
||||
| `0x40C8E0` | Busy-wait frame limiter (~60/sec via timeGetTime), used in palette-fade / state transitions (`0x40CAE0`), NOT in the gameplay loop |
|
||||
| `0x40b5b0` | Win32 WndProc dispatch (msg 2 WM_DESTROY -> `0x40b680` sets state=0xf0 + PostQuitMessage; focus msgs 7/0x1c handled) |
|
||||
| `0x409FC0` | DirectSound init |
|
||||
| `0x42D970` | show_error_dialog (patched to `RET` so failures do not popup) |
|
||||
| `0x4824E4` | timeGetTime IAT; `[0x47e56c]` holds a 60Hz tick baseline (set once, not used as a per-frame delta) |
|
||||
|
||||
Reproduce any disassembly with r2:
|
||||
```sh
|
||||
python3 -c "import r2pipe; r=r2pipe.open('HAVOC_NOCD.EXE',['-e','bin.relocs.apply=true']); print(r.cmd('pd 60 @ 0x43ac40'))"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 6. File formats: status
|
||||
|
||||
Cracked (see `docs/FORMATS.md` for specs and `tools/` for extractors):
|
||||
- `.FF` "FlashFile" archive (`tools/ff_extract.py`).
|
||||
- MUSIC.FF: 97 WAV segments + 17 PLST-sequenced songs + FL Studio pack.
|
||||
- INTRFACE.FF: bitmap system (BitB/BitR headers, XPAL palettes, STR# string tables).
|
||||
- **GRAFIX0N.DAT** (`tools/grafix_extract.py`): BE header (size@0x08, count@0x0c), 16-byte entry
|
||||
records (BE offset + id), per-graphic blocks (BE w/h/frames/id + 8-bit indexed pixels). 64x64.
|
||||
|
||||
TODO (the asset-cracking stream):
|
||||
- **MAP<lvl>.MAP**: 2-byte-per-cell grid, ~192x144 confirmed. Next: disassemble its loader to get
|
||||
exact dims + which byte is tile vs height. Find the loader by searching for the code that opens
|
||||
"MAP" filenames (same pattern as GRAFIX via `0x42a480`).
|
||||
- **STUF<lvl>.DAT**: entity placement. Contains ASCII `"Item"` records - very parseable. Start
|
||||
here for a quick win.
|
||||
- **LAND<lvl>.DAT**: terrain geometry (hardest). BE fields; music-track id is a BE u16 at offset
|
||||
0x12 (per prior notes). Has 4-byte records around 0x20 (`17 70 00 c0` ...).
|
||||
- **OBJECTS.FF**: 3D models (vehicles, enemies, pickups) - not yet touched.
|
||||
- **World palette:** GRAFIX tiles currently render with a UI XPAL palette. The true per-world 3D
|
||||
palette is TBD - likely loaded during `0x418600` init or embedded per-world. Finding it makes
|
||||
all extracted art color-correct.
|
||||
|
||||
---
|
||||
|
||||
## 7. Ghidra decompilation stream (primary Linux task)
|
||||
|
||||
Load `HAVOC.EXE` (or `HAVOC_NOCD.EXE`) into Ghidra (x86 32-bit, image base 0x400000). Import the
|
||||
key addresses from section 5 as labels to bootstrap. High-value targets, roughly in order:
|
||||
|
||||
1. **The FPS-coupling / render modes.** The user reports cockpit mode is FPS-coupled but the
|
||||
minimal HUD mode is not. Find the mode toggle and the two render/update paths. Determine
|
||||
whether minimal mode uses a frame limiter (Sleep/timeGetTime) or a fixed-timestep/delta update.
|
||||
That path is the template for the remaster's fixed-timestep, and possibly a cleaner in-binary
|
||||
fix than the dgVoodoo FPS cap. Start from the main loop `0x43AC40` and its per-frame methods
|
||||
`0x436170` / `0x4361d0`, and the busy-wait limiter `0x40C8E0`.
|
||||
2. **Cheat code handler.** Codes `mmm` (max heatseeker), `vvv` (free vehicle), `sss` (shield),
|
||||
`hhh` (health), `aaa` (ammo). Find the keyboard-buffer matcher; it will reveal the full cheat
|
||||
list and the player/inventory structs.
|
||||
3. **Weapon system.** 6 primary classes (Laser, Disc, Shockwave, Fireball, Plasma, VARG) infinite
|
||||
ammo; 14 secondary lines (see `docs/WEAPONS.md`, from string tables STR#7D02/7D03). Recover the
|
||||
weapon data tables, upgrade tiers, and firing logic.
|
||||
4. **Player / vehicle physics** (movement constants, the FPS-coupled step) and **enemy AI**.
|
||||
5. **Level + entity loaders** (MAP/LAND/STUF) - decompiling these also directly finishes the
|
||||
asset-cracking stream (get exact record layouts instead of guessing).
|
||||
6. **World table**: 6 worlds stored in reverse (Overlord, Wasteland, Malterra, Tyrak, Fallout,
|
||||
Badlands), 3 levels + boss + bonus each. See `docs/FORMATS.md` and game-mechanics notes.
|
||||
|
||||
---
|
||||
|
||||
## 8. Methodology playbook (how the cracking was done)
|
||||
|
||||
- **Crack a format via its loader, not guesswork.** Find where the game opens the file
|
||||
(filenames are in `.data`; the loader `0x42a480` builds `WORLDS\<name>` paths), then read the
|
||||
loader's parse code. GRAFIX was solved by reading `0x444db0` (the 16-byte record stride and
|
||||
BE->LE byte-swap gave the record layout exactly).
|
||||
- **These WORLDS files are big-endian** (Mac-origin data; the loaders byte-swap). Always try BE
|
||||
first for headers/records.
|
||||
- **Find grid dimensions** by row autocorrelation (minimize mean abs diff between byte j and
|
||||
j+stride over candidate strides); then render grayscale at that width and eyeball for coherent
|
||||
structure. That is how MAP's 384-byte row (192x2) was found.
|
||||
- **Verify visually.** Render extracted pixel data to PNG and build contact sheets; correct
|
||||
decodes look like real art. Pillow `Image.frombytes('P', (w,h), data); putpalette(...)`.
|
||||
- **r2pipe pattern** for scanning: open with `bin.relocs.apply=true`, iterate `.text` FOs looking
|
||||
for opcode patterns (e.g. `ff 15` CALL [IAT], `a1`/`8b 05` mem reads) and convert FO->VA.
|
||||
|
||||
---
|
||||
|
||||
## 9. Tool inventory (`tools/`)
|
||||
|
||||
- `ff_extract.py` - `.FF` archive extractor (archives, songs, FL pack). `--help` for flags.
|
||||
- `grafix_extract.py` - GRAFIX0N.DAT -> PNG (`-p <768-byte palette>`).
|
||||
- `dbg_path.py` - Windows-only ctypes debugger (used for the boot crash; keep for reference).
|
||||
- Many `r2*.py` / `find_*.py` / `analyze*.py` - one-off RE probes from the crash hunt and format
|
||||
work. Useful as r2pipe examples; not all are load-bearing.
|
||||
- `run_probe.bat` / `run_and_log.bat` - Windows launch helpers (repro/testing on Windows only).
|
||||
|
||||
Palettes: extract with `python3 tools/ff_extract.py INTRFACE.FF -o tools/out_intrface`, palettes
|
||||
land in `tools/out_intrface/raw/XPAL*.DAT` (768-byte RGB).
|
||||
|
||||
---
|
||||
|
||||
## 10. Immediate next steps (suggested)
|
||||
|
||||
1. (Asset, quick win) Crack `STUF<lvl>.DAT` entity records (ASCII `"Item"` markers).
|
||||
2. (Asset) Disassemble the MAP loader to lock MAP dims + cell meaning; render a proper level map.
|
||||
3. (Ghidra) Stand up the project, label the section-5 addresses, and chase the render-mode /
|
||||
FPS-coupling path (item 7.1) - it informs both the exe fix and the Godot physics.
|
||||
4. (Remaster) Once STUF + MAP + a world palette are done, there is enough to scaffold the Godot
|
||||
project (empty today) and load a real level.
|
||||
|
||||
The FPS workaround (dgVoodoo `FPSLimit = 30`) is a stopgap; the user is tuning the value. A proper
|
||||
fix is either a frame limiter patched into `0x43AC40` (the game imports `Sleep`) or, better,
|
||||
adopting the minimal-mode timing path once Ghidra reveals it.
|
||||
Loading…
Reference in a new issue