docs: add full RE methodology for blog post
Documents the process (not just the results) for each format cracked: - Binary identification: why 32-bit PE not 16-bit, what actually broke - .FF archive discovery: speculative parsing from file count heuristic - MUSIC.FF playlists: big-endian payload in LE container, segment ID math - INTRFACE.FF bitmaps: BitB/BitR/XPAL system, index-2 black patch discovery Includes tools, dead ends, corrections (2D->3D), and what's next. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
parent
acd4057e72
commit
534c41af73
1 changed files with 189 additions and 0 deletions
189
docs/METHODOLOGY.md
Normal file
189
docs/METHODOLOGY.md
Normal file
|
|
@ -0,0 +1,189 @@
|
|||
# Havoc RE Methodology: How We Cracked the Formats
|
||||
|
||||
This document records the *process* of reverse-engineering Havoc (Reality Bytes, Inc., 1995),
|
||||
including the wrong turns, so the blog post can tell the full story.
|
||||
|
||||
---
|
||||
|
||||
## 1. Starting point: what do we have?
|
||||
|
||||
The game files had been sitting on a network share untouched. The top level:
|
||||
|
||||
```
|
||||
HAVOC.EXE 485 KB
|
||||
LAUNCHER.EXE 17 KB
|
||||
SETUP.EXE 110 KB
|
||||
BIGFILE.DAT 64 MB
|
||||
MUSIC.FF ~6 MB
|
||||
INTRFACE.FF ~12 MB
|
||||
OBJECTS.FF ~2 MB
|
||||
SOUND.FF ~1 MB
|
||||
WORLDS/ terrain data
|
||||
DIRECTX/ bundled DirectX 5
|
||||
AUTORUN.INF OPEN=LAUNCHER.EXE
|
||||
.windows-serial contains "0"
|
||||
```
|
||||
|
||||
The first question: **why did it stop running?** The common assumption for old Windows games
|
||||
is "16-bit" or "DOS". That assumption was wrong.
|
||||
|
||||
---
|
||||
|
||||
## 2. Identifying the binaries (correcting the 16-bit assumption)
|
||||
|
||||
Approach: read the PE header directly in Python rather than trusting assumptions.
|
||||
|
||||
Every Windows executable starts with an MZ stub. At offset `0x3C` is a uint32 pointing to
|
||||
the PE signature. We read that offset for all three EXEs:
|
||||
|
||||
```
|
||||
HAVOC.EXE PE offset = 0x80
|
||||
LAUNCHER.EXE PE offset = 0x80
|
||||
SETUP.EXE PE offset = 0x80
|
||||
```
|
||||
|
||||
At `0x80` in each: `50 45 00 00` = `PE\x00\x00`. Confirmed 32-bit PE. The Machine field
|
||||
(`0x014C`) = i386. The Magic field (`0x010B`) = PE32 (not PE32+). Subsystem = 2 (GUI).
|
||||
|
||||
**Key finding:** these are 32-bit Win32 applications. They will run under WOW64 on 64-bit
|
||||
Windows today. The game itself is not the problem.
|
||||
|
||||
The import table confirmed the real story: `DDRAW.dll`, `DSOUND.dll`, `WINMM.dll`,
|
||||
`WSOCK32.dll`, `GDI32.dll`, `USER32.dll`, `KERNEL32.dll`. Notably: **no Direct3D at all**.
|
||||
|
||||
**What actually broke:** the bundled `DIRECTX/` folder ships `DSETUP16.DLL` and `DDRAW16.DLL`
|
||||
-- genuine 16-bit components used by the installer path. Those cannot run on 64-bit Windows.
|
||||
But the game doesn't need the installer to run once files are in place.
|
||||
|
||||
The second problem: legacy DirectDraw palettized fullscreen (Mode X / 8-bit indexed) is
|
||||
unreliable on modern GPUs without a compatibility wrapper like dgVoodoo2.
|
||||
|
||||
**Correction during process:** we initially described Havoc as "2D". Looking at the
|
||||
screenshots on myabandonware corrected this -- it is a first-person software-rendered 3D
|
||||
vehicular combat game. The software rasterizer blits into a DirectDraw surface, explaining
|
||||
why there is no Direct3D import despite full 3D gameplay.
|
||||
|
||||
---
|
||||
|
||||
## 3. Identifying the archive format (.FF files)
|
||||
|
||||
The `.FF` files are the main asset containers. No magic signature at the start, so we
|
||||
read the first few bytes as a uint32: `116` for `MUSIC.FF`. That looked like a file count.
|
||||
|
||||
We then parsed speculatively: if byte 4 is an offset and the following bytes are a NUL-
|
||||
terminated filename, does it make sense?
|
||||
|
||||
```python
|
||||
count = struct.unpack_from("<I", data, 0)[0] # 116 for MUSIC.FF
|
||||
pos = 4
|
||||
offset = struct.unpack_from("<I", data, pos)[0] # first file offset
|
||||
name = data[pos+4 : data.index(b"\x00", pos+4)] # "FFCREATE.EXE"
|
||||
```
|
||||
|
||||
It did. The first entry was `FFCREATE.EXE` at a reasonable offset, and its first bytes were
|
||||
`MZ` -- a real DOS executable. The format was clear:
|
||||
|
||||
```
|
||||
uint32 file_count (little-endian)
|
||||
repeat file_count:
|
||||
uint32 absolute_data_offset (little-endian)
|
||||
char[] filename (NUL-terminated)
|
||||
<concatenated file payloads>
|
||||
```
|
||||
|
||||
Finding `FFCREATE.EXE` (a DOS MZ binary) embedded inside the archive was a nice confirmation:
|
||||
this was the tool Reality Bytes used to *build* the `.FF` files, shipped alongside the data it
|
||||
created. The name "FF" almost certainly stands for "FlashFile."
|
||||
|
||||
Gotcha: the last TOC entry may have an empty name and offset pointing past the end. Skip it.
|
||||
|
||||
---
|
||||
|
||||
## 4. Cracking MUSIC.FF: the playlist format
|
||||
|
||||
`MUSIC.FF` contains 97 `WAVE*.DAT` entries and 17 `PLST*.DAT` entries, plus `FFCREATE.EXE`.
|
||||
|
||||
The WAVE entries were straightforward: `RIFF` signature, standard PCM WAV, mono 22050 Hz
|
||||
8-bit. Directly playable with any audio tool.
|
||||
|
||||
The PLST (playlist) entries were more interesting. Initial read (little-endian):
|
||||
|
||||
```
|
||||
first two uint16s of PLST07D1: 0x0004, 0x0014 = (4, 20)
|
||||
```
|
||||
|
||||
A 4-segment song with 20 playback steps. Then 20 uint16s for the sequence. But when we
|
||||
tried to unpack them little-endian, the step count came out as `10240` -- clearly wrong for
|
||||
a 44-byte file. **The PLST payload is big-endian** despite the `.FF` container being
|
||||
little-endian. Swapping to `">HH"` gave the correct values.
|
||||
|
||||
Segment resolution: a playlist `PLST07D1` references segments by 1-based index into a group.
|
||||
First attempt mapped by the last hex digit of the filename (`07D1` -> segment 1, `07D2` -> 2).
|
||||
This failed for multi-digit segment counts. Correct rule: **step N maps to wave ID
|
||||
`base_id + N - 1`**, where `base_id = int("07D1", 16) = 2001`. Step 2 -> wave ID 2002 =
|
||||
`WAVE07D2`. Verified by assembling all 17 songs and confirming they sound correct.
|
||||
|
||||
---
|
||||
|
||||
## 5. Cracking INTRFACE.FF: the bitmap system
|
||||
|
||||
`INTRFACE.FF` (547 entries) contained several entry types by name prefix:
|
||||
|
||||
| Prefix | Count | Hypothesis |
|
||||
|---|---|---|
|
||||
| `BitR` | 239 | "Bit Raster" -- raw pixel data |
|
||||
| `BitB` | 239 | "Bit header/Border" -- paired with BitR by same hex ID |
|
||||
| `XPAL`/`xpal` | ~47 | "eXtended PALette" -- 768-byte palette = 256 * RGB |
|
||||
| `STR#` | 16 | String tables |
|
||||
| `RTbl` | 3 | "Remap Table" (function TBD) |
|
||||
| `FFCREATE.EXE` | 1 | Archive builder |
|
||||
|
||||
The `BitB` entries were all exactly 4 bytes. Reading as two big-endian uint16s gave
|
||||
dimensions: `(48, 48)` for `BitB4E21`. Checking `BitR4E21`: 2304 bytes = 48 * 48. Every
|
||||
single BitB/BitR pair (239 pairs) matched exactly. Format confirmed.
|
||||
|
||||
The `BitR` data is raw 8-bit palette-indexed pixels, row-major, no header. The `XPAL` files
|
||||
are 768 bytes = 256 RGB triplets, 8-bit per channel (NOT 6-bit VGA -- we verified by
|
||||
checking that ~67% of values exceed 63).
|
||||
|
||||
**The index-2 black patch (key discovery):**
|
||||
|
||||
First renders of most screens showed a salmon/pink background (#e78484) instead of the
|
||||
expected dark industrial look. One screen, `7A12`, rendered correctly with a dark background.
|
||||
|
||||
Comparing palettes `xpal7A12` vs `xpal7A1C` (same screen layout, different background color):
|
||||
only 3 of 256 palette entries differed -- indices 0, 1, and 2. In `7A12`: index 2 = `#000000`
|
||||
(black). In `7A1C`: index 2 = `#e78484` (salmon). **The game engine forces palette index 2 to
|
||||
black at runtime.** The XPAL files store development/authoring-time values; the runtime
|
||||
patches index 2 before display. One-line fix: `pal[6], pal[7], pal[8] = 0, 0, 0` after
|
||||
loading any XPAL.
|
||||
|
||||
Assets extracted from INTRFACE.FF include: 32 full-screen menu/UI backgrounds, cockpit
|
||||
windshield frame layers (three color variants), vehicle selection screens with pre-rendered
|
||||
3D vehicle thumbnails, cockpit side panel strips, menu button strips in multiple states,
|
||||
and 57 vehicle HUD icons in the game's monochrome green HUD style.
|
||||
|
||||
The "minorly off" colors in the renders vs reference screenshots are consistent with
|
||||
CRT gamma -- the palette values were tuned for a 1995 CRT display (~2.2 gamma) and display
|
||||
directly on a modern LCD without gamma compensation. This is a Godot shader concern, not
|
||||
an extraction accuracy issue.
|
||||
|
||||
---
|
||||
|
||||
## 6. Tools used
|
||||
|
||||
- **Python 3** (stdlib only for format parsing: `struct`, `os`, `collections`)
|
||||
- **Pillow** (`PIL`) for bitmap rendering and PNG export
|
||||
- **ffmpeg** (via scoop on Windows) for WAV to OGG/MP3 conversion
|
||||
- **No disassembler yet** -- all format work done from data patterns alone
|
||||
- **Ghidra** (planned) for game logic / binary RE
|
||||
|
||||
---
|
||||
|
||||
## 7. What's next
|
||||
|
||||
- `SOUND.FF`: same format as MUSIC.FF, just raw SFX WAVs (no playlists)
|
||||
- `WORLDS/`: terrain data -- `MAP*.MAP`, `LAND*.DAT`, `GRAFIX*.DAT`, `STUF*.DAT`
|
||||
- `OBJECTS.FF`: 3D models -- the hard one; `ASND*` entries TBD
|
||||
- `BIGFILE.DAT`: 64 MB bulk data pool (starts with copyright string then RIFF data)
|
||||
- Ghidra decompilation of `HAVOC.EXE` for game logic, AI, cheat codes, weapon system
|
||||
Loading…
Reference in a new issue