Developer tools: unattended runs, captures and render diagnostics

The game can run with nobody at the controls: hidden window, a clock as fast as the machine allows, frames captured straight from the renderer, and an attract-mode demo on any track. Runs are bit-exact, so two runs with the same settings give identical frames. Use this to reproduce visual bugs. Never drive the game with OS keystrokes or wait on a real-time demo.

Quick start

# Build, then capture Dingo Canyon's demo every 100 frames, at 2x render scale:
tools\decrypted\devrun.ps1 -Track 0 -Frames 4800 -From 1500 -Every 100 -Config @{render_scale=2}

The script prints the capture folder (%TEMP%\ctrdev\shots-HHmmss unless -Out is given; the next run deletes default folders older than a day): one fNNNNNN.png per captured frame and sheet.png, a labelled contact sheet. A race renders about 1,300 frames a second, so a whole demo takes about ten seconds. The race starts about 1,200 frames into a demo run.

devrun.ps1 runs from its own folder (%TEMP%\ctrdev\run) with a baseline config.ini (60 FPS, PGXP FULL, 16:9, render scale ORIGINAL), so the owner's install and saves are never touched.

Environment variables

All are read once, by platform/native_dev.c (include/platform/native_dev.h). With none set, the game behaves exactly as shipped.

Frame numbers count every presented frame, boot splash included. Frames shown straight from VRAM (boot splash, scrapbook, other movies) are captured from VRAM.

devrun.ps1 clears earlier captures from the output folder, and warns when the game exits with an error code before -Frames (a crash). For a stack, run the same switches on the Archipelago build (-Exe build-ap\ctr_native_ap.exe): its crash handler writes ctr-ap-crash.txt in the run folder, and addr2line -f -e build-ap\ctr_native_ap.exe <0x400000 + offset> names each frame. Release builds inline heavily, so check neighbouring symbols with nm -n too.

Capturing a frame while playing

For bugs that only show where a player drives: press F7 (every build has it). Each press makes its own folder, debug/captures/<date>-<time> next to the game:

• state.ctrstates: a savestate of that moment. Replay it with CTR_DEV_STATE_FILE=<it> and CTR_DEV_LOAD_STATE=1300 on the exact exe that made it (keep a copy); the next frame is pixel-identical. A rebuilt exe crashes loading it, so to test a change there, warp with where.txt instead.
• where.txt: the level, player 1's position and heading as a ready CTR_DEV_WARP= line, the camera's position and rotation, and the view settings (camera_far, aspect, draw distance, render scale, PGXP).
• polys-NNNNNN.txt, fNNNNNN.bmp, vramNNNNNN.bin: the next frame's polygon dump (CTR_DEV_DUMP_FRAME format), image and VRAM. pgxp_cracks.py, vram_png.py and a short script over the dump show which polygons drew it.
• For sound bugs (press right after the sound): the last 20 seconds of mixed audio (audio-NNNNNN.raw, CTR_DEV_AUDIO format, read by audio_noise.py) and the SPU audit's last 128 rows (spu-NNNNNN.csv, then N,vblank with the VBlank of the press).

To compare a spot with the original game, set retail's Driver.posCurr (+0x2d4, world × 256) and angle (+0x39a) from where.txt in an emulator.

Rounding detector

The game's camera is in whole units, and its rounding steps every few frames while it moves. Under the precise camera nothing drawn should follow those steps; anything that does jumps up to a pixel against the scene (wobble, judder). The detector finds those draw paths in any frame:

# The Adventure hub at frame 4000, then a race while the autopilot drives:
tools\decrypted\nudge.ps1 -Frame 4000 -Exe build-mrg\ctr_native_ap.exe -Env @{CTR_DEV_RACE=26; CTR_DEV_MODE='adv'}
tools\decrypted\nudge.ps1 -Frame 2600 -Env @{CTR_DEV_RACE=0; CTR_DEV_AUTOPILOT=1}

It runs the frame once as is and once per camera axis with CTR_DEV_NUDGE, in parallel, and nudge_diff.py compares each polygon dump with the base one, matching polygons by who drew them, texture and nearest position. Polygons whose retail position moved but drawn position did not are precise. Those whose drawn position moved too follow the integer camera, and are listed by @name with how far they moved and where. --png paints them red on the frame. The frame must show the follow camera (a race, or a hub after the intro).

Reading it: precise polygons move at most a few hundredths of a pixel. A whole group following is a draw path that copies GTE results into primitives without the precise positions (NativePgxp_StoreGteSXY, NativePgxp_CopyXY), takes positions relative to the integer camera outside a NativePgxp_ViewBeginRelative scope, or is an instance whose model-view is not registered (split and screen-space models are not); see PGXP.md. Two kinds of entry are noise, not rounding: one or two polygons of a precise model whose drawn move is about its retail move (look-alike triangles mismatched; their drawn positions barely move in the dump), and depth-less vertices at x 1023 (a skid mark corner behind the camera, clamped by the GTE as in retail). The two example frames above show only noise. After a fix, the group must leave the list and find_holes.py totals must not rise.

Analysis scripts

• tools/decrypted/find_holes.py <dir> [--ignore-top N] [--max-run N] finds see-through holes in a CTR_PGXP_DEBUG=1 capture, where the sky draws pure blue: a short vertical run of blue with geometry above and below is a hole. Prints a count per frame and a total, and writes holes-*.png with holes in magenta. Use --ignore-top 110 --max-run 4 at 2x render scale to skip HUD text. HUD text, boost flames and gaps between distant rocks also count, so compare against a baseline run (same settings with CTR_PGXP_DEBUG=3).
• tools/decrypted/audio_noise.py <audio.raw> [--flat F] [--wav out.wav] finds white noise in a CTR_DEV_AUDIO dump or F7 audio: loud, spectrally flat windows (white noise scores about 0.58, game audio under 0.2). Prints noisy spans with their VBlank and can write a .wav.
• tools/decrypted/howl_banks.py <disc.bin> [--bank N ...] [--samples] lists the sound banks in KART.HWL with the SPU bytes each uploads, to name a W row of the SPU audit and check that a set of banks fits under the SPU ceiling (include/platform/native_spu_memory.h). The largest hub set (common, hub, player's, three podium banks) ends at 0x75c00, under the retail 0x7e000.
• tools/decrypted/filter_check.py <baseline.png> <capture.png>... [--lines 216] compares CTR_DEV_SHOT_PRESENTED captures of one frame against a baseline (normally SMOOTH): mean brightness, 99th percentile of the step between neighbouring pixels (anti-aliasing lowers it), and the share of vertical variation at the 216-line frequency (scanlines raise it). See SCREEN-FILTERS.md.
• tools/decrypted/quad_dump.py <dump.csv> [--level N] lists a level's turbo pad triggers and the visible quads over each, with their texture slots (*10 marks a 10-frame animation).
• tools/decrypted/nudge.ps1 and nudge_diff.py <base polys> <nudged polys> [--png out.png] [--all]: the rounding detector.
• tools/decrypted/pgxp_cracks.py <polys-NNNNNN.txt> [--level-only] reads a polygon dump. It reports level vertices drawn at more than one position (a shared corner that disagrees with itself), then T-junction candidates: vertices inside another polygon's edge in retail but not when drawn.
• tools/decrypted/driver_motion.py <drivers.bin> <trace.csv> [--ms 3000] [--drivers 1-7] splits every racer's movement in a CTR_DEV_DRIVER_DUMP into along and across its heading, from the green light (taken from the same run's trace): forward speed reached, sideways speed (max and mean) and net sideways distance. Compare 30 and 60 FPS runs for racers sliding at the start.
• tools/decrypted/driver_diff.py <A.bin> <B.bin> [--delta N] compares two CTR_DEV_DRIVER_DUMP files and prints the first race frame where any driver differs: driver, offset, both values. Words that look like host pointers in both runs are skipped, or, with --delta, words that differ by exactly the load-base difference.

Determinism and load addresses

Runs are bit-exact only while nothing depends on where the executable is loaded. Level data, statics and function pointers all sit at the load address, so code that reads past the end of level data, or tests a pointer's value as retail tests for 0x80xxxxxx, behaves differently between builds, and between runs if the base moves.

The game is linked without ASLR (--disable-dynamicbase on Windows, -no-pie on Linux, in CMakeLists.txt), so every run of one build loads at the same address. Savestates and input replays store raw pointers and depend on this. A state loads only in the exact build that made it, since any code change moves addresses.

To check for address dependence, build at two fixed bases and compare races with CTR_DEV_DRIVER_DUMP and driver_diff.py --delta:

cmake -S . -B /c/t/fb400000 -G Ninja -DCMAKE_BUILD_TYPE=Release "-DCMAKE_EXE_LINKER_FLAGS=-Wl,--disable-dynamicbase -Wl,--image-base=0x400000"
cmake -S . -B /c/t/fb10000000 -G Ninja -DCMAKE_BUILD_TYPE=Release "-DCMAKE_EXE_LINKER_FLAGS=-Wl,--disable-dynamicbase -Wl,--image-base=0x10000000"

All 18 race tracks' attract demos match across the two bases for 4,500 frames of all eight drivers, so any difference is a regression.

Unattended races

The race switches live in platform/native_dev_race.c. They check gameplay timing with nobody driving, mainly 60 FPS against 30:

# Every track, a Time Trial at 30 and at 60 FPS, plus the 30 FPS ghost raced at both:
tools\decrypted\fps-parity.ps1 -Ghosts
# One track:
tools\decrypted\fps-parity.ps1 -Tracks 3

tools/decrypted/race_trace.py reads the traces:

• laps A.csv B.csv: lap splits.
• jumps A.csv: airborne spells, airtime and apex height.
• parity A.csv B.csv: finish times, and jumps launched from the same spot in both runs.
• fps A.csv B.csv [--until MS]: two runs of the same CTR_DEV_DRIVE script, normally 30 and 60 FPS. Per kart and camera field, the RMS difference as a share of the field's spread, and the race time it first strays by a tenth of that. tools/decrypted/fps-drive.ps1 -Track 3 -Drive '0:X:5000,1500:L:1500' -Until 4000 runs both and prints this; -Env @{CTR_DEV_WARP='5838,2374,7579,2683'} adds switches to both (here, Dingo Canyon's run-up to the canyon jump).
• smooth A.csv ...: how unevenly player 1's camera moves: per channel, the median and 90th percentile of each frame's distance from the midpoint of its neighbours (zero at a steady rate), and alt, the share that flips sign every frame (something stepped on odd frames only). Integer and precise camera channels sit side by side.
• ghost SOURCE.csv PLAY.csv: how far the ghost in PLAY strays from the SOURCE player who recorded it.

A Time Trial takes 10 to 20 seconds at 30 FPS, twice that at 60. The autopilot never boosts, hops or drifts, so on N. Gin Labs and Polar Pass it keeps falling into the same pits and the run ends at the frame cap. Per-track expectations are in 60FPS.md. Input replays (--record, --replay, REPLAYS.md) work with these switches; set the same CTR_DEV_RACE and CTR_DEV_MODE when playing back, because they change the title menu's flow.

Saves and progress

tools/decrypted/ctr_save.py reads and edits the save, memcards\slot0\BASCUS-94426-SLOTS next to the exe: the four Adventure profiles and game-wide progress (struct MemcardProfile, include/namespace_Memcard.h). It rewrites the checksum so the game accepts the file. python tools\decrypted\ctr_save.py -h lists every name.

$s = 'tools\decrypted\ctr_save.py'
python $s show <save>                    # what each profile has; --json for scripts
python $s new C:\t\sv\test.sav           # an empty card
# Profile 0 as Coco, saved in Glacier Park: 8 trophies, 2 keys, one relic, the doors
# to Glacier Park open, every hint already seen (no Aku Aku popups in the run).
python $s edit C:\t\sv\test.sav --character coco --hub glacier --give trophy:0,1,2,3,4,5,6,7 key:2 sapphire:blizzard door:beach_gemstone,beach_glacier,ruins_glacier hint
python $s edit C:\t\sv\test.sav --profile 1 --give everything racer cup ghost   # 100% and every unlock
tools\decrypted\devrun.ps1 -Track -1 -Frames 2600 -From 2400 -Save C:\t\sv\test.sav -Env @{CTR_DEV_PROFILE=0}

--give and --take take KIND[:WHICH]. Per profile: trophies, relics per tier, tokens, keys, gems, purple tokens, hub doors, the two Oxide endings and hints. Per save: racers, bonus tracks and arenas, Arcade cups, the scrapbook and beaten Time Trial ghosts. An edited empty profile becomes DEV, Crash, in N. Sanity Beach. A door stands open as if opened with keys (game/232/AH_Door.c); warp pads and boss garages still count trophies, relics and keys. In the Archipelago build doors follow the slot's door history instead.

Archipelago progress lives on the server, so the same script talks to a local server (127.0.0.1, any room; ap-give needs the item cheat, on by default in host.yaml):

python $s ap-show 38300 Decrypt                         # items received, locations checked
python $s ap-give 38300 Decrypt "Sapphire Relic" 6      # the server's !getitem
python $s ap-check 38300 Decrypt "Crash Cove: *" "*: Trophy Race"   # names or wildcards

ap-check connects as a game client of the slot (the server ignores checks from text-only clients) and checks the matching locations, so their items go out as if found in game. Names are in apworld/ctr/data/locations.json and items.json. A killed server does not write its .apsave; stop it from its console (/exit) to keep what was given.

README screenshots

tools/decrypted/readme-shots.ps1 retakes the three README images in docs/images (1920x1080, 16:9, PGXP, increased draw distance): N. Oxide on Hot Air Skyway, the Archipelago item feed in a Crash Cove race, and the Mods screen's track browser. Retake them after a visible change to any of those; -Only oxide (or archipelago, mods) does one.

• The Oxide shot is fast-forwarded, so its frame is always the same.
• The Archipelago shot generates a throwaway two-player seed (Decrypt and Friend, item boxes as checks) with this repository's apworld, in the Archipelago checkout apworld-test.ps1 sets up (run that once first). It hosts the seed on 127.0.0.1:38300 and races in real time while tools/decrypted/ap_feed_demo.py connects as Friend and checks Friend's locations holding Decrypt's items, so the feed shows <ITEM> FROM FRIEND.
• The Mods shot needs the network for the Project Saphi catalogue.

Real-time shots can land a little differently each run. Every captured frame stays under C:\t\shots\<name>; pick another with -ApFrame or -ModsFrame.

Archipelago server load

tools/decrypted/ap_backlog.py PORT SLOT COUNT [ITEM] gives a slot a large item backlog on a local server through the item cheat (!getitem, allowed by default in host.yaml), so the next connect receives it all in one packet. To measure a connect:

# A throwaway seed: a copy of a player YAML with its own name, in its own folder.
C:\ProgramData\Archipelago\ArchipelagoGenerate.exe --player_files_path C:\t\ap\players --outputpath C:\t\ap\out
# Host it on this machine only, then queue the items:
C:\ProgramData\Archipelago\ArchipelagoServer.exe --host 127.0.0.1 --port 38300 C:\t\ap\out\AP_<seed>.zip
python tools\decrypted\ap_backlog.py 38300 Bench 20000
tools\decrypted\devrun.ps1 -Exe build-ap\ctr_native_ap.exe -RealTime -Track -1 -Frames 2400 -From 100000000 -Config @{ mute_when_unfocused='true'; 'Connection.uri'='ws://localhost:38300'; 'Connection.slot'='Bench' } -GameArgs '--perf-dir','C:\t\ap\perf'

The client logs every frame over 250 ms as [AP PERF] frame stall in the run folder's log, and frame_times.csv has every frame.

The same server drives connected Adventure runs (CTR_DEV_MODE=adv, CTR_DEV_RACES) with the same -Config connection keys. An already-checked location changes the flow (a won trophy skips its podium), so for a first win stop the server, delete the .apsave next to the seed zip, restart it, and use a new -RunDir (the client keeps ap-state.json there).

Frame timing

ctr_native.exe --perf writes debug/perf/perf-latest/frame_times.csv and summary.txt. A frame's budget is the VBlanks it waits for times 16.68 ms (33.37 ms at 30 FPS, 16.68 at 60); the summary counts frames over budget, over 1.5 times it (on screen a refresh too long) and over twice it. In a hidden window the swap does not wait for the display, so measure VSync On and Adaptive with the window visible.

tools/decrypted/vsync-check.ps1 does that for every VSync option at 30 and 60 FPS, windowed and fullscreen: it plays the attract demo in a visible window in real time with --perf-dir, reads the display's refresh rate from Windows, and runs tools/decrypted/perf_pacing.py on each result. Per run it prints frame interval percentiles, how many refreshes each frame stayed on screen, the uneven frames (an unusual refresh count) and the long ones (over 1.5 times nominal). It takes about 25 minutes, and the screen must stay on and unlocked, so run it when nobody is using the machine.