Skip to content

Check it yourself

stepmaniay-verification Public

Tests anyone can run against StepManiaY's published builds and ITGmania 1.3.0, the release it grew from.

Clone

Clone with HTTPS

Plain git over HTTPS: no account, no app, nothing else to install.

Download the source tarball 3.0 MB
docker
results
tests
tools
.gitignore 31 bytes
engines.json 765 bytes
README.md 5.6 KB
verify.sh 1.5 KB

README.md

StepManiaY verification kit

Tests anyone can run to check what stepmaniay.com/why says StepManiaY fixed, against ITGmania 1.3.0, the release StepManiaY's engine grew from. You don't need StepManiaY's source: every test runs the public downloads, after checking each one's SHA-256 against the published value, and records what each engine did.

git clone https://stepmaniay.com/verification.git stepmaniay-verification
cd stepmaniay-verification
./verify.sh stability

Without git, the same files are a tarball at stepmaniay.com/verification, with its SHA-256 and the commit it was made from.

What you need

  • Docker (Docker Desktop on macOS and Windows; on Windows, run the scripts from WSL or Git Bash).
  • git, for the ITGmania build.
  • About 3 GB of disk: both engines' builds, and ITGmania's source for the timing test.

Every run is headless (a virtual display, software OpenGL, the Null sound driver), offline, and in a fresh portable copy of the engine holding only the test's songs, so nothing touches a game you already have installed.

The tests

Command Time What it checks
./verify.sh stability 5 min Damaged files that broke the old song scan: a WAV whose chunk size is -8, an .ssc with #SPEEDS:=;, a .dwi with #single:, an .ssc with a 1/256 time signature. Each engine loaded the song list, crashed, or hung (no song list after 180 s).
./verify.sh playback 3 min The 1/256 song played through a minimal theme on each engine's own _fallback, the same Lua on both: played, crashed or hung.
tests/timing/build_itgmania.sh then ./verify.sh timing 30 min the first time StepManiaY's 653-note test chart played by a self-test that presses the panels through the engine's own input system at exact moments, with a recorder that writes every judged step and the note it was meant for.

tests/stability/run.py lists, for every damaged file, the line of ITGmania 1.3.0's source it reaches. tests/stability/make_songs.py and tests/timing/make_song.py make every file from nothing (the timing chart is StepManiaY's own, its music a generated click track), so the kit carries no one else's music or charts.

Stability and playback run the official x86_64 builds (ITGmania publishes no other Linux build), so an Apple silicon Mac runs them under emulation. Timing runs on your machine's own architecture, so an emulator can't stand in for the long frames it makes on purpose.

The ITGmania side of the timing test

ITGmania has no self-test or recorder, so tests/timing/build_itgmania.sh builds ITGmania 1.3.0 (tag v1.3.0, commit 8a3bf8884a, the official release's build) from its own repository, in its own build container (Utils/Dockerfile-linux), with tests/timing/itgmania-1.3.0-timing-harness.patch added. The patch only adds lines; ITGmania's judging is untouched and every judgment is its own:

git apply --numstat tests/timing/itgmania-1.3.0-timing-harness.patch   # the second column, lines removed, is 0 throughout

To build it without the script, follow ITGmania's own build instructions after:

git clone --branch v1.3.0 --recurse-submodules https://github.com/itgmania/itgmania.git
cd itgmania && git apply ../stepmaniay-verification/tests/timing/itgmania-1.3.0-timing-harness.patch

then point path in tests/timing/engines-<arch>.json at your itgmania binary. The script prints the binary's SHA-256; each run's results name the binaries they ran.

The self-test's flags are the same on both engines: --smytest=timing turns it on when gameplay starts, --timingoffsets=16 presses every step 16 ms late (late positive), --timingrate=1.5 sets the music rate, --timingstutter=60 makes one frame 60 ms long every second.

Results so far

From results/ (November 2026, StepManiaY 1.3.1.2611032302 against ITGmania 1.3.0):

Test ITGmania 1.3.0 StepManiaY
WAV with a chunk size of -8 hung loaded (the file refused, with the reason in userlog.txt)
.ssc with #SPEEDS:=; crashed (segmentation fault) loaded (the entry skipped and logged)
.dwi with #single: crashed (Unrecognized DWI notes format) loaded
.ssc with a 1/256 time signature, scanned loaded loaded (the signature skipped and logged)
The same song, played hung played
653-note chart, exact steps, rate 1.0 and 0.75 653 W1 653 W1
Exact steps, a 60 ms frame every second (3 runs) 4–5 notes missed 653 W1
Every step 16 ms late, rate 1.5 (3 runs) 48–55 steps on the wrong note, 12–13 missed 653 W1
Every step 16 ms early, rate 1.5 16 missed 653 W1
10 ms late, a 60 ms frame every second 5 missed 653 W1

python3 tools/summarize.py results/*/ prints every run's table. Each results folder keeps every run's log.txt, userlog.txt, crash report, console output and recorder CSV beside results.json.

Not tested here yet

Claims that need hardware: XInput controllers polled every millisecond on Windows, the sound clock's noise and drift over hours on real sound cards, Play Together's sync between machines, and frame times on a Raspberry Pi 5. Those need a capture of the sound or the controller from outside the game, so this kit will carry their procedures and analysis rather than run them in a container.

Sharing what you found

Run it on your machine and publish your results/ folder wherever you like, or send it through stepmaniay.com/feedback. Results that disagree with ours are the ones we most want to see.