Check it yourself
stepmaniay-verification Public
Tests anyone can run against StepManiaY's published builds and ITGmania 1.3.0, the release it grew from.
Clone
Clone with HTTPS
Plain git over HTTPS: no account, no app, nothing else to install.
Download the source tarball 3.0 MBfeat: anyone can clone and run it: each machine's own architecture for timing, a README for testers, and the website's release
12 files changed: +265 −79
modified README.md +88 −40
@@ -1,52 +1,100 @@ # StepManiaY verification kit -Tests anyone can run against the published builds of StepManiaY and of ITGmania 1.3.0, the-release StepManiaY's engine grew from, to check what stepmaniay.com/why says StepManiaY fixed.-No source code is needed: each test runs the public downloads, after checking their SHA-256-against the published one (`engines.json`), and records what each engine did.--## Running--Docker on any machine (an Apple silicon Mac runs the x86_64 builds under emulation):+Tests anyone can run to check what [stepmaniay.com/why](https://stepmaniay.com/why/) says StepManiaY+fixed, against ITGmania 1.3.0, the release StepManiaY's engine grew from. You don't need+StepManiaY's source: every test runs the public downloads, after checking each one's SHA-256+against the published value, and records what each engine did. ```sh-./verify.sh stability # files that broke the old song scan-./verify.sh playback # files that only break a song once it's played-tests/timing/build_itgmania.sh && ./verify.sh timing # judging, with the same self-test on both-python3 tools/summarize.py results/*/+git clone https://stepmaniay.com/verification.git stepmaniay-verification+cd stepmaniay-verification+./verify.sh stability ``` -Every run is headless (a virtual display, software OpenGL, the Null sound driver), offline, and in a-fresh portable copy of the engine with only the test's songs, so nothing touches your own setup.-Each result folder keeps every run's `log.txt`, `userlog.txt`, crash report and console output-beside `results.json`.+Without git, the same files are a tarball at [stepmaniay.com/verification](https://stepmaniay.com/verification/),+with its SHA-256 and the commit it was made from. -## Tests+## What you need -`tests/stability` puts a control song and one damaged file in the song folder and starts the game:-**loaded** (the song list finished loading and the game kept running), **crashed** or **hung** (no-finished song list within 180 s). `tests/playback` plays the song through a minimal theme on the-engines' own `_fallback`, the same Lua on both: **played**, **crashed** or **hung** (no gameplay-screen within 120 s).+- Docker (Docker Desktop on macOS and Windows; on Windows, run the scripts from WSL or Git Bash).+- git, for the ITGmania build.+- About 3 GB of disk: both engines' builds, and ITGmania's source for the timing test. -| Case | The file | In ITGmania 1.3.0's source |+Every run is headless (a virtual display, software OpenGL, the Null sound driver), offline, and in+a fresh portable copy of the engine holding only the test's songs, so nothing touches a game you+already have installed.++## The tests++| Command | Time | What it checks | | --- | --- | --- |-| `01-wav-negative-chunk` | A WAV whose LIST chunk has a size of -8 | `RageSoundReader_WAV.cpp:497` adds the size to the read position unchecked |-| `02-ssc-empty-speeds` | An `.ssc` with `#SPEEDS:=;` | `NotesLoaderSM.cpp:834` reads `vs2[0]` before checking the size |-| `03-ssc-timesig-1-256` | An `.ssc` with a 1/256 time signature | `Background.cpp:499` steps a measure (0 rows) at a time when the song is played |-| `04-dwi-lowercase-mode` | A `.dwi` with `#single:` | `NotesLoaderDWI.cpp:278` compares the mode in upper case only, then asserts |+| `./verify.sh stability` | 5 min | Damaged files that broke the old song scan: a WAV whose chunk size is -8, an `.ssc` with `#SPEEDS:=;`, a `.dwi` with `#single:`, an `.ssc` with a 1/256 time signature. Each engine **loaded** the song list, **crashed**, or **hung** (no song list after 180 s). |+| `./verify.sh playback` | 3 min | The 1/256 song played through a minimal theme on each engine's own `_fallback`, the same Lua on both: **played**, **crashed** or **hung**. |+| `tests/timing/build_itgmania.sh` then `./verify.sh timing` | 30 min the first time | StepManiaY's 653-note test chart played by a self-test that presses the panels through the engine's own input system at exact moments, with a recorder that writes every judged step and the note it was meant for. | -`tests/timing` plays StepManiaY's 653-note test chart (BPM changes, stops, a delay, 16th and 32nd-jacks, streams, holds, rolls and mines) with a self-test that presses the panels through the engine's-own input system at exact moments, and a flight recorder that writes every judged step with the note-it was meant for. ITGmania has neither, so `build_itgmania.sh` builds ITGmania 1.3.0 from its own-source and build container with only `itgmania-1.3.0-timing-harness.patch` added: 885 lines added,-none of ITGmania's changed, so every judgment is still ITGmania's own. The runs: exact steps at 1.0x-and 0.75x, exact steps with a 60 ms frame every second, every step 16 ms late or early at 1.5x, and-10 ms late with the long frames. Each counts the steps judged against the wrong note or none, the-misses, and the judgments. It runs on arm64 (`tests/timing/platform`) so a slow emulated frame can't-stand in for the long frames the test makes on purpose.+`tests/stability/run.py` lists, for every damaged file, the line of ITGmania 1.3.0's source it+reaches. `tests/stability/make_songs.py` and `tests/timing/make_song.py` make every file from+nothing (the timing chart is StepManiaY's own, its music a generated click track), so the kit+carries no one else's music or charts. -`tests/stability/make_songs.py` makes every file from nothing, so the kit carries no one else's-music or charts. Add a case there, with the line of the old engine it reaches, and in the test's-`CASES`.+Stability and playback run the official x86_64 builds (ITGmania publishes no other Linux build), so an+Apple silicon Mac runs them under emulation. Timing runs on your machine's own architecture, so an+emulator can't stand in for the long frames it makes on purpose.++## The ITGmania side of the timing test++ITGmania has no self-test or recorder, so `tests/timing/build_itgmania.sh` builds ITGmania 1.3.0+(tag `v1.3.0`, commit `8a3bf8884a`, the official release's build) from its own repository, in its+own build container (`Utils/Dockerfile-linux`), with `tests/timing/itgmania-1.3.0-timing-harness.patch`+added. The patch only adds lines; ITGmania's judging is untouched and every judgment is its own:++```sh+git apply --numstat tests/timing/itgmania-1.3.0-timing-harness.patch # the second column, lines removed, is 0 throughout+```++To build it without the script, follow ITGmania's own build instructions after:++```sh+git clone --branch v1.3.0 --recurse-submodules https://github.com/itgmania/itgmania.git+cd itgmania && git apply ../stepmaniay-verification/tests/timing/itgmania-1.3.0-timing-harness.patch+```++then point `path` in `tests/timing/engines-<arch>.json` at your `itgmania` binary. The script prints+the binary's SHA-256; each run's results name the binaries they ran.++The self-test's flags are the same on both engines: `--smytest=timing` turns it on when gameplay+starts, `--timingoffsets=16` presses every step 16 ms late (late positive), `--timingrate=1.5` sets+the music rate, `--timingstutter=60` makes one frame 60 ms long every second.++## Results so far++From `results/` (November 2026, StepManiaY 1.3.1.2611032302 against ITGmania 1.3.0):++| Test | ITGmania 1.3.0 | StepManiaY |+| --- | --- | --- |+| WAV with a chunk size of -8 | hung | loaded (the file refused, with the reason in `userlog.txt`) |+| `.ssc` with `#SPEEDS:=;` | crashed (segmentation fault) | loaded (the entry skipped and logged) |+| `.dwi` with `#single:` | crashed (`Unrecognized DWI notes format`) | loaded |+| `.ssc` with a 1/256 time signature, scanned | loaded | loaded (the signature skipped and logged) |+| The same song, played | hung | played |+| 653-note chart, exact steps, rate 1.0 and 0.75 | 653 W1 | 653 W1 |+| Exact steps, a 60 ms frame every second (3 runs) | 4–5 notes missed | 653 W1 |+| Every step 16 ms late, rate 1.5 (3 runs) | 48–55 steps on the wrong note, 12–13 missed | 653 W1 |+| Every step 16 ms early, rate 1.5 | 16 missed | 653 W1 |+| 10 ms late, a 60 ms frame every second | 5 missed | 653 W1 |++`python3 tools/summarize.py results/*/` prints every run's table. Each results folder keeps every+run's `log.txt`, `userlog.txt`, crash report, console output and recorder CSV beside `results.json`.++## Not tested here yet++Claims that need hardware: XInput controllers polled every millisecond on Windows, the sound+clock's noise and drift over hours on real sound cards, Play Together's sync between machines, and+frame times on a Raspberry Pi 5. Those need a capture of the sound or the controller from outside the+game, so this kit will carry their procedures and analysis rather than run them in a container.++## Sharing what you found++Run it on your machine and publish your `results/` folder wherever you like, or send it through+[stepmaniay.com/feedback](https://stepmaniay.com/feedback/). Results that disagree with ours are+the ones we most want to see. added tests/playback/platform +1 −0
@@ -0,0 +1 @@+linux/amd64 added tests/stability/platform +1 −0
@@ -0,0 +1 @@+linux/amd64 modified tests/timing/build_itgmania.sh +13 −6
@@ -2,14 +2,21 @@ # Builds ITGmania 1.3.0 (github.com/itgmania/itgmania, tag v1.3.0, commit 8a3bf8884a, the # official release's build) with only the timing harness (itgmania-1.3.0-timing-harness.patch, # which adds lines and changes none), in ITGmania's own build container (Utils/Dockerfile-linux),-# into .build/itgmania-1.3.0-harness/, and prints the binary's SHA-256.+# into .build/itgmania-1.3.0-harness-<arch>/, and prints the binary's SHA-256. It builds for this+# machine's architecture, as ./verify.sh timing runs (PLATFORM=linux/amd64 or linux/arm64 for another). #-# tests/timing/build_itgmania.sh (PLATFORM=linux/amd64 for an x86_64 build)+# tests/timing/build_itgmania.sh set -eu KIT=$(cd "$(dirname "$0")/../.." && pwd) SRC="$KIT/.build/itgmania-1.3.0"-OUT="$KIT/.build/itgmania-1.3.0-harness"-PLATFORM=${PLATFORM:-linux/arm64}+case "$(uname -m)" in+ x86_64 | amd64) NATIVE=linux/amd64 ;;+ arm64 | aarch64) NATIVE=linux/arm64 ;;+ *) NATIVE="linux/$(uname -m)" ;;+esac+PLATFORM=${PLATFORM:-$NATIVE}+ARCH=${PLATFORM#linux/}+OUT="$KIT/.build/itgmania-1.3.0-harness-$ARCH" COMMIT=8a3bf8884acdcb69363f7f0ea592138e398c71e4 [ -d "$SRC/.git" ] || git clone -q --branch v1.3.0 --depth 1 https://github.com/itgmania/itgmania.git "$SRC"@@ -36,7 +43,7 @@ docker build --platform "$PLATFORM" -q -t itgmania-1.3.0-build -f Utils/Dockerfile-linux Utils >/dev/null docker run --rm --platform "$PLATFORM" -v "$SRC:/src" -w /src itgmania-1.3.0-build sh -c \- 'cmake -B Build-harness -DCMAKE_BUILD_TYPE=Release >/dev/null && cmake --build Build-harness -j "$(nproc)" >/dev/null'+ "cmake -B Build-harness-$ARCH -DCMAKE_BUILD_TYPE=Release >/dev/null && cmake --build Build-harness-$ARCH -j \$(nproc) >/dev/null" rm -rf "$OUT" mkdir -p "$OUT"@@ -44,4 +51,4 @@ cp -R "$SRC/$d" "$OUT/" done cp "$SRC/itgmania" "$OUT/"-shasum -a 256 "$OUT/itgmania"+if command -v sha256sum >/dev/null; then sha256sum "$OUT/itgmania"; else shasum -a 256 "$OUT/itgmania"; fi added tests/timing/engines-amd64.json +13 −0
@@ -0,0 +1,13 @@+{+ "stepmaniay-1.3.1.2611032302-amd64": {+ "url": "https://data.stepmaniay.com/library/games/a3/a3e4dd28f2604b366883224adace9a920fbf0d383f9b7db1a13c5e403a40fc34/StepManiaY-1.3.1-2611032302-linux-x86_64.tar.gz",+ "sha256": "a3e4dd28f2604b366883224adace9a920fbf0d383f9b7db1a13c5e403a40fc34",+ "binary": "StepManiaY/stepmaniay",+ "published": "https://stepmaniay.com/download/downloads.json"+ },+ "itgmania-1.3.0+timing-harness": {+ "path": "/build/itgmania-1.3.0-harness-amd64/itgmania",+ "build": "tests/timing/build_itgmania.sh",+ "source": "https://github.com/itgmania/itgmania/releases/tag/v1.3.0 (commit 8a3bf8884a) with tests/timing/itgmania-1.3.0-timing-harness.patch"+ }+} added tests/timing/engines-arm64.json +13 −0
@@ -0,0 +1,13 @@+{+ "stepmaniay-1.3.1.2611032302-arm64": {+ "url": "https://data.stepmaniay.com/library/games/40/40d499e59b372bd0f3e5ec6a62d9dc7d676e6046a92bdf769cf5fa6a0bad181b/StepManiaY-1.3.1-2611032302-linux-arm64.tar.gz",+ "sha256": "40d499e59b372bd0f3e5ec6a62d9dc7d676e6046a92bdf769cf5fa6a0bad181b",+ "binary": "StepManiaY/stepmaniay",+ "published": "https://stepmaniay.com/download/downloads.json"+ },+ "itgmania-1.3.0+timing-harness": {+ "path": "/build/itgmania-1.3.0-harness-arm64/itgmania",+ "build": "tests/timing/build_itgmania.sh",+ "source": "https://github.com/itgmania/itgmania/releases/tag/v1.3.0 (commit 8a3bf8884a) with tests/timing/itgmania-1.3.0-timing-harness.patch"+ }+} deleted tests/timing/engines.json +0 −13
@@ -1,13 +0,0 @@-{- "stepmaniay-1.3.1.2611032302-arm64": {- "url": "https://data.stepmaniay.com/library/games/40/40d499e59b372bd0f3e5ec6a62d9dc7d676e6046a92bdf769cf5fa6a0bad181b/StepManiaY-1.3.1-2611032302-linux-arm64.tar.gz",- "sha256": "40d499e59b372bd0f3e5ec6a62d9dc7d676e6046a92bdf769cf5fa6a0bad181b",- "binary": "StepManiaY/stepmaniay",- "published": "https://stepmaniay.com/download/downloads.json"- },- "itgmania-1.3.0+timing-harness": {- "path": "/build/itgmania-1.3.0-harness/itgmania",- "build": "tests/timing/build_itgmania.sh",- "source": "https://github.com/itgmania/itgmania/releases/tag/v1.3.0 (commit 8a3bf8884a) with tests/timing/itgmania-1.3.0-timing-harness.patch"- }-} deleted tests/timing/platform +0 −1
@@ -1 +0,0 @@-linux/arm64 added tools/publish.sh +37 −0
@@ -0,0 +1,37 @@+#!/bin/sh+# Packs this repository's committed history for stepmaniay.com, where anyone can clone it with+# plain git over HTTPS: a bare copy in one pack with the files a client reads first (info/refs,+# objects/info/packs), a source tarball for anyone without git, and release.json naming the commit,+# the tarball's SHA-256 and the latest results. One archive, for the server's+# `manage.py verification_publish`.+#+# tools/publish.sh <archive.tar>+set -eu+KIT=$(cd "$(dirname "$0")/.." && pwd)+OUT=$(cd "$(dirname "$1")" && pwd)/$(basename "$1")+WORK=$(mktemp -d)+trap 'rm -rf "$WORK"' EXIT++[ -z "$(git -C "$KIT" status --porcelain)" ] || { echo "commit first: what's published is the committed history" >&2; exit 1; }++git clone -q --bare --no-local "$KIT" "$WORK/bare.git"+git -C "$WORK/bare.git" repack -q -a -d -f+git -C "$WORK/bare.git" update-server-info++mkdir -p "$WORK/out/repo/info" "$WORK/out/repo/objects/info" "$WORK/out/repo/objects/pack"+cp "$WORK/bare.git/HEAD" "$WORK/out/repo/HEAD"+cp "$WORK/bare.git/info/refs" "$WORK/out/repo/info/refs"+cp "$WORK/bare.git/objects/info/packs" "$WORK/out/repo/objects/info/packs"+cp "$WORK/bare.git"/objects/pack/pack-*.pack "$WORK/bare.git"/objects/pack/pack-*.idx "$WORK/out/repo/objects/pack/"++COMMIT=$(git -C "$KIT" rev-parse HEAD)+TARBALL=stepmaniay-verification-$(git -C "$KIT" rev-parse --short=12 HEAD).tar.gz+git -C "$KIT" archive --format=tar.gz --prefix=stepmaniay-verification/ -o "$WORK/out/$TARBALL" HEAD++# Each test's latest results folder.+LATEST=$(for t in "$KIT"/tests/*/; do ls -d "$KIT/results/$(basename "$t")"-* 2>/dev/null | sort | tail -n 1; done)+# shellcheck disable=SC2086+python3 "$KIT/tools/release.py" "$COMMIT" "$WORK/out/$TARBALL" $LATEST > "$WORK/out/release.json"++tar -cf "$OUT" -C "$WORK/out" .+echo "$OUT: commit $COMMIT, $(du -h "$OUT" | cut -f1)" added tools/release.py +32 −0
@@ -0,0 +1,32 @@+"""release.json for the website: the commit published, its source tarball (name, size, SHA-256)+and the latest results of every test (summarize.py's tables).++ python3 release.py <commit> <tarball> <results folder> [...]+"""++from __future__ import annotations++import hashlib+import json+import sys+from datetime import datetime, timezone+from pathlib import Path++sys.path.insert(0, str(Path(__file__).resolve().parent))+import summarize # noqa: E402+++def main() -> None:+ commit, tarball, *folders = sys.argv[1:]+ data = Path(tarball).read_bytes()+ release = {+ "commit": commit,+ "published": datetime.now(timezone.utc).isoformat(timespec="seconds"),+ "tarball": {"name": Path(tarball).name, "size": len(data), "sha256": hashlib.sha256(data).hexdigest()},+ "results": [summarize.summary(Path(f)) for f in folders],+ }+ print(json.dumps(release, indent=2, ensure_ascii=False))+++if __name__ == "__main__":+ main() modified tools/summarize.py +56 −16
@@ -1,6 +1,8 @@-"""A results folder's results.json as a Markdown table: one row per case, one column per engine.+"""Results folders as tables: one row per case, one column per engine. A timing folder's repeats+(<folder>/repeats/*/) count with it, so each cell gives the range over every run. - python3 summarize.py <results folder> [...]+ python3 summarize.py <results folder> [...] Markdown+ python3 summarize.py --json <results folder> [...] the same as JSON (the website's release) """ from __future__ import annotations@@ -10,21 +12,59 @@ from pathlib import Path +def runs_of(folder: Path) -> tuple[dict, list[dict]]:+ data = json.loads((folder / "results.json").read_text())+ runs = list(data["runs"])+ for repeat in sorted((folder / "repeats").glob("*/results.json")) if (folder / "repeats").is_dir() else []:+ runs += json.loads(repeat.read_text())["runs"]+ return data["meta"], runs+++def span(values: list[int]) -> str:+ return f"{min(values)}" if min(values) == max(values) else f"{min(values)}–{max(values)}"+++def cell(runs: list[dict]) -> str:+ if not runs:+ return "not run"+ recorded = [r["recorded"] for r in runs if r.get("recorded")]+ times = f" ({len(runs)} runs)" if len(runs) > 1 else ""+ if not recorded:+ return " / ".join(sorted({r["verdict"] for r in runs})) + times+ perfect = all(not (r["wrong_note"] or r["no_note"] or r["misses"]) and set(r["judgments"]) == {"W1"} for r in recorded)+ if perfect:+ return f"{span([r['steps'] for r in recorded])} W1, none on the wrong note or missed{times}"+ return (f"{span([r['wrong_note'] for r in recorded])} on the wrong note, "+ f"{span([r['misses'] for r in recorded])} missed{times}")+++def summary(folder: Path) -> dict:+ meta, runs = runs_of(folder)+ engines = list(meta["engines"])+ rows = []+ for case, description in meta["cases"].items():+ rows.append({+ "case": case,+ "about": description,+ "cells": {e: cell([r for r in runs if r["engine"] == e and r["case"] == case]) for e in engines},+ })+ return {"test": folder.name.split("-", 1)[0], "folder": folder.name, "when": meta["when"],+ "platform": meta["platform"], "engines": engines, "rows": rows}++ def main() -> None:- for folder in map(Path, sys.argv[1:]):- data = json.loads((folder / "results.json").read_text())- engines = list(data["meta"]["engines"])- cases = list(data["meta"]["cases"])- cell = {(r["engine"], r["case"]): r for r in data["runs"]}- print(f"### {folder.name} ({data['meta']['platform']}, {data['meta']['when']})\n")- print("| Case | " + " | ".join(engines) + " | What it reaches in the old engine |")- print("| --- |" + " --- |" * (len(engines) + 1))- for case in cases:- row = []- for engine in engines:- r = cell.get((engine, case))- row.append(f"{r['verdict']} ({r['seconds']} s)" if r else "not run")- print(f"| {case} | " + " | ".join(row) + f" | {data['meta']['cases'][case]} |")+ args = sys.argv[1:]+ as_json = args[:1] == ["--json"]+ tables = [summary(Path(a)) for a in (args[1:] if as_json else args)]+ if as_json:+ print(json.dumps(tables, indent=2, ensure_ascii=False))+ return+ for t in tables:+ print(f"### {t['folder']} ({t['platform']}, {t['when']})\n")+ print("| Case | " + " | ".join(t["engines"]) + " |")+ print("| --- |" + " --- |" * len(t["engines"]))+ for row in t["rows"]:+ print(f"| {row['case']} | " + " | ".join(row["cells"][e] for e in t["engines"]) + " |") print() modified verify.sh +11 −3
@@ -7,9 +7,17 @@ TEST=${1:?usage: verify.sh <test> (tests: $(ls tests | tr '\n' ' '))} KIT=$(cd "$(dirname "$0")" && pwd) OUT="$KIT/results/$TEST-$(date -u +%Y%m%dT%H%MZ)"-# A test can name its own platform and engines (tests/<test>/platform, tests/<test>/engines.json).-PLATFORM=$(cat "$KIT/tests/$TEST/platform" 2>/dev/null || echo linux/amd64)-ENGINES=tests/$TEST/engines.json+# A test can pin its platform (tests/<test>/platform); otherwise it runs on this machine's own, so no+# emulator stands between the engine and its frames. Its engines are tests/<test>/engines-<arch>.json,+# tests/<test>/engines.json or engines.json, the first there is.+case "$(uname -m)" in+ x86_64 | amd64) NATIVE=linux/amd64 ;;+ arm64 | aarch64) NATIVE=linux/arm64 ;;+ *) NATIVE="linux/$(uname -m)" ;;+esac+PLATFORM=$(cat "$KIT/tests/$TEST/platform" 2>/dev/null || echo "$NATIVE")+ENGINES=tests/$TEST/engines-${PLATFORM#linux/}.json+[ -f "$KIT/$ENGINES" ] || ENGINES=tests/$TEST/engines.json [ -f "$KIT/$ENGINES" ] || ENGINES=engines.json mkdir -p "$OUT" "$KIT/.engines" "$KIT/.build" docker build --platform "$PLATFORM" -q -t "smy-verification:${PLATFORM#linux/}" "$KIT/docker" >/dev/null