mirror of
https://github.com/blitzkit/blitzkit.git
synced 2026-11-13 20:56:11 +01:00
Mark Greene
c5fea49f5d
## Why In CI, `models_legendary` was the slowest chunk of the website build (65 min on 2026-11-08), slower than `models_tanks` (49 min) even though there are far fewer camouflages (443) than tanks (738). In the CI log, each camouflage takes longer than the last, from about 8 s at the start to about 50 s near the end, which looked like a leak. It isn't a leak: - Nothing is kept between routes (no module-level caches) and the file reader caches nothing. - The camouflages later in the list are heavier: newer tanks with more parts and textures. - All the work is JavaScript on one thread, so Astro's 4 routes at a time take turns on one core. Each route's time also includes waiting for the other three. The build only kept about 1 core busy. A CPU profile of one heavy camouflage (1224, 52 s) showed where the time goes: - **About 60% in `readTexture`:** decoding DDS textures in our own decoder (`core/src/streams/dds.ts`). - **About 16% in `crc32`:** checksumming every `.dvpl` game file with the pure-JavaScript `crc` package. - **Repeated work:** that camouflage decoded 32 textures, but only 8 different files. Every part decodes the camouflage's shared atlas again (4 textures × 7 parts). ## What changed ### DDS decoder (`packages/core/src/streams/dds.ts`) The old decoder allocated at almost every step: - for every pixel, a new 4-value array (`[...color, alpha]`), then `forEach` + `Math.round`; - for every block, a closure plus palette objects built with `.map()` and spread; - for every small read (2–8 bytes), a byte copy through `consumeUint8Array`; - every loop through lodash `times()`, adding a function call per block and per pixel. A 2048×2048 texture has 4 million pixels in 262,000 blocks, so that meant tens of millions of allocations and calls per texture, all on the single JavaScript thread. `decodeBlocks` now: - reads the file's bytes in place; - works out each block's palette once into two reused 16-byte buffers; - writes the pixels straight into the output with plain loops. The output is unchanged. Palette values use the same formulas in the same order (for example `(2 * c0 + c1) / 3`) and are rounded with the same `Math.round(value * 255)`. Rounding once per palette entry gives the same bytes as rounding once per pixel. Two old quirks are kept on purpose: - BC1 is treated as transparent when `color0 < color1` (the standard says `<=`). - BC3's 6-alpha mode stores `255` for index 7, which wraps to `1` as a byte (the standard value is opaque). This is marked in a comment and left for a separate fix, because fixing it changes some textures. ### Texture reuse (`[...chunk_models_tanks]/[id]/model.glb.ts`) `cachedImage` keeps the last 64 finished textures (encoded images), keyed by reader and file path, and wraps `readBaseColor`, `readOcclusion`, `readRoughnessMetallic` and `readNormal`. Repeat requests reuse the same image instead of decoding and encoding again. The cap keeps memory from growing over the build. The logic inside the four readers is unchanged; they're only re-indented. ### DVPL checksum (`packages/core/src/blitz/readDVPL.ts`) `Bun.hash.crc32` replaces the `crc` package and gives the same values. `readDVPL` only runs under Bun (every caller reads through `LocalVFS`, which already uses `Bun.file`). `crc` is removed from core's dependencies; the scripts package still uses it. ## Timings All local timings are on the same Mac, one after another, with nothing else heavy running. ### Full builds "Before" for a local build is CI time ÷ 1.5, which matches the real ~40-minute local camouflage build of main. | Build | CI on main (2026-11-08) | Before, local (CI ÷ 1.5) | After, local | Speed-up | |---|---|---|---|---| | models_tanks | 49 min | ~33 min | 6 min 37 s | ~4.9× | | models_legendary | 65 min | ~43 min | 4 min 52 s | ~8.9× | With the 2026-11-10 production run (58 and 57 min), it's ~5.8× for tanks and ~7.8× for camouflages. The camouflage build after each step, locally: | Stage | models_legendary | |---|---| | main (real local build) | ~40 min | | Texture reuse + checksum | 32 min 24 s | | + DDS decoder | 4 min 52 s | ### Samples (main against this change, same machine) | Sample | main | After | Speed-up | Identical | |---|---|---|---|---| | 30 DDS textures (BC1/BC2/BC3, incl. sRGB) | 6.1 s | 0.3 s | ~20× | 30/30 | | 5 heavy late camouflages from CI | 124.6 s | 13.4 s | ~9.3× | 5/5 | | 45 camouflages (every 10th in build order) | 258.5 s | 77.2 s | ~3.35× | 45/45 | | 29 tanks (15 slowest in CI + 14 spread) | 105.3 s | 43.6 s | ~2.4× | 29/29 | The 5 heavy camouflages after each step: | Camouflage | main | Texture reuse | + checksum | + DDS decoder | |---|---|---|---|---| | 1224 | 49.7 s | 6.2 s | 6.4 s | 3.2 s | | 2146 | 17.2 s | 6.3 s | 6.4 s | 2.9 s | | 2122 | 31.4 s | 7.5 s | 5.9 s | 2.6 s | | 2143 | 16.9 s | 10.1 s | 6.1 s | 2.8 s | | 2059 | 9.6 s | 5.8 s | 4.1 s | 1.9 s | | **Total** | **124.6 s** | **36.0 s** | **28.9 s** | **13.4 s** | The single-camouflage speed-up (3.35× on the 45-camouflage sample) is smaller than the full-build one (~8×). With decoding no longer holding the JavaScript thread, Astro's 4 routes at a time overlap instead of queuing, and sharp's threads run alongside them. ### Full site build (`CHUNK=*`, all chunks one after another) 36 min 36 s, 0 errors, 21,613 routes, peak memory 4.6 GB. | Chunk | Time | |---|---| | api | ~4.3 min | | models_tanks | ~6.1 min | | models_legendary | ~4.6 min | | default (pages, 9 locales) | ~21 min | ### CPU and memory - The tank build used 1,183 s of CPU in 397 s, so about 3 cores busy on average; before, it was about 1. - Peak memory was 3.5 GB for the tank build and 4.6 GB for the full site. - CI runners have 4 cores and 16 GB, so this fits. No threads or processes were added: Astro's concurrency (4) and sharp's threading are unchanged. ## What this means for CI The chunks run as parallel jobs, so the website build takes as long as its slowest chunk. Before, that was `models_legendary` at about 65 min. Using CI ≈ 1.5× local, tank models should take about 10 min and camouflages about 7–8 min. `default` (about 36 min in CI) becomes the slowest chunk, so the whole build should drop from about 65 min to about 36. This is an estimate until the next CI run. ## Checked - Byte-identical GLBs (SHA-1) for 45 camouflages and 29 tanks against main, and decoded pixels for 30 DDS textures. - A full site build finishes with no errors. - Animations and particles are unaffected: the camouflage GLBs, including their animation data, are byte-identical.