Commit Graph
5035 Commits
Author SHA1 Message Date
TrèsAbhi 26f3245257 Merge pull request #437 from Hitjack007/faster-texture-exports
Fix: Faster tank and legendary camouflage model builds
2026-11-13 13:40:20 -06:00
Mark Greene c5fea49f5d Fix: Faster tank and legendary camouflage model builds
## Why

In CI, `models_legendary` was the slowest chunk of the website build (65 min on 2026-11-08), slower than `models_tanks` (49 min) even though there are far fewer camouflages (443) than tanks (738). In the CI log, each camouflage takes longer than the last, from about 8 s at the start to about 50 s near the end, which looked like a leak.

It isn't a leak:

- Nothing is kept between routes (no module-level caches) and the file reader caches nothing.
- The camouflages later in the list are heavier: newer tanks with more parts and textures.
- All the work is JavaScript on one thread, so Astro's 4 routes at a time take turns on one core. Each route's time also includes waiting for the other three. The build only kept about 1 core busy.

A CPU profile of one heavy camouflage (1224, 52 s) showed where the time goes:

- **About 60% in `readTexture`:** decoding DDS textures in our own decoder (`core/src/streams/dds.ts`).
- **About 16% in `crc32`:** checksumming every `.dvpl` game file with the pure-JavaScript `crc` package.
- **Repeated work:** that camouflage decoded 32 textures, but only 8 different files. Every part decodes the camouflage's shared atlas again (4 textures × 7 parts).

## What changed

### DDS decoder (`packages/core/src/streams/dds.ts`)

The old decoder allocated at almost every step:

- for every pixel, a new 4-value array (`[...color, alpha]`), then `forEach` + `Math.round`;
- for every block, a closure plus palette objects built with `.map()` and spread;
- for every small read (2–8 bytes), a byte copy through `consumeUint8Array`;
- every loop through lodash `times()`, adding a function call per block and per pixel.

A 2048×2048 texture has 4 million pixels in 262,000 blocks, so that meant tens of millions of allocations and calls per texture, all on the single JavaScript thread.

`decodeBlocks` now:

- reads the file's bytes in place;
- works out each block's palette once into two reused 16-byte buffers;
- writes the pixels straight into the output with plain loops.

The output is unchanged. Palette values use the same formulas in the same order (for example `(2 * c0 + c1) / 3`) and are rounded with the same `Math.round(value * 255)`. Rounding once per palette entry gives the same bytes as rounding once per pixel. Two old quirks are kept on purpose:

- BC1 is treated as transparent when `color0 < color1` (the standard says `<=`).
- BC3's 6-alpha mode stores `255` for index 7, which wraps to `1` as a byte (the standard value is opaque). This is marked in a comment and left for a separate fix, because fixing it changes some textures.

### Texture reuse (`[...chunk_models_tanks]/[id]/model.glb.ts`)

`cachedImage` keeps the last 64 finished textures (encoded images), keyed by reader and file path, and wraps `readBaseColor`, `readOcclusion`, `readRoughnessMetallic` and `readNormal`. Repeat requests reuse the same image instead of decoding and encoding again. The cap keeps memory from growing over the build. The logic inside the four readers is unchanged; they're only re-indented.

### DVPL checksum (`packages/core/src/blitz/readDVPL.ts`)

`Bun.hash.crc32` replaces the `crc` package and gives the same values. `readDVPL` only runs under Bun (every caller reads through `LocalVFS`, which already uses `Bun.file`). `crc` is removed from core's dependencies; the scripts package still uses it.

## Timings

All local timings are on the same Mac, one after another, with nothing else heavy running.

### Full builds

"Before" for a local build is CI time ÷ 1.5, which matches the real ~40-minute local camouflage build of main.

| Build | CI on main (2026-11-08) | Before, local (CI ÷ 1.5) | After, local | Speed-up |
|---|---|---|---|---|
| models_tanks | 49 min | ~33 min | 6 min 37 s | ~4.9× |
| models_legendary | 65 min | ~43 min | 4 min 52 s | ~8.9× |

With the 2026-11-10 production run (58 and 57 min), it's ~5.8× for tanks and ~7.8× for camouflages.

The camouflage build after each step, locally:

| Stage | models_legendary |
|---|---|
| main (real local build) | ~40 min |
| Texture reuse + checksum | 32 min 24 s |
| + DDS decoder | 4 min 52 s |

### Samples (main against this change, same machine)

| Sample | main | After | Speed-up | Identical |
|---|---|---|---|---|
| 30 DDS textures (BC1/BC2/BC3, incl. sRGB) | 6.1 s | 0.3 s | ~20× | 30/30 |
| 5 heavy late camouflages from CI | 124.6 s | 13.4 s | ~9.3× | 5/5 |
| 45 camouflages (every 10th in build order) | 258.5 s | 77.2 s | ~3.35× | 45/45 |
| 29 tanks (15 slowest in CI + 14 spread) | 105.3 s | 43.6 s | ~2.4× | 29/29 |

The 5 heavy camouflages after each step:

| Camouflage | main | Texture reuse | + checksum | + DDS decoder |
|---|---|---|---|---|
| 1224 | 49.7 s | 6.2 s | 6.4 s | 3.2 s |
| 2146 | 17.2 s | 6.3 s | 6.4 s | 2.9 s |
| 2122 | 31.4 s | 7.5 s | 5.9 s | 2.6 s |
| 2143 | 16.9 s | 10.1 s | 6.1 s | 2.8 s |
| 2059 | 9.6 s | 5.8 s | 4.1 s | 1.9 s |
| **Total** | **124.6 s** | **36.0 s** | **28.9 s** | **13.4 s** |

The single-camouflage speed-up (3.35× on the 45-camouflage sample) is smaller than the full-build one (~8×). With decoding no longer holding the JavaScript thread, Astro's 4 routes at a time overlap instead of queuing, and sharp's threads run alongside them.

### Full site build (`CHUNK=*`, all chunks one after another)

36 min 36 s, 0 errors, 21,613 routes, peak memory 4.6 GB.

| Chunk | Time |
|---|---|
| api | ~4.3 min |
| models_tanks | ~6.1 min |
| models_legendary | ~4.6 min |
| default (pages, 9 locales) | ~21 min |

### CPU and memory

- The tank build used 1,183 s of CPU in 397 s, so about 3 cores busy on average; before, it was about 1.
- Peak memory was 3.5 GB for the tank build and 4.6 GB for the full site.
- CI runners have 4 cores and 16 GB, so this fits. No threads or processes were added: Astro's concurrency (4) and sharp's threading are unchanged.

## What this means for CI

The chunks run as parallel jobs, so the website build takes as long as its slowest chunk. Before, that was `models_legendary` at about 65 min. Using CI ≈ 1.5× local, tank models should take about 10 min and camouflages about 7–8 min. `default` (about 36 min in CI) becomes the slowest chunk, so the whole build should drop from about 65 min to about 36. This is an estimate until the next CI run.

## Checked

- Byte-identical GLBs (SHA-1) for 45 camouflages and 29 tanks against main, and decoded pixels for 30 DDS textures.
- A full site build finishes with no errors.
- Animations and particles are unaffected: the camouflage GLBs, including their animation data, are byte-identical.
2026-11-13 17:06:17 +13:00
tresabhi ad0c8b3431 open source WOTB_GLOSSARY 2026-11-08 18:38:46 -06:00
tresabhi e50cc6cf5d only trigger on repo context 2026-11-08 18:33:35 -06:00
tresabhi be9dd9721d non transparent ground 2026-11-08 18:30:07 -06:00
tresabhi 4686877dd1 use wgc by default 2026-11-08 18:13:52 -06:00
tresabhi 47cb0f36f9 download wg parts 2026-11-08 18:10:00 -06:00
tresabhi 7918a88666 fetch urls 2026-11-08 18:07:12 -06:00
tresabhi cd2ef98e22 code split download client 2026-11-08 17:38:55 -06:00
tresabhi f2d3c59e97 lint 2026-11-07 20:52:38 -06:00
tresabhi 1cb760e227 heheheha 2026-11-07 20:51:53 -06:00
tresabhi ce27388c00 organize 2026-11-07 20:48:08 -06:00
tresabhi e8b2651595 move to global scripts 2026-11-07 20:44:23 -06:00
tresabhi 3695e2cd5e add scripts 2026-11-07 20:42:52 -06:00
tresabhi 01445cf845 move download-smart-dlc to closed 2026-11-07 20:20:36 -06:00
tresabhi b121147b43 undo optional dep 2026-11-07 20:17:10 -06:00
tresabhi b3b56462d1 move @blitzkit/closed to optional 2026-11-07 20:07:30 -06:00
tresabhi d42cccc3c2 partial submodules 2026-11-07 20:01:33 -06:00
tresabhi c592501f48 ignore submodules on pr's 2026-11-07 19:59:26 -06:00
TrèsAbhi 5f4cf15500 Merge pull request #405 from Hitjack007/animated-legendary-camos 2026-11-07 18:32:38 -06:00
Mark Greene 232b3e7770 Fix: Address karelpak1's review on animated legendary camouflages
- Sockets bound to a joint now sit right on the joint, as in game. Only the skeleton's own children are bound. This fixes the WZ-114 gun barrels collapsing.
- Clips keep their full duration when it runs past their last key.
- Part states are kept per entity and exported on each part's own switchers, so parts with matching names can't overwrite each other.
- A finished one-shot animation now stops on whatever it switched to, as in game. It only keeps looping if it changes nothing.
- Delayed actions run in delay order.
2026-11-07 15:35:12 +13:00
Mark Greene 99373872ed Merge branch 'main' into animated-legendary-camos 2026-11-07 15:02:21 +13:00
Mark Greene fddf407cc0 Fix: Address review on animated legendary camouflages
- Inline the camouflage preset lookup instead of a one-off `camouflagePreset` helper
- Move the `primitive` override into the `jsxTree` mergers, so `skinTree` takes plain mergers and falls back to `<primitive>` like the mesh and group defaults
2026-11-05 17:26:56 +13:00
tresabhi 88346308cb more hacks for this stupid ai slop wargaming is shoving down the client 2026-11-04 16:14:42 -06:00
tresabhi ebedac0642 some hacks to get some tier 11's compiling 2026-11-04 16:04:02 -06:00
TrèsAbhi a2b97380ac Update countdown text to reflect 3 tanks 2026-11-04 03:50:49 +00:00
tresabhi de4094554e countdown 2026-11-03 09:26:32 -06:00
tresabhi cdcec8548a logs 2026-11-02 22:17:18 -06:00
TrèsAbhi 5ffe5a0a9f Merge pull request #283 from Hitjack007/tankopedia-antagonist-selector 2026-11-01 13:30:18 -06:00
tresabhi 7fb73edb87 Merge branch 'main' into pr/Hitjack007/283 2026-11-01 13:29:12 -06:00
TrèsAbhi 1bc0561dbc Merge pull request #378 from Hitjack007/compare-switch-all-shells 2026-11-01 13:27:49 -06:00
tresabhi c82d2ccf1c Merge branch 'main' into pr/Hitjack007/378-2 2026-11-01 13:26:16 -06:00
TrèsAbhi 15a08b934b Merge pull request #285 from Hitjack007/tankopedia-scale 2026-11-01 13:23:50 -06:00
tresabhi f4894655b0 yeah this is fine 2026-11-01 13:23:35 -06:00
tresabhi e4b49e1600 use canvas 2026-11-01 13:20:02 -06:00
tresabhi 7bfd930cea Revert "remove granuality"
This reverts commit b65229dc80.
2026-11-01 13:15:06 -06:00
tresabhi 71e9e6fd5b Revert "use canvas"
This reverts commit ff507ed4e6.
2026-11-01 13:15:06 -06:00
tresabhi be0be90bdb Revert "no gray picking"
This reverts commit 62637418e5.
2026-11-01 13:15:06 -06:00
tresabhi 62637418e5 no gray picking 2026-11-01 13:07:22 -06:00
tresabhi ff507ed4e6 use canvas 2026-11-01 13:05:23 -06:00
tresabhi b65229dc80 remove granuality 2026-11-01 13:03:37 -06:00
tresabhi 4a6a5b3d34 Merge branch 'main' into pr/Hitjack007/285-1 2026-11-01 12:55:25 -06:00
tresabhi 2150fb3758 Merge branch 'main' into pr/Hitjack007/405 2026-11-01 12:44:23 -06:00
Mark Greene 682f039c6c Fix: Hide unused states of legendary camouflage parts (fixes maus) 2026-11-01 14:46:21 +13:00
tresabhi 0632899ecf disable halloween 2026-10-31 18:14:10 -05:00
TrèsAbhi 72758e9ff7 Merge pull request #423 from Hitjack007/legendary-camos-fix
Fix: Legendary camouflage parts, materials, names and attachments
2026-10-31 13:26:16 -05:00
Mark Greene 8566b2cc2f Feature: Looping garage animations for legendary camouflages
Legendary camo add-ons now loop their garage animation in the Tankopedia viewer. It's always on, with no settings.

- The garage replay now also works out which motions end up looping, and the export writes only those clips
- Customization parts export skeletons, skin weights and sockets
- Parts render as primitives so their bones can animate, with attachments layered on top
- Base tank models are unchanged
2026-10-31 22:15:57 +13:00
tresabhi 8028446788 fall back to github.token for pr runs 2026-10-30 16:59:31 -05:00
tresabhi e216b4cd6b fix 1 pixel lights 2026-10-30 16:49:43 -05:00
Mark Greene c535811c1d Fix: Fade glowing camouflage parts with the lights
Unlit `Textured` materials loaded as `MeshBasicMaterial`, so they stayed bright while the lights receded during a camouflage switch (E 100 legendary camouflage).

- `Textured` materials now export as standard materials with the base color texture as emissive and a black base color, so they look the same as before.
- The Tankopedia model fades their emissive intensity with the lighting transition.
- Additive blending applies to these parts again, since they are standard materials.
2026-10-30 17:26:50 +13:00