enricobuehler
077db416ec
Wave-2 PW5 stage 5. Depth is STILL 1 — the handles alternate per frame, one in flight. PyroWave's `Encoder` cannot hold two frames. Not "probably not" — structurally not. `Encoder::Impl` owns ONE each of `wavelet_img_high_res`, `bucket_buffer`, `meta_buffer`, `block_stat_buffer`, `payload_data` and `quant_buffer`, and `Impl::encode` OPENS by discarding them: an image barrier with `VK_IMAGE_LAYOUT_UNDEFINED` as the old layout — a written promise that nothing else is reading it — plus three `fill_buffer` clears. Two encodes recorded into two command buffers and submitted to one queue have no execution dependency in Vulkan (submission order orders the START, not the completion), so N+1's DWT would overwrite the wavelet bands and zero the RDO buckets while N's block packing still reads them. Content-dependent, silent. So overlap means TWO handles on one device, alternated — one per slot. Every resource above is then private per handle, and within a handle the encodes stay strictly serialized (a slot's next frame is recorded only after that slot's previous one retired), which leaves patch 0004's scratch-pool invariant intact without touching it. THE LANDMINE, and it is the reason this stage is its own commit: `sequence_count` ALSO lives on `Impl`, and it is the 3-bit counter stamped into every block header. Two handles each count 1,2,3... alone, so the wire sees 1,1,2,2,3,3.... The decoder restarts a frame only when the value CHANGES (`diff = (hdr.sequence - last_seq) & 0x7; restart = diff != 0`), so a repeat reads as MORE BLOCKS OF THE SAME FRAME: `clear()` never runs, `decoded_frame_for_current_sequence` stays true, and the second frame of each pair is swallowed. Half frame rate, occasional mixed-frame blocks, no error anywhere — on every client, since pf-client-core and the Apple Metal hand-port parse the same field. `patches/0007-encoder-sequence-override.patch` (new, ~38 lines) exposes `Encoder::set_next_sequence` + a `pyrowave_encoder_set_next_sequence` C entry + a `PYROWAVE_SEQUENCE_MASK` define, so ONE monotonic counter on the Rust side is stamped regardless of which handle encodes. The setter stores `(seq - 1) & mask` because `Impl::encode` pre-increments — its contract is about the next ENCODE, not the next store. Inert when unused, so the whole Windows backend is untouched. No `.def` change: the C API is a static archive. PREDICTED, THEN OBSERVED. A negative control on .21 (the override call removed, nothing else) reads the wire out at exactly: [1, 1, 2, 2, 3, 3, 4, 4, 5, 5, 6, 6, 7, 7, 0, 0, 1, 1, 2, 2] which is the analysis's prediction character for character, and with the override: +1 mod 8, all 20 frames, through the 3-bit wrap. THE GATE, `wire_sequence_increments_across_alternating_handles`, checks three things over 20 frames because any one alone could pass while the stream is broken: the wire counter advances by 1 mod 8; ONE persistent decoder (its `last_seq` carried across every push, exactly like a client's) reports every AU decodable; and consecutive decoded pictures DIFFER. Content moves every frame — and the first run caught a trap in the harness itself rather than the encoder: `test_card` starts its LCG at `seed | 1`, so seeds 2 and 3 build a byte-identical card and the test faked the very repeat it hunts. Odd seeds only now, with the reason written down. A runtime self-check backs the test up where the test cannot reach: after packetize, the stamped sequence is compared against what we asked for, and a mismatch logs once per process naming patch 0007. A re-vendor that loses the patch would not fail to build — it would fail on glass, subtly, and this makes it loud instead. Two byte reads per frame. `reset()` rebuilds both handles and `Drop` destroys both, each with the same null-immediately discipline the single handle had (`pyrowave_encoder_destroy` is a bare `delete` with no null check, so a stale pointer left in the field is a double free). Vendored-patch discipline: patch 0007 re-applies clean to a pristine vendor checkout (verified by stashing the vendor tree and re-applying), and `git diff crates/pyrowave-sys/vendor/` touches exactly the four intended files. VERIFIED ON GLASS (.21, RTX 5070 Ti, GPU idle at 180 MHz of 3090): all 8 `#[ignore]`d GPU tests pass, including the new gate and the 4:2:0 / 4:4:4 / 24-bpp PSNR smokes. Gates green at CI parity.
PyroWave
PyroWave is an intra-only video codec (practially speaking a still-image codec) that is optimized for extremely fast GPU encode (< ~0.1 ms encode and decode at 1080p, < ~0.2 ms at 4K). It is fully implemented in Vulkan compute shaders.
The targeted bit-rates are quite high (~200+ mbit/s) and the intended use case is local network game streaming over ethernet with absolute minimum latency where bandwidth is less of a concern.
It is integrated as part of my pyrofling project.
It is similar in scope to my master thesis from 2014, except the use case now is game streaming instead of adaptation of raw video to ethernet links.
Overview
Colorspace
Currently implemented YCbCr 4:2:0 and 4:4:4.
Wavelets
The images are transformed with the Discrete Wavelet Transform using CDF 9/7 filter. This is basically the exact same as JPEG2000.
Exact rate control
The encoder can target exact maximum bitrate for an encoded image.
Trivial "entropy" coding
The encoding of coefficients is trivial compared to normal image codecs, and is responsible for a large increase in bit-rate, especially at higher compression ratios. This simplicity massively improves encode/decode performance, since it's extremely parallelizable. It also makes it possible to do a single-pass exact rate control for an entire image.
It should be possible in theory to add a more proper entropy coder to the encoded bit-planes to achieve somewhat competent compression, but that has not been explored and is out of scope. High Throughput JPEG2000 is likely a good place to look for that anyway.
Robustness against packet loss
Being intra-only and encoding 64x64 blocks of coefficients in isolation ensures great recovery against packet loss. PyroWave has been battled tested over long distance streaming over fiber links.
Bitstream definition
Building
PyroWave is intended to be built alongside PyroFling with Granite in the normal case.
Standalone C API
NOTE: This API is still under development and the API/ABI is not yet quite stable.
A small portion of Granite needs to be checked out.
bash checkout_granite.sh
Build normally with CMake and a C API is installed.
$ mkdir build
$ cd build
$ cmake .. -DCMAKE_INSTALL_PREFIX=output -DCMAKE_BUILD_TYPE=Release -G Ninja
$ ninja install
[0/1] Install the project...
-- Install configuration: "Release"
-- Installing: ...../build/output/include/pyrowave/pyrowave.h
-- Installing: ...../build/output/lib/libpyrowave-shared.so.0.0.0
-- Installing: ...../build/output/lib/libpyrowave-shared.so.0
-- Installing: ...../build/output/lib/libpyrowave-shared.so
-- Installing: ...../build/output/share/pyrowave-shared/cmake/pyrowave-sharedConfig.cmake
-- Installing: ...../build/output/share/pyrowave-shared/cmake/pyrowave-sharedConfig-release.cmake
-- Installing: ...../build/output/share/pkgconfig/pyrowave-shared.pc
The build is tested on Linux, MinGW, msys2 and MSVC.
See pyrowave-c-test which unit tests the shared C API.
That test also serves as a basic user guide for the API.
build-steamrt.sh builds against the Sniper SDK and is also supported.
Local development and CLI
For the sample and test applications in this repo however, check out
the full https://github.com/Themaister/Granite before invoking CMake.
Build with -DPYROWAVE_DEVEL=ON to get the "full" build.
git clone --depth 1 --recursive --shallow-submodules https://github.com/Themaister/Granite Granite
Basic encoder/decoder CLI
The basic CLI takes a y4m and dumps out a raw bitstream. This is just intended for local testing.
pyrowave-encode test.y4m out.wave 400000
This encodes a raw bitstream where each frame consumes maximum 400 KB. To decode back to y4m:
pyrowave-decode out.wave out.y4m