enricobuehler
ac0e73321c
PyroWave's wavelet encode runs on the GPU's compute/shader cores, so a GPU-bound game starves it: submit spikes from ~2 ms to ~15 ms under a 95%+ game load and the stream fps collapses. NVENC is immune (separate encoder ASIC). Two levers to let the encode get scheduled ahead of the game's rendering: - Windows process GPU scheduling: D3DKMTSetProcessSchedulingPriorityClass, env PUNKTFUNK_GPU_PRIORITY = off|above-normal|high (default)|realtime. Best-effort, once per process, non-fatal on refusal (enc/windows/pyrowave.rs). - Global-priority Vulkan encode queue (Granite patch 0005): request a VK_KHR_global_priority queue (PYROWAVE_QUEUE_PRIORITY = off|high|realtime, default realtime), downgrading REALTIME→HIGH→none on NOT_PERMITTED so a refused class never regresses the encoder to HEVC. HONEST STATUS: on an RTX 4090 / Windows / WDDM neither moved the ~15 ms spikes — the graphics-vs-compute preemption granularity is the wall, not the priority level. Kept because both are correct, harmless (graceful fallback), and may help other GPUs/drivers. For a GPU-saturated game the working levers are reducing the encode's GPU cost (4:2:0/8-bit) or H.265; PyroWave holds full rate on the desktop and in games that leave the GPU headroom. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
PyroWave
PyroWave is an intra-only video codec (practially speaking a still-image codec) that is optimized for extremely fast GPU encode (< ~0.1 ms encode and decode at 1080p, < ~0.2 ms at 4K). It is fully implemented in Vulkan compute shaders.
The targeted bit-rates are quite high (~200+ mbit/s) and the intended use case is local network game streaming over ethernet with absolute minimum latency where bandwidth is less of a concern.
It is integrated as part of my pyrofling project.
It is similar in scope to my master thesis from 2014, except the use case now is game streaming instead of adaptation of raw video to ethernet links.
Overview
Colorspace
Currently implemented YCbCr 4:2:0 and 4:4:4.
Wavelets
The images are transformed with the Discrete Wavelet Transform using CDF 9/7 filter. This is basically the exact same as JPEG2000.
Exact rate control
The encoder can target exact maximum bitrate for an encoded image.
Trivial "entropy" coding
The encoding of coefficients is trivial compared to normal image codecs, and is responsible for a large increase in bit-rate, especially at higher compression ratios. This simplicity massively improves encode/decode performance, since it's extremely parallelizable. It also makes it possible to do a single-pass exact rate control for an entire image.
It should be possible in theory to add a more proper entropy coder to the encoded bit-planes to achieve somewhat competent compression, but that has not been explored and is out of scope. High Throughput JPEG2000 is likely a good place to look for that anyway.
Robustness against packet loss
Being intra-only and encoding 64x64 blocks of coefficients in isolation ensures great recovery against packet loss. PyroWave has been battled tested over long distance streaming over fiber links.
Bitstream definition
Building
PyroWave is intended to be built alongside PyroFling with Granite in the normal case.
Standalone C API
NOTE: This API is still under development and the API/ABI is not yet quite stable.
A small portion of Granite needs to be checked out.
bash checkout_granite.sh
Build normally with CMake and a C API is installed.
$ mkdir build
$ cd build
$ cmake .. -DCMAKE_INSTALL_PREFIX=output -DCMAKE_BUILD_TYPE=Release -G Ninja
$ ninja install
[0/1] Install the project...
-- Install configuration: "Release"
-- Installing: ...../build/output/include/pyrowave/pyrowave.h
-- Installing: ...../build/output/lib/libpyrowave-shared.so.0.0.0
-- Installing: ...../build/output/lib/libpyrowave-shared.so.0
-- Installing: ...../build/output/lib/libpyrowave-shared.so
-- Installing: ...../build/output/share/pyrowave-shared/cmake/pyrowave-sharedConfig.cmake
-- Installing: ...../build/output/share/pyrowave-shared/cmake/pyrowave-sharedConfig-release.cmake
-- Installing: ...../build/output/share/pkgconfig/pyrowave-shared.pc
The build is tested on Linux, MinGW, msys2 and MSVC.
See pyrowave-c-test which unit tests the shared C API.
That test also serves as a basic user guide for the API.
build-steamrt.sh builds against the Sniper SDK and is also supported.
Local development and CLI
For the sample and test applications in this repo however, check out
the full https://github.com/Themaister/Granite before invoking CMake.
Build with -DPYROWAVE_DEVEL=ON to get the "full" build.
git clone --depth 1 --recursive --shallow-submodules https://github.com/Themaister/Granite Granite
Basic encoder/decoder CLI
The basic CLI takes a y4m and dumps out a raw bitstream. This is just intended for local testing.
pyrowave-encode test.y4m out.wave 400000
This encodes a raw bitstream where each frame consumes maximum 400 KB. To decode back to y4m:
pyrowave-decode out.wave out.y4m