A worker that died before the Hello landed said EPIPE, not "handshake" #271

Merged

enricobuehler merged 1 commits from worktree-pyrowave-handshake-epipe into main 2026-08-16 19:58:57 +00:00

Conversation 0 Commits 1 Files Changed 1

+35 -3

enricobuehler commented 2026-08-16 19:03:05 +00:00

Owner

Copy Link

Copy Source

CI red on pyrowave_remote::tests::a_worker_that_exits_immediately_is_a_handshake_failure:

the rung must name the handshake: send Hello: Broken pipe (os error 32)

What was wrong

The test pinned a premise that is only usually true. spawn_link execs a binary that exits at once, then races it: the parent writes Hello while the child is exiting.

• Win the race — the death surfaces as the EOF the recv reads, which carries .context("encode worker handshake (died on startup?)"). ✅ names the handshake.
• Lose it — the peer is already closed, so the send takes EPIPE, and that arm carried only .context("send Hello"). ❌ never contains the word the test looks for.

Why this is not merely a test problem

Both arms are one cause — a worker that died during startup — and the operator was getting two diagnoses for it. On the EPIPE arm the fallback warn in open_preferring_worker degraded to a bare send Hello: Broken pipe (os error 32): it names neither the worker nor the stage that failed, on the one code path whose entire job is to explain why the session just fell back to the in-process encoder.

The send now names the handshake too, and the race becomes harmless because both outcomes satisfy the same contract.

The new deterministic rung is the part worth keeping

The spawn-driven test cannot be trusted to catch this. Backed out, the unfixed code did not fail once in 60 runs in a container here, because an idle box always wins the race — it is CI's load that loses it, which is exactly why this reached main.

a_worker_that_died_before_hello_still_names_the_handshake closes the peer before the handshake starts, so the Hello send cannot succeed. No scheduler dependence at all. Backed out, it fails with CI's exact message:

the rung must name the handshake even when the death beats the Hello: send Hello: Broken pipe (os error 32)

Verification

CI's own command, in the amd64 container:

cargo clippy -p pf-encode --all-targets --locked --features nvenc,vulkan-encode,pyrowave -- -D warnings
cargo test  -p pf-encode --locked --features nvenc,vulkan-encode,pyrowave

• clippy clean under -D warnings
• 123 passed / 0 failed (50 ignored)
• 100 consecutive passes of each of the two rungs
• cargo fmt --all -- --check clean
• non-vacuity confirmed by Checking pf-encode / Compiling pf-encode in the log

Also swept the rest of the module for the same class of fragility and found none: the other thread-based tests use blocking request/response with no sleeps, and the single 200 ms timeout is an upper bound on an EOF that is already pending. Round 1's failure (distrust_references, #268) is already on main, so this was the only outstanding red.

CI red on `pyrowave_remote::tests::a_worker_that_exits_immediately_is_a_handshake_failure`: ``` the rung must name the handshake: send Hello: Broken pipe (os error 32) ``` ## What was wrong The test pinned a premise that is only *usually* true. `spawn_link` execs a binary that exits at once, then races it: the parent writes `Hello` while the child is exiting. - **Win the race** — the death surfaces as the EOF the recv reads, which carries `.context("encode worker handshake (died on startup?)")`. ✅ names the handshake. - **Lose it** — the peer is already closed, so the **send** takes EPIPE, and that arm carried only `.context("send Hello")`. ❌ never contains the word the test looks for. ## Why this is not merely a test problem Both arms are **one cause** — a worker that died during startup — and the operator was getting two diagnoses for it. On the EPIPE arm the fallback warn in `open_preferring_worker` degraded to a bare `send Hello: Broken pipe (os error 32)`: it names neither the worker nor the stage that failed, on the one code path whose entire job is to explain why the session just fell back to the in-process encoder. The send now names the handshake too, and the race becomes harmless because both outcomes satisfy the same contract. ## The new deterministic rung is the part worth keeping The spawn-driven test cannot be trusted to catch this. Backed out, the unfixed code did **not fail once in 60 runs** in a container here, because an idle box always wins the race — it is CI's load that loses it, which is exactly why this reached main. `a_worker_that_died_before_hello_still_names_the_handshake` closes the peer *before* the handshake starts, so the Hello send cannot succeed. No scheduler dependence at all. Backed out, it fails with CI's exact message: ``` the rung must name the handshake even when the death beats the Hello: send Hello: Broken pipe (os error 32) ``` ## Verification CI's own command, in the amd64 container: ``` cargo clippy -p pf-encode --all-targets --locked --features nvenc,vulkan-encode,pyrowave -- -D warnings cargo test -p pf-encode --locked --features nvenc,vulkan-encode,pyrowave ``` - clippy clean under `-D warnings` - **123 passed / 0 failed** (50 ignored) - **100 consecutive passes of each of the two rungs** - `cargo fmt --all -- --check` clean - non-vacuity confirmed by `Checking pf-encode` / `Compiling pf-encode` in the log Also swept the rest of the module for the same class of fragility and found none: the other thread-based tests use blocking request/response with no sleeps, and the single 200 ms timeout is an upper bound on an EOF that is already pending. Round 1's failure (`distrust_references`, #268) is already on main, so this was the only outstanding red.

enricobuehler added 1 commit 2026-08-16 19:03:06 +00:00

fix(encode): a worker that died before the Hello landed said EPIPE, not "handshake" ... ab3ef5b208

CI red on `a_worker_that_exits_immediately_is_a_handshake_failure`:

    the rung must name the handshake: send Hello: Broken pipe (os error 32)

The test was pinning a premise that is only usually true. `spawn_link` execs a binary
that exits at once, then races it: the parent writes Hello while the child is exiting.
Win the race and the death surfaces as the EOF the recv reads, which carries
`.context("encode worker handshake (died on startup?)")`. Lose it and the peer is
already closed, so the SEND takes EPIPE — and that arm carried only
`.context("send Hello")`, which never contains the word the test looks for.

So this is not merely a test problem. Both arms are one cause — a worker that died
during startup — and the operator was getting two diagnoses for it. On the EPIPE arm the
fallback warn in `open_preferring_worker` degraded to a bare "send Hello: Broken pipe
(os error 32)": it names neither the worker nor the stage that failed, on the one code
path whose entire job is to explain why the session just fell back to the in-process
encoder. The send now names the handshake too, and the race becomes harmless because
both outcomes satisfy the same contract.

The new deterministic rung is the part worth keeping. The spawn-driven test cannot be
trusted to catch this: it did not fail ONCE in 60 runs of the unfixed code in a
container here, because an idle box always wins the race — it is CI's load that loses
it, which is exactly why this reached main. Closing the peer before the handshake starts
reproduces the EPIPE arm with no scheduler dependence at all; backed out, it fails with
CI's exact message.

Verified with CI's own command, `cargo clippy -p pf-encode --all-targets --locked
--features nvenc,vulkan-encode,pyrowave -- -D warnings` + the matching `cargo test`, in
the amd64 container: clippy clean, 123 passed / 0 failed, and 100 consecutive passes of
each of the two rungs. `cargo fmt --all --check` clean. Non-vacuity confirmed by
`Checking pf-encode` / `Compiling pf-encode` in the log.

enricobuehler merged commit aa2287ce15 into main 2026-08-16 19:58:57 +00:00

enricobuehler deleted branch worktree-pyrowave-handshake-epipe 2026-08-16 19:58:59 +00:00

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

No items

No labels

Milestone

No items

No Milestone

Projects

Clear projects

No projects

Assignees

Clear assignees

FernandoMX HitFrostbite enricobuehler (Enrico Bühler) kszlim luxus nannku neoney sassycrown

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: unom/punktfunk#271