# vLLM v9 B12X TP3 graph-capture diagnosis

## Scope

- vLLM commit: `45c1582e9`
- B12X TP3 eager and standalone CUDA-graph tests pass.
- Server capture completes 67 PIECEWISE descriptors, then does not make visible
  progress into FULL capture.
- Production service and configuration are out of scope for this diagnostic.

## Primary hypothesis

`CudaGraphManager.capture()` holds `CustomAllreduce.capture()` open across both
PIECEWISE and FULL phases. This sets `_IS_CAPTURING` during every eager warmup,
even when the CUDA stream is not currently capturing.

`CustomAllreduce.custom_all_reduce()` returns an uninitialized
`torch.empty_like(input)` for those non-PIECEWISE warmups. That preserves the
legacy custom-AR allocation pattern, but B12X PCIe oneshot uses preallocated
eager buffers and can safely run the real collective. Feeding rank-local
uninitialized tensors through DeepSeek's warmup can make data-dependent kernel
selection or routing differ between TP ranks before FULL capture.

The minimal candidate correction is B12X-specific:

```python
if self._pcie_runtime is not None or _is_piecewise_cudagraph_runtime():
    return self.all_reduce(input, registered=False)
```

Legacy custom all-reduce retains its placeholder behavior.

## Evidence

- The placeholder branch was added in the same commit as B12X integration.
- No v9 test covers B12X while `_IS_CAPTURING=True` and the stream is not
  capturing.
- Two isolated unit tests pass against the exact v9 image:
  - B12X runs a real eager all-reduce in this state.
  - Legacy custom AR still returns a placeholder.
- Patch: `provisioning/patches/vllm-b12x-capture-warmup.patch`.

This is not hardware-confirmed. A successful startup alone is weaker evidence
than matching per-rank collective sequences at the phase boundary.

## Instrumentation

Gate all diagnostics behind `VLLM_DEBUG_B12X_CAPTURE_SEQUENCE=1`.

In `CudaGraphManager.capture()`, log on every TP rank:

- `mode`, full descriptor, and stage: `warmup_begin`, `warmup_end`,
  `capture_begin`, `capture_end`.
- current CUDA stream ID and `torch.cuda.is_current_stream_capturing()`.
- the B12X custom-AR monotonic call counter.

In `CustomAllreduce.custom_all_reduce()`, increment a monotonic counter and log:

- TP rank, counter, tensor shape/bytes.
- `_IS_CAPTURING`, stream-capture state, and forward-context mode.
- selected path: `captured`, `pcie_eager_warmup`, `piecewise_eager`,
  `placeholder`, or `outside_capture`.

Do not add a synchronization barrier initially; it can hide a timing bug. The
last line from each rank identifies the first divergent descriptor/call.

## Falsification

Reject the primary hypothesis if the patched run still stalls and all three
ranks report identical custom-AR counters and paths through the first FULL
descriptor. The next target is then repeated graph capture on one shared B12X
signal/channel: reproduce 67 breakable captures followed by one monolithic
capture in a small three-process harness.