flashinfer.cake_mhc_det_train.cake_mhc_det_train_forward_workspace_bytes¶
- flashinfer.cake_mhc_det_train.cake_mhc_det_train_forward_workspace_bytes(T: int, device: device, *, hidden: int = 4096) int¶
Bytes of the caller-owned forward workspace for up to
Ttokens of hidden sizehidden.The forward routes (
forward_prepareand theforward_statsreplay) hold one 256-byte ticket region (first, at the same offset for every token count), the BF16 pieces offnin the mapping operand layout and the FP32 per-K-unit partials of the mapping pass. The ticket words must be zero on first use (allocate withtorch.zerosorcake_mhc_det_train_forward_workspace()); the kernels leave them zero.- Parameters:
T (int) – Largest token count the buffer serves. The size is the maximum over every token count up to
T, so the buffer also serves every smaller one (and CUDA graph replays at a smallerT);T == 0needs no workspace (nothing is launched).device (torch.device) – CUDA device of the calls.
hidden (int) – Hidden size
Hof the calls the buffer serves (X.shape[2]; default4096): a hidden size the device’s architecture exports (4096, 5120 or 7168 on SM100 / SM103 / SM107), elseValueError. Thefnpieces and the partials scale with4 H; every call checks the buffer against its ownHandT, so a buffer sized for oneHis refused (ValueError) by a call of a largerHit is too small for.
- Returns:
The size of the forward workspace in bytes.
- Return type:
int