flashinfer.cake_mhc_nondet_train.cake_mhc_nondet_train_forward_workspace_bytes¶
- flashinfer.cake_mhc_nondet_train.cake_mhc_nondet_train_forward_workspace_bytes(T: int, device: device, *, hidden: int = 4096) int¶
Bytes of the caller-owned forward workspace for up to
Ttokens ondevice.The forward workspace has one layout in both packages (see
cake_mhc_det_train_forward_workspace_bytes()).- Parameters:
T (int) – Largest token count the buffer serves (the size is the maximum over every token count up to
T);T == 0needs no workspace.device (torch.device) – CUDA device of the calls.
hidden (int) – Hidden size
Hof the calls the buffer serves (X.shape[2]; default4096): 4096, 5120 or 7168 on SM100 / SM103, 4096 on SM107 (any other value raisesValueError).
- Returns:
The size of the forward workspace in bytes.
- Return type:
int