flashinfer.cake_mhc_head_det_train.cake_mhc_head_det¶
- flashinfer.cake_mhc_head_det_train.cake_mhc_head_det(X: Tensor) Tensor¶
Autograd-enabled
cake_mhc_head_det_train_forward():Y = mean_i X[:, i, :].- Parameters:
X (torch.Tensor) – BF16
[T, 4, H]contiguous residual streams (may require gradients),Hin 4096, 5120 or 7168.- Returns:
BF16
Yof shape[T, H]. Its backward iscake_mhc_head_det_train_backward()(bit for bit torch’s gradient ofX.mean(dim=1)); the node saves no tensor, so an activation checkpoint around it keeps nothing alive and its backward needs no recomputation.- Return type:
torch.Tensor