flashinfer.cake_kda_det_train.cake_kda_det_train_forward¶
- flashinfer.cake_kda_det_train.cake_kda_det_train_forward(q: Tensor, k: Tensor, v: Tensor, g: Tensor, beta: Tensor, cu_seqlens: Tensor, *, scale: float | None = None, save_for_backward: bool = True, workspace: Any | None = None, save_mode: str | None = None, arena: Any | None = None, doc_lengths: Sequence[int] | None = None, use_gate_in_kernel: bool = False, A_log: Tensor | None = None, dt_bias: Tensor | None = None, lower_bound: float | None = None, use_qk_l2norm_in_kernel: bool = False) tuple[Tensor, Any | None]¶
Packed KDA chunked training forward.
- Parameters:
q (torch.Tensor) – BF16
[1, S, H, 128](or[S, H, 128]), contiguous.k (torch.Tensor) – BF16
[1, S, H, 128](or[S, H, 128]), contiguous.v (torch.Tensor) – BF16
[1, S, H, 128](or[S, H, 128]), contiguous.g (torch.Tensor) – FP32
[1, S, H, 128]natural-log decay gate (preactivated); withuse_gate_in_kernel=Truethe raw gate, BF16[1, S, H, 128], which may be a strided view (unit innermost stride, token and head strides multiples of 8 elements, rows that do not overlap, a 16-byte aligned address; never copied).beta (torch.Tensor) – BF16
[1, S, H]delta-rule gate after the sigmoid.cu_seqlens (torch.Tensor) – int32
[num_docs + 1]document offsets on the device (first 0, lastS).scale (float, optional) – Output scale;
128 ** -0.5whenNone.save_for_backward (bool) – Write the saved set (
Aqk/Akkon the token rows plus the device chunk tables).workspace (CakeKDADetWorkspace, optional) – A reused workspace from
cake_kda_det_train_workspace(); without one this call allocates its own.save_mode (str, optional) –
"light"(saved set version 4:Aqk/Akkand the tables, 256 B per token-head, the forward’s gated queriesqg, 256 B per internal token-head, plus the segment-boundary states on the segmented route whenS >= 384 * NSEG, so the backward skips the forward half of its recurrence recompute, and the WY representation on the rows whose backward reads it from the set) or"tape"(version 2: also the forward’s state tape andv_new, 768 B more per token-head, so the backward skips the whole recompute);Nonetakes the workspace’s default ("light"without one).arena (CakeKDADetArena, optional) – The lower-memory arena option (
cake_kda_det_train_arena(); off by default): the forward’s layout is the same under every option, and a batched option (batch_tokens) runs the forward once per document group;Nonetakes the workspace’s option, a differing option on a given workspace is refused.doc_lengths (sequence of int, optional) – The documents’ token counts in
cu_seqlensorder (host integers;len == num_docs,sum == S), required under a batched option (the host reads no tensor) and checked againstcu_seqlenson the device.use_gate_in_kernel (bool) –
False(the default):gis the activated FP32 gate andA_log,dt_bias,lower_boundmust beNone.True:gis the raw BF16 gate and the kernels activate it aslower_bound * sigmoid(exp(A_log) * (g + dt_bias)); the workspace must be allocated withuse_gate_in_kernel=Trueand the saved set records the mode.A_log (torch.Tensor, optional) – FP32
[H], contiguous (raw gate mode).dt_bias (torch.Tensor, optional) – FP32
[H * 128]or[H, 128], contiguous (raw gate mode).lower_bound (float, optional) – The gate’s lower bound, required in raw gate mode: finite,
-5.0 <= lower_bound < 0.0.use_qk_l2norm_in_kernel (bool) –
False(the default):q/kare L2-normalized by the caller and contiguous.True:q/kare the raw rows, normalized per token and head inside the kernels (bf16(x * r),r = 1 / sqrt(sum(x^2) + 1e-6)in FP32); they may be strided views (unit innermost stride, head stride 128, token stride a multiple of 8 elements and at least128 * H, a 16-byte aligned address; never copied). The saved set adds the FP32 reciprocal norms[2, S, H](no normalized rows are kept) and records the mode; the workspace must be allocated withuse_qk_l2norm_in_kernel=True.
- Returns:
(out, saved):outBF16 likev;savedaCakeKDADetChunkSaved(versioned) orNone.- Return type:
tuple[torch.Tensor, Optional[Any]]