flashinfer.cake_kda_det_train_kcp.cake_kda_det_train_kcp_forward¶
- flashinfer.cake_kda_det_train_kcp.cake_kda_det_train_kcp_forward(q: Tensor, k: Tensor, v: Tensor, g: Tensor, beta: Tensor, ctx: Any, *, scale: float | None = None, save_for_backward: bool = True, save_mode: str = 'light', boundary_mode: str = 'saved', workspace: Any | None = None, use_qk_l2norm_in_kernel: bool = False, use_gate_in_kernel: bool = False, A_log: torch.Tensor | None = None, dt_bias: torch.Tensor | None = None, lower_bound: float | None = None, save_transfer: bool | None = None, exchange: str = 'auto', head_groups: int = 1, offset_class: str = 'fla', debug_sequence: bool = False, spec3_probe: str = 'off', graph: str = 'off', _kcp_test_knobs: dict[str, Any] | None = None) tuple[Tensor, Any | None]¶
Context-parallel packed KDA chunked training forward of this rank’s fragments.
- Parameters:
q (torch.Tensor) – This rank’s shard of the operands of
cake_kda_det_train_forward()(the tokens of its fragments, in the partition’s order): BF16[1, S_local, H, 128]q, k, v, FP32g, BF16[1, S_local, H]beta.k (torch.Tensor) – This rank’s shard of the operands of
cake_kda_det_train_forward()(the tokens of its fragments, in the partition’s order): BF16[1, S_local, H, 128]q, k, v, FP32g, BF16[1, S_local, H]beta.v (torch.Tensor) – This rank’s shard of the operands of
cake_kda_det_train_forward()(the tokens of its fragments, in the partition’s order): BF16[1, S_local, H, 128]q, k, v, FP32g, BF16[1, S_local, H]beta.g (torch.Tensor) – This rank’s shard of the operands of
cake_kda_det_train_forward()(the tokens of its fragments, in the partition’s order): BF16[1, S_local, H, 128]q, k, v, FP32g, BF16[1, S_local, H]beta.beta (torch.Tensor) – This rank’s shard of the operands of
cake_kda_det_train_forward()(the tokens of its fragments, in the partition’s order): BF16[1, S_local, H, 128]q, k, v, FP32g, BF16[1, S_local, H]beta.ctx (CakeKDADetKcpContext) – From
cake_kda_det_train_kcp_build_context().scale (float, optional) – Output scale;
128 ** -0.5whenNone.save_for_backward (bool) – Write the saved set (the local saved set of
cake_kda_det_train_forward()plus the boundary states the backward starts from).save_mode (str) –
"light"or"tape"as forcake_kda_det_train_forward().boundary_mode (str) –
"saved"keeps the received boundary states in the saved set;"replay"saves nothing and lets the backward replay the forward’s exchanges.workspace (CakeKDADetKcpWorkspace, optional) – A reused workspace of this rank; without one the call allocates its own. One workspace may serve every prepared pack of the rank (
CakeKDADetKcpWorkspace.reserve(ctxs, H)sizes it for all of them up front).use_qk_l2norm_in_kernel (bool) – Keyword-only, default
False(unchanged behavior).Truenormalizesqandkinside the kernels: per token and per 128-channel head,q * rsqrt(sum(q * q) + 1e-6)in FP32, rounded once to BF16 (likewisek;vis not normalized), with the FP32 pullback fused into the BF16dq/dk.q/kare then this rank’s raw rows and may be strided views (for example columns of a packed projection or convolution output: innermost stride 1, head stride 128 elements, token stride and base address 16-byte aligned, token stride >= H*128 (no overlapping rows); never copied). The saved set keeps the rawq/k(the caller’s tensors) and adds the FP32 per-token rstd[2, S_local, H]; normalizedq/kare never saved. The workspace must be of the same mode (CakeKDADetKcpWorkspace(use_qk_l2norm_in_kernel=True)). Until the single-rank in-kernel normalization programs are available on the device,TrueraisesNotImplementedErroron every rank alike. WithFalseevery launch, saved tensor, key and byte is unchanged.**options – Further KCP options (
save_transfer,exchange,head_groups,offset_class,graph), forwarded. The raw gate mode ofcake_kda_det_train_forward()(keywordsuse_gate_in_kernel,A_log,dt_bias,lower_bound, forwarded like the options) applies to this rank’s shard: withuse_gate_in_kernel=Truegis the rank’s raw gate (BF16, any view under the single-rank stride rule, never copied),A_logFP32[H]anddt_biasFP32[H * 128]or[H, 128]are the full parameters (the same tensors on every rank) andlower_boundis required (-5 <= lower_bound < 0); the workspace must be of the same mode (CakeKDADetKcpWorkspace(use_gate_in_kernel=True)).
- Returns:
(out, saved):outBF16 likev(this rank’s tokens);savedaCakeKDADetKcpSavedorNone.- Return type:
tuple[torch.Tensor, Optional[Any]]