flashinfer.cake_kda_nondet_train_kcp.cake_kda_nondet_train_kcp_forward¶
- flashinfer.cake_kda_nondet_train_kcp.cake_kda_nondet_train_kcp_forward(q: Tensor, k: Tensor, v: Tensor, g: Tensor, beta: Tensor, ctx: Any, *, scale: float | None = None, save_for_backward: bool = True, save_mode: str = 'light', boundary_mode: str = 'saved', workspace: Any | None = None, use_qk_l2norm_in_kernel: bool = False, saved_rows: str = 'auto', use_gate_in_kernel: bool = False, A_log: torch.Tensor | None = None, dt_bias: torch.Tensor | None = None, lower_bound: float | None = None, save_transfer: bool | None = None, exchange: str = 'auto', head_groups: int = 1, offset_class: str = 'fla', debug_sequence: bool = False, spec3_probe: str = 'off', graph: str = 'off', _kcp_test_knobs: dict[str, Any] | None = None) tuple[Tensor, Any | None]¶
Context-parallel packed KDA chunked training forward of this rank’s fragments.
- Parameters:
q (torch.Tensor) – This rank’s shard of the operands of
cake_kda_nondet_train_forward()(the tokens of its fragments, in the partition’s order): BF16[1, S_local, H, 128]q, k, v, FP32g, BF16[1, S_local, H]beta.k (torch.Tensor) – This rank’s shard of the operands of
cake_kda_nondet_train_forward()(the tokens of its fragments, in the partition’s order): BF16[1, S_local, H, 128]q, k, v, FP32g, BF16[1, S_local, H]beta.v (torch.Tensor) – This rank’s shard of the operands of
cake_kda_nondet_train_forward()(the tokens of its fragments, in the partition’s order): BF16[1, S_local, H, 128]q, k, v, FP32g, BF16[1, S_local, H]beta.g (torch.Tensor) – This rank’s shard of the operands of
cake_kda_nondet_train_forward()(the tokens of its fragments, in the partition’s order): BF16[1, S_local, H, 128]q, k, v, FP32g, BF16[1, S_local, H]beta.beta (torch.Tensor) – This rank’s shard of the operands of
cake_kda_nondet_train_forward()(the tokens of its fragments, in the partition’s order): BF16[1, S_local, H, 128]q, k, v, FP32g, BF16[1, S_local, H]beta.ctx (CakeKDANondetKcpContext) – From
cake_kda_nondet_train_kcp_build_context().scale (float, optional) – Output scale;
128 ** -0.5whenNone.save_for_backward (bool) – Write the saved set (the local saved set of
cake_kda_nondet_train_forward()plus the boundary states the backward starts from).save_mode (str) –
"light"or"tape"as forcake_kda_nondet_train_forward().boundary_mode (str) –
"saved"keeps the received boundary states in the saved set;"replay"saves nothing and lets the backward replay the forward’s exchanges.workspace (CakeKDANondetKcpWorkspace, optional) – A reused workspace of this rank; without one the call runs on a workspace the package keeps per device, stream, modes and problem geometry: its host objects are reused, while its device memory is allocated at the call’s start and freed at its end (the same bytes as a new workspace per call; the package keeps host objects only). One workspace may serve every prepared pack of the rank (
CakeKDANondetKcpWorkspace.reserve(ctxs, H)sizes it for all of them up front). A workspace of the deterministic package raisesValueErrornaming the mode.use_qk_l2norm_in_kernel (bool) – Keyword-only, default
False(unchanged behavior).Truenormalizesqandkinside the kernels: per token and per 128-channel head,q * rsqrt(sum(q * q) + 1e-6)in FP32, rounded once to BF16 (likewisek;vis not normalized), with the FP32 pullback fused into the BF16dq/dk.q/kare then this rank’s raw rows and may be strided views (for example columns of a packed projection or convolution output: innermost stride 1, head stride 128 elements, token stride and base address 16-byte aligned, token stride >= H*128 (no overlapping rows); never copied). The autograd node keeps the rawq/k(the caller’s tensors, not members of the saved set); the saved set adds the FP32 per-token rstd[2, S_local, H]; normalizedq/kare never saved. The workspace must be of the same mode (CakeKDANondetKcpWorkspace(use_qk_l2norm_in_kernel=True)). Until the single-rank in-kernel normalization programs are available on the device,TrueraisesNotImplementedErroron every rank alike. WithFalseevery launch, saved tensor, key and byte is unchanged.saved_rows (str) – Keyword-only,
"auto"(default),"none","intra"or"wy"(another value raisesValueError)."auto"and"none"keep the light saved set (unchanged behavior)."intra"keeps the forward’s internalAqk/Akkrows and"wy"also its WY rowsw/uin the saved set, so the backward skips its intra program (and with"wy"its WY program); the forward runs the same launches and collectives. The saved set grows by256 * R * H/768 * R * Hbytes (cake_kda_nondet_train_kcp_saved_bytes()saved_rows). The gradients stay within the light path’s FP64 error budget; they are not bitwise those of the light set. Refused withspec3_probe.**options – Further KCP options (
save_transfer,exchange,head_groups,offset_class,graph), forwarded. The raw gate mode ofcake_kda_nondet_train_forward()(keywordsuse_gate_in_kernel,A_log,dt_bias,lower_bound, forwarded like the options) applies to this rank’s shard: withuse_gate_in_kernel=Truegis the rank’s raw gate (BF16, any view under the single-rank stride rule, never copied),A_logFP32[H]anddt_biasFP32[H * 128]or[H, 128]are the full parameters (the same tensors on every rank) andlower_boundis required (-5 <= lower_bound < 0); the workspace must be of the same mode (CakeKDANondetKcpWorkspace(use_gate_in_kernel=True)).
- Returns:
(out, saved):outBF16 likev(this rank’s tokens);savedaCakeKDANondetKcpSavedorNone.- Return type:
tuple[torch.Tensor, Optional[Any]]