flashinfer.cake_kda_nondet_train_kcp.cake_kda_nondet_train_kcp_forward

flashinfer.cake_kda_nondet_train_kcp.cake_kda_nondet_train_kcp_forward(q: Tensor, k: Tensor, v: Tensor, g: Tensor, beta: Tensor, ctx: Any, *, scale: float | None = None, save_for_backward: bool = True, save_mode: str = 'light', boundary_mode: str = 'saved', workspace: Any | None = None, use_qk_l2norm_in_kernel: bool = False, saved_rows: str = 'auto', use_gate_in_kernel: bool = False, A_log: torch.Tensor | None = None, dt_bias: torch.Tensor | None = None, lower_bound: float | None = None, save_transfer: bool | None = None, exchange: str = 'auto', head_groups: int = 1, offset_class: str = 'fla', debug_sequence: bool = False, spec3_probe: str = 'off', graph: str = 'off', _kcp_test_knobs: dict[str, Any] | None = None) → tuple[Tensor, Any | None]

Context-parallel packed KDA chunked training forward of this rank’s fragments.

Parameters:
  • q (torch.Tensor) – This rank’s shard of the operands of cake_kda_nondet_train_forward() (the tokens of its fragments, in the partition’s order): BF16 [1, S_local, H, 128] q, k, v, FP32 g, BF16 [1, S_local, H] beta.

  • k (torch.Tensor) – This rank’s shard of the operands of cake_kda_nondet_train_forward() (the tokens of its fragments, in the partition’s order): BF16 [1, S_local, H, 128] q, k, v, FP32 g, BF16 [1, S_local, H] beta.

  • v (torch.Tensor) – This rank’s shard of the operands of cake_kda_nondet_train_forward() (the tokens of its fragments, in the partition’s order): BF16 [1, S_local, H, 128] q, k, v, FP32 g, BF16 [1, S_local, H] beta.

  • g (torch.Tensor) – This rank’s shard of the operands of cake_kda_nondet_train_forward() (the tokens of its fragments, in the partition’s order): BF16 [1, S_local, H, 128] q, k, v, FP32 g, BF16 [1, S_local, H] beta.

  • beta (torch.Tensor) – This rank’s shard of the operands of cake_kda_nondet_train_forward() (the tokens of its fragments, in the partition’s order): BF16 [1, S_local, H, 128] q, k, v, FP32 g, BF16 [1, S_local, H] beta.

  • ctx (CakeKDANondetKcpContext) – From cake_kda_nondet_train_kcp_build_context().

  • scale (float, optional) – Output scale; 128 ** -0.5 when None.

  • save_for_backward (bool) – Write the saved set (the local saved set of cake_kda_nondet_train_forward() plus the boundary states the backward starts from).

  • save_mode (str) – "light" or "tape" as for cake_kda_nondet_train_forward().

  • boundary_mode (str) – "saved" keeps the received boundary states in the saved set; "replay" saves nothing and lets the backward replay the forward’s exchanges.

  • workspace (CakeKDANondetKcpWorkspace, optional) – A reused workspace of this rank; without one the call runs on a workspace the package keeps per device, stream, modes and problem geometry: its host objects are reused, while its device memory is allocated at the call’s start and freed at its end (the same bytes as a new workspace per call; the package keeps host objects only). One workspace may serve every prepared pack of the rank (CakeKDANondetKcpWorkspace.reserve(ctxs, H) sizes it for all of them up front). A workspace of the deterministic package raises ValueError naming the mode.

  • use_qk_l2norm_in_kernel (bool) – Keyword-only, default False (unchanged behavior). True normalizes q and k inside the kernels: per token and per 128-channel head, q * rsqrt(sum(q * q) + 1e-6) in FP32, rounded once to BF16 (likewise k; v is not normalized), with the FP32 pullback fused into the BF16 dq / dk. q / k are then this rank’s raw rows and may be strided views (for example columns of a packed projection or convolution output: innermost stride 1, head stride 128 elements, token stride and base address 16-byte aligned, token stride >= H*128 (no overlapping rows); never copied). The autograd node keeps the raw q / k (the caller’s tensors, not members of the saved set); the saved set adds the FP32 per-token rstd [2, S_local, H]; normalized q / k are never saved. The workspace must be of the same mode (CakeKDANondetKcpWorkspace(use_qk_l2norm_in_kernel=True)). Until the single-rank in-kernel normalization programs are available on the device, True raises NotImplementedError on every rank alike. With False every launch, saved tensor, key and byte is unchanged.

  • saved_rows (str) – Keyword-only, "auto" (default), "none", "intra" or "wy" (another value raises ValueError). "auto" and "none" keep the light saved set (unchanged behavior). "intra" keeps the forward’s internal Aqk / Akk rows and "wy" also its WY rows w / u in the saved set, so the backward skips its intra program (and with "wy" its WY program); the forward runs the same launches and collectives. The saved set grows by 256 * R * H / 768 * R * H bytes (cake_kda_nondet_train_kcp_saved_bytes() saved_rows). The gradients stay within the light path’s FP64 error budget; they are not bitwise those of the light set. Refused with spec3_probe.

  • **options – Further KCP options (save_transfer, exchange, head_groups, offset_class, graph), forwarded. The raw gate mode of cake_kda_nondet_train_forward() (keywords use_gate_in_kernel, A_log, dt_bias, lower_bound, forwarded like the options) applies to this rank’s shard: with use_gate_in_kernel=True g is the rank’s raw gate (BF16, any view under the single-rank stride rule, never copied), A_log FP32 [H] and dt_bias FP32 [H * 128] or [H, 128] are the full parameters (the same tensors on every rank) and lower_bound is required (-5 <= lower_bound < 0); the workspace must be of the same mode (CakeKDANondetKcpWorkspace(use_gate_in_kernel=True)).

Returns:

(out, saved): out BF16 like v (this rank’s tokens); saved a CakeKDANondetKcpSaved or None.

Return type:

tuple[torch.Tensor, Optional[Any]]