flashinfer.cake_kda_nondet_train_kcp.cake_kda_nondet_train_kcp_backward

flashinfer.cake_kda_nondet_train_kcp.cake_kda_nondet_train_kcp_backward(q: Tensor, k: Tensor, v: Tensor, g: Tensor, beta: Tensor, ctx: Any, dout: Tensor, *, saved: Any | None = None, scale: float | None = None, save_mode: str = 'light', workspace: Any | None = None, use_qk_l2norm_in_kernel: bool = False, use_gate_in_kernel: bool = False, A_log: torch.Tensor | None = None, dt_bias: torch.Tensor | None = None, lower_bound: float | None = None, save_transfer: bool | None = None, exchange: str = 'auto', head_groups: int = 1, offset_class: str = 'fla', debug_sequence: bool = False, spec3_probe: str = 'off', graph: str = 'off', _kcp_test_knobs: dict[str, Any] | None = None) → tuple[Tensor, ...]

Context-parallel packed KDA chunked training backward of this rank’s fragments.

Parameters:
  • q – The forward’s inputs (same contracts as cake_kda_nondet_train_kcp_forward()).

  • k – The forward’s inputs (same contracts as cake_kda_nondet_train_kcp_forward()).

  • v – The forward’s inputs (same contracts as cake_kda_nondet_train_kcp_forward()).

  • g – The forward’s inputs (same contracts as cake_kda_nondet_train_kcp_forward()).

  • beta – The forward’s inputs (same contracts as cake_kda_nondet_train_kcp_forward()).

  • ctx – The forward’s inputs (same contracts as cake_kda_nondet_train_kcp_forward()).

  • dout (torch.Tensor) – BF16 gradient of this rank’s out.

  • saved (CakeKDANondetKcpSaved, optional) – The forward’s saved set; without one the forward is replayed (checkpointing). A saved set of the deterministic package raises ValueError naming the mode.

  • scale – As for cake_kda_nondet_train_kcp_forward(); the backward reads its own options, among them spec3_probe: the debug probe of the first saved-mode backward, "off" by default, "record", "warn" or "assert" (the same value on every rank); and the forward’s raw gate keywords.

  • save_mode – As for cake_kda_nondet_train_kcp_forward(); the backward reads its own options, among them spec3_probe: the debug probe of the first saved-mode backward, "off" by default, "record", "warn" or "assert" (the same value on every rank); and the forward’s raw gate keywords.

  • workspace – As for cake_kda_nondet_train_kcp_forward(); the backward reads its own options, among them spec3_probe: the debug probe of the first saved-mode backward, "off" by default, "record", "warn" or "assert" (the same value on every rank); and the forward’s raw gate keywords.

  • **options – Additional options and raw-gate keywords of cake_kda_nondet_train_kcp_forward().

  • use_qk_l2norm_in_kernel (bool) – Keyword-only, default False: the forward’s value. With True dq / dk are the gradients of the raw q / k; a saved set or a workspace of the other mode raises ValueError.

Returns:

Five tensors in the explicit gate mode (use_gate_in_kernel=False): (dq, dk, dv, dbeta, dg) of this rank’s tokens, BF16 dq, dk, dv, dbeta, FP32 dg. Seven tensors in raw gate mode (use_gate_in_kernel=True): (dq, dk, dv, dbeta, dg, dA_log, ddt_bias), dg the BF16 gradient of the rank’s raw gate, dA_log FP32 [H] and ddt_bias FP32 (dt_bias’s shape) this rank’s partial sums – the gradients through its own tokens. Sum them over the context-parallel group like the gradients of any replicated parameter, or with cake_kda_nondet_train_kcp_reduce_gate_param_grads() for a bitwise reproducible sum (an NCCL all-reduce does not fix its summation order). Bitwise reproducible at the same topology; dg of the explicit gate mode differs from the deterministic package’s by FP32 rounding only (module docstring).

Return type:

tuple of torch.Tensor