flashinfer.cake_kda_det_train_kcp.cake_kda_det_train_kcp_backward¶
- flashinfer.cake_kda_det_train_kcp.cake_kda_det_train_kcp_backward(q: Tensor, k: Tensor, v: Tensor, g: Tensor, beta: Tensor, ctx: Any, dout: Tensor, *, saved: Any | None = None, scale: float | None = None, save_mode: str = 'light', workspace: Any | None = None, use_qk_l2norm_in_kernel: bool = False, use_gate_in_kernel: bool = False, A_log: torch.Tensor | None = None, dt_bias: torch.Tensor | None = None, lower_bound: float | None = None, save_transfer: bool | None = None, exchange: str = 'auto', head_groups: int = 1, offset_class: str = 'fla', debug_sequence: bool = False, spec3_probe: str = 'off', graph: str = 'off', _kcp_test_knobs: dict[str, Any] | None = None) tuple[Tensor, ...]¶
Context-parallel packed KDA chunked training backward of this rank’s fragments.
- Parameters:
q – The forward’s inputs (same contracts as
cake_kda_det_train_kcp_forward()).k – The forward’s inputs (same contracts as
cake_kda_det_train_kcp_forward()).v – The forward’s inputs (same contracts as
cake_kda_det_train_kcp_forward()).g – The forward’s inputs (same contracts as
cake_kda_det_train_kcp_forward()).beta – The forward’s inputs (same contracts as
cake_kda_det_train_kcp_forward()).ctx – The forward’s inputs (same contracts as
cake_kda_det_train_kcp_forward()).dout (torch.Tensor) – BF16 gradient of this rank’s
out.saved (CakeKDADetKcpSaved, optional) – The forward’s saved set; without one the forward is replayed (checkpointing).
scale – As for
cake_kda_det_train_kcp_forward(); the backward reads its own options, among themspec3_probe: the debug probe of the first saved-mode backward,"off"by default,"record","warn"or"assert"(the same value on every rank); and the forward’s raw gate keywords.save_mode – As for
cake_kda_det_train_kcp_forward(); the backward reads its own options, among themspec3_probe: the debug probe of the first saved-mode backward,"off"by default,"record","warn"or"assert"(the same value on every rank); and the forward’s raw gate keywords.workspace – As for
cake_kda_det_train_kcp_forward(); the backward reads its own options, among themspec3_probe: the debug probe of the first saved-mode backward,"off"by default,"record","warn"or"assert"(the same value on every rank); and the forward’s raw gate keywords.**options – Additional options and raw-gate keywords of
cake_kda_det_train_kcp_forward().use_qk_l2norm_in_kernel (bool) – Keyword-only, default
False: the forward’s value. WithTruedq/dkare the gradients of the rawq/k; a saved set or a workspace of the other mode raisesValueError.
- Returns:
Five tensors in the explicit gate mode (
use_gate_in_kernel=False):(dq, dk, dv, dbeta, dg)of this rank’s tokens, BF16dq, dk, dv, dbeta, FP32dg. Seven tensors in raw gate mode (use_gate_in_kernel=True):(dq, dk, dv, dbeta, dg, dA_log, ddt_bias),dgthe BF16 gradient of the rank’s raw gate,dA_logFP32[H]andddt_biasFP32 (dt_bias’s shape) this rank’s partial sums – the gradients through its own tokens. Sum them over the context-parallel group like the gradients of any replicated parameter, or withcake_kda_det_train_kcp_reduce_gate_param_grads()for a bitwise reproducible sum (an NCCL all-reduce does not fix its summation order).- Return type:
tuple of torch.Tensor