flashinfer.cake_gdn_nondet_train_kcp.cake_gdn_nondet_train_kcp_backward¶
- flashinfer.cake_gdn_nondet_train_kcp.cake_gdn_nondet_train_kcp_backward(q: Tensor, k: Tensor, v: Tensor, g: Tensor, beta: Tensor, ctx: Any, dout: Tensor, *, saved: Any | None = None, scale: float | None = None, workspace: Any | None = None, options: Any | None = None, use_qk_l2norm_in_kernel: bool = False, exchange: str = 'a2a', save_transfer: bool = True, save_mode: str | None = None, boundary_mode: str = 'saved', precision_mode: int = 0, transfer_walk: str = 'tf32_hilo', offset_class: str = 'fla', layout: str = 'tma', group_reduce: str = 'fused', graph: str = 'off', reuse_recompute_tapes: bool | str | None = None, recompute_output: str = 'auto', backend: str = 'kernels', segments: int | None = None, test_knobs: dict[str, Any] | None = None) tuple[Tensor, Tensor, Tensor, Tensor, Tensor]¶
Context-parallel packed GDN training backward.
- Parameters:
q – The forward’s operands and context.
k – The forward’s operands and context.
v – The forward’s operands and context.
g – The forward’s operands and context.
beta – The forward’s operands and context.
ctx – The forward’s operands and context.
dout (torch.Tensor) – BF16 gradient of
out, laid out likev.saved (optional) – The forward’s saved set;
Nonerecomputes it from the inputs (checkpoint replay: the summaries and the exchange are re-run). A saved set of the deterministic package is refused (ValueError).scale – As in
cake_gdn_nondet_train_kcp_forward()(the same options as the forward’s).workspace – As in
cake_gdn_nondet_train_kcp_forward()(the same options as the forward’s).options – As in
cake_gdn_nondet_train_kcp_forward()(the same options as the forward’s).**knobs – Additional backend options, as in
cake_gdn_nondet_train_kcp_forward().
- Returns:
(dq, dk, dv, dbeta, dg): BF16 gradients ofq, k, v(theHV / H_qkvalue heads’ contributions todq/dkreduced in FP32 and rounded once), FP32 gradients ofbetaandg; fresh tensors, ranked like the inputs; within the deterministic package’s accuracy budgets, not bitwise equal to its results.- Return type:
tuple[torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor]