flashinfer.cake_kda_nondet_train_kcp.cake_kda_nondet_kcp¶
- flashinfer.cake_kda_nondet_train_kcp.cake_kda_nondet_kcp(q: Tensor, k: Tensor, v: Tensor, g: Tensor, beta: Tensor, ctx: Any, scale: float | None = None, save_mode: str = 'light', boundary_mode: str = 'saved', workspace: Any | None = None, *, recompute_handle: Any | None = None, checkpoint_keep: str | None = None, checkpoint_handoff: str = 'auto', saved_rows: str = 'auto', use_qk_l2norm_in_kernel: bool = False, use_gate_in_kernel: bool = False, A_log: torch.Tensor | None = None, dt_bias: torch.Tensor | None = None, lower_bound: float | None = None, save_transfer: bool | None = None, exchange: str = 'auto', head_groups: int = 1, offset_class: str = 'fla', debug_sequence: bool = False, spec3_probe: str = 'off', graph: str = 'off', _kcp_test_knobs: dict[str, Any] | None = None) Tensor¶
Autograd-enabled context-parallel packed KDA chunked training step of this rank’s fragments (the forward of
cake_kda_nondet_train_kcp_forward(); the backward ofcake_kda_nondet_train_kcp_backward()from its saved set).- Parameters:
q (torch.Tensor) – This rank’s input rows, with the contracts of
cake_kda_nondet_train_kcp_forward().k (torch.Tensor) – This rank’s input rows, with the contracts of
cake_kda_nondet_train_kcp_forward().v (torch.Tensor) – This rank’s input rows, with the contracts of
cake_kda_nondet_train_kcp_forward().g (torch.Tensor) – This rank’s input rows, with the contracts of
cake_kda_nondet_train_kcp_forward().beta (torch.Tensor) – This rank’s input rows, with the contracts of
cake_kda_nondet_train_kcp_forward().ctx – This rank’s context from
cake_kda_nondet_train_kcp_build_context().scale (float, optional) – Output scale;
128 ** -0.5whenNone.save_mode (str, default="light") – Local saved-set mode of
cake_kda_nondet_train_kcp_forward().boundary_mode (str, default="saved") – Keep boundary states with
"saved"or recompute them with"replay".workspace (optional) – Caller-owned workspace of this package and the same gate and normalization modes.
recompute_handle (CakeKDANondetKcpRecomputeHandle, optional) – Keyword-only, default
None(unchanged behavior). A handle (flashinfer.cake_kda_nondet_train_kcp.CakeKDANondetKcpRecomputeHandleor any object with anentriesdict, such as the convolution’s handle of the same layer) makes thetorch.utils.checkpointrecompute of this call run without the forward state exchange: the original forward exchanges as always and stores its incoming states (the saved set’s own tensors, key"kda") in the handle; the recompute inside the backward reads them back, sooutand the gradients are bitwise those of the same checkpoint without a handle. The backward still exchanges. Create one handle per layer and micro-step, outside the outermost checkpoint region, and pass it as an argument of the checkpointed function; a forward outside the backward that finds an entry raises. Refused withgraph="replay", inside a CUDA-graph capture, withdebug_sequence,spec3_probeandboundary_mode="replay". The exchange is a collective: every rank ofctx.grouppasses a handle for the same call or none, created at the same checkpoint nesting level. The handle’s presence is part of the rank-agreement check of the call’s first original forward (a mismatch trips a device-side assert); a mismatch of the nesting level is not checked and hangs or corrupts data.checkpoint_keep (str, optional) – With a handle only.
"states"(default): the handle keepsS_inand the call runs withsave_transfer=False(bitwise the default form; the backward re-sends the maps)."transfer": it also keeps the owner’s c1 receive tensor (save_transfer=True; not withexchange="ring"). An explicitsave_transfermust agree with the tier ("states"->False,"transfer"->True); a contradicting value raisesValueError. The retained bytes arecake_kda_nondet_train_kcp_saved_bytes(ctx, H=..., checkpoint_keep=..., with_conv=False).checkpoint_handoff (str) – Keyword-only,
"auto"(default),"on"or"off"(another value raisesValueError): the checkpoint hand-off. Undertorch.utils.checkpointthe forward keeps the formsaved_rowsnames in the saved set and the backward skips that part of the local recompute (saved_rows="auto": the"wy"rows;"tape": the tape set). With the tape set, on the devices ofKCP_HANDOFF_VERIFIEDthe outputs and gradients are bitwise those of"off";"on"elsewhere has no such evidence."auto"engages wherecake_kcp_plan.kcp_handoff_auto_refusaladmits the call (a verified device,HinKCP_HANDOFF_HEADS,H * (total tokens // world size) >= 393216, and on a device with its own bound inKCP_HANDOFF_MIN_HEAD_TOKENS_BY_KEYat least that bound). An engaged call raises the checkpoint peak of this rank by at mostcake_kda_nondet_train_kcp_handoff_bytes()(its saved set iscake_kda_nondet_train_kcp_saved_bytes(..., checkpoint_handoff=True): no carry plane);"off"opts out. In raw gate mode (use_gate_in_kernel=True) and with the in-kernel q / k L2 norm (use_qk_l2norm_in_kernel=True) the hand-off engages as with the activated gate (the same tape set). The devices verified in that mode (cake_kcp_plan.KCP_HANDOFF_VERIFIED_MODES) replace those ofKCP_HANDOFF_VERIFIED: there the raw gate’s gradient, the parameter partial sums and the rawq/kgradients are bitwise those of"off"too, and"auto"admits only those devices. Counters:cake_kda_nondet_train_kcp_handoff_stats().saved_rows (str) – Keyword-only,
"auto"(default),"none","intra","wy"or"tape"(another value raisesValueError): which forward rows the saved set keeps (cake_kda_nondet_train_kcp_forward())."auto"keeps the"wy"rows where the hand-off engages (faster than the tape set and 528 MiB less per layer at 16384 tokens per rank, H 64) and nothing elsewhere;"tape"keeps the tape set where the hand-off engages and nothing elsewhere."none"keeps no rows in any pass (withcheckpoint_handoff="on"it raisesValueError, in a checkpoint pass or not)."intra"/"wy": inside a checkpoint pass, the rows form of the hand-off where the hand-off engages (instead of its tape set; nothing where it is refused,"off"included); outside one, the rows where the call builds an autograd node outside a backward and no saved-tensor hooks of the caller are active (there"off"does not stop them). No rows withboundary_mode="replay"or on a rank without tokens. The rows add256 * R * H/768 * R * Hbytes per layer (cake_kda_nondet_train_kcp_saved_bytes()saved_rows) until the node’s backward; the gradients stay within the light path’s FP64 error budget.use_qk_l2norm_in_kernel (bool) – Keyword-only, default
False. As forcake_kda_nondet_train_kcp_forward(); the gradients go to the rawq/k, and a recompute handle’s fingerprint records the mode (its entry does not change: the recompute writes its rstd afresh).**options – As for
cake_kda_nondet_train_kcp_forward(), the raw gate keywords included: in raw gate mode the gradients go to the raw gate (BF16),A_loganddt_bias(FP32; this rank’s partial sums, seecake_kda_nondet_train_kcp_backward()), and a recompute handle’s entry records the mode andlower_bound.
- Returns:
BF16
outlikev(this rank’s tokens).- Return type:
torch.Tensor