flashinfer.cake_gdn_nondet_train_kcp.cake_gdn_nondet_kcp

flashinfer.cake_gdn_nondet_train_kcp.cake_gdn_nondet_kcp(q: Tensor, k: Tensor, v: Tensor, g: Tensor, beta: Tensor, ctx: Any, *, scale: float | None = None, save_mode: str | None = None, boundary_mode: str = 'saved', workspace: Any | None = None, graph: str = 'off', options: Any | None = None, recompute_handle: Any | None = None, use_qk_l2norm_in_kernel: bool = False, exchange: str = 'a2a', save_transfer: bool = True, precision_mode: int = 0, transfer_walk: str = 'tf32_hilo', offset_class: str = 'fla', layout: str = 'tma', group_reduce: str = 'fused', reuse_recompute_tapes: bool | str | None = None, recompute_output: str = 'auto', backend: str = 'kernels', segments: int | None = None, test_knobs: dict[str, Any] | None = None) → Tensor

Context-parallel packed GDN training as an autograd node (torch.utils.checkpoint compatible).

Returns out BF16 like v with gradients to q, k, v, g, beta; every saved tensor travels through save_for_backward only, so a checkpointed forward retains nothing. See cake_gdn_nondet_train_kcp_forward() for the operands and options and cake_gdn_nondet_train_kcp_backward() for the gradients.

reuse_recompute_tapes (keyword-only, default "auto"; True | False): under torch.utils.checkpoint(use_reentrant=False) the save_mode="light" backward binds the forward tapes that torch’s recompute of the forward (inside .backward()) just wrote, instead of replaying the prep program and the state walk itself; the gradients are bitwise the same, the checkpointed forward still retains nothing, and the direct (non-checkpoint) autograd path, boundary_mode="replay" and save_mode="full" are unchanged. The kept tapes live from the recompute to this node’s backward: when a later operation of the checkpointed region saves a tensor (an output projection, an MLP), they live through that operation’s backward and raise the region’s backward peak by the full tape set (one GB300, 261,987 tokens, (2, 6), GDN + a 768x768 projection + a SwiGLU MLP: 9250.54 MiB against 6927.31 MiB replayed, 21.29 against 22.70 ms per step of the region); "auto" (the default) keeps them only when this node is the region’s last saver (the checkpoint-frame proof of recompute_output) and takes the replay form otherwise – a region this node closes keeps today’s speed and memory; True always keeps them, False never (the replay form). One kept set is live at a time: in a checkpoint segment holding several GDN layers only the last layer skips the replay (the earlier layers replay as before). Under graph="replay" a caller-held workspace keeps its own tapes under graph="replay" and the backward binds them only while the workspace made no other call in between (else the replay form); the kept tapes add no memory there, while the recompute’s own hand-off graph holds its static results like every graph key (one more saved set per checkpointed layer and, when that recompute launches the output program – recompute_output="always", or True with a later saver – a full static output); there "auto" decides only from cake_kcp.GRAPH_RECOMPUTE_MIN_ROWS token x value-head rows per rank (a smaller recompute replays the original forward’s graph: its host-bound step gains nothing from the hand-off).

recompute_output (keyword-only, "auto" default | "always"): torch’s recompute of a checkpointed forward returns nothing, so when this node is the last operation of the checkpointed region that saves a tensor, the recompute’s output is read by nothing; "auto" leaves that output program out after reading the proof from torch’s checkpoint frame (the node is executable, the remaining saved-tensor holders are exactly this node’s, early stop is on, and the torch release – torch.__version__ without its local +cuNNN build tag – is one of the package’s verified releases, cake_kcp.RECOMPUTE_OUTPUT_TORCH_VERSIONS) and launches it as before in every other state – a later operation of the region that saves the output (another layer, a norm), a reentrant checkpoint, a forward outside a recompute, a non-executable node, another torch build, and under graph="replay" with the default reuse_recompute_tapes="auto" a rank below cake_kcp.GRAPH_RECOMPUTE_MIN_ROWS (reason "small"; an explicit True / False reads the proof at every size) (cake_kcp.LAST_CALL["fwd"]["fwd_o_reason"] records the decision and its reason). The gradients and the retained bytes are unchanged either way; "always" is the unconditional launch.

recompute_handle (keyword-only, default None; a CakeGDNNondetKcpRecomputeHandle): the checkpoint recompute of this forward runs without cross-rank communication. The original forward exchanges as always and stores this rank’s incoming states S_in (the saved set’s own tensor) in the handle; torch’s recompute of the forward inside .backward() reads them back and runs no exchange, no transfer-summary walk and no merge – only the prep program, at P > 1 the local segment walks and carry merges, and the state walk from S_in, the launches of the light backward’s replay – so the gradients are bitwise those of the same checkpoint without a handle. The backward still exchanges. The handle retains S_in (cake_gdn_nondet_train_kcp_saved_bytes(ctx, ..., save_mode="light", save_transfer=False)) from the original forward until the checkpoint frame holding it is freed, instead of retaining nothing (with_conv=False in that query: the convolution keeps its own rows). Create one handle per layer and micro-step, outside the outermost checkpoint region, and pass it as an argument of the checkpointed function (CakeGDNNondetKcpRecomputeHandle of this module):

def layer(x, handle):
    ...
    return cake_gdn_nondet_kcp(q, k, v, g, beta, ctx, recompute_handle=handle)

y = torch.utils.checkpoint.checkpoint(layer, x, CakeGDNNondetKcpRecomputeHandle(), use_reentrant=False)

A handle holds one entry per key: it serves this call and at most one context-parallel convolution call (whose halo rows go under the key "conv"); a layer with separate Q / K / V convolutions uses one handle per convolution call, and this call shares one of them. Needs boundary_mode="saved" and graph="off"; implies save_transfer=False. A forward outside the backward that finds an entry raises (a handle reused across micro-steps).

q / k / v that are head views of one packed [..., T, (2 H_qk + HV) * 128] row (the convolution’s output) get the full backward speed as the views of cake_gdn_nondet_train_kcp_packed_qkv_views() (or of the deterministic package’s cake_gdn_det_train_kcp_packed_qkv_views): both helpers’ split nodes pass a packed gradient row on without a copy. The package writes that row only for the outputs of either helper. Plain split / view views of the row give the same results, but torch’s split backward concatenates dq / dk / dv into a new row: about 0.2 ms (6-7 %) more backward per call at (16, 48) and 32,768 tokens on B200 and GB300 and 0.17 ms (7-8 %) on R200, with the in-kernel norm off or on (flashinfer/experimental/cake/cake_gdn_nondet_train_kcp/README.md, “Packed Q/K/V”).

torch.use_deterministic_algorithms changes neither the forward nor its checkpoint recompute (see the module docstring).