flashinfer.cake_gdn_det_train_kcp.cake_gdn_det_kcp¶
- flashinfer.cake_gdn_det_train_kcp.cake_gdn_det_kcp(q: Tensor, k: Tensor, v: Tensor, g: Tensor, beta: Tensor, ctx: Any, *, scale: float | None = None, save_mode: str = 'light', boundary_mode: str = 'saved', workspace: Any | None = None, graph: str = 'off', options: Any | None = None, recompute_handle: Any | None = None, use_qk_l2norm_in_kernel: bool = False, exchange: str = 'a2a', save_transfer: bool = True, precision_mode: int = 0, transfer_walk: str = 'tf32_hilo', offset_class: str = 'fla', layout: str = 'tma', group_reduce: str = 'fused', reuse_recompute_tapes: bool | str | None = None, recompute_output: str = 'auto', backend: str = 'kernels', segments: int | None = None, test_knobs: dict[str, Any] | None = None) Tensor¶
Context-parallel packed GDN training as an autograd node (
torch.utils.checkpointcompatible).Returns
outBF16 likevwith gradients toq, k, v, g, beta; every saved tensor travels throughsave_for_backwardonly, so a checkpointed forward retains nothing. Seecake_gdn_det_train_kcp_forward()for the operands and options andcake_gdn_det_train_kcp_backward()for the gradients.reuse_recompute_tapes(keyword-only, default"auto";True|False): undertorch.utils.checkpoint(use_reentrant=False)thesave_mode="light"backward binds the forward tapes that torch’s recompute of the forward (inside.backward()) just wrote, instead of replaying the prep program and the state walk itself; the gradients are bitwise the same, the checkpointed forward still retains nothing, and the direct (non-checkpoint) autograd path,boundary_mode="replay"andsave_mode="full"are unchanged. The kept tapes live from the recompute to this node’s backward: when a later operation of the checkpointed region saves a tensor (an output projection, an MLP), they live through that operation’s backward and raise the region’s backward peak by the full tape set (one GB300, 261,987 tokens,(2, 6), GDN + a 768x768 projection + a SwiGLU MLP: 9250.54 MiB against 6927.31 MiB replayed, 21.29 against 22.70 ms per step of the region);"auto"(the default) keeps them only when this node is the region’s last saver (the checkpoint-frame proof ofrecompute_output) and takes the replay form otherwise – a region this node closes keeps today’s speed and memory;Truealways keeps them,Falsenever (the replay form). One kept set is live at a time: in a checkpoint segment holding several GDN layers only the last layer skips the replay (the earlier layers replay as before). Not available withgraph="replay": there the default resolves toFalseand an explicitTrue/"auto"is refused.recompute_output(keyword-only,"auto"default |"always"): torch’s recompute of a checkpointed forward returns nothing, so when this node is the last operation of the checkpointed region that saves a tensor, the recompute’s output is read by nothing;"auto"leaves that output program out after reading the proof from torch’s checkpoint frame (the node is executable, the remaining saved-tensor holders are exactly this node’s, early stop is on, and the torch release –torch.__version__without its local+cuNNNbuild tag – is one of the package’s verified releases,cake_kcp.RECOMPUTE_OUTPUT_TORCH_VERSIONS) and launches it as before in every other state – a later operation of the region that saves the output (another layer, a norm), a reentrant checkpoint, a forward outside a recompute, a non-executable node, another torch build (cake_kcp.LAST_CALL["fwd"]["fwd_o_reason"]records the decision and its reason). The gradients and the retained bytes are unchanged either way;"always"is the unconditional launch.recompute_handle(keyword-only, defaultNone; aCakeGDNDetKcpRecomputeHandle): the checkpoint recompute of this forward runs without cross-rank communication. The original forward exchanges as always and stores this rank’s incoming statesS_in(the saved set’s own tensor) in the handle; torch’s recompute of the forward inside.backward()reads them back and runs no exchange, no transfer-summary walk and no merge – only the prep program, atP > 1the local segment walks and carry merges, and the state walk fromS_in, the launches of the light backward’s replay – so the gradients are bitwise those of the same checkpoint without a handle. The backward still exchanges. The handle retainsS_in(cake_gdn_det_train_kcp_saved_bytes(ctx, ..., save_mode="light", save_transfer=False)) from the original forward until the checkpoint frame holding it is freed, instead of retaining nothing (with_conv=Falsein that query: the convolution keeps its own rows). Create one handle per layer and micro-step, outside the outermost checkpoint region, and pass it as an argument of the checkpointed function:def layer(x, handle): ... return cake_gdn_det_kcp(q, k, v, g, beta, ctx, recompute_handle=handle) y = torch.utils.checkpoint.checkpoint(layer, x, CakeGDNDetKcpRecomputeHandle(), use_reentrant=False)
A handle holds one entry per key: it serves this call and at most one context-parallel convolution call (whose halo rows go under the key
"conv"); a layer with separate Q / K / V convolutions uses one handle per convolution call, and this call shares one of them. Needsboundary_mode="saved"andgraph="off"; impliessave_transfer=False. A forward outside the backward that finds an entry raises (a handle reused across micro-steps).use_qk_l2norm_in_kernel(keyword-only, defaultFalse): the in-kernel per-head L2 norm ofq/k(module docstring); it is part of the saved set and of the recompute handle’s entry, so a checkpoint, a handle and a CUDA-graph replay see it like any other option.q/k/vthat are head views of one packed[..., T, (2 H_qk + HV) * 128]row (the convolution’s output) get the full backward speed only ascake_gdn_det_train_kcp_packed_qkv_views()views. Plainsplit/viewviews of the row give bitwise the same results, but torch’s split backward concatenatesdq/dk/dvinto a new row: about 1.1 ms (33-36 %) more backward per call at(16, 48)and 32,768 tokens on GB200 and GB300, with the in-kernel norm off or on (package README, “Packed Q/K/V”).