flashinfer.cake_gdn_nondet_train_kcp.cake_gdn_nondet_kcp¶
- flashinfer.cake_gdn_nondet_train_kcp.cake_gdn_nondet_kcp(q: Tensor, k: Tensor, v: Tensor, g: Tensor, beta: Tensor, ctx: Any, *, scale: float | None = None, save_mode: str | None = None, boundary_mode: str = 'saved', workspace: Any | None = None, graph: str = 'off', options: Any | None = None, recompute_handle: Any | None = None, use_qk_l2norm_in_kernel: bool = False, exchange: str = 'a2a', save_transfer: bool = True, precision_mode: int = 0, transfer_walk: str = 'tf32_hilo', offset_class: str = 'fla', layout: str = 'tma', group_reduce: str = 'fused', reuse_recompute_tapes: bool | str | None = None, recompute_output: str = 'auto', backend: str = 'kernels', segments: int | None = None, test_knobs: dict[str, Any] | None = None) Tensor¶
Context-parallel packed GDN training as an autograd node (
torch.utils.checkpointcompatible).Returns
outBF16 likevwith gradients toq, k, v, g, beta; every saved tensor travels throughsave_for_backwardonly, so a checkpointed forward retains nothing. Seecake_gdn_nondet_train_kcp_forward()for the operands and options andcake_gdn_nondet_train_kcp_backward()for the gradients.reuse_recompute_tapes(keyword-only, default"auto";True|False): undertorch.utils.checkpoint(use_reentrant=False)thesave_mode="light"backward binds the forward tapes that torch’s recompute of the forward (inside.backward()) just wrote, instead of replaying the prep program and the state walk itself; the gradients are bitwise the same, the checkpointed forward still retains nothing, and the direct (non-checkpoint) autograd path,boundary_mode="replay"andsave_mode="full"are unchanged. The kept tapes live from the recompute to this node’s backward: when a later operation of the checkpointed region saves a tensor (an output projection, an MLP), they live through that operation’s backward and raise the region’s backward peak by the full tape set (one GB300, 261,987 tokens,(2, 6), GDN + a 768x768 projection + a SwiGLU MLP: 9250.54 MiB against 6927.31 MiB replayed, 21.29 against 22.70 ms per step of the region);"auto"(the default) keeps them only when this node is the region’s last saver (the checkpoint-frame proof ofrecompute_output) and takes the replay form otherwise – a region this node closes keeps today’s speed and memory;Truealways keeps them,Falsenever (the replay form). One kept set is live at a time: in a checkpoint segment holding several GDN layers only the last layer skips the replay (the earlier layers replay as before). Undergraph="replay"a caller-held workspace keeps its own tapes undergraph="replay"and the backward binds them only while the workspace made no other call in between (else the replay form); the kept tapes add no memory there, while the recompute’s own hand-off graph holds its static results like every graph key (one more saved set per checkpointed layer and, when that recompute launches the output program –recompute_output="always", orTruewith a later saver – a full static output); there"auto"decides only fromcake_kcp.GRAPH_RECOMPUTE_MIN_ROWStoken x value-head rows per rank (a smaller recompute replays the original forward’s graph: its host-bound step gains nothing from the hand-off).recompute_output(keyword-only,"auto"default |"always"): torch’s recompute of a checkpointed forward returns nothing, so when this node is the last operation of the checkpointed region that saves a tensor, the recompute’s output is read by nothing;"auto"leaves that output program out after reading the proof from torch’s checkpoint frame (the node is executable, the remaining saved-tensor holders are exactly this node’s, early stop is on, and the torch release –torch.__version__without its local+cuNNNbuild tag – is one of the package’s verified releases,cake_kcp.RECOMPUTE_OUTPUT_TORCH_VERSIONS) and launches it as before in every other state – a later operation of the region that saves the output (another layer, a norm), a reentrant checkpoint, a forward outside a recompute, a non-executable node, another torch build, and undergraph="replay"with the defaultreuse_recompute_tapes="auto"a rank belowcake_kcp.GRAPH_RECOMPUTE_MIN_ROWS(reason"small"; an explicitTrue/Falsereads the proof at every size) (cake_kcp.LAST_CALL["fwd"]["fwd_o_reason"]records the decision and its reason). The gradients and the retained bytes are unchanged either way;"always"is the unconditional launch.recompute_handle(keyword-only, defaultNone; aCakeGDNNondetKcpRecomputeHandle): the checkpoint recompute of this forward runs without cross-rank communication. The original forward exchanges as always and stores this rank’s incoming statesS_in(the saved set’s own tensor) in the handle; torch’s recompute of the forward inside.backward()reads them back and runs no exchange, no transfer-summary walk and no merge – only the prep program, atP > 1the local segment walks and carry merges, and the state walk fromS_in, the launches of the light backward’s replay – so the gradients are bitwise those of the same checkpoint without a handle. The backward still exchanges. The handle retainsS_in(cake_gdn_nondet_train_kcp_saved_bytes(ctx, ..., save_mode="light", save_transfer=False)) from the original forward until the checkpoint frame holding it is freed, instead of retaining nothing (with_conv=Falsein that query: the convolution keeps its own rows). Create one handle per layer and micro-step, outside the outermost checkpoint region, and pass it as an argument of the checkpointed function (CakeGDNNondetKcpRecomputeHandleof this module):def layer(x, handle): ... return cake_gdn_nondet_kcp(q, k, v, g, beta, ctx, recompute_handle=handle) y = torch.utils.checkpoint.checkpoint(layer, x, CakeGDNNondetKcpRecomputeHandle(), use_reentrant=False)
A handle holds one entry per key: it serves this call and at most one context-parallel convolution call (whose halo rows go under the key
"conv"); a layer with separate Q / K / V convolutions uses one handle per convolution call, and this call shares one of them. Needsboundary_mode="saved"andgraph="off"; impliessave_transfer=False. A forward outside the backward that finds an entry raises (a handle reused across micro-steps).q/k/vthat are head views of one packed[..., T, (2 H_qk + HV) * 128]row (the convolution’s output) get the full backward speed as the views ofcake_gdn_nondet_train_kcp_packed_qkv_views()(or of the deterministic package’scake_gdn_det_train_kcp_packed_qkv_views): both helpers’ split nodes pass a packed gradient row on without a copy. The package writes that row only for the outputs of either helper. Plainsplit/viewviews of the row give the same results, but torch’s split backward concatenatesdq/dk/dvinto a new row: about 0.2 ms (6-7 %) more backward per call at(16, 48)and 32,768 tokens on B200 and GB300 and 0.17 ms (7-8 %) on R200, with the in-kernel norm off or on (flashinfer/experimental/cake/cake_gdn_nondet_train_kcp/README.md, “Packed Q/K/V”).torch.use_deterministic_algorithmschanges neither the forward nor its checkpoint recompute (see the module docstring).