flashinfer.cake_kda_nondet_train_kcp

Cake non-deterministic KDA KCP training helpers and entry points. The module is a stable API; its backend package, flashinfer.experimental.cake.cake_kda_nondet_train_kcp, is experimental and is imported on first use.

cake_kda_nondet_train_kcp_partition(...[, ...])

The context-parallel partition of one packed batch over world_size ranks.

cake_kda_nondet_train_kcp_build_context(...)

This rank's context of a partition: its device tables, the collectives of the state and halo exchanges and the segment plan of its fragments.

cake_kda_nondet_train_kcp_forward(q, k, v, ...)

Context-parallel packed KDA chunked training forward of this rank's fragments.

cake_kda_nondet_train_kcp_backward(q, k, v, ...)

Context-parallel packed KDA chunked training backward of this rank's fragments.

cake_kda_nondet_kcp(q, k, v, g, beta, ctx[, ...])

Autograd-enabled context-parallel packed KDA chunked training step of this rank's fragments (the forward of cake_kda_nondet_train_kcp_forward(); the backward of cake_kda_nondet_train_kcp_backward() from its saved set).

cake_kda_nondet_train_kcp_workspace_bytes(ctx, *)

Bytes of this rank's KCP workspace for H heads: the local arena (that of cake_kda_nondet_train_workspace_bytes() for a backward workspace; a forward-only one takes the KCP forward layout, 2432 B per internal token-head) plus the boundary pools of the context's collectives, as CakeKDANondetKcpWorkspace allocates them (the backward's send pools share the forward's storage; the ring exchange's hop pools, allocated by the first exchange="ring" call, are not counted).

cake_kda_nondet_train_kcp_reduce_gate_param_grads(...)

The raw gate mode's full (dA_log, ddt_bias) from every rank's partial sums (cake_kda_nondet_train_kcp_backward()), bitwise reproducible: one all-gather on ctx.group, then an FP32 sum in rank order on every rank, so every rank gets the same bits under every schedule.

cake_kda_nondet_train_kcp_saved_bytes(ctx, *)

Bytes of this rank's KCP saved set for H heads: the local saved set (cake_kda_nondet_train_saved_bytes()), the boundary states, and (with_conv) the convolution's history rows of C channels.

cake_kda_nondet_train_kcp_handoff_bytes(ctx, *)

Bytes of the checkpoint hand-off's set for one engaged call with H heads on this rank, the most by which it raises this rank's checkpoint peak (see cake_kda_nondet_kcp()).

cake_kda_nondet_train_kcp_handoff_stats()

A copy of the checkpoint hand-off counters of this process ("save": forwards that kept a tape set, "bound": backwards that read one, "refused_<reason>": requests the static keys refused).

cake_kda_nondet_train_kcp_comm_bytes(ctx, *)

Bytes this rank sends and receives in one training step (forward and backward) for H heads and C convolution channels: the state summaries, the boundary states and the halo rows of the context's collectives, as a table per collective and per boundary (cake_kcp_plan.cake_kda_nondet_train_kcp_comm_bytes).