cake_kda_nondet_train_kcp_partition(...[, ...])
|
The context-parallel partition of one packed batch over world_size ranks. |
cake_kda_nondet_train_kcp_build_context(...)
|
This rank's context of a partition: its device tables, the collectives of the state and halo exchanges and the segment plan of its fragments. |
cake_kda_nondet_train_kcp_forward(q, k, v, ...)
|
Context-parallel packed KDA chunked training forward of this rank's fragments. |
cake_kda_nondet_train_kcp_backward(q, k, v, ...)
|
Context-parallel packed KDA chunked training backward of this rank's fragments. |
cake_kda_nondet_kcp(q, k, v, g, beta, ctx[, ...])
|
Autograd-enabled context-parallel packed KDA chunked training step of this rank's fragments (the forward of cake_kda_nondet_train_kcp_forward(); the backward of cake_kda_nondet_train_kcp_backward() from its saved set). |
cake_kda_nondet_train_kcp_workspace_bytes(ctx, *)
|
Bytes of this rank's KCP workspace for H heads: the local arena (that of cake_kda_nondet_train_workspace_bytes() for a backward workspace; a forward-only one takes the KCP forward layout, 2432 B per internal token-head) plus the boundary pools of the context's collectives, as CakeKDANondetKcpWorkspace allocates them (the backward's send pools share the forward's storage; the ring exchange's hop pools, allocated by the first exchange="ring" call, are not counted). |
cake_kda_nondet_train_kcp_reduce_gate_param_grads(...)
|
The raw gate mode's full (dA_log, ddt_bias) from every rank's partial sums (cake_kda_nondet_train_kcp_backward()), bitwise reproducible: one all-gather on ctx.group, then an FP32 sum in rank order on every rank, so every rank gets the same bits under every schedule. |
cake_kda_nondet_train_kcp_saved_bytes(ctx, *)
|
Bytes of this rank's KCP saved set for H heads: the local saved set (cake_kda_nondet_train_saved_bytes()), the boundary states, and (with_conv) the convolution's history rows of C channels. |
cake_kda_nondet_train_kcp_handoff_bytes(ctx, *)
|
Bytes of the checkpoint hand-off's set for one engaged call with H heads on this rank, the most by which it raises this rank's checkpoint peak (see cake_kda_nondet_kcp()). |
cake_kda_nondet_train_kcp_handoff_stats()
|
A copy of the checkpoint hand-off counters of this process ("save": forwards that kept a tape set, "bound": backwards that read one, "refused_<reason>": requests the static keys refused). |
cake_kda_nondet_train_kcp_comm_bytes(ctx, *)
|
Bytes this rank sends and receives in one training step (forward and backward) for H heads and C convolution channels: the state summaries, the boundary states and the halo rows of the context's collectives, as a table per collective and per boundary (cake_kcp_plan.cake_kda_nondet_train_kcp_comm_bytes). |