cake_kda_det_train_kcp_partition(...[, ...])
|
The context-parallel partition of one packed batch over world_size ranks. |
cake_kda_det_train_kcp_build_context(part, ...)
|
This rank's context of a partition: its device tables, the collectives of the state and halo exchanges and the segment plan of its fragments. |
cake_kda_det_train_kcp_forward(q, k, v, g, ...)
|
Context-parallel packed KDA chunked training forward of this rank's fragments. |
cake_kda_det_train_kcp_backward(q, k, v, g, ...)
|
Context-parallel packed KDA chunked training backward of this rank's fragments. |
cake_kda_det_kcp(q, k, v, g, beta, ctx[, ...])
|
Autograd-enabled context-parallel packed KDA chunked training step of this rank's fragments (the forward of cake_kda_det_train_kcp_forward(); the backward of cake_kda_det_train_kcp_backward() from its saved set). |
cake_kda_det_train_kcp_workspace_bytes(ctx, *)
|
Bytes of this rank's KCP workspace for H heads: the local arena (that of cake_kda_det_train_workspace_bytes() for a backward workspace; a forward-only one takes the KCP forward layout, 2432 B per internal token-head) plus the boundary pools of the context's collectives, as CakeKDADetKcpWorkspace allocates them (the backward's send pools share the forward's storage; the ring exchange's hop pools, allocated by the first exchange="ring" call, are not counted). |
cake_kda_det_train_kcp_reduce_gate_param_grads(...)
|
The raw gate mode's full (dA_log, ddt_bias) from every rank's partial sums (cake_kda_det_train_kcp_backward()), bitwise reproducible: one all-gather on ctx.group, then an FP32 sum in rank order on every rank, so every rank gets the same bits under every schedule. |
cake_kda_det_train_kcp_saved_bytes(ctx, *[, ...])
|
Bytes of this rank's KCP saved set for H heads: the local saved set (cake_kda_det_train_saved_bytes()), the boundary states, and (with_conv) the convolution's history rows of C channels. |
cake_kda_det_train_kcp_comm_bytes(ctx, *[, ...])
|
Bytes this rank sends and receives in one training step (forward and backward) for H heads and C convolution channels: the state summaries, the boundary states and the halo rows of the context's collectives, as a table per collective and per boundary (cake_kcp_plan.cake_kda_det_train_kcp_comm_bytes). |