flashinfer.cake_kda_nondet_train_kcp.cake_kda_nondet_train_kcp_reduce_gate_param_grads¶
- flashinfer.cake_kda_nondet_train_kcp.cake_kda_nondet_train_kcp_reduce_gate_param_grads(ctx: Any, dA_log: Tensor, ddt_bias: Tensor) tuple[Tensor, Tensor]¶
The raw gate mode’s full
(dA_log, ddt_bias)from every rank’s partial sums (cake_kda_nondet_train_kcp_backward()), bitwise reproducible: one all-gather onctx.group, then an FP32 sum in rank order on every rank, so every rank gets the same bits under every schedule. An NCCL all-reduce gives the same sum to FP32 rounding but does not fix its summation order. The deterministic package’scake_kda_det_train_kcp_reduce_gate_param_gradsis the same algorithm (the same bits from the same partials). A collective: every rank ofctx.groupcalls it. Withworld_size == 1copies of the inputs; withworld_size > 1and no initialized process group it raisesRuntimeError.- Parameters:
ctx – This rank’s KCP context; every rank of
ctx.groupmust call the collective.dA_log (torch.Tensor) – This rank’s FP32 partial gradient of
A_logfrom the KCP backward.ddt_bias (torch.Tensor) – This rank’s FP32 partial gradient of
dt_biasfrom the KCP backward.