flashinfer.cake_kda_nondet_train_kcp.cake_kda_nondet_train_kcp_reduce_gate_param_grads

flashinfer.cake_kda_nondet_train_kcp.cake_kda_nondet_train_kcp_reduce_gate_param_grads(ctx: Any, dA_log: Tensor, ddt_bias: Tensor) → tuple[Tensor, Tensor]

The raw gate mode’s full (dA_log, ddt_bias) from every rank’s partial sums (cake_kda_nondet_train_kcp_backward()), bitwise reproducible: one all-gather on ctx.group, then an FP32 sum in rank order on every rank, so every rank gets the same bits under every schedule. An NCCL all-reduce gives the same sum to FP32 rounding but does not fix its summation order. The deterministic package’s cake_kda_det_train_kcp_reduce_gate_param_grads is the same algorithm (the same bits from the same partials). A collective: every rank of ctx.group calls it. With world_size == 1 copies of the inputs; with world_size > 1 and no initialized process group it raises RuntimeError.

Parameters:
  • ctx – This rank’s KCP context; every rank of ctx.group must call the collective.

  • dA_log (torch.Tensor) – This rank’s FP32 partial gradient of A_log from the KCP backward.

  • ddt_bias (torch.Tensor) – This rank’s FP32 partial gradient of dt_bias from the KCP backward.