flashinfer.cake_causal_conv1d_nondet_train_kcp.cake_causal_conv1d_nondet_kcp_multi¶
- flashinfer.cake_causal_conv1d_nondet_train_kcp.cake_causal_conv1d_nondet_kcp_multi(xs: Sequence[Tensor], weights: Sequence[Tensor], ctx: Any, *, recompute_handle: Any | None = None, cooperative: bool | None = None, bwd_reserve_sms: int | None = None) tuple[Tensor, ...]¶
Autograd-enabled context-parallel convolutions of several tensors of one layer (for example its q, k and v) with one halo exchange per direction, non-deterministic backward.
Outputs and
dxare bitwisetuple(cake_causal_conv1d_nondet_kcp(x, w, ctx) for x, w in zip(xs, weights)); eachdweightis that tensor’s own unordered FP32 partial. Under torch’s deterministic mode the backward alerts like the single call’s.- Parameters:
xs (sequence of torch.Tensor) –
n >= 1BF16[S_local, C_i]or[1, S_local, C_i]contiguous packed activations of this rank’s fragments; eachC_iis a multiple of 8 (they may differ).weights (sequence of torch.Tensor) – FP32
[C_i, 4]contiguous taps, one per tensor.ctx (CakeKDADetKcpContext) – As in
cake_causal_conv1d_nondet_kcp(). Every rank of the group calls this function once per layer, in the same order, with the same tensor count and channel counts.recompute_handle (CakeCausalConv1dNondetKcpRecomputeHandle, optional) – One handle for the whole call (its
"conv"entry holds thenhistory pools), under the rules ofcake_causal_conv1d_nondet_kcp().cooperative (bool, optional) – As in
cake_causal_conv1d_nondet_kcp().bwd_reserve_sms (int, optional) – As in
cake_causal_conv1d_nondet_kcp(), one value for every tensor of the call.
- Returns:
The
nBF16 outputs, each of its input’s shape.- Return type:
tuple of torch.Tensor