0
Fork 0
mirror of https://github.com/zml/zml.git synced 2026-11-16 19:11:09 +00:00

zml/Compiler: remove conditionals from command buffer (#913)

Due to recent changes in XLA (introduced in
3cf18a01c7), the way child are
counted to decide if a conditional is expressed as a graph conditional
or kept outside has been changed.

With the new corrected way, the body of the conditional used in the
paged attention StableHLO backend was recorded as a child graph. The
problem is that when we are sharded, there are collectives inside this
child graph and XLA adds event wait/record around the actual collective
kernel. This is explicitly forbidden because
https://docs.nvidia.com/cuda/cuda-programming-guide/04-special-topics/cuda-graphs.html#conditional-node-body-graph-requirements
states:

> The graph can only contain kernel nodes, empty nodes, memcpy nodes,
memset nodes, child graph nodes, and conditional nodes.

The temporary solution is to remove `CONDITIONAL` from the
`xla_gpu_enable_command_buffer` XLA flag. In practice this reverts to
the old behavior we had when the counting was done differently.

Long term, I don't have a solution yet and. There is an environment
variable that solves it
[NCCL_GRAPH_MIXING_SUPPORT](https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/env.html#nccl-graph-mixing-support)
but I don't know if it's safe to disable it.

Maybe we could also check if the events introduced by NCCL are actually
required.
This commit is contained in:
Corentin Godeau 2026-10-08 12:33:34 +02:00 • committed by GitHub
commit 08391af750
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
2 changed files with 6 additions and 2 deletions