mirror of
https://github.com/zml/zml.git
synced 2026-11-16 19:11:09 +00:00
zml/Compiler: remove conditionals from command buffer (#913)
Due to recent changes in XLA (introduced in
3cf18a01c7), the way child are
counted to decide if a conditional is expressed as a graph conditional
or kept outside has been changed.
With the new corrected way, the body of the conditional used in the
paged attention StableHLO backend was recorded as a child graph. The
problem is that when we are sharded, there are collectives inside this
child graph and XLA adds event wait/record around the actual collective
kernel. This is explicitly forbidden because
https://docs.nvidia.com/cuda/cuda-programming-guide/04-special-topics/cuda-graphs.html#conditional-node-body-graph-requirements
states:
> The graph can only contain kernel nodes, empty nodes, memcpy nodes,
memset nodes, child graph nodes, and conditional nodes.
The temporary solution is to remove `CONDITIONAL` from the
`xla_gpu_enable_command_buffer` XLA flag. In practice this reverts to
the old behavior we had when the counting was done differently.
Long term, I don't have a solution yet and. There is an environment
variable that solves it
[NCCL_GRAPH_MIXING_SUPPORT](https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/env.html#nccl-graph-mixing-support)
but I don't know if it's safe to disable it.
Maybe we could also check if the events introduced by NCCL are actually
required.
This commit is contained in:
parent
b774b7ad16
commit
08391af750
2 changed files with 6 additions and 2 deletions
Loading…
Add table
Add a link
Reference in a new issue