0
Fork 0
mirror of https://github.com/zml/zml.git synced 2026-11-16 19:11:09 +00:00
Commit graph

3 commits

Author SHA1 Message Date
Corentin Godeau
08391af750
zml/Compiler: remove conditionals from command buffer (#913)
Due to recent changes in XLA (introduced in
3cf18a01c7), the way child are
counted to decide if a conditional is expressed as a graph conditional
or kept outside has been changed.

With the new corrected way, the body of the conditional used in the
paged attention StableHLO backend was recorded as a child graph. The
problem is that when we are sharded, there are collectives inside this
child graph and XLA adds event wait/record around the actual collective
kernel. This is explicitly forbidden because
https://docs.nvidia.com/cuda/cuda-programming-guide/04-special-topics/cuda-graphs.html#conditional-node-body-graph-requirements
states:

> The graph can only contain kernel nodes, empty nodes, memcpy nodes,
memset nodes, child graph nodes, and conditional nodes.

The temporary solution is to remove `CONDITIONAL` from the
`xla_gpu_enable_command_buffer` XLA flag. In practice this reverts to
the old behavior we had when the counting was done differently.

Long term, I don't have a solution yet and. There is an environment
variable that solves it
[NCCL_GRAPH_MIXING_SUPPORT](https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/env.html#nccl-graph-mixing-support)
but I don't know if it's safe to disable it.

Maybe we could also check if the events introduced by NCCL are actually
required.
2026-10-08 12:33:34 +02:00
Benjamin Rabier
289c29f365
workspace: update AGENTS.md (#548)
More concise AGENTS.md and add a command to help the agent find the
sources of the Zig library it regularly struggles with it
2026-05-25 17:14:28 +02:00
Benjamin Rabier
82c9c25836
examples/lfm: Add LiquidAI LFM2.5 (#376) 2026-02-19 09:27:10 +01:00