mirror of
https://github.com/zml/zml.git
synced 2026-11-16 19:11:09 +00:00
This implement support for OpenAI "gpt-oss" models. It's mostly a take over of #292 branch, but MoE code is kept inside examples folder cause it's still a bit warm. In particular the routing logic can break apart when there are undefined memory in the inputs, where it wasn't an issue for normal MLP. I only tested with the 20B version. |
History
|
|
|---|---|---|
| .. | ||
| BUILD.bazel | ||
| GptOss.zig | ||
| main.zig | ||
| README.md | ||
GptOss
Download the model (you can change 20b to 120b for the bigger version if you have the required hardware).
bazel run //tools/hf -- download openai/gpt-oss-20b --local-dir /tmp/models/openai/gpt-oss-20b --exclude metal/ original/
Run the model, use --@zml//runtimes:xxx flags to target the relevant platform for your hardware.
bazel run --config=release //examples/gpt_oss \
--@zml//runtimes:cuda=true \
--@zml//runtimes:cpu=false \
--@zml//runtimes:tpu=false \
--@zml//runtimes:rocm=false \
-- --hf-model-path=/tmp/models/openai/gpt-oss-20b/ \
--prompt='What are some spooky funfacts about animal for a Halloween party ?'