# LLM `//examples/llm` runs interactive or one-shot text generation from a model repository from HuggingFace. We support the following models, automatically detected from the `model_type` in the `config.json`: - Llama 3.1 - Qwen 3.5 - LFM 2.5 ## Run To load a model from HuggingFace directly: ```bash # CPU bazel run //examples/llm -- --model=hf://meta-llama/Llama-3.1-8B-Instruct # CUDA bazel run //examples/llm --@zml//platforms:cuda=true -- --model=hf://meta-llama/Llama-3.1-8B-Instruct # ROCm bazel run //examples/llm --@zml//platforms:rocm=true -- --model=hf://meta-llama/Llama-3.1-8B-Instruct ``` From a local directory: ```bash bazel run //examples/llm --@zml//platforms:cuda=true -- --model=/var/models/meta-llama/Llama-3.1-8B-Instruct/ ``` For a single non-interactive prompt: ```bash bazel run //examples/llm --@zml//platforms:cuda=true -- --model=hf://meta-llama/Llama-3.1-8B-Instruct --prompt="Write a haiku about Zig" ``` ## Options - `--model=`: Required. Model repository to load. This can be a local path or a huggingface/S3 URI such as `hf://...` or `s3://...`. - `--prompt=`: Optional. Runs a single prompt instead of opening the interactive chat loop. - `--seqlen=`: Optional. Maximum sequence length. Defaults to `2048`. - `--backend=`: Optional. Attention backend. `cute` uses CuTe flash attention on CUDA GPUs with compute capability 8.0 or newer. If omitted, the program auto-selects one for the current platform, including `cute` on supported CUDA GPUs.