Local LLMs at the metal.

Models for Mila — a C++23 and CUDA runtime for open LLMs, built from explicit components you can read and understand.

Gemma 4 and Llama 3.x, quantized to FP4 so they run on the consumer hardware you already own.

Compatibility

These models are built for Mila and load with Mila. The quantization is Mila's own, matched to its CUDA kernels rather than to a portable interchange format.

The files are ordinary safetensors. Open one in any reader and the tensors, shapes and metadata are all there to read.

Each model page names the original it was built from, so the upstream weights are always one link away.

Installing a model

Install once, then load as often as you like. Installing is the only step that touches the network — loading reads what is already on your machine, so nothing downloads behind your back.

In Mila's chat harness:

/install Llama-3.2-3B-Instruct-fp4
/model Llama-3.2-3B-Instruct-fp4

From Python (mila-llm):

import mila

mila.ModelStore().pull("Llama-3.2-3B-Instruct-fp4", mila.default_hub_owner())

tokenizer = mila.BpeTokenizer.from_store("Llama-3.2-3B-Instruct-fp4")
model = mila.LlamaModel.from_store("Llama-3.2-3B-Instruct-fp4", 4096)

From C++:

ModelStore store;
ModelResolver resolver( store, *makeDefaultModelHub() );

resolver.pull( "Llama-3.2-3B-Instruct-fp4", std::string( kDefaultHubOwner ) );

const auto model = store.locate( "Llama-3.2-3B-Instruct-fp4" );

Every file is checked against a published checksum as it downloads, and the chat harness, the Python package and the inference server all share one local store — install a model once and all three can use it.

---

mila.toddt.me · GitHub · PyPI