Inference & LLM

Architecture

InferenceSession

The main entry point for LLM inference. Loads weights from a SafeTensors file, sets up the KV-cache, and exposes token-step and generation APIs.

$cfg     = ModelConfig::llama3_8b();
$session = InferenceSession::load('models/llama3-8b.safetensors', $cfg);

foreach ($session->generate($promptIds, maxNewTokens: 512, temperature: 0.8) as $id) {
    echo $tok->decode([$id]);
    flush();
}

KVCache

Pml\KVCache wraps a C MultiKVCache* struct that manages nLayers × nHeads key/value cache slots. Eliminates the O(T²) attention recompute during autoregressive decode.

$kv = new KVCache(
    nLayers:   32,
    nHeads:    8,    // nKvHeads for GQA
    maxSeqLen: 4096,
    headDim:   128,
);
printf("Cache memory: %.1f MB\n", $kv->memoryBytes() / 1e6);

Memory estimate

// LLaMA-3 8B: 32 layers, 8 KV-heads, 4096 max context, 128 head-dim
$kv = new KVCache(32, 8, 4096, 128);
// = 32 × 8 × 4096 × 2 × 128 × 4 bytes = 1,073,741,824 bytes = 1.07 GB

// Reduce context to 2048 for lower memory:
$kv = new KVCache(32, 8, 2048, 128);
// = 536 MB

Tokenizer (BPE)

Pml\Inference\Tokenizer wraps the C BPE tokenizer (tokenizer.c). Supports HuggingFace tokenizer.json format and raw vocabulary + merges files. Zero-copy: the PHP Tokenizer holds a C Tokenizer* pointer.

$tok = Tokenizer::fromJson('models/llama3/tokenizer.json');

$ids = $tok->encode("Hello world", addBos: true);
// → [1, 22557, 1879]  (example)

ModelConfig

Pml\Inference\ModelConfig is a pure value object describing the model architecture. Passed to InferenceSession::load(). Pre-built configs are available as static factory methods.

Pre-built configs

Constructor properties

Generation Examples

Streaming generation

use Pml\Inference\{InferenceSession, Tokenizer, ModelConfig};

$tok     = Tokenizer::fromJson('models/smollm2/tokenizer.json');
$session = InferenceSession::load('models/smollm2/model.safetensors', ModelConfig::smollm2_135m());

$prompt  = "def fibonacci(n):\n";
$ids     = $tok->encode($prompt, addBos: true);

// Stream token by token
foreach ($session->generate($ids, maxNewTokens: 200, temperature: 0.7, topP: 0.9) as $tokenId) {
    if ($tokenId === $tok->eosId()) break;
    echo $tok->decode([$tokenId]);
    flush();
}

Chat interface

$response = $session->chat([
    ['role' => 'system',    'content' => 'You are a helpful assistant.'],
    ['role' => 'user',      'content' => 'What is gradient descent?'],
], maxNewTokens: 256, temperature: 0.8);

echo $response;

INT8-quantized LLM

// Load weights and quantize all Dense layers to INT8
$session = InferenceSession::load('models/llama3-8b.safetensors', ModelConfig::llama3_8b());
// $session internally uses Sequential — call quantize() on its model:
$session->getModel()->quantize(groupSize: 32);
// fp32 weights freed immediately → ~7 GB instead of ~28 GB

foreach ($session->generate($ids, maxNewTokens: 500) as $id) {
    echo $tok->decode([$id]);
}

Performance Notes