Single-host CUDA profiles add werk gpus (JSON inventory),
werk deployment-plan <profiles.json> [--inventory <fixture.json>] (dry-run), and
werk --deployments <profiles.json> serve. See
deployment profiles for configuration,
native-build selection, replicas, budgets and compatibility limits.
This page describes the public command groups and their operational semantics.
The reference for the canonical Linux build (--no-default-features, no optional
backend features) also includes:
The command and schema catalogues are generated from a fresh build of the current checkout. Other platforms and feature builds can expose a different parser; see conditional build options. For an installed binary, inspect its exact flags and accepted values:
werk --help
werk COMMAND --help
Examples use MODEL for an installed model ID and PATH for a filesystem path.
Angle brackets in usage lines denote required arguments; square brackets denote
optional ones. A -- separator ends option parsing, for example
werk run model-id -- "--explain this text". Shell quoting is required for
whitespace and structured JSON. Task/layout names accept hyphens or underscores;
JSON serialization uses underscores.
Compatibility spellings remain accepted:
| Spelling | Canonical form / scope |
|---|---|
werk rm, werk delete |
werk remove |
werk auth hf |
werk auth huggingface |
werk auth api-keys |
werk auth api-key |
serve --verbose-lite |
serve --verbose-pure |
run/chat --single-turn |
--no-history |
bench --tokens |
--max-tokens |
Media --output-path |
--output |
Image/video --cfg |
--guidance |
Music/sound --tempo |
--bpm |
werk audio speak |
werk audio generate speech; hidden compatibility command with the same options. |
werk audio separate |
werk audio transform separate; hidden compatibility command with the same options. |
werk audio generate MODEL |
Legacy model/task inference using the music/sound options; see task selection. |
--help / -h and --version / -V report help and version without executing
inference. werk help COMMAND … is equivalent to command help. Numeric ranges,
required option combinations and accepted enum values are checked before
execution; command errors return a nonzero exit status. Output-format switches
are command-specific: --json is not a global option. Ordinary operational logs
go to stderr; structured command results go to stdout.
The checked-in catalogues use Linux, --no-default-features, no optional
backend features. The CLI Reference PR workflow builds this profile from
the checked-out revision in target/cli-reference-neutral, then runs --check.
Changes to sources, schemas, Cargo configuration, the generator, tests or reference
trigger it. Drift fails the check, prints a diff and a regeneration command, and
does not rewrite files.
The GitHub Release workflow builds the same profile after preparing the new
product version, regenerates and checks the catalogues before its release PR,
and checks again before tagging. Existing-tag retries only check; they never
regenerate tagged content. Generated release commits use [skip ci]; the inline
Release checks cover reference validation without requiring a duplicate PR run.
Existing repository rules still apply.
To regenerate or check locally on Linux:
cargo +stable build --locked --bin werk --no-default-features --target-dir target/cli-reference-neutral
python3 scripts/generate-cli-reference.py target/cli-reference-neutral/debug/werk
python3 scripts/generate-cli-reference.py target/cli-reference-neutral/debug/werk --check
Use this fresh build even if another binary reports the same version: a version
match does not establish source revision or build features. The generator rejects
version mismatches and the known noncanonical burn parser value; these guards
are not a universal binary provenance check. Plain Cargo builds and aliases do
not invoke the generator.
The script recursively discovers public command help, including nested runtime
state actions, and queries all canonical task schemas in a temporary empty store
with automatic backend installation disabled. It strips environment defaults and
does not perform inference, download models or install backends. Python 3.9+ is
required. Review the prose here and in cli-media.md when behavior, aliases,
conflicts or support boundaries change: help/schema comparison alone cannot
prove that prose is semantically current or that inference works.
scripts/package-release.sh and scripts/build-windows.ps1 validate generation
after a successful build using --profile discovery and a temporary output
directory, then discard that output. Windows packaging through
scripts/package-release.ps1 uses the same hook. Generation failures stop
packaging. These runs cannot overwrite the canonical catalogues. With
--target TRIPLE, native builds inspect their new artifact; cross-builds compile a neutral
host discovery tool in target/cli-reference-host and use the exact Cargo-reported
executable. That host parser/schema snapshot does not validate foreign-target
inference or optional backend execution.
--backend burn is accepted only when compiled with burn-experimental. The
canonical neutral tables intentionally omit this value. Parser acceptance does
not enable the separate burn-runtime implementation or promise model/hardware
support. The PR workflow runs the existing parser test in both configurations,
using separate build directories and no models or inference. As in the existing
CI test setup, the large CLI test needs a 16 MiB thread stack:
RUST_MIN_STACK=16777216 cargo +stable test --locked --no-default-features --target-dir target/cli-reference-neutral --lib cli::tests::parses_cli_commands -- --exact
RUST_MIN_STACK=16777216 cargo +stable test --locked --no-default-features --features burn-experimental --target-dir target/cli-reference-burn --lib cli::tests::parses_cli_commands -- --exact
If generating discovery output from an optional-feature binary, use
--profile discovery --output-dir TEMP_DIRECTORY. Do not compare that output
with neutral pages or use it to replace the checked-in reference.
Interactive commands share the Werk logo palette: cyan for commands and ready
states, blue for table headers and fields, indigo for timing, violet for borders
and activity, and pink for titles, warnings and errors. Help, model/backend lists,
installation, downloads, diagnostics, media reports, chat and top use the same
theme. Normal commands use scrollback-friendly panels; top is the full-screen
dashboard. Narrow terminals use a compact banner and wrap diagnostic prose.
serve displays its actual bound address, default model, authentication status
and a werk top --url … command. Its request feed shows HTTP status and elapsed
response-body time, including streaming, in terminals and redirected logs.
Monitoring polls are available at TRACE. Request bodies and credentials are not
included in HTTP access events.
Set NO_COLOR=1 to remove colors while retaining layout. TERM=dumb and redirected
streams use plain output for ordinary commands. Interactive top instead requires
a TTY and uses raw mode and an alternate screen, even with TERM=dumb.
NO_COLOR changes its colors, not that mode; use werk top --once or
werk top --once --json for scripts. JSON/JSONL, inspect, protocol commands and temp path
remain unadorned even in a terminal. Streamed model content is not rewritten.
Backend installer output is streamed through the same presentation; failures
retain nonzero exit codes and their diagnostic details.
For a local regression check without model downloads or package installation:
cargo build
python3 scripts/smoke-console.py target/debug/werk
The default threshold is INFO. Log levels and presentation are independent:
| Level | Events |
|---|---|
error |
Failed inference, HTTP 5xx/stream errors, command failures. |
warn |
HTTP 4xx, abandoned requests, cancelled inference/jobs, warnings. |
info |
Server/process lifecycle, completed HTTP requests and inference, job state changes. |
debug |
Request/inference start and phase details, backend diagnostics. |
trace |
Also include successful monitoring polls. No per-token logging. |
off |
Disable operational events. |
serve --verbose defaults to DEBUG and keeps the colored terminal presentation
and detailed analysis statistics. serve --verbose-pure defaults to DEBUG
and writes raw JSONL to stderr, without a banner, panels or ANSI escapes.
--verbose-lite is an alias of --verbose-pure; the two verbose modes conflict.
run/chat and image/video/audio commands also default to DEBUG with their
supported --verbose or --debug flags, as does bench --debug. An explicit
--log-level (or WERK_LOG_LEVEL when no flag is given) wins over these defaults.
# Readable console plus an independent structured file
werk serve --model convaiinnovations/laya --verbose \
--log-file ./logs/werk.jsonl
# Raw events for collectors; stdout remains separate
werk serve --model convaiinnovations/laya --verbose-pure \
--log-file ./logs/werk.jsonl
# Production INFO JSONL without debug phases
werk serve --log-format json --log-level info
These global options also apply to other CLI commands:
| Option | Default / behavior |
|---|---|
--log-level error\|warn\|info\|debug\|trace\|off |
INFO; DEBUG for verbose serve modes, run/chat and media --verbose/--debug, or bench --debug. An explicit level wins. |
--log-format auto\|text\|json |
auto: terminal palette when interactive, plain text in pipes. text: undecorated text. json: raw JSONL. Pure mode forces JSON. |
--log-file PATH |
Optional append-only JSONL destination, independent of console rendering. No file is created unless configured. |
--log-max-size-mb N |
Rotate before exceeding 10 MiB; one oversized event remains intact. Range 1–4096. |
--log-retention N |
Keep 5 rotated files, PATH.1 newest through PATH.N oldest. Range 1–100. |
Command results (for example werk list --json) remain on stdout. Operational
logs use stderr. Interactive top suppresses console logs to protect its screen;
file logging remains available. A file path has one writer per process, protected
by a persistent PATH.lock sidecar; concurrent Werk processes need separate paths.
New log and lock files have mode 0600 on Unix. Rotation preserves complete JSONL
records. Appending does not change existing file permissions.
Each event has schema_version, UTC timestamp, level, stable event name,
message, pid, optional request_id, and typed fields. HTTP responses carry a
server-generated x-request-id; the same ID appears in request events, correlated
inference and background jobs. Werk and Anthropic request-ID headers use this ID
as well. inference_id connects completion events to top’s numeric request rows;
job_id connects later job state changes. Model names, runtime, device, precision,
cache hits, token/result counts and measured durations are recorded when available.
A missing measurement remains null rather than becoming an invented zero.
http.request.completed is emitted when the response body finishes; its
headers_seconds measures time to headers and duration_seconds includes the
body stream. Dropping an unfinished body produces http.request.cancelled; a
body error produces http.request.failed. This measures server-side body delivery,
not client receipt. Routes use templates, never raw paths, queries or headers.
Structured request/inference events exclude prompts, responses and tool payloads.
Known configured credentials and sensitive structured field names are redacted.
Free-form native/legacy diagnostics can include runtime-specific information;
review them before sharing logs. Native llama.cpp/vLLM output remains opt-in via
WERK_LLAMA_LOG / WERK_VLLM_LOG and is recorded at DEBUG when enabled.
File writes use a bounded asynchronous queue (4096 events). A slow/full disk never
creates an unbounded backlog: dropped records and write failures produce stderr
errors and Prometheus counters werk_log_records_dropped_total and
werk_log_write_errors_total. Shutdown waits up to three seconds for the queue.
This is operational logging, not a durable audit journal; SIGKILL or a crash can
lose queued records. Existing Prometheus metric names, Grafana queries and top
statistics are unchanged by the log level or format.
Validate without model downloads:
python3 scripts/smoke-logging.py target/debug/werk
werk setup MODEL checks the advertised model features and offers one dialog
for missing backends and components, including vision projectors. pull runs
this check after saving the main model; local run, chat, and configured
serve models run it before execution. Choose Set up all to install the listed
components, or Later to retain the current setup. Without a terminal, use
--auto-install-backends for explicit approval. --offline permits only
existing-asset setup, and --text-only excludes vision from this review.
See complete model setup.
Global options can appear before or after a subcommand; examples put them first:
werk --model-home /srv/werk \
--backend auto \
chat model-id
The most important global controls are:
| Option | Meaning |
|---|---|
--model-home PATH |
Select the complete Werk store; overrides WERK_HOME. |
--backend BACKEND |
Select auto or constrain routing to a named runtime/backend family. |
--device DEVICE |
Legacy Candle-only device override. Prefer --backend or typed media accelerator controls. |
--auto-install-backends |
Approve the complete model setup plan, including missing backends and model-specific feature components. |
--no-auto-install-backends |
Skip model setup review and prohibit managed provisioning. |
--ctx-size, --batch-size, --ubatch-size |
Advanced llama.cpp context and batching controls. |
--gpu-layers, --main-gpu |
Advanced llama.cpp accelerator placement controls. |
--kv-cache-type, --flash-attn, --kv-offload |
Advanced llama.cpp cache and attention controls. |
--threads, --threads-batch |
CPU helper-thread controls for generation and prompt evaluation. |
--parallel N |
llama-server active sequence slots; default 1, positive integer. Each slot receives --ctx-size tokens, so additional slots increase memory demand. |
--warmup-tokens N |
Native llama-server startup warmup control: 0 disables it (default); a positive value enables native warmup rather than specifying its token count. Separate from serve --preload. |
--fp4-kernel auto\|native\|marlin\|ggml |
NVFP4 CUDA kernel policy; default auto. See NVFP4 requirements and limits. |
Backend availability, explicit constraints and fallback semantics are covered in Backends.
import copies a local file or repository directory into the managed store:
werk import /absolute/path/to/model --name local-model
The original path is not used as mutable runtime state after a successful copy. Removing the managed model does not remove the original source.
Add --link to register existing files without copying them or creating an
operating-system symlink:
werk import /mnt/f/Werk1112/models/wan22-ti2v-5b --name wan22-ti2v-5b --link
The source can be a file, a repository directory, or an existing Werk model
directory containing manifest.json and files/. Existing model metadata is
retained. The active store holds the registration and optimized artifacts,
while model files stay at their external location. This supports local and
external models together; --model-home and WERK_HOME still select the
complete store. import without --link continues to copy files.
Import a collection containing several models with --all:
# Register every model from the RAID in the active store:
werk import /mnt/f/Werk1112/models --all --link
# Copy a collection into the active store instead:
werk import /path/to/model-collection --all
--all examines the collection’s immediate children for model directories and
supported model files, ignoring unrelated entries. Each repository remains one
model; its components and weight shards are not registered separately. The
collection directory must not itself be a model repository. Discovery is not
recursive.
Existing Werk model directories retain their model IDs and metadata, including
when copied. Other models use the directory name or file stem as their ID.
--name is required for a single model and cannot be combined with --all.
Before importing a collection, Werk checks for duplicate IDs, destination name
collisions and models already installed in the active store. A conflict stops
the operation before any models are imported; existing models are not
overwritten. Successful imports print each model followed by the total count.
werk pull organization/repository --name local-name
Select a variant from a multi-quant repository:
werk pull organization/model-GGUF \
--file model.Q4_K_M.gguf \
--name model-q4
For a split GGUF, select its first shard. Werk downloads every matching shard in the same directory, validates that the set is complete, and uses the first shard as the runtime entry point. Other quantization variants are excluded:
werk pull ggml-org/DeepSeek-V4-Flash-GGUF \
--file DeepSeek-V4-Flash-Q2_K_S-00001-of-00002.gguf \
--name deepseek-v4-flash-q2-k-s
Automatic GGUF selection also includes the complete shard set. A separate
--name lets multiple variants coexist; pulling into an existing model ID
does not replace that model.
Pull currently uses Git plus Git LFS. Gated repositories require accepted
upstream conditions and a token from werk auth huggingface login, an accepted
environment variable or the standard Hugging Face token cache.
werk list
werk list --task image-generation --layout diffusers
werk list --input-modality audio --output-modality text
werk list --family flux --json
werk inspect model-id
list shows model metadata, a managed, external or remote storage label,
and the absolute local path in PATH. For managed models, this is the directory
containing manifest.json and files/; for external bindings, it is the
external files’ source path. Hosted registrations use remote, with PATH
pointing to their local registration directory, not weights or a provider URL.
It supports metadata filters and sizes table columns to their contents. When the
table would exceed the terminal width, each model is shown as a labeled block with
its full name and path. list --json prints the stored/enriched manifests.
External bindings include a storage record with kind: "external" and their
absolute path. They remain listed if the external disk is unmounted; loading
them requires the recorded location to be accessible.
inspect prints the full manifest as JSON. Declared task support is not proof
that an installed runtime can execute the model.
For an installed repository with several model files:
werk select-file model-id model.Q4_K_M.gguf
Both a path relative to the model’s files directory and a files/... path
are accepted. Inspect the manifest to obtain exact filenames.
werk remove model-id
werk rm model-id
werk delete model-id
This removes the managed model directory and its local optimized artifacts. For external bindings, only the local registration and artifacts are removed; external model files are never deleted. Backend environments, unrelated models and original import sources are separate.
The directory layout and retention rules are documented in Models, manifests and the store.
For an installed, compatible Hugging Face NVFP4 checkpoint:
werk convert source-nvfp4 --to gguf --name converted-nvfp4 \
--python /path/to/converter-venv/bin/python
The model IDs and interpreter path are placeholders. --to accepts only gguf
(the default); --name is required and must identify a new destination.
--converter PATH selects a compatible convert_hf_to_gguf.py; otherwise Werk
discovers the installed NVFP4 profile’s converter. --python defaults to
python3, and --verbose shows conversion diagnostics. Provision the converter
and its dependencies first.
This explicit conversion preserves the source and validates the output before registering a separate model. It does not download missing files, install Python packages or overwrite an existing model. Only supported ModelOpt or compressed-tensors inputs with declared NVFP4 metadata qualify; it is not a general format converter. See checkpoint and projector requirements.
werk support exposes the evidence catalog, official hosted registrations and
model-specific provider/action commands:
werk support list
werk support register MODEL [--name NAME]
werk support register-action NAME --checkpoint PATH
werk support image --request PATH --output PATH
werk support speech --request PATH --output PATH
werk support action --request PATH --output PATH
Uppercase values are placeholders; brackets mark optional arguments.
list prints the support/evidence catalog as JSON. register accepts the
official hosted IDs mistral-large-4, flux-3-image and eleven_v4, with an
optional local alias via --name. It does not register arbitrary provider IDs.
Listing and registration need no provider credentials and do not establish
successful inference. See discovery and registration.
register-action links a complete local DROID package without copying weights.
Keep its config, checkpoint and saved manifest/encoder references available at
that path; registration alone does not validate execution. See the
DROID prerequisites and request contract.
| Command | Input and output |
|---|---|
support image |
Read the FLUX 3 Image JSON contract, write returned image bytes and print metadata. |
support speech |
Read the Eleven v4 dialogue JSON contract, write the completed audio file and print metadata. HTTP audio streaming does not make this CLI command stream audio incrementally. |
support action |
Read the DROID JSON contract and write prediction JSON with executed: false; no hardware is actuated. |
All three --output paths must not already exist; files are created without
overwriting. Hosted inference requires MISTRAL_API_KEY, BFL_API_KEY or
ELEVENLABS_API_KEY as appropriate, provider access and potentially paid usage.
The new Kolibri, Mistral Large 4, FLUX 3 Image, Eleven v4 and DROID paths remain
IMPLEMENTED_UNTESTED for actual inference. Consult the
model support matrix for exact mode-specific status
and JSON fields; these commands do not change that verification status.
Print or list the temporary directory for the active Werk store, preview its cleanup or purge it:
werk temp path
werk temp list
werk temp purge --dry-run
werk temp purge
These commands resolve the same active WERK_HOME as the rest of the CLI,
including a global --model-home PATH override. temp list prints each direct
child path in sorted order, including hidden entries, without creating or
changing the temporary directory. It does not recursively expand directories or
follow child symlinks. Purging removes every child of that store’s tmp
directory, including any concurrently active temporary work, but preserves the
tmp directory itself. The --dry-run form reports the planned cleanup without
changing the filesystem.
Models, artifacts, managed outputs, jobs, authentication data, backends, files at the store root and output paths outside the store are persistent boundaries and are not touched by temporary-file purging.
Inspect local persistence storage and remove one listed entry or all eligible
caches in the active WERK_HOME (or global --model-home PATH):
werk cache list
werk cache list --kind chat-kv
werk cache list --json
werk cache purge <CACHE-ID>
werk cache purge --all --dry-run
werk cache purge --all
werk cache purge --all --kind chat-kv
werk cache purge --all --include-history
Copy the exact ID from cache list; paths and wildcard selectors are not
accepted. purge requires either that ID or --all. Deletion takes effect
immediately unless --dry-run is supplied. Both subcommands support --json;
list JSON separates entries from blocked storage-provider errors so one
unavailable catalog does not hide the remaining caches.
| Kind | Storage beneath the Werk home | Cleanup scope |
|---|---|---|
chat-kv |
chat-sessions/<session-hash>.cache/ |
Rebuildable native KV caches for one persistent conversation, across its runtime namespaces. |
chat-history |
chat-sessions/<session-hash>.json |
Saved conversation messages; excluded from --all unless --include-history or --kind chat-history is supplied. An exact history ID also explicitly selects its deletion. |
omlx-worker |
backends/omlx/workers/<worker>/cache/ |
Inactive worker cache files only; settings, credentials and logs are retained. |
runtime-state |
runtime-state/v1/ |
Individual disk states from the managed runtime catalog; pinned or in-use states are protected. |
The inventory reports each entry’s kind, backend when known, size and protection
status. Active chats and workers are skipped; runtime-state cleanup is blocked
while a runtime holds the catalog open. An explicitly selected protected entry
returns an error; bulk cleanup reports skipped entries and continues. Unknown
or unsafe storage is protected. For legacy oMLX worker caches without a lifetime
lock, Werk can remove the known empty directory layout when its native
_boundary_snapshots process markers identify owners that have all exited.
This removes directories only: any file (including zero-byte files), live or
unverifiable owner, or unknown layout blocks the legacy cleanup. Existing
lifetime locks continue to protect active workers even when their caches are
empty. Legacy runtime-state catalogs without a usage lease
also remain protected: use werk runtime prune on their running server, or
initialize the lease through a state mutation in a current runtime and stop
that runtime before offline cleanup. Stop older Werk processes and their
workers before using cache cleanup; they predate the new runtime usage locks.
Chat KV and history entries share a session hash but can be removed separately.
Deleting KV data retains the conversation and triggers prompt recomputation on
the next run. Deleting history removes the saved messages. Lock files remain
to preserve process coordination. Model weights, original SSD-backed MoE expert
tensors, installed backends, remote/runtime-owned RAM caches and authentication
data are outside this command’s scope. Temporary downloads remain managed by
werk temp; live named state operations remain available through werk runtime.
werk runtime is a quiet, pretty-JSON client for the versioned /werk/v1
control plane of an already-running server. Begin with discovery:
werk runtime info
werk runtime capabilities
werk runtime memory
For another trusted HTTP endpoint, put the connection options before the runtime subcommand:
export WERK_API_KEY="replace-with-generated-key"
werk runtime --url http://werk-host:11434 capabilities
werk runtime --timeout-seconds 75 info
--url defaults to http://127.0.0.1:11434, must contain an explicit port and
currently accepts only plain http:// with no path or embedded credentials.
--api-key overrides the WERK_API_KEY default, but the environment variable
avoids placing a key directly in shell history. The bundled client does not
follow redirects or provide TLS; use it only over loopback or a trusted private
hop. --timeout-seconds defaults to 30 and accepts 1 through 86400. The shared
request deadline starts before DNS resolution and bounds TCP connection attempts
and I/O. Synchronous DNS resolution itself can block past that deadline; after
it returns, the client checks the remaining time before connecting. It is not a
hard timeout on DNS. Like --url and --api-key, the option must appear before
the runtime subcommand.
The active store belongs to the server. A client-side global --model-home
does not redirect remote runtime state; start the server with the intended
WERK_HOME/--model-home instead.
For a server using deployment profiles, select an exact instance with
werk runtime --deployment INSTANCE_ID capabilities. Replica aliases are not
runtime-control targets; see deployment profiles.
werk runtime states
werk runtime states --model my-model --tier disk --limit 50
werk runtime states --cursor OPAQUE_CURSOR
Filters are optional. Tier is vram, ram, disk or external; limit is 1
through 100. The JSON result contains opaque state IDs and an optional next
cursor. It never prints a backend handle, snapshot path, prompt or API key.
Every mutation previews by default. Repeat it with --execute only after
checking the JSON response:
werk runtime state st_OPAQUE_ID pin
werk runtime state st_OPAQUE_ID pin --execute
werk runtime state st_OPAQUE_ID unpin --execute
werk runtime state st_OPAQUE_ID evict --execute
werk runtime state st_OPAQUE_ID promote ram --allow-experimental
werk runtime state st_OPAQUE_ID promote ram --allow-experimental --execute
werk runtime state st_OPAQUE_ID demote disk --allow-experimental --execute
Promotion targets are ram or vram; demotion targets are ram or disk.
The requested direction must be valid for the state’s current tier. Supplying
--allow-experimental acknowledges the backend capability status for that one
request; it does not enable experimental behavior globally.
Prune also defaults to preview and requires exactly one selector form.
purge is a visible alias with identical safety semantics:
# One or more exact IDs
werk runtime prune --id st_FIRST --id st_SECOND
werk runtime prune --id st_FIRST --execute
# A non-empty model/tier/time filter
werk runtime prune \
--model my-model \
--tier disk \
--older-than-unix-ms 1788444000000
# Every state visible to this authenticated principal
werk runtime prune --all --confirm-all
werk runtime prune --all --confirm-all --execute
--all is rejected without --confirm-all; --confirm-all is invalid for
the ID and filter forms. --execute changes dry_run from true to false.
The selector determines which live runtime states are explicitly deleted. Execute
also performs maintenance in the same principal namespace: it expires volatile
states, reconciles TTL/catalog remnants and can quarantine corrupt entries. This
can affect entries outside a narrow ID/model/time selector. Dry-run does not run
that additional maintenance, and matched/removed do not account for all of it.
Pruning cannot affect other principals and does not purge temporary files or remove
models, model artifacts, outputs, jobs, authentication data, backend installations
or external paths.
To clear every runtime state visible to the current authenticated principal, preview and then execute the explicit all-selector:
werk runtime purge --all --confirm-all
werk runtime purge --all --confirm-all --execute
This is the normal recovery path when persisted state is no longer useful. With multiple API keys, each key has a separate opaque namespace and can purge only its own states. Handoff values cannot be listed: they are intentionally short-lived, single-use secrets held only in server memory.
If the running process or its backend is too unhealthy to complete that
operation, stop werk serve first. As a local administrator, move the exact
active server store’s runtime-state/v1 directory to a separately named backup
and restart Werk. Moving it instead of deleting it keeps recovery possible;
Werk recreates an empty catalog. Do not move the surrounding store or
auth/runtime-namespace.key, and never do this while the server is running.
This offline recovery clears disk state for every principal in that store;
models, artifacts, outputs, jobs, credentials, backends and tmp are siblings
and remain untouched.
The CLI currently exposes info, capabilities, memory and state maintenance. Prefill/decode and expert contracts are HTTP/SDK surfaces; the ComfyUI package provides typed prefill/decode nodes plus capability-gated expert telemetry and dry-run-first expert-control nodes. The nodes do not imply production backend support. See the Werk Protocol 1.0 reference, the runtime architecture and capability matrix, and the ComfyUI custom-node guide.
Hugging Face credentials:
werk auth huggingface login
werk auth huggingface status
werk auth huggingface logout
login --token TOKEN supplies the token noninteractively; omitting it prompts
for a token. auth hf is the short command alias. Token discovery order and
storage are described in environment variables.
Generate an API key file for werk serve:
werk auth api-key generate
werk auth api-key generate --name comfyui --path /tmp/comfyui-key.toml
One API-key file can contain multiple [[keys]] entries. Generation does not
append to an existing file; merge a newly generated block deliberately.
--force overwrites the complete target file and must not be used as an append
operation.
estimate has two intentionally different modes.
Model-fit estimation does not require a canonical task:
werk estimate model-id
werk estimate organization/repository --file model.Q4_K_M.gguf --verbose
werk estimate model-id --json
For an installed model it accounts for selected weights, runtime overhead and a text KV-cache estimate. A repository-looking ID can be estimated remotely from Hugging Face metadata without downloading the weights.
Workload estimation includes a canonical typed task and runs parameter resolution first:
werk estimate flux-dev --task image-generation \
--width 1024 --height 1024 --steps 28
werk estimate wan-i2v --task image-to-video \
--width 832 --height 480 --frames 81
Task estimates require an installed model. They report estimated accelerator, host and output demand, fit status, confidence, assumptions, warnings and recommendations. An estimate is a planning aid, not a guarantee that a third-party backend can load the architecture.
Workload geometry options are --width, --height, --frames, --duration
(seconds), --batch-size, --sample-rate (Hz), --channels and --steps.
They populate the selected task’s namespace; they are not all valid for every
task. For audio, for example:
werk estimate music-model --task music-generation \
--duration 30 --sample-rate 44100 --channels 2 --json
werk parameters --task image-generation
werk parameters flux-dev --backend auto --json
werk parameters flux-dev --example
werk parameters flux-dev --sources
werk doctor
werk doctor --task image-generation
werk doctor --model flux-dev
werk doctor --runtime RUNTIME
werk doctor perf model-id
werk backend doctor --debug
parameters describes the typed schema and, with a model, model/runtime
support. doctor checks the host and can add a non-executing model/task probe.
Neither command performs a full cold model load.
Replace RUNTIME with the runtime name to inspect. doctor perf MODEL reports
the selected backend and available llama.cpp runtime/performance configuration;
it does not run a throughput benchmark. doctor has no --debug flag; use
backend doctor --debug for detailed discovery or chat MODEL --debug for
inference diagnostics.
werk artifacts build model-id
werk artifacts list model-id
werk artifacts rebuild model-id
Artifacts are runtime-specific derivatives stored separately from source model files. An explicit ONNX route may attempt an artifact build when no usable ONNX artifact exists; normal automatic safetensors routing does not require ONNX.
Start an interactive chat:
werk chat model-id --max-tokens 128
--max-tokens defaults to 256; it is a hard completion cap and can stop text
mid-sentence. Terminal chat streams decoded pieces by default. Use
--stream-granularity chunk to reduce terminal flushes and --verbose for
prompt/decode timing and throughput.
--chat-template model|generic|phi3|llama3|gemma|chatml|qwen-chatml|none
overrides prompt formatting on supported local routes. Model-native formatting
is preferred for GGUF/native-server and external tokenizer-based backends;
none requests untemplated text. The backend can reject an incompatible
override. Tool requests through run require the model template, and
run --server rejects local template overrides because the server owns them.
Inside interactive chat, /exit and /quit end the conversation.
Text embeddings, reranking and classification use dedicated HTTP endpoints.
Their CLI lifecycle is pull/import, inspect, backend install text-analysis,
doctor --model … and serve. There is no dedicated text-analysis inference
subcommand; chat cannot turn a non-generative analysis model into a chat model.
See text-analysis endpoints and the
task table.
For oMLX run and chat, the first completed turn includes worker/model
preparation in load duration and total duration, including persistent-session
startup. Later turns do not count that startup again. Prompt/decode timings and
first-token latency describe the request to the prepared backend.
runrun shares chat’s conversation/session implementation and the media inference
service used by serve. It exits after one completed response:
werk run model-id "Explain Rust ownership" --max-tokens 256
werk run model-id "Remember: my project is called Atlas" --session project
werk run model-id "What is my project called?" --session project --stream
werk chat model-id --session project
A model/session pair uses the same archive in run and chat. Backend, device,
llama.cpp tuning, sampling, templates, image inputs, verbose diagnostics and
supported native KV persistence use the existing chat implementation. Ordinary
output is buffered; --stream streams text, and --stream-granularity token|chunk
also enables streaming. --no-history (alias --single-turn) conflicts with
persistence, as it does in chat.
Werk-managed llama-server workers disable native synthetic warmup by default
when the runtime exposes --no-warmup. This is shared by all models and CPU/GPU
modes, including workers used by run, chat and serve. --warmup-tokens 0
keeps it disabled; a positive value explicitly enables native warmup via
--warmup when supported (the server uses a boolean, not an exact token count).
Trailing WERK_LLAMA_ARGS retain native argument precedence. Verbose local
session output reports the effective warmup setting; unsupported controls are
reported as the runtime default. Local llama.cpp run/chat never evaluate
a separate test prompt for session persistence. A new session runs the user
request and saves its state. Existing snapshots are checked on restore, and
successful real requests report observed reuse from backend cache counts.
Verbose output distinguishes unverified restore from historical observed reuse.
Worker loading still occurs on every local process start; cold file reads may
remain expensive, including during the first real prefill.
Runtime inspection avoids repeated help/version commands. Session library and
snapshot file work runs after native startup, avoiding additional I/O during
model loading. Verbose output separates runtime inspection, native readiness
and final binary validation. Compatibility checks and native slot restore still
complete before user inference.
On Linux CUDA workers, WERK_LLAMA_PREFETCH=auto (the default) optionally
prepares the explicitly CPU-placed MoE expert pages before native startup.
It recognizes --cpu-moe and --n-cpu-moe N, inventories only the selected
GGUF shard family, avoids rereading cached pages, and uses at most two bounded readers.
It does not intentionally traverse dense weights or lazy PLE/ngram tables.
Unsupported metadata/kernel support, ambiguous placement, explicit alternative
load modes and insufficient host/container memory budgets retain native loading.
Set WERK_LLAMA_PREFETCH=off to compare against native loading alone. This
preparation does not change native arguments, expert placement, or KV-cache
identity. Its time and checked/missing/prefaulted byte counts appear in verbose
diagnostics and are included in preparation and total wall time. Read-only
shared mappings retain the prepared page-table entries during the worker’s
lifetime, so cache-only trimming cannot immediately undo that preparation.
This adds no weight copy or memory lock; memory pressure can still reclaim pages.
The mappings are removed before file-cache release on normal or interrupted
shutdown. Verbose first requests log the prepared worker PID and startup
diagnostics before entering native inference.
On WSL, WERK_MODEL_CACHE_RELEASE=auto (the default) requests release of the
loaded model files’ clean OS cache pages when their local model owner or worker
is destroyed on backend paths that acquire a CacheReleaseGuard file lease.
Those paths can release pages after run, after leaving chat, or when serve
stops or unloads the worker. Subprocesses participating in ManagedChild cleanup
are stopped and reaped first, including CLI interruption. These guarantees do not
cover every local provider: the DROID worker uses a normal Child with kill/wait
on drop, without a CacheReleaseGuard lease or central ManagedChild interrupt
registration. An active chat/server worker retains its model; this is not an
idle timer between requests.
This file-scoped operation does not delete model files, chat history or saved
KV snapshots, and does not restart WSL or clear the system-wide cache. It can
make the next standalone run slower because model weights need reading again.
Set WERK_MODEL_CACHE_RELEASE=off to retain those pages, or on to request the
same release on other Linux hosts. Other operating systems keep their native
cache behavior. Release is advisory: diagnostics report the file ranges for
which advice was accepted, not measured bytes returned to the host. Concurrent
Werk readers with an acquired lease protect their files; unavailable leases
disable scoped cleanup for that owner. Remote servers are externally managed.
To reuse a worker across separate text/vision/tool run processes, start the
existing server once and point the client at it:
# Terminal 1: worker settings belong to serve.
WERK_LLAMA_ARGS='--n-cpu-moe 38 --reasoning off' \
werk --backend cuda --threads 24 --threads-batch 24 --ctx-size 4096 \
serve --model vumpt/Qwen3.8-Flash-Next-GGUF --persistence \
--host 127.0.0.1 --port 11434 --allow-unauthenticated
# Terminal 2: reuse the running worker; each invocation exits normally.
werk run vumpt/Qwen3.8-Flash-Next-GGUF "Explain Rust ownership briefly." \
--server http://127.0.0.1:11434 --session qwen-test --persistence \
--stream --verbose --max-tokens 128
This Qwen example uses 38 CPU expert layers out of the installed GGUF’s 48
blocks. It retains the measured 24-thread configuration; it is not a claim of
optimal settings for every machine or quantization. --n-cpu-moe is static CPU
expert placement, not an adaptive expert cache.
--server uses the existing /v1/chat/completions API, including images, tools,
sampling and request-scoped werk.omlx controls. It starts no local worker and
never silently falls back to local inference. The model must be registered in
both client and server stores under the same ID. --model-home on the client
selects its model metadata and conversation archive; it does not redirect the
server store. CLI --image files are read on the client and sent as data URLs.
Paths inside JSON requests refer to the server filesystem; use data URLs for
separate hosts. Media run requests continue to use the local media
service; combining them with --server is rejected.
Set backend/device/context/thread options and WERK_LLAMA_ARGS on the server.
Worker CLI options on a remote run are rejected. For authenticated servers,
provide --api-key or WERK_API_KEY to the client. The example explicitly binds
an unauthenticated development server to loopback; HTTP URLs require host and
port. There is no automatic server discovery or background daemon.
Sessions still save the transcript locally. Prefix reuse now depends on the
server’s live cache: restarting the server or another request replacing that
prefix may require prefill again. This route does not upload or restore local
KV snapshots. Local run without --server retains native disk snapshots where
supported. Neither transcript persistence nor live cache hits imply persistent
MoE expert residency.
Verbose text diagnostics now include preparation duration (worker preparation
and optional capability probe), plus llama.cpp worker startup, cache probe,
restore and save durations. Preparation is included once in load duration
and total duration. first token continues to measure the generation request;
run first token including preparation measures from entry to run until the
first nonempty text/tool delta arrives. With --stream, the first text delta is
flushed immediately. run elapsed through completion also includes snapshot
saving; final transcript publication and process exit follow it. Interactive
chat waiting for input is excluded. Native prefill/decode rates are unchanged.
CLI JSON completions also include timings and backend_diagnostics; the HTTP
API includes native values under werk.timings and werk.backend_diagnostics.
An available native cache count appears as usage.prompt_tokens_details.cached_tokens;
unknown counts remain absent.
For structured conversations and tool calls, pass the existing OpenAI chat
request schema. --request - reads JSON from stdin:
werk run model-id --request request.json --json --session tools
{
"messages": [{"role": "user", "content": "What is the weather in Berlin?"}],
"tools": [{
"type": "function",
"function": {
"name": "weather",
"parameters": {
"type": "object",
"properties": {"city": {"type": "string"}},
"required": ["city"]
}
}
}],
"tool_choice": "auto",
"max_completion_tokens": 256
}
This accepts messages (including image parts and tool results), tools,
tool_choice, parallel_tool_calls, stop strings, sampling, streaming, and
supported werk.omlx controls, as in /v1/chat/completions. The positional model
selects the installed model; an optional JSON model must match its canonical
ID. Explicit CLI sampling options override JSON values. Without an explicit
limit, the JSON limit wins, otherwise the default is 256 tokens.
Tool calls are returned to the caller, as with the HTTP API; Werk does not execute
the declared functions. With a persistent session, the next invocation can send
only new tool-result messages with the corresponding tool_call_id. Do not
resubmit the saved transcript, since new messages are appended to that session.
--json prints a completion object containing model, message, finish_reason
and usage. Combined with streaming it emits newline-delimited text_delta and
tool_call_delta events followed by that completion object. These are CLI JSON
events, not HTTP SSE. Diagnostics go to stderr, with no startup banner on stdout.
For media, run accepts all canonical inference tasks and parameters supported
by the existing inference service and selected model/runtime:
werk run image-model "A mountain lake" --task image-generation \
--set image.width=1024 --set image.height=1024 --set image.steps=20 \
--output lake.png
werk run speech-model "Hello from Werk" --task text-to-speech --output hello.wav
werk run whisper-model --task speech-to-text --input audio=recording.wav --json
werk run video-model "Clouds drifting across mountains" --task video-generation \
--output clouds.mp4
werk run model-id --request media-request.json --json
--input MODALITY[:ROLE]=PATH_OR_URL is repeatable; use roles such as
image:mask_image, image:initial_image, or audio:reference_audio for conditioned
tasks. --set PATH=VALUE accepts canonical parameters and routing.* overrides
(e.g. routing.precision=float16). A media request file uses the existing
InferenceRequest schema:
{
"task": "image_generation",
"prompt": "A mountain lake",
"parameters": {"image.width": 1024, "image.height": 1024, "image.steps": 20}
}
If no task is supplied, text/vision models use conversation inference; a model
with exactly one media task uses that task. Ambiguous media models require
--task. werk parameters MODEL --task TASK lists applicable parameters.
JSON task names use snake_case (image_generation); CLI names use hyphens
(image-generation), following the existing schemas.
Media outputs use the existing output store and CLI publication rules; --json
returns the canonical inference result. Conversation persistence and text
streaming flags do not apply to media tasks and produce an explicit error.
Server transport settings and named Werk Protocol state management remain on
serve and werk runtime; run does not start an HTTP listener.
--persistence saves completed conversation turns and resumes them on the
next invocation of run or chat. It is available for every chat backend, including automatic
routing, and preserves the normal streaming path:
werk chat model-id --persistence
werk --backend mlx chat model-id --persistence --session project
werk --backend omlx chat model-id --session project --verbose
The model ID and session name identify the conversation; changing the backend
does not create a different transcript. The default session name is default.
--session NAME implies persistence. --persistence-reuse prefer resumes if
the conversation exists (the default), required fails if it does not exist,
and disabled starts a fresh conversation that replaces the previous archive
only after a completed answer. The reuse option also implies persistence.
These options conflict with --no-history and its --single-turn alias.
The reuse policy controls the saved conversation and the default prefix-cache
settings for managed vLLM/oMLX, as in serve. Explicit native runtime cache
settings still take precedence; this flag does not purge existing caches.
Private conversation archives live under $WERK_HOME/chat-sessions/, or the
equivalent default Werk home. Only one process may open a given model/session
at once. Writes are atomic; failed or interrupted streams do not replace the
last completed conversation. Context-window trimming affects the model input
and preserves the full saved transcript. Archives are limited to 4096 messages
and 16 MiB; corrupt or oversized archives fail explicitly. Image parts remain
in the saved conversation and require an image-capable backend when resumed.
Conversation history is portable text/message data, not a model or KV snapshot.
Werk reports whether the selected route also enables persistent native KV
caching. Unsupported native caches leave conversation persistence operational
and the backend recomputes the prompt. Local vLLM and oMLX receive the same
prefix-cache defaults as serve --persistence; explicit runtime
arguments still win. This cache remains vLLM-owned and does not survive its
process restart. Exiting run or chat still stops its owned backend workers.
The existing llama.cpp route also saves native slot snapshots for persistent
text chats on Unix when the server exposes the required private slot operations.
It restores a compatible snapshot on the next process start and reports
actual prefix hits as prompt cached count in --verbose output. Snapshots are
namespaced by model files, runtime executable/libraries, native environment and
effective arguments; incompatible or corrupt snapshots are rebuilt from the
conversation. Each runtime namespace retains its latest completed snapshot,
with a 2 GiB snapshot limit. This does not persist loaded model weights or add
cross-restart named Werk Protocol states.
For the optional CUDA expert-cache runtime and an offload test command, see CUDA expert offload.
Archives and supported native KV caches survive exit. Use
werk cache list and werk cache purge to inspect
or remove them; deleting just the KV entry preserves the conversation.
run and chat also accept --persistence-mode disk|auto|memory|ephemeral
and --persistence-ttl-seconds N (1–2592000). Disk/auto keeps the conversation
across process restarts; memory/ephemeral does not save a transcript or native
snapshot to disk. TTL expires the saved conversation. These options imply
persistence. --persistence-pin remains a server option for named Werk Protocol
states; terminal conversations do not create /werk/v1/prefill state handles.
Attach one or more images to a compatible vision-language model with repeatable
--image values:
werk --backend auto run vision-model \
"Inspect this render for clipped text and alignment defects." \
--image /absolute/path/to/render.png \
--max-tokens 512 --debug
werk --backend auto chat vision-model \
--image /absolute/path/to/render.png \
--no-history --debug
The model manifest must advertise image understanding and an image-capable runtime must pass its probe. For GGUF, the llama.cpp server path additionally requires a manifest-listed multimodal projector. See Vision and visual quality assurance.
Benchmark an installed model with a fixed prompt and sampling settings:
werk --backend omlx bench model-id \
--prompt "Write one English sentence about Rust." \
--max-tokens 64 --runs 3 --warmups 1 \
--temperature 0 --top-p 0.95 --seed 42 \
--json --include-output --debug
--runs defaults to 5 measured runs after 1 unmeasured --warmups run.
--max-tokens defaults to 256. Temperature defaults to 0 and seed to 42;
omitted --top-p inherits the backend’s behavior. A warmup does not guarantee that every later request
reuses its prompt cache. Keep runtime settings and system load consistent
between comparisons.
--compare defaults to none; --compare legacy adds the legacy llama backend
for GGUF runs using automatic or llama-server selection, subject to compiled
backend availability. Other routes keep their selected backend. Use
--print-native-info for available resolved llama.cpp settings in the text report.
JSON samples contain token counts, timings, finish_reason and
backend_diagnostics. Add --include-output with --json to retain generated
text in each sample’s output field for quality review. Without that flag,
the field is omitted. Inspect finish_reason: "length" for truncation before
comparing answer quality or total duration. --debug requests additional
backend diagnostics, including available oMLX expert counters.
The benchmark sends the original structured user message through the selected
backend’s prompt handling, as run and chat do. Runtime-owned chat templates
therefore receive the original message instead of an already formatted prompt.
For HTTP streaming and multi-turn conversations, use the separate HTTP chat benchmark. It records time to first visible text, final API usage when supplied, answers and declared quality checks through the same endpoint used by Open WebUI and werkStation.
Top-level media command groups are:
werk image generate|edit|upscale
werk video generate|animate|transform|upscale
werk audio generate|transcribe|translate|detect|analyze|transform|embed
The model is explicit. Typed commands share task schemas, resolution, estimation, planning, output publication and diagnostic behavior with the HTTP service, but do not invoke the HTTP routes internally.
Prompt-capable commands resolve text from an explicit value, a text file,
piped standard input and then an interactive prompt where supported. Generated
outputs go to the requested --output destination or Werk’s managed output
store. Prompt text is not included in automatic filenames.
See media CLI rules for task selection, input precedence, structured lists, profiles, routing and output contracts. Every option is listed in the image, video and audio catalogues. Media inference provides execution support and runnable examples.
werk serve \
--host 127.0.0.1 --port 11434 \
--model chat-model \
--image-model image-model
The default address is 127.0.0.1:11434. Authentication is enabled by default
through --api-key, WERK_API_KEY, --api-keys or the default key file.
--allow-unauthenticated is intended only for deliberate local development.
Browser CORS is disabled by default. Add exact trusted origins with repeatable
--cors-origin; wildcard and opaque null origins are rejected.
--image-model selects the default for compatible image requests that omit a
model or use an OpenAI image alias. Startup checks its declared image-generation
support; it does not warm the image pipeline. See HTTP model selection.
Enable persistence defaults and supported backend prefix caching:
werk serve --model chat-model --persistence
--persistence supplies auto retention, prefer reuse, no TTL and no pinning
when a POST /werk/v1/prefill request omits its top-level policy member. It
independently supplies allow_experimental: true when the top-level
allow_experimental member is omitted, even if policy: {} is present. Each
explicit top-level value wins over its respective server default. For a local
vLLM process started by this server, it defaults vLLM’s native automatic
prefix cache on. Werk verifies that the installed vLLM help advertises the
generated flag before starting the process. A remote vLLM endpoint remains
externally managed and receives no generated launch argument.
For local oMLX 0.6.4 and 0.7.0, persistence mode auto or disk with reuse other
than disabled also enables verified short exact-prefix caching. This applies to
ordinary OpenAI chat and supported tool requests, including when auto
routing selects oMLX. The native SSD cache is limited to 4 GB in the private
worker’s cache/prefix-cache directory and lasts for that worker. It does not
save chat history or guarantee reuse after restart. Use chat --persistence for
durable CLI conversation history and its supported native KV cache.
For compatible oMLX expert offload, set the expert cache and thinking default when starting the service:
WERK_OMLX_THINKING=0 \
WERK_OMLX_EXPERT_CACHE_MB=8192 \
werk --backend omlx serve --model chat-model --persistence --verbose
The expert cache budget is in MiB and is separate from the native prompt/KV
cache. Expert execution defaults to grouped; set
WERK_OMLX_EXPERT_EXECUTION=serial before starting the service for a comparison.
Grouped execution batches tensor materialization, retains active expert groups
through GPU evaluation and reduces allocator clearing. These settings apply to
ordinary HTTP chat clients as well as CLI generation. See
experimental oMLX expert offload
for runtime and model compatibility requirements.
With serve --verbose, available expert diagnostics describe the whole
worker’s measurement interval. Overlapping requests can contribute to those
counters. Logical read bytes include reads served by the OS file cache and do
not establish physical SSD throughput; nested timing counters are not
independent phases to add together.
The defaults can be selected individually; any granular option implies
--persistence:
werk serve --model chat-model \
--persistence-mode disk \
--persistence-reuse prefer \
--persistence-ttl-seconds 3600 \
--persistence-pin
Persistence mode is ephemeral, memory, disk or auto; reuse is
disabled, prefer or required; TTL is 1 through 2592000 seconds. If the
request contains policy, that complete object wins, including protocol
defaults for fields omitted inside it. An explicitly supplied
allow_experimental value also wins, including false.
For a local vLLM launch, --persistence-reuse disabled defaults native prefix
caching off. An explicit --enable-prefix-caching or
--no-enable-prefix-caching in WERK_VLLM_ARGS wins over the generated
default. These backend-native cache entries remain opaque: they are not named
Werk state and cannot be listed, moved, persisted or pruned by Werk.
For oMLX, modes memory and ephemeral, or reuse disabled, leave the
additional exact-prefix helper off. TTL and pinning apply to named Prefill
state; required reuse does not make an ordinary OpenAI cache miss an error.
Existing native backend caching remains independent of this helper.
These flags do not redirect OpenAI-compatible /v1 or media
requests through Prefill, add semantic output caching, or enable cross-restart
restore. Exact model/pipeline residency is already automatic in supported
Werk-owned in-process and resident-worker paths. Current named state/prefill
support is experimental and limited to a functionally validated, Werk-managed
llama-server process for the exact installed GGUF model. The backend owns the
opaque runtime state; Werk owns its policy, lifecycle, accounting and
compatibility checks. Inspect werk runtime capabilities before relying on it.
The separate model-, pipeline-, and backend-owned reuse paths are listed in the
execution lifetime and reuse matrix.
The route inventory and request contracts are documented in the HTTP API reference.
--preload MODEL... accepts multiple values and is repeatable:
werk serve --preload model-a model-b model-c --preload model-d --model model-a
Replace the names with already installed compatible models, or configured
deployment aliases/instance IDs. Matching builds, runtimes and resources must
be available; authentication uses the ordinary serve configuration.
Werk processes targets sequentially in first-occurrence order and deduplicates them. The CLI imposes no fixed model count, but RAM, VRAM and runtime capacity still limit residency. Request handling begins after the preload run; the listener can already be bound while preparation is in progress.
--model independently selects the default for chat/text-analysis requests
that omit a model. Preload does not choose that default. With a nonempty preload
list, a default outside the list receives no separate startup preparation and
remains requestable on demand. Without --preload, existing startup behavior
is retained: outside deployment mode, Werk prepares the default model; for
text-analysis models this can be a runtime check with weights loaded on the
first request.
| Target | Preparation and residency boundary |
|---|---|
| Supported local text model | Internal synthetic warmup with at most 32 output tokens, without a public HTTP request or saved conversation. Compatible resident llama.cpp, oMLX, local vLLM, Candle, experimental Burn, Transformers compatibility and embedded ONNX GenAI routes participate, subject to model/build/runtime requirements. |
| Separately operated vLLM endpoint | Werk can retain its client configuration and send the synthetic warmup, but the external service owns weights and residency. This is not Werk-guaranteed remote residency. |
| Optional hosted provider | Check credential presence/header validity locally and prepare an HTTP client. No provider call, inference, billing or local weight residency; account access and remote readiness are not verified. These model paths remain IMPLEMENTED_UNTESTED. |
| FLUX 3 Action DROID | Load a separate resident policy worker and warm it with a synthetic prediction. Real model inference remains IMPLEMENTED_UNTESTED. See model-specific requirements. |
One-shot MLX/MLX-VLM and nonresident ONNX runners do not support resident preload. Explicitly disabled residency is respected. Preload does not generally warm image, video or audio generation.
Successful local instances are retained until server shutdown. Pinned runtime
entries sit outside ordinary cache capacity/eviction limits and cannot be
evicted by later requests. Earlier targets claim resources first. Missing
models, incompatible paths, resource shortages and warmup failures stop startup
at the first failed target with a nonzero exit status. Later targets are not
attempted. The CLI does not report “Server ready” or restart itself after a
setup/preload failure. Installed components remain available; resolve the reported
cause, then rerun the complete werk serve command with all original parameters.
Setup checks, backend installation, projector validation and model loading show
a spinner with the current phase and elapsed time in an interactive terminal.
Projector downloads also show bytes, transfer speed and ETA when the size is
known. At the normal log level, running phases update the spinner in place;
completion messages, warnings and errors remain visible. --verbose also prints
the running status events. Log files retain those events in either mode. Status
events are emitted every five seconds during long operations, including when
the backend produces no output. Redirected output and explicit
text/JSON logging use status events without terminal animation. The event names
are operation.started, operation.phase, operation.progress and
operation.finished; fields include phase, elapsed seconds and download bytes
when available. A spinner indicates activity, not a measured percentage of model
loading. Native llama.cpp startup also shows its PID and readiness timeout.
Native llama.cpp readiness allows 900 seconds by default for large-model cold
starts. Set WERK_LLAMA_HEALTH_TIMEOUT_SECONDS to a positive integer to change
this limit, for example 1200 for 20 minutes. An individual health probe cannot
wait longer than one second or the remaining startup budget. This limit covers
native model/projector loading and native warmup, not later inference. Runtime
termination and memory release can take additional time and have their own
visible phase. Increasing the timeout does not change GPU placement or make an
oversized configuration fit; explicit --gpu-layers settings remain unchanged.
If native common_fit_params reports a failed memory fit or an error during
fitting, Werk stops the worker promptly instead of allowing the subsequent load
to continue until timeout. The native reason is retained even if later log lines
overwrite the output tail. This also applies with WERK_LLAMA_LOG enabled.
Correct placement/context/parallel settings and rerun the complete command;
Werk does not silently reduce settings or restart. This check relies on native
fitting diagnostics, and is not a general RAM/VRAM limit or proof that a model
fits when a backend skips fitting or does not report a failure.
In deployment mode, an alias expands to all replicas in instance-ID order; individual IDs also work. Deduplication uses the expanded target instance IDs. Existing placement/admission rules still apply. This adds no multi-node cluster management or guarantee that all resident models decode simultaneously; see deployment profiles and inference concurrency.
Startup events are serve.preload.started, serve.preload.ready,
serve.preload.failed and serve.preload.finished. Hosted-client preparation
also emits serve.preload.remote_prepared. “Ready” means preparation succeeded
for that target type, not that hosted inference was verified. Residency,
conversation history, prefix/KV reuse and expert caches are
separate mechanisms.
Prompt processing, cache misses and dynamic expert offload can still add latency;
preload guarantees neither KV reuse nor a latency-free answer.
werk backend list
werk backend doctor --debug
werk backend install TARGET
werk backend install omlx # Apple Silicon; explicit installation only
werk backend install mlx
werk backend install mlx-vlm
werk backend install transformers
werk backend install media
werk backend install onnx-cpu
TARGET is a placeholder for a supported installer name, such as llama-cpu.
The supported install targets, operating-system matrix and manual cleanup
procedure are documented in Backends. There is currently no
managed werk backend uninstall command.
werk top opens the responsive terminal dashboard. It connects to
http://127.0.0.1:11434 by default; --url selects another server, and
WERK_API_KEY or --api-key supplies its credential. Loopback targets also try
Werk’s standard local API-key file. The updated server exposes observability
without an additional serve flag. Older running servers need a planned restart
with the updated binary; opening the dashboard never restarts them.
| Option | Meaning |
|---|---|
--interval-ms N |
Poll interval, 500–60000 ms; default 2000 ms |
--once |
Print one snapshot and exit |
--once --json |
Structured snapshot for scripts |
--no-animation |
Disable decorative animation |
--demo |
Simulated preview without a server connection; conflicts with --once, --json and --api-key |
Keys: q/Escape/Ctrl-C quit; Space pauses the display; arrows select a request;
Tab/Enter toggles details; b selects a native worker; a toggles animation. The dashboard is read-only.
v opens the session-local field picker (Space toggles, Enter applies, Escape
cancels). e enables placement: Tab/Shift-Tab selects a field and arrows move it;
Enter/Escape finishes. c cycles columns, r toggles row/column order, s cycles
the request-list split, o reverses field order, and R restores the automatic
view. PageUp/PageDown scroll custom cards. The automatic view adapts to generation,
embedding, reranking or classification requests; unavailable metrics stay absent.
See observability for metrics, backend
coverage, Prometheus setup and the importable Grafana dashboard.