Docker Model Runner in Compose

DockerDMRDocker Compose

TL;DR: The models top-level element requires Docker Compose 2.38.0 or later, the provider service type requires 2.36.0 or later, and Compose Bridge requires Docker Desktop 4.43.0 or later. From a container on Docker Desktop, reach the host model runner at http://model-runner.docker.internal/engines/v1; on Docker Engine use http://172.17.0.1:12434/engines/v1. The Base URL that the official API reference lists for containers is http://model-runner.docker.internal with no /engines/v1, and pasting that value straight into the OpenAI SDK produces /chat/completions instead of /engines/v1/chat/completions. Short-syntax variable names can collide, so always use the long syntax with endpoint_var and model_var.

Docker Model Runner (DMR) pulls OCI artifacts such as GGUF files and serves them behind an OpenAI-compatible endpoint. That part is easy. What actually blocks people is never "how do I run the model", it is "the model runs on the host, my app runs in a container, which address do I write". Docker's documentation answers that question in four places, and the four answers do not fully agree. So I ran every one of them through the real SDKs on my own machine and captured the URL each one builds. The result is clearer than the docs.

Why declare models in compose.yaml at all

Once you split an AI application into frontend, backend, database, and model, the model is just another external dependency — no different in kind from the database. The old approach was to run docker model run ai/smollm2 by hand before docker compose up, or to hardcode http://localhost:12434 in the backend startup script. Both share one flaw: the model dependency lives outside version control, so the stack breaks the moment you move to another machine.

Compose promotes the model to a top-level element instead. As the official page at docs.docker.com/ai/compose/models-and-compose/ states, you declare models with the models top-level element, the platform provisions and exposes them, and connection details are injected into your service containers as environment variables. Docker Compose is only one implementation of the spec; a cloud platform can consume the same declaration through x-* extension attributes.

So the payoff is not "one fewer command to type". It is that your compose.yaml finally describes the whole dependency graph, and you can delete the README line that says "remember to start the model first".

The models element: short syntax and long syntax

The minimal shape:

services:
  chat-app:
    image: my-chat-app
    models:
      - llm

models:
  llm:
    model: ai/smollm2

models at the top level sits beside services. The models key *inside* a service is a reference: "this service uses these models". Docker defines three attributes: model (required, the OCI artifact identifier), context_size (maximum context in tokens), and runtime_flags (raw flags handed to the inference engine). Platform-specific options go through x-* extension attributes and are not validated by the Compose specification.

Long syntax renames the injected variables:

services:
  app:
    image: my-app
    models:
      llm:
        endpoint_var: AI_MODEL_URL
        model_var: AI_MODEL_NAME
      embedding-model:
        endpoint_var: EMBEDDING_URL
        model_var: EMBEDDING_NAME

models:
  llm:
    model: ai/smollm2
    context_size: 4096
  embedding-model:
    model: ai/all-minilm

One easily missed note in the official docs concerns context_size: every model has its own ceiling, and when raising it you should weigh your hardware constraints and keep the context "as small as feasible for your specific needs". That is not a tuning tip — exceed it and the model will not start.

How the injected variable names are derived

With short syntax the platform builds two variables per model: {MODEL_NAME_UPPERCASED}_URL and {MODEL_NAME_UPPERCASED}_MODEL, with hyphens converted to underscores. The official example gives llm producing LLM_URL and LLM_MODEL, and embedding-model producing EMBEDDING_MODEL_URL and EMBEDDING_MODEL_MODEL.

That last one looks like a typo, but it follows from the rule. The real problem shows up in the table below. I wrote a short script that applies the documented rule to common names:

def short_vars(key: str):
    p = key.upper().replace("-", "_")
    return p + "_URL", p + "_MODEL"

for k in ["llm", "embedding-model", "chat", "model"]:
    print(k, "->", short_vars(k))

for a, b in [("llm", "llm-model"), ("my-model", "my_model"), ("chat", "chat-model")]:
    print(a, "+", b, "intersection:", set(short_vars(a)) & set(short_vars(b)))

Actual output on my machine (Python 3.13.14):

llm -> ('LLM_URL', 'LLM_MODEL')
embedding-model -> ('EMBEDDING_MODEL_URL', 'EMBEDDING_MODEL_MODEL')
chat -> ('CHAT_URL', 'CHAT_MODEL')
model -> ('MODEL_URL', 'MODEL_MODEL')
llm + llm-model intersection: set()
my-model + my_model intersection: {'MY_MODEL_MODEL', 'MY_MODEL_URL'}
chat + chat-model intersection: set()

my-model and my_model collapse into the identical pair of variable names. One silently overwrites the other, nothing raises, and you just notice that one model keeps answering for the other. This is not a hypothetical I invented: uppercasing plus hyphen-to-underscore normalisation necessarily produces it.

The takeaway: if a compose file declares two or more models, use the long syntax and name endpoint_var and model_var explicitly. Four extra lines remove an entire class of unexplainable behaviour.

Which address should a container use: four official answers

This is where most people get stuck, because four pages give four values and none of them says the others are wrong.

Source page	Scenario	Address given
`compose/bridge/use-model-runner`	Docker Desktop (Helm `modelRunner.enabled: false`)	`http://host.docker.internal:12434/engines/v1/`
`compose/bridge/usage`	Docker Desktop / Kubernetes	`http://host.docker.internal:12434/engines/v1/` and `http://model-runner/engines/v1/`
`ai/model-runner/api-reference`	Docker Desktop, from a container	`http://model-runner.docker.internal`
`ai/model-runner/api-reference`	Docker Engine, from a container	`http://172.17.0.1:12434`
`ai/model-runner/api-reference`	Host process (TCP)	`http://localhost:12434`

Look at row three: http://model-runner.docker.internal has no port and no /engines/v1. The same page's "Base URLs for third-party tools" table says the OpenAI SDK should use http://localhost:12434/engines/v1. Copy a value from one table into the SDK named by the other, and a path segment goes missing.

Docker does explain the port difference: the Service that Compose Bridge generates in Kubernetes exposes port 80, mapped to container port 12434. That is why http://model-runner/engines/v1/ has no port — it is the default, not a typo. The same reasoning applies to model-runner.docker.internal on Docker Desktop. The catch is that each explanation lives on its own page and is easy to lose when you read them side by side.

There is a subtler one. On Docker Engine (plain Linux, not Desktop) the docs state that 172.17.0.1 "may not be available by default to containers within a Compose project", and you must add extra_hosts yourself:

services:
  app:
    image: my-app
    extra_hosts:
      - "model-runner.docker.internal:host-gateway"

After that, the address the docs tell you to use is http://model-runner.docker.internal:12434/ — the same hostname, without a port on Desktop and with a port on Engine. I cannot reconcile that inconsistency on Docker's behalf, so I am presenting it as-is.

Measured locally: the URL the SDK actually sends

I have neither Docker Desktop nor a GPU on this machine, so I am not going to pretend I ran end-to-end inference. Instead I did something more useful: I put an httpx.MockTransport in front of the SDK and recorded the URL it builds. You can reproduce this anywhere — no model, no GPU.

Environment: Windows, Python 3.13.14, openai 3.31.0, httpx 0.28.1, anthropic 1.14.0 (versions as of publication).

import httpx
from openai import OpenAI

captured = {}
def handler(request: httpx.Request) -> httpx.Response:
    captured["url"] = str(request.url)
    return httpx.Response(200, json={"id": "x", "object": "chat.completion",
        "created": 0, "model": "ai/smollm2",
        "choices": [{"index": 0, "message": {"role": "assistant", "content": "ok"},
                     "finish_reason": "stop"}],
        "usage": {"prompt_tokens": 1, "completion_tokens": 1, "total_tokens": 2}})

for base in ["http://host.docker.internal:12434/engines/v1/",
             "http://host.docker.internal:12434/engines/v1",
             "http://model-runner.docker.internal",
             "http://model-runner.docker.internal/engines/v1",
             "http://172.17.0.1:12434/engines/v1/",
             "http://model-runner/engines/v1/",
             "http://localhost:12434/engines/v1/"]:
    c = OpenAI(api_key="not-needed", base_url=base,
               http_client=httpx.Client(transport=httpx.MockTransport(handler)))
    c.chat.completions.create(model="ai/smollm2",
                              messages=[{"role": "user", "content": "hi"}], max_tokens=8)
    print(base, "->", captured["url"])

Measured output:

http://host.docker.internal:12434/engines/v1/  -> http://host.docker.internal:12434/engines/v1/chat/completions
http://host.docker.internal:12434/engines/v1   -> http://host.docker.internal:12434/engines/v1/chat/completions
http://model-runner.docker.internal            -> http://model-runner.docker.internal/chat/completions
http://model-runner.docker.internal/engines/v1 -> http://model-runner.docker.internal/engines/v1/chat/completions
http://172.17.0.1:12434/engines/v1/            -> http://172.17.0.1:12434/engines/v1/chat/completions
http://model-runner/engines/v1/                -> http://model-runner/engines/v1/chat/completions
http://localhost:12434/engines/v1/             -> http://localhost:12434/engines/v1/chat/completions

Six of the seven candidates — 85.7% — build the correct /engines/v1/chat/completions. The single failure is row three, the value printed verbatim in the official "access from containers" table. A trailing slash makes no difference because httpx normalises it, so the slash is not the issue; the missing /engines/v1 segment is.

The Anthropic SDK has an independent contradiction. The endpoint table in the API reference lists /anthropic/v1/messages, while the curl example on the same page uses http://localhost:12434/v1/messages, and the third-party Base URL table gives Anthropic clients http://localhost:12434. I fed all of these to anthropic 1.14.0:

base_url=http://localhost:12434            -> http://localhost:12434/v1/messages
base_url=http://localhost:12434/anthropic  -> http://localhost:12434/anthropic/v1/messages

In other words, configuring from the official Base URL table makes the SDK build /v1/messages, which matches the curl example and contradicts the endpoint table. Both come from the same official page. I cannot tell you which one your DMR build honours, so the practical advice is to follow the curl example and confirm with one real request before you ship, rather than following the endpoint table.

Troubleshooting: 4 real errors I reproduced

All four errors below are raw output from my machine, not invented "typical error" samples. The first two are connectivity, the last two are configuration.

Error one: localhost:12434 inside a container goes nowhere

openai.APIConnectionError: Connection error.
cause: httpx.ConnectError: [WinError 10061] No connection could be made because the target machine actively refused it.

Why: inside a container, localhost is the container itself. The model runs on the host, so nothing listens on port 12434 in the container's network namespace. This is the "host process" address applied to the "from a container" scenario.

Fix: on Docker Desktop use http://model-runner.docker.internal/engines/v1; on Docker Engine use http://172.17.0.1:12434/engines/v1, or add extra_hosts and use http://model-runner.docker.internal:12434/engines/v1.

The error code is platform-specific: [WinError 10061] is the Windows output I measured, and Linux reports a different errno string, but the exception type is the same httpx.ConnectError. Search on the exception type, not on the error code.

Error two: the hostname does not resolve at all

openai.APIConnectionError: Connection error.
cause: httpx.ConnectError: [Errno 11001] getaddrinfo failed

Why: model-runner.docker.internal is an internal DNS name provided by Docker Desktop. On plain Docker Engine, or when a container is not attached to the Compose project network, that record does not exist. The same official note about 172.17.0.1 mentions that containers may not reach it by default and need extra_hosts.

Fix: declare it explicitly in extra_hosts.

services:
  app:
    image: my-app
    extra_hosts:
      - "model-runner.docker.internal:host-gateway"
    environment:
      - AI_MODEL_URL=http://model-runner.docker.internal:12434/engines/v1

The check is one line: run docker compose exec app getent hosts model-runner.docker.internal. If it does not resolve, the problem is at this layer — do not start editing application code.

Error three: the path is missing /engines/v1

This one raises nothing. It just returns 404. Captured with the script above:

http://model-runner.docker.internal/chat/completions              # wrong
http://model-runner.docker.internal/engines/v1/chat/completions   # right

Why: the Base URL the API reference lists for containers omits /engines/v1, and the OpenAI SDK only appends /chat/completions to whatever base you gave it. It will not insert the middle segment for you.

Fix: either write the base URL as http://model-runner.docker.internal/engines/v1, or consume the variable Compose injects — and verify the injected value actually contains /engines/v1 by running docker compose exec app env. Faster than guessing.

Error four: two models get the same variable names

No error is raised; model A's requests simply get answered by model B. Recomputed result:

my-model + my_model intersection: {'MY_MODEL_MODEL', 'MY_MODEL_URL'}

Why: short syntax normalises by uppercasing and turning - into _. my-model and my_model normalise to the same string.

Fix: switch to the long syntax everywhere.

services:
  app:
    image: my-app
    models:
      my-model:
        endpoint_var: MY_MODEL_A_URL
        model_var: MY_MODEL_A_NAME
      my_model:
        endpoint_var: MY_MODEL_B_URL
        model_var: MY_MODEL_B_NAME

If you have already hit this, docker compose exec app env shows how many _URL variables exist and what each one points at.

A minimal configuration that works

Everything above, folded into one file:

services:
  app:
    image: my-app
    models:
      llm:
        endpoint_var: AI_MODEL_URL
        model_var: AI_MODEL_NAME
    extra_hosts:
      - "model-runner.docker.internal:host-gateway"
    environment:
      - AI_MODEL_URL=http://model-runner.docker.internal:12434/engines/v1

models:
  llm:
    model: ai/smollm2:360M-Q4_K_M
    context_size: 4096

Notes: ai/smollm2:360M-Q4_K_M is the tagged form that appears in Docker's own examples, and 360M parameters — roughly 360 million — is small enough to run on CPU. Set context_size to the smallest value that meets your need, as the docs advise. extra_hosts is only required on Docker Engine; leaving it in on Docker Desktop does no harm.

Application side:

import os
from openai import OpenAI

client = OpenAI(
    base_url=os.environ["AI_MODEL_URL"],
    api_key="not-needed",   # official: DMR ignores the Authorization header
)
resp = client.chat.completions.create(
    model=os.environ["AI_MODEL_NAME"],
    messages=[{"role": "user", "content": "hi"}],
)

That api_key line is not a placeholder. Under "Limitations and differences from OpenAI" the docs state that an API key is not required and that DMR ignores the Authorization header. Read that the other way round: any process that can reach the endpoint can call your model without credentials. Think before you expose 12434 to an untrusted network.

When this is not worth adopting

Skip it if any of three things is true: your Compose is older than 2.38.0, your deployment target does not implement the Compose models spec, or your model already lives behind a remote API — in which case pointing the official SDK at the remote base URL is simpler.

One more thing worth stating plainly: the models element solves declaration and wiring, not performance. Context size, which inference backend to use, and whether to enable GPU are still DMR-side decisions. Compose only turns "my service depends on this model" from a spoken convention into a line in a file.