Docker Model Runner

Canonical version: Docker Model Runner.

Docker Model Runner is a feature of Docker Desktop that lets you download and run AI/ML models locally using Docker CLI commands. It provides a simple interface to pull models from registries (including Docker Hub and Hugging Face), run inference, and expose an OpenAI-compatible API endpoint for integration with existing tools and applications.

Models run natively on the host machine (using GPU acceleration when available), not inside containers. This means near-bare-metal performance for inference. The familiar Docker UX (docker model pull, docker model run, docker model ls) makes it accessible to developers already comfortable with Docker workflows.

Key Commands

# List available models
docker model ls

# Pull a model
docker model pull ai/llama3.2

# Run a model interactively
docker model run ai/llama3.2

# Serve a model via OpenAI-compatible API
docker model serve ai/llama3.2

OpenAI-Compatible API

When a model is served, it exposes an API at http://localhost (via Docker Desktop's built-in proxy) that follows the OpenAI chat completions format. Any tool or library that supports the OpenAI API can point to this endpoint, making it a drop-in local replacement for cloud-based LLM APIs.

Use Cases

• Local AI development and testing without cloud API costs
• Privacy-sensitive workloads where data cannot leave the machine
• Offline development environments
• Prototyping AI-powered features before committing to a cloud provider
• Running Model Context Protocol (MCP) tool servers alongside local models

References

• https://docs.docker.com/model-runner/
• https://www.docker.com/blog/docker-model-runner/

Related

• Docker AI
• Docker Desktop
• Docker
• Docker MCP Catalog
• Docker Compose for AI
• Ollama
• Model Context Protocol (MCP)

About Sébastien

I'm Sébastien Dubois, and I'm on a mission to help knowledge workers escape information overload. After 20+ years in IT and seeing too many brilliant minds drowning in digital chaos, I've decided to help people build systems that actually work. Through the Knowii Community, my courses, products & services and my Website/Newsletter, I share practical and battle-tested systems.

I write about Knowledge Work, Personal Knowledge Management, Note-taking, Lifelong Learning, Personal Organization, Productivity, and more. I also craft lovely digital products and tools.

If you want to follow my work, then become a member and join our community.

Ready to get to the next level?

If you're tired of information overwhelm and ready to build a reliable knowledge system:

• 📚 KM for Beginners — 10+ hours of structured video lessons
• 🚀 Obsidian Starter Kit — Ready-made vault with 40+ templates
• 💼 Knowledge Worker Kit — Complete guides + lifetime community
• 🦉 1-on-1 Coaching — Personalized guidance
• 🎯 Join Knowii — Community + ALL courses & tools