Used for feedback · quests · score card
Replace the cloud LLM with a local model using Ollama. Run your entire AI agent stack on your own machine with no API keys required.
Add a local LLM as a Docker Compose service
Pull and manage models inside the container
Switch from OpenAI to local model with env vars
Ollama provides an OpenAI-compatible API out of the box. Add it as a service
in your docker-compose.yml alongside
your existing MCP server and agent.
ollama: image: ollama/ollama:latest container_name: ollama ports: - "11434:11434" volumes: - ollama_data:/root/.ollama healthcheck: test: ["CMD", "curl", "-f", "http://localhost:11434/api/tags"] interval: 10s timeout: 5s retries: 5 restart: unless-stopped # Add to the volumes section at the bottom: volumes: ollama_data:
Why a volume? The ollama_data volume persists
downloaded models between container restarts. Without it, you'd need to re-download models every time.
Start the stack, then pull a model into the Ollama container. We'll use
llama3.2 — a compact 3B parameter
model that supports tool calling.
# Start the stack (Ollama will start alongside your other services) docker compose up -d # Pull the model (this downloads ~2GB, may take a few minutes) docker exec ollama ollama pull llama3.2 # Verify the model is available docker exec ollama ollama list # Optional: test the model directly curl http://localhost:11434/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "llama3.2", "messages": [{"role": "user", "content": "Hello!"}] }'
Model sizes: llama3.2 (3B, ~2GB) is a good
starting point. If you have more RAM, try llama3.2:latest or
mistral. For tool calling,
llama3.2 and
mistral have native support.
Ollama exposes an OpenAI-compatible endpoint at /v1.
Point your agent at it by changing three environment variables. No code changes needed.
# Comment out the OpenAI settings: # OPENAI_API_KEY=sk-... # OPENAI_BASE_URL=https://api.openai.com/v1 # OPENAI_MODEL=gemini-3.5-flash-lite # Use Ollama instead: OPENAI_API_KEY=ollama # Ollama ignores the key, but the SDK requires one OPENAI_BASE_URL=http://ollama:11434/v1 OPENAI_MODEL=llama3.2
Make sure the agent service depends on Ollama in your compose file:
agent: build: ./services/agent depends_on: ollama: condition: service_healthy mcp-server: condition: service_healthy environment: - OPENAI_API_KEY=${OPENAI_API_KEY} - OPENAI_BASE_URL=${OPENAI_BASE_URL} - OPENAI_MODEL=${OPENAI_MODEL} - MCP_SERVER_URL=http://mcp-server:8000
Key insight: Because the agent uses the OpenAI SDK with configurable
base_url and model,
switching between OpenAI, Ollama, or any OpenAI-compatible API is just an env var change.
Restart the agent so it picks up the new environment variables, then test a query that triggers tool calling.
# Restart the agent to pick up new env vars docker compose restart agent # Test a simple query (no tools) curl -s http://localhost:8001/query \ -H "Content-Type: application/json" \ -d '{"query": "What is MCP?"}' | jq . # Test a query that requires tool calling curl -s http://localhost:8001/query \ -H "Content-Type: application/json" \ -d '{"query": "What is the weather in Oslo?"}' | jq . # Check the agent logs for the tool-calling flow docker compose logs agent --tail 50
Expect slower responses. Local models run on CPU by default and are much slower
than cloud APIs. A simple response may take 10-30 seconds depending on your hardware.
If you have an NVIDIA GPU, add deploy.resources.reservations.devices
to enable GPU acceleration.
If you have an NVIDIA GPU with Docker GPU support installed, you can dramatically speed up inference by passing the GPU through to the container.
ollama: image: ollama/ollama:latest container_name: ollama ports: - "11434:11434" volumes: - ollama_data:/root/.ollama deploy: resources: reservations: devices: - driver: nvidia count: all capabilities: [gpu]
On macOS with Apple Silicon, Ollama automatically uses the Metal GPU — no extra configuration needed. Just use the standard Ollama image.
Local models are powerful for development and privacy, but come with trade-offs you should understand before choosing them for production.
Smaller models (3B-7B) are less reliable at generating correct tool call arguments compared to a frontier model. They may hallucinate parameter names or produce malformed JSON. Expect to handle more parsing errors.
CPU-only inference is 10-50x slower than cloud APIs. For a workshop or development this is fine, but production workloads need GPU acceleration or a cloud API.
Most local models have smaller context windows (4K-8K tokens) than cloud models (128K+). Long conversations or large tool responses may get truncated.
A 7B model needs ~4GB RAM, a 13B model needs ~8GB, and a 70B model needs ~40GB. Make sure your Docker host has enough memory allocated.
Best practice: Use local models for development and testing, cloud models for production. The env-var-based switching pattern you've built makes this trivial.
Your feedback helps us improve the workshop.
Your handle is sent with this feedback; leave it blank in the header and the submission stays anonymous. Please keep personal data out of the comment too — no name, e-mail address or employer, yours or anyone else's.