Browser-Use with Local AI: Why Llama.cpp Beats Ollama (And How to Set It Up in Minutes)

---

After hours of fighting with Ollama, GPU compatibility issues, and broken vision models, I finally have a working local browser-use setup. It’s fast, reliable, and costs nothing beyond your hardware. Here’s what I learned—and why I switched to Llama.cpp.

The Problem: Why Local Browser Automation Feels Broken

If you’ve tried to get browser-use working with a local AI model, you’ve probably hit some combination of these walls:

Ollama’s Vision Model Nightmares

Getting multimodal models to work with Ollama is a gamble. The issue isn’t just getting the model loaded—it’s the mmproj (vision projector) files that often refuse to cooperate.

The common failure pattern:

# This works perfectly with llama-server

llama-server -m Qwen3.5-35B-A3B-heretic-v2-Q5_K_M.gguf --mmproj Qwen3.5-35B-A3B-mmproj-BF16.gguf -c 4096

# But Ollama chokes on the same files

ollama run qwen3.5-custom-vision # → "error loading model architecture: 'qwen35moe'"

The problem: imported GGUF models with separate mmproj files often fail in Ollama with “unknown model architecture” errors, even though they work perfectly via llama.cpp’s --mmproj flag. Community GGUF models from HuggingFace frequently fall into this category. (Source)

Ollama on AMD GPUs (Strix Halo): A Reliability Problem

On AMD hardware—particularly the Strix Halo platform—the situation gets worse. The Ollama Vulkan backend is unreliable:

• Qwen3.5 hangs on Vulkan after working initially with other models
• ROCm without proper overrides doesn’t detect the GPU at all
• Even when configured, Vulkan produces unstable performance compared to native ROCm

Users report spending full days troubleshooting Ollama on Strix Halo, only to end up with unreliable performance or crashes. The workaround requires specific environment variables, and even then, it’s hit-or-miss. (Source)

The Performance Tax

Even when Ollama works, it comes with a performance cost. Running the same model through Ollama produces fewer tokens per second than llama.cpp directly. In one benchmark test, llama.cpp achieved 161 tokens/second while Ollama on the same hardware reached only ~89 tokens/second—nearly 1.8x difference. (Source)

The abstraction layer adds overhead that matters for real-time tasks like browser automation.

Browser Automation Frustrations

The browser-use community is full of posts from people who:

• Spent hours configuring API keys and cloud endpoints
• Dealt with slow response times from remote APIs
• Found that cloud AI services don’t understand web page layouts
• Couldn’t get vision models to reliably analyze screenshots

The setup friction is mostly an authentication problem masquerading as a documentation problem—most web-facing MCP servers need fresh credentials per service, and that’s what kills the quick-test loop. (Source)

The promise of “local AI browser automation” often falls apart at the configuration stage.

The Solution: Llama.cpp with browser-use

I found a setup that works reliably. The key insight: skip Ollama entirely and use llama.cpp server directly.

Why Llama.cpp Wins

1. Vision models work out of the box — The --mmproj flag handles multimodal models correctly, including community GGUF files that Ollama rejects (Source)
2. No GPU compatibility headaches — Works with CUDA, ROCm, and Vulkan without the specific workarounds Ollama requires (Source)
3. Better performance — Direct model execution means higher tokens/second (Source)
4. Simpler debugging — When something breaks, you’re debugging one component, not Ollama’s abstraction layer

What You Get

• Privacy: All processing happens on your machine
• Speed: 827+ tokens/second prompt processing, 31 tokens/second generation—real-time browser interaction without delays on Strix Halo
• Reliability: No API rate limits or network dependencies
• Cost: Zero ongoing costs after hardware investment

Setup Guide: Under 5 Minutes

Prerequisites

• Consumer GPU (RTX 4070/3080, AMD 7800 XT, or similar)
• 15GB+ VRAM for the model
• llama.cpp installed

Step 1: Install llama.cpp server

On most systems:

# Clone and build

git clone https://github.com/ggerganov/llama.cpp

cd llama.cpp

cmake -B build && cmake --build build --config Release

# Or use a package manager (brew, apt, etc.)

Step 2: Download the model

Get Qwen 3.5 9B with its vision projector from HuggingFace or your preferred GGUF source. You’ll need:

• *Qwen3.5-9B*.gguf (or Qwen3.5-9B-Q4_K_M.gguf)
• *mmproj*.gguf (e.g., Qwen3.5-9B-mmproj-Q4_K_M.gguf)

Step 3: Start the server

./llama-server \

-m Qwen3.5-9B-Q4_K_M.gguf \

--mmproj Qwen3.5-9B-mmproj-Q4_K_M.gguf \

--host 0.0.0.0 \

-port 8080

That’s it. You now have an OpenAI-compatible API at localhost:8080.

Step 4: Configure browser-use

Point your browser-use MCP configuration to the local server:

{

"mcpServers": {

"browser-use": {

"command": "npx",

"args": ["-y", "@browser-use/mcp-server"],

"env": {

"OPENAI_API_BASE": "http://localhost:8080/v1",

"OPENAI_API_KEY": "local",

"ANTHROPIC_API_KEY": "local"

}

}

}

}

Step 5: Use it

Your AI agent can now:

• Navigate websites autonomously
• Take and analyze screenshots
• Complete multi-step web tasks
• Understand page layouts through vision

Hardware Requirements

Strix Halo Performance Numbers

When running Qwen 3.5 9B on AMD Strix Halo via llama.cpp server, the benchmark results speak for themselves:

The prompt prefill speed of 827 tokens/second means page analysis happens almost instantaneously. Token generation at 31 tokens/second is fast enough for real-time browser interaction without frustrating delays.

These numbers vary by model quantization level and context size, but even at the lower end, the Strix Halo handles browser automation tasks with ease thanks to its 40GB unified memory architecture.

Repository

I’ve packaged this into a ready-to-use setup:

https://github.com/damianhunziker/browser-use-llama.cpp

It includes everything you need to get started in under 5 minutes—no more fighting with broken configurations.

Sources

1. GitHub Issue #14730 – unknown model architecture ‘qwen35moe’ with mmproj
2. GitHub Issue #14855 – AMD Strix Halo ROCm/Vulkan problems
3. Reddit r/LocalLLaMA – llama.cpp runs 1.8 times faster than ollama
4. Reddit r/LocalLLaMA – AI agent tools MCP servers fragmentation
5. Benchmark: prompt eval 827.51 tok/s, generation 31.43 tok/s on Strix Halo

The Bottom Line

Ollama’s simplicity comes at a cost: reliability and performance. Llama.cpp’s server approach gives you:

• Reliable vision model support that doesn’t break on community GGUF files
• Consistent performance without Ollama’s abstraction overhead
• No AMD GPU headaches — it just works
• Complete privacy — nothing leaves your machine

If you’ve been frustrated by local AI browser automation, this is the setup that actually delivers on the promise.