Ollama on ARM64 works out of the box, no special build or workaround required. If you’ve got a Hetzner CAX instance or another Ampere-based ARM VPS sitting idle, you can have a model running in about ten minutes. This is the hands-on follow-up to our self-hosted LLM cost breakdown — here’s the actual setup.
Before you start
You’ll need SSH access to an ARM64 VPS running Ubuntu or Debian. A Hetzner CAX instance works well for this: the CAX21 (4 vCPU, 8 GB RAM) is a sensible starting point for a small model, and the CAX31 (8 vCPU, 16 GB RAM) gives you room for a 7-8B model. Nothing here needs root beyond the install step, and nothing needs a GPU.
Step 1: Update the system
SSH in and update packages first, as with any fresh VPS:
sudo apt update && sudo apt upgrade -y
Step 2: Install Ollama on ARM64
The official install script auto-detects your architecture, so the same command works whether you’re on x86 or ARM64:
curl -fsSL https://ollama.com/install.sh | sh
If you’d rather install manually, or the script is blocked by a firewall, the ARM64 build is also available directly:
curl -fsSL https://ollama.com/download/ollama-linux-arm64.tar.zst | sudo tar x -C /usr
Either method sets up Ollama as a systemd service, so it starts automatically on reboot. Confirm it’s running with:
systemctl status ollama
Step 3: Pick a model that fits your RAM
This is the part people get wrong, and it’s almost always a RAM problem, not a CPU one. As a rule of thumb for Q4-quantized models:
- 8 GB RAM (CAX21): comfortable for a 3-4B model, e.g. Qwen3.5 4B
- 16 GB RAM (CAX31): comfortable for a 7-8B model, e.g. Llama 3.3 8B or Qwen 3 7B
We covered the full cost and sizing table in the break-even piece if you want the reasoning behind these numbers.
Step 4: Pull and run a model
Pulling a model downloads it; running it starts an interactive session:
ollama run qwen2.5:7b
The first run downloads the model, which takes a few minutes depending on your VPS’s network speed. After that it loads from disk each time you run it. Type a prompt directly at the resulting prompt to test it, or exit with /bye.
Step 5: Keep it off the public internet
By default, Ollama’s API only listens on localhost (127.0.0.1), which is the right default for a single-user setup — nothing outside the box can reach it. Leave it that way unless you specifically need remote access, and if you do open it up, put it behind authentication and a firewall rule scoped to known IPs rather than exposing port 11434 to the world.
Where this goes next
A raw terminal session is fine for testing, but not for daily use. The next practical step is putting a proper chat interface in front of it, which is what we’ll cover in the next guide.