Key takeaways
- ollama run <model> downloads the model if needed, loads it and opens a chat; the same model is then available on the local API.
- Ollama runs quantised models with the GGML engine on your CPU or GPU, entirely on your machine.
- After a model is downloaded, Ollama works fully offline.
- CPU-only works with small models; laptops with 16 GB RAM handle 7B to 8B models at 4-bit.
- Raspberry Pi can run small models; phones need dedicated on-device apps rather than Ollama.
To run a model with Ollama, install it and type ollama run followed by a model name, such as ollama run llama3.2; Ollama downloads the model the first time, loads it, and opens a chat in your terminal, while also serving it on a local API for your apps. Everything runs on your own machine, offline once the model is downloaded. This page explains the command, how local inference works, offline and CPU-only use, laptops and Macs, Raspberry Pi, Android and iPhone, and running GGUF models from Hugging Face.
How do I use ollama run?
Run ollama run <model> in a terminal. If the model is not installed, Ollama pulls it first; then it loads the model into memory and opens an interactive chat. Type your message, and /bye to exit.
Also asked as: how to ollama run
Useful variations:
ollama run llama3.2 # chat
ollama run qwen2.5:7b "Explain BM25 in one line" # one-off prompt
ollama run llama3.2 --verbose # shows tokens per second
Inside the chat, /set parameter num_ctx 8192 raises the context window and /show info displays model details. After the first run, the same model is available to apps through http://localhost:11434. Use ollama ps to see whether it is running on GPU or CPU.
How does Ollama run models locally?
Ollama loads a model file, usually a quantised GGUF model, into your RAM or GPU memory and runs inference with the GGML engine, the same family of code behind llama.cpp, using your CPU, GPU or both.
When you send a prompt, Ollama applies the model's chat template, tokenises the text, runs the model to generate tokens one by one, and streams them back. It keeps the model loaded for a few minutes between requests so follow-ups are fast, then unloads it to free memory. Quantisation, storing weights in 4 or 8 bits instead of 16, is what lets billion-parameter models fit on ordinary laptops. Nothing is sent to external servers unless you use Ollama's optional cloud models.
Can Ollama run without internet?
Yes. Once a model has been downloaded, Ollama runs completely offline; prompts and responses never leave your machine. You need internet only to install Ollama and pull or update models.
This makes it useful for travel, poor connectivity, privacy-sensitive documents and air-gapped environments. To prepare, pull the models you need while online, and if you use tools on top of Ollama, such as a web UI or LangChain, make sure their dependencies are installed too. Models can be copied between machines by copying the models folder, which helps when internet is slow. Cloud models in Ollama, by contrast, always need a connection.
Can Ollama run on CPU only?
Yes. Ollama runs on CPU when no supported GPU is available, though more slowly. Small models, of 1B to 4B parameters, are comfortable on modern CPUs; 7B to 8B models work but respond more slowly.
Tips for CPU-only use: pick small, quantised models; keep the context window modest; close memory-heavy apps; and prefer machines with more RAM and newer CPUs, which have faster vector instructions. Apple Silicon Macs are a special case: their unified memory and GPU make local models fast even without a separate graphics card. For tasks such as embeddings, classification and short answers, small CPU models are often good enough.
Can I run Ollama on my laptop?
Yes. Most modern laptops can run Ollama. With 8 GB of RAM, use 1B to 4B models; with 16 GB, 7B to 8B models at 4-bit run well; with 32 GB or a GPU with 8 GB or more of video memory, larger models become practical.
Check your setup with ollama run llama3.2 --verbose and look at tokens per second: above roughly 10 tokens per second feels responsive for chat. Plug in the charger for long sessions, since power-saving modes slow inference. If your laptop has an NVIDIA GPU, keep drivers updated so Ollama uses it. For heavier models, consider Ollama's cloud models or a remote GPU server.
How do I run Ollama on a Mac?
Download the Mac app from ollama.com, move it to Applications and open it; the ollama command is then available in Terminal. Run ollama run llama3.2 to start.
Also asked as: how to run ollama on mac
Apple Silicon Macs, with M-series chips, run local models efficiently because the GPU shares fast unified memory with the CPU; a Mac with 16 GB can run 7B to 8B models smoothly, and more memory allows larger models. Intel Macs work on CPU, more slowly. Keep the app running in the menu bar so the local API stays available for other tools. Models are stored in ~/.ollama/models; set OLLAMA_MODELS to move them to an external drive if space is tight.
Can you run Ollama on a Raspberry Pi?
Yes, on recent 64-bit Raspberry Pi models with a 64-bit operating system, using the Linux install script. Expect it to be slow: use very small models, around 1B to 3B parameters, and short contexts.
Also asked as: can you run ollama on raspberry pi
A Raspberry Pi 5 with 8 GB of RAM is the practical minimum for a usable experience with tiny models. It is a great learning project and can power simple offline assistants, home automation commands or classification tasks, but not fast chat with larger models. Cooling matters, as sustained inference heats the board. For anything heavier, run Ollama on a laptop or server and let the Pi call it over the network.
Can Ollama run on Android?
Not officially. Ollama provides installers for Windows, macOS and Linux, not Android. Some enthusiasts run it on Android through Linux environments such as Termux, but performance and support vary.
For on-device AI on Android phones, apps built for mobile, which run small models with frameworks optimised for phones, are a better choice, and Google's on-device Gemini Nano powers some built-in features. A practical alternative is to run Ollama on your laptop or a home server and connect to it from your phone over your local network through a web UI, keeping the API protected and off the public internet.
Can you run Ollama on an iPhone?
No. Ollama does not run on iOS. To use local models on an iPhone, use apps designed to run small models on-device, or connect from your iPhone to Ollama running on a Mac or server on your network.
Also asked as: can you run ollama on iphone
Apple's own on-device models power Apple Intelligence features on supported iPhones. Third-party apps can run small open models on recent iPhones, with limits on model size because of memory. The remote approach often gives the best experience: run Ollama with a capable model on your Mac, put a chat UI in front of it, and open that UI in Safari on your iPhone over Wi-Fi.
Can Ollama run GGUF models?
Yes. GGUF is Ollama's native format. You can run GGUF models from Hugging Face directly by reference, or import a local GGUF file with a Modelfile and ollama create.
ollama run hf.co/<user>/<repo> # GGUF repo on Hugging Face
# Modelfile
FROM ./my-model.Q4_K_M.gguf
PARAMETER temperature 0.3
ollama create my-model -f Modelfile && ollama run my-model
Choose a quantisation that fits your memory, such as Q4_K_M for a balance of size and quality. Make sure the chat template matches the model, or answers will be odd; models from Ollama's library and many Hugging Face repos include it.
Sources
GenAI engineer and AI product builder with 2+ years shipping production-grade AI systems: RAG pipelines, fine-tuned LLMs, hybrid retrieval and multi-modal apps across vision, speech and OCR, architected end to end from ingestion to deployment. Leads engineering for SCULT INDIA's 14-member team, founded the 500+ member TechVerse Enclave community and has mentored 200+ students. Three hackathon first prizes: Changethon 2025 (IIT Roorkee), Product Genesis at Vividhotsava 2025 (CSJMU) and BYTEBATTLE (MeetKats).
Open to GenAI roles, hackathon judging, mentorship sessions and guest talks at colleges. For campus sessions the inviting institution covers the visit end to end: travel (a First AC train ticket for journeys up to five hours, a flight for anything longer), accommodation, meals, and pickup and drop; the honorarium and workshop fees are negotiable on WhatsApp or on the call, and requests are welcome from institutions that can cover the visit this way. Email pranjulrathour41@gmail.com.
