Key takeaways
- People run local LLMs for privacy, offline use, cost control and customisation.
- Ollama, LM Studio and llama.cpp run quantised models on Mac, Windows and Linux.
- CPU-only inference works for small models, and everything runs offline once downloaded.
- Apple Silicon's unified memory makes MacBook Airs and Mac minis capable local LLM machines.
- AMD GPUs, phones and Raspberry Pi boards can all run suitably sized quantised models.
Running a large language model locally means downloading open weights and generating text on your own hardware instead of calling a cloud API. Quantised models and tools such as Ollama, LM Studio and llama.cpp have made it practical on ordinary laptops. FaceVision, which I built, runs its models locally in the browser, so this page covers why people run local LLMs, Mac and Windows setup, CPU-only and offline use, the MacBook Air, the Mac mini, AMD GPUs, phones and the Raspberry Pi.
Why do people run local LLMs?
Privacy is the main reason: prompts and documents never leave your machine. Others include working offline, avoiding per-token costs and rate limits, customising or fine-tuning models freely, and learning how models work hands-on.
Companies handling sensitive data, such as legal, medical or source code, use local models to meet data protection requirements. Developers prototype without API bills and test fine-tuned models directly. Hobbyists experiment with specialised community models. The trade-offs are real: local models are usually less capable than the strongest cloud models, need capable hardware, and require you to manage updates and safety yourself. Many teams use a hybrid approach, running local models for private or routine work and cloud models for the hardest tasks.
How do you run a local LLM on a Mac?
Install Ollama or LM Studio, download a model sized for your memory, and start chatting. On Apple Silicon, both use the GPU through Metal, and LM Studio also supports Apple's MLX framework for fast inference.
Also asked as: how to run local llm on mac
With Ollama, run a command such as ollama run followed by a model name, and it downloads and serves the model with a local API compatible with OpenAI clients. LM Studio offers a graphical interface for browsing, downloading and chatting with models. For developers, llama.cpp and the MLX-LM package give finer control. Choose quantised 4-bit models, which need a little over half a gigabyte of memory per billion parameters, so an 8 billion parameter model fits comfortably in 16 GB alongside your other apps.
How do you run a local LLM on Windows?
Install Ollama for Windows or LM Studio, pick a model that fits your GPU memory or RAM, and run it. An NVIDIA GPU with CUDA gives the best speed, while CPU-only machines can still run smaller quantised models.
Also asked as: how to run local llm on windows
Ollama runs as a background service with a local API, so tools such as Open WebUI, coding assistants and your own scripts can use it. LM Studio and Jan provide desktop chat interfaces. For server-grade frameworks such as vLLM, use WSL2 with a Linux environment. Microsoft's Foundry Local also runs models on Windows using available GPUs or NPUs. Keep GPU drivers updated, check that models load onto the GPU rather than falling back to the CPU, and close memory-heavy applications before loading larger models.
Can you run a local LLM without a GPU?
Yes. llama.cpp and tools built on it, such as Ollama and LM Studio, run quantised models on the CPU using system RAM. Small models with 1 to 8 billion parameters give usable speeds on modern laptops and desktops.
Also asked as: can i run local llm without gpu
Speed depends mostly on memory bandwidth and CPU cores: expect a few to perhaps a couple of dozen tokens per second for small models, much slower than a GPU. Choose 4-bit quantised GGUF models and keep context lengths moderate to save memory. Larger models, such as 70 billion parameters, technically run on CPUs with enough RAM but become too slow for interactive use. For batch tasks such as summarising documents overnight, CPU inference is perfectly practical. Apple Silicon and recent integrated GPUs offer a middle ground.
Can a local LLM run without internet?
Yes. Once the model weights and software are downloaded, a local LLM runs entirely offline. Ollama, LM Studio and llama.cpp need internet only to download models or updates, not to generate text.
Also asked as: can local llm run without internet
This makes local models useful on flights, in secure facilities with air-gapped networks, in areas with weak connectivity, and for sensitive work that must not leave a device. Download several models in advance, since you cannot fetch new ones offline. Features that depend on the web, such as browsing or online search tools, will not work, but retrieval over local documents still can if your embeddings and vector store are local too. Verify offline behaviour by disconnecting before you rely on it.
Can you run a local LLM on a MacBook Air?
Yes. Apple Silicon MacBook Airs run small and mid-sized quantised models well thanks to unified memory and the GPU. With 16 GB or more, 7 to 8 billion parameter models run comfortably at conversational speeds.
Also asked as: can i run local llm on macbook air
With 8 GB, stick to models of about 3 to 4 billion parameters, since macOS and apps need memory too. Recent models such as the M4 MacBook Air start with more memory, which helps. Because the Air has no fan, long generation sessions can cause thermal throttling and slower speeds, though short chats are fine. Use Ollama, LM Studio or MLX-based tools for the best performance. For heavier local workloads, such as larger models or long batch jobs, a MacBook Pro or Mac mini sustains performance better.
Why is the Mac mini good for local LLMs?
Apple Silicon Mac minis share one pool of fast unified memory between the CPU and GPU, so models can use most of the system's RAM, unlike PC graphics cards with separate, limited video memory. That lets a compact, quiet machine run large models.
Also asked as: why is mac mini good for local llm
Higher-memory configurations, especially with M4 Pro chips and their greater memory bandwidth, can run models that would need multiple consumer GPUs on a PC. The Mac mini is also power-efficient and quiet enough to run continuously as a home or office model server with Ollama. Its limits are lower raw speed than high-end NVIDIA GPUs and memory that cannot be upgraded later, so buy enough upfront. Some enthusiasts cluster several Mac minis with tools such as exo to run even larger models.
Can you run a local LLM on an AMD GPU?
Yes. Ollama supports many AMD Radeon GPUs on Linux and Windows, llama.cpp runs on AMD through ROCm or Vulkan, and LM Studio uses Vulkan. Support depends on the specific card, so check compatibility lists.
Also asked as: can you run local llm on amd gpu
AMD's ROCm platform provides a CUDA-like stack for PyTorch and frameworks such as vLLM, best supported on Linux with recent Radeon and Instinct cards. Vulkan backends work on a wider range of GPUs with less setup, though sometimes more slowly. AMD cards with large video memory can be good value for running bigger models. Keep drivers updated, confirm the model actually loads onto the GPU, and expect some tools to support NVIDIA first. AMD's Ryzen AI chips also run smaller models on integrated GPUs and NPUs.
Can you run a local LLM on a phone?
Yes, small models with roughly 1 to 4 billion parameters run on recent phones. Apps such as PocketPal AI, MLC Chat and Google's AI Edge Gallery let you download and chat with models like Gemma, Llama, Phi and Qwen on-device.
Also asked as: can you run local llm on phone
Phone makers also ship on-device models: Apple Intelligence runs a compact model on supported iPhones, and Gemini Nano runs on supported Android phones for features such as summaries. Performance depends on the chip and memory, so flagship phones with 8 GB or more of RAM work best. Expect slower speeds than laptops, battery drain and heating during long sessions. On-device models suit private notes, offline translation and quick drafting, while complex reasoning still benefits from larger cloud models.
Can you run a local LLM on a Raspberry Pi?
Yes, within limits. A Raspberry Pi 5 with 8 GB or more of RAM can run small quantised models of about 1 to 3 billion parameters with llama.cpp or Ollama, generating a few tokens per second.
Also asked as: can you run local llm on raspberry pi
Larger models of around 7 billion parameters can load on higher-memory boards but run slowly. Use 4-bit models, keep context short, add active cooling and use fast storage such as an SSD for loading models. Practical uses include offline voice assistants, home automation that interprets commands, and educational projects about how LLMs work. For better speed in small form factors, consider a mini PC or an NVIDIA Jetson board. A Pi can also act as a lightweight client for a more powerful local model server.
Sources
GenAI engineer and AI product builder with 2+ years shipping production-grade AI systems: RAG pipelines, fine-tuned LLMs, hybrid retrieval and multi-modal apps across vision, speech and OCR, architected end to end from ingestion to deployment. Leads engineering for SCULT INDIA's 14-member team, founded the 500+ member TechVerse Enclave community and has mentored 200+ students. Three hackathon first prizes: Changethon 2025 (IIT Roorkee), Product Genesis at Vividhotsava 2025 (CSJMU) and BYTEBATTLE (MeetKats).
Open to GenAI roles, hackathon judging, mentorship sessions and guest talks at colleges. For campus sessions the inviting institution covers the visit end to end: travel (a First AC train ticket for journeys up to five hours, a flight for anything longer), accommodation, meals, and pickup and drop; the honorarium and workshop fees are negotiable on WhatsApp or on the call, and requests are welcome from institutions that can cover the visit this way. Email pranjulrathour41@gmail.com.
