What are small language models? Examples, size, CPUs, training, and building one

Small language models (SLMs) are compact language models, usually from a few hundred million to around ten billion parameters, designed to run quickly and cheaply, often on laptops, phones or a single GPU. They trade some breadth of knowledge for speed, lower cost and privacy, and become very capable when fine-tuned for a specific task. I built FineTune Studio for QLoRA fine-tuning with live loss tracking and base-versus-tuned evaluation, so this page covers what SLMs are, their size, examples, how they work, CPUs, their future, and training or building one.

What is a small language model in AI?

A small language model is a language model with far fewer parameters than frontier models, built to be efficient enough for local devices, edge hardware or high-volume, low-cost serving. It generates and understands text like larger models, within narrower limits.

Also asked as: what is a small language model ai

SLMs suit focused tasks: classification, extraction, summarisation, routing, autocomplete, on-device assistants and domain chatbots. Their advantages are low latency, low cost, privacy when run locally and easy fine-tuning. Their limits are less world knowledge and weaker complex reasoning than the largest models. Many production systems combine them: a small model for routine requests, a large model for hard ones.

How big are small language models?

There is no strict cutoff, but "small" usually means under about 10 billion parameters, with many popular SLMs between 0.5 and 8 billion. Quantised to 4 bits, a 3-billion-parameter model needs only about 2 GB of memory.

For context, frontier models are believed to have hundreds of billions of parameters or more. Within SLMs, sub-1-billion models run on phones and even browsers; 1 to 4 billion models run comfortably on laptops; 7 to 9 billion models need a decent GPU or a fast CPU with enough RAM. Size is only one factor: training data quality strongly affects capability, which is why recent small models outperform much larger older ones.

What are examples of small language models?

Well-known small language models include Microsoft's Phi series, Google's Gemma models, Meta's Llama 3.2 1B and 3B, Alibaba's small Qwen models, Mistral's smaller models, and Hugging Face's SmolLM family. TinyLlama and research models like those trained on TinyStories show how small useful models can be.

Each family offers different sizes, licences and strengths: Phi models emphasise reasoning from curated, synthetic-heavy data; Gemma includes variants designed for phones; Qwen's small models are strong multilingual options; SmolLM targets very small, fully open models. Check each model's licence before commercial use, and compare them on your own task, since benchmark rankings do not always reflect domain performance.

How do small language models work?

They work like large language models: a transformer network predicts the next token from the previous ones, trained on large text datasets and then instruction-tuned to follow requests. The difference is scale: fewer layers, smaller hidden sizes and fewer parameters.

To stay capable despite size, SLM builders use high-quality and synthetic training data, distillation from larger teacher models, longer training on more tokens per parameter, and efficient architectures. At inference, quantisation shrinks them further for fast local use. Because they know less, SLMs benefit greatly from retrieval: giving a small model the right passages through RAG often matters more than model size for factual tasks.

Can small language models run on a CPU?

Yes. Quantised small models run well on ordinary CPUs using llama.cpp, Ollama or similar runtimes. A 1 to 4 billion parameter model in 4-bit format can generate text at usable speeds on a modern laptop CPU.

Also asked as: can small language models run on cpu

Speed depends on model size, quantisation, CPU cores, memory bandwidth and context length. Apple silicon and recent x86 laptops handle small models comfortably; older machines may be slow with larger SLMs. For browsers, smaller models can run through WebAssembly or WebGPU. Local CPU inference keeps data private and works offline, which suits personal tools, education in low-connectivity settings and prototypes.

Are small language models the future?

They are a big part of it. The trend is towards using the smallest model that meets the quality bar for each task, which is often a small, fine-tuned model, with large models reserved for complex reasoning. On-device AI in phones and laptops also relies on SLMs.

Reasons: lower cost at scale, faster responses, privacy, offline use and energy efficiency. Agent systems increasingly route simple steps to small models. Large models will keep advancing, and distillation from them makes small models better each year. For builders, the practical lesson is to measure: try a small model first, fine-tune if needed, and escalate to a larger one only where quality demands it.

Can I train a small language model?

Yes. Fine-tuning an existing small model on your data is very achievable with LoRA or QLoRA on a single consumer or free cloud GPU. Training a tiny model from scratch is also possible for learning, though useful general models need far more data and compute.

For most projects, fine-tune rather than pretrain: start from a strong small base model, prepare a few thousand high-quality examples, and train adapters. FineTune Studio does exactly this with Transformers, PEFT and TRL, showing loss live over WebSockets and comparing base and tuned outputs side by side. Always evaluate against the base model, since a well-prompted base model sometimes matches a poorly fine-tuned one.

How do I train small language models?

Pick a base model, prepare clean instruction-response data in the model's chat format, fine-tune with LoRA or QLoRA using libraries such as Hugging Face TRL, monitor training and validation loss, and evaluate on held-out examples before deploying.

Also asked as: how to train small language models

Steps:

• Define the task and success metric.
• Collect and clean examples; quality beats quantity.
• Format with the model's chat template.
• Train adapters with conservative learning rates for one to three epochs.
• Watch for overfitting: validation loss rising while training loss falls.
• Compare base and tuned outputs on the same prompts.
• Merge or serve adapters, optionally quantise for deployment.

How do I make a small language model from scratch?

Build one to learn: write or adapt a minimal transformer, such as Andrej Karpathy's nanoGPT, train a tokenizer, prepare a text dataset, and train on a GPU for a few hours. The TinyStories research showed that very small models can write coherent simple stories.

Also asked as: how to make small language model

A from-scratch project teaches tokenisation, attention, training loops, learning rate schedules and sampling. Start with a character-level model on a small dataset, then move to subword tokenisation and a larger corpus. Expect a model useful for narrow text generation, not a general assistant; general capability requires billions of tokens and large compute budgets. For real applications, fine-tune an existing small model instead.

Do small language models exist?

Yes, many. Small language models are widely available as open-weight models on platforms like Hugging Face and Ollama, and companies ship them inside phones, laptops and browsers for on-device features such as summarisation and smart replies.

You can try one in minutes: install Ollama and run a small model locally, or use Hugging Face's Transformers library in Python. Developers build products on them for offline assistants, document classification and domain chatbots. They are a practical, accessible way to work with language models, especially for students without access to large compute budgets.

Pranjul Rathour

GenAI Engineer · India

GenAI engineer and AI product builder with 2+ years shipping production-grade AI systems: RAG pipelines, fine-tuned LLMs, hybrid retrieval and multi-modal apps across vision, speech and OCR, architected end to end from ingestion to deployment. Leads engineering for SCULT INDIA's 14-member team, founded the 500+ member TechVerse Enclave community and has mentored 200+ students. Three hackathon first prizes: Changethon 2025 (IIT Roorkee), Product Genesis at Vividhotsava 2025 (CSJMU) and BYTEBATTLE (MeetKats).

Open to GenAI roles, hackathon judging, mentorship sessions and guest talks at colleges. For campus sessions the inviting institution covers the visit end to end: travel (a First AC train ticket for journeys up to five hours, a flight for anything longer), accommodation, meals, and pickup and drop; the honorarium and workshop fees are negotiable on WhatsApp or on the call, and requests are welcome from institutions that can cover the visit this way. Email pranjulrathour41@gmail.com.

PortfolioInvite me to your campusHire meLinkedInGitHubXInstagramThreadsFacebookBlueskyDev.toBloggerHashnodeSCULTEmail