Small language models: hallucination, size, training, fine-tuning, local use, agents

Small language models trade breadth for efficiency: they run on laptops, phones and browsers, cost little to serve and can be fine-tuned cheaply, but they know less and hallucinate more on facts. I fine-tune small models with QLoRA in FineTune Studio and run models in the browser for FaceVision, so this page covers hallucination, how small is small, training and building one, fine-tuning, running locally, retrieval, why researchers care, the hype and agents.

Do small language models hallucinate?

Yes, and generally more than large models on factual questions, because fewer parameters store less world knowledge. They may confidently invent names, dates or details, especially for rare topics.

Reduce hallucination by giving small models the facts through retrieval, constraining outputs with schemas, fine-tuning them on domain data with examples of declining when unsure, and keeping tasks narrow, such as classification or extraction where answers come from the input. Evaluation should measure faithfulness to provided context, not just fluency. In RAG designs such as RAG.NextUpgrad, a confidence gate that declines when evidence is weak helps whichever model size you use. Always test hallucination rates on your own questions.

How many parameters does a small language model have?

There is no strict definition, but small language models usually range from a few hundred million to around ten or fifteen billion parameters, small enough to run on a single consumer GPU, laptop or phone.

Also asked as: how many parameters in small language models

The boundary keeps moving as hardware improves and large models grow. Mixture-of-experts models blur it further: a model may have thirty billion total parameters but activate only three billion per token, running nearly as fast as a small dense model. Practically, small often means a model you can run locally with acceptable speed. Examples include Qwen3 at 0.6B to 8B, Gemma 3 at 1B to 12B, Llama 3.2 at 1B and 3B, and Phi-4-mini.

How are small language models trained?

Small language models are pretrained on trillions of tokens of filtered text, often including high-quality synthetic data, then instruction-tuned and aligned with preference methods. Many also learn from larger models through distillation.

Data quality matters more at small scale: Microsoft's Phi models showed that carefully curated, textbook-style data lets small models punch above their weight. Google's Gemma models used distillation, training on a larger teacher's predictions. Training small models longer than older scaling rules suggested improves quality per parameter. After pretraining, supervised fine-tuning teaches instruction following, and preference tuning improves helpfulness and safety. Open projects such as SmolLM document these recipes publicly.

How do you create a small language model?

You can train a tiny language model from scratch for learning, following Andrej Karpathy's nanoGPT or nanochat repositories: build a tokenizer, prepare text data, define a small transformer, train on a GPU and evaluate. Useful models usually start from existing ones instead.

Also asked as: how to create a small language model

Training a capable small model from scratch needs huge datasets and substantial compute, so practitioners almost always fine-tune an existing open model for their task. Building one from scratch remains an excellent way to understand transformers, tokenization and training dynamics. Start with a character-level model on a small corpus, then scale up. Sebastian Raschka's book Build a Large Language Model (From Scratch) walks through the full process in PyTorch.

How do you fine-tune small language models?

Collect a few hundred to a few thousand high-quality examples of your task, choose an instruction-tuned base model, and train with LoRA or QLoRA using tools such as Unsloth, TRL or Axolotl, then compare the tuned model with the base model.

Also asked as: how to fine tune small language models

Small models make fine-tuning cheap: a 1B to 4B model can be fine-tuned on a free Colab GPU or a consumer card in hours. Use the correct chat template, a small learning rate and few epochs, watching validation loss for overfitting. Test on held-out examples and check general skills for regressions. FineTune Studio follows this workflow, streaming live loss charts and showing base-versus-tuned outputs side by side.

How do you run a small language model locally?

Use Ollama or LM Studio on a laptop or desktop, llama.cpp for maximum control, MLX on Apple silicon Macs, or browser runtimes such as WebLLM and Transformers.js with WebGPU. Phones can run small models through on-device frameworks.

Also asked as: how to run small language model locally

Download a quantized version, such as 4-bit GGUF, to reduce memory use; a 3B to 8B model runs comfortably on most modern laptops. On phones, Google's AI Edge tools and Apple's Foundation Models framework give apps access to on-device models. Browser deployment, similar to how FaceVision runs vision models with ONNX Runtime Web, keeps data on the user's device. Measure tokens per second and memory before choosing a model size.

Can small language models use what they retrieve?

Yes, but less reliably than large models. Small models sometimes ignore retrieved passages, get distracted by irrelevant ones, or blend context with invented details. Careful retrieval, clear prompts and fine-tuning improve their faithfulness.

Send fewer, better passages using hybrid search and reranking, so the model sees mainly relevant evidence. Instruct it to answer only from the context and cite sources. Fine-tuning methods such as RAFT train models to use relevant documents and ignore distractors, improving grounded answering. Measure faithfulness directly by checking whether answers are supported by retrieved text. Small embedding models and rerankers also run locally alongside them. With good retrieval, small models can match larger ones on domain question answering.

Why do researchers care about small language models?

Small models are cheaper to train and study, run on accessible hardware, protect privacy on devices, and reveal how data quality, distillation and training choices affect capability, making AI research more reproducible and inclusive.

Universities and independent researchers can experiment with small models without massive compute budgets, testing ideas that later scale up. Small models also enable applications in low-resource settings, such as offline tools for regions with limited connectivity, and languages underserved by large commercial models. They lower energy use per query. Work on small models has shown that curated and synthetic data can substitute for sheer scale in many tasks. Open releases with documented recipes accelerate this work.

Why are small language models called the next big thing in AI?

Small models are gaining attention because they cut costs and latency, run privately on devices, can be specialised cheaply, and handle many real tasks well enough, especially combined with retrieval and larger models for hard cases.

Also asked as: why small language models are the next big thing in ai

As companies scale AI features to millions of users, per-request costs matter, and small models can be many times cheaper to serve. On-device AI in phones and laptops depends on them. Specialised fine-tuned small models often beat general large models on narrow tasks. The claim is partly hype, since frontier models remain essential for complex reasoning, but the trend toward mixing model sizes is real and growing. Expect hybrid architectures to become the default.

How are small language models key to scalable agentic AI?

Agents make many model calls per task, so cost and latency multiply quickly. Small models can handle most routine steps, such as routing, extraction and tool calls, while larger models handle the hardest reasoning, making agent systems affordable at scale.

Also asked as: how small language models are key to scalable agentic ai

NVIDIA researchers argued in 2025 that small language models are sufficient for many agent subtasks and more economical to deploy. Fine-tuning a small model on an agent's real tool-calling logs can improve reliability for that workflow. Heterogeneous systems route each step to the cheapest capable model, escalating when confidence is low. This design also enables on-device agents that keep data private and work offline, which matters for sensitive personal and business data.

Pranjul Rathour

GenAI Engineer · India

GenAI engineer and AI product builder with 2+ years shipping production-grade AI systems: RAG pipelines, fine-tuned LLMs, hybrid retrieval and multi-modal apps across vision, speech and OCR, architected end to end from ingestion to deployment. Leads engineering for SCULT INDIA's 14-member team, founded the 500+ member TechVerse Enclave community and has mentored 200+ students. Three hackathon first prizes: Changethon 2025 (IIT Roorkee), Product Genesis at Vividhotsava 2025 (CSJMU) and BYTEBATTLE (MeetKats).

Open to GenAI roles, hackathon judging, mentorship sessions and guest talks at colleges. For campus sessions the inviting institution covers the visit end to end: travel (a First AC train ticket for journeys up to five hours, a flight for anything longer), accommodation, meals, and pickup and drop; the honorarium and workshop fees are negotiable on WhatsApp or on the call, and requests are welcome from institutions that can cover the visit this way. Email pranjulrathour41@gmail.com.

PortfolioInvite me to your campusHire meLinkedInGitHubXInstagramThreadsFacebookBlueskyDev.toBloggerHashnodeSCULTEmail