Running vision inference on-device — a phone, a browser, a small board — avoids the latency and privacy cost of a cloud round trip, but the model has to actually fit and run fast enough on far weaker hardware.
The two levers
- Quantization: converting model weights from 32-bit to 8-bit (or lower) shrinks size and speeds up inference, usually with a small, measurable accuracy cost.
- ONNX: an intermediate model format that runs across many devices and runtimes without depending on the original training framework, which is what actually makes 'runs on this specific phone' practical.
A realistic workflow
Train and validate normally, export to ONNX, quantize the ONNX model, then re-run your evaluation set on the quantized version before shipping — the accuracy drop from quantization is usually acceptable but should be measured, not assumed.
See ONNX Runtime Web: running ML in the browser for the specific case of shipping this to a browser rather than a native app.
GenAI engineer and AI product builder with 2+ years shipping production-grade AI systems: RAG pipelines, fine-tuned LLMs, hybrid retrieval and multi-modal apps across vision, speech and OCR, architected end to end from ingestion to deployment. Leads engineering for SCULT INDIA's 14-member team, founded the 500+ member TechVerse Enclave community and has mentored 200+ students. Three hackathon first prizes: Changethon 2025 (IIT Roorkee), Product Genesis at Vividhotsava 2025 (CSJMU Kanpur) and BYTEBATTLE (MeetKats).
Open to GenAI roles, hackathon judging, mentorship sessions and guest talks at colleges. Email pranjulrathour41@gmail.com.
