How does object detection work? Models, CNNs, LLMs, Gemini, OpenCV, and coding one

Object detection works by running an image through a neural network that predicts, for each object it finds, a class label such as "person" or "car", a confidence score and a bounding box around it. It answers both "what" and "where", unlike classification, which only labels the whole image. Face detection is the first step in FaceVision's browser pipeline, before recognition and liveness checks. This page covers how detection works, what it achieves, models, CNNs, LLMs and Gemini, OpenCV, and how to code and train a detector.

How does object detection work?

A detection network extracts features from the image with a backbone, then a detection head predicts boxes and class scores at many positions and scales. Overlapping predictions for the same object are merged with non-maximum suppression, leaving one box per object.

Also asked as: is object detection cnn · how to make object detection

One-stage detectors such as YOLO predict boxes and classes in a single pass, which makes them fast for real-time video. Two-stage detectors such as Faster R-CNN first propose candidate regions, then classify and refine them, often with higher accuracy on small objects. Transformer detectors such as DETR predict a set of objects directly using attention. Training uses images with labelled boxes and losses for both location and class.

What is object detection in AI?

In AI, object detection is the computer vision task of locating and identifying multiple objects in images or video. Its output is a list of objects, each with a category, a bounding box and a confidence score.

It sits between classification, which labels a whole image, and segmentation, which labels every pixel. Variants include face detection, pedestrian detection, text detection for OCR, and 3D detection for autonomous vehicles using depth or lidar. Detection is often the first stage of larger systems: detect faces, then recognise them; detect products, then read their labels; detect vehicles, then track them.

What does object detection accomplish?

Object detection lets machines find and count things in visual data automatically, enabling decisions such as alerting, measuring or acting. It turns raw images into structured information: what objects exist, where they are, and how many.

Applications include safety monitoring for helmets and protective gear, retail shelf analysis, traffic counting, wildlife monitoring with camera traps, defect detection in manufacturing, medical findings in scans, document layout analysis, and robotics grasping. Combined with tracking, it follows objects across video frames, supporting analytics such as footfall counting. Its value depends on accuracy in your specific conditions, which is why custom training is common.

What are the main object detection models?

Popular families include the YOLO series for fast real-time detection, Faster R-CNN and Mask R-CNN for accuracy and instance masks, SSD and RetinaNet as earlier one-stage designs, and transformer-based models such as DETR, RT-DETR and open-vocabulary detectors such as Grounding DINO and OWL-ViT.

Also asked as: what are object detection models

Choosing a model: YOLO variants for speed on edge devices and video; two-stage models where small-object accuracy matters most; open-vocabulary detectors when you need to detect objects described in text without training on them. Benchmarks on the COCO dataset compare accuracy and speed, but test on your own images. Check licences carefully: some popular implementations use copyleft licences with obligations for commercial products.

Is object detection based on CNNs?

Often, but not always. Many detectors, including most YOLO versions and Faster R-CNN, use convolutional neural network backbones to extract features. Newer detectors use vision transformers or hybrid designs, and some use transformers in the detection head even with CNN backbones.

CNNs remain popular because they are efficient and run well on phones and edge hardware. Transformer detectors such as DETR simplify the pipeline by removing hand-designed steps like anchor boxes, and perform strongly with large-scale pretraining. In practice, the choice depends on available models, hardware and accuracy needs, not on CNN versus transformer ideology.

Can LLMs do object detection?

Text-only LLMs cannot see images, so they cannot detect objects. Multimodal models that accept images can describe objects and, in some cases, return bounding box coordinates when prompted, making zero-shot detection possible without training.

This is useful for prototyping, rare categories and flexible queries such as "find every red helmet". Trade-offs: slower and more expensive per image than dedicated detectors, less precise boxes, and less consistent results. A practical pattern is using a multimodal model to pre-label images, reviewing labels by hand, then training a fast dedicated detector for production use.

Can Gemini do object detection?

Yes. Gemini models can return bounding boxes for objects when asked, in coordinates normalised to the image size, and some versions can also return segmentation masks. Google's documentation shows how to request boxes with a prompt and parse the output.

Ask for structured JSON output listing labels and box coordinates, then convert them to pixel positions for drawing. Gemini works well for flexible, descriptive queries, such as "detect all items that are not in their packaging". For real-time video, high frame rates or offline use, a dedicated detector on the device is faster and cheaper. Validate Gemini's detections on a sample of your images before relying on them.

Can OpenCV do object detection?

Yes. OpenCV's DNN module runs pretrained detection models, such as YOLO or SSD in ONNX format, and older classical detectors such as Haar cascades for faces and HOG for pedestrians. OpenCV also handles the surrounding work: reading video, resizing frames and drawing boxes.

Haar cascades are fast but less accurate than deep learning detectors; use them only for simple, controlled cases. For modern accuracy, export a trained detector to ONNX and run it with OpenCV DNN or ONNX Runtime. OpenCV's face detection now includes a neural network detector that is far more robust than older cascades.

How do I code object detection?

The quickest start is a pretrained YOLO model with the Ultralytics Python library: load the model, run it on an image or webcam, and read the boxes.

Also asked as: how to code object detection

from ultralytics import YOLO
model = YOLO("yolo11n.pt")             # small pretrained model
results = model("street.jpg")
for box in results[0].boxes:
    print(model.names[int(box.cls)], float(box.conf), box.xyxy.tolist())

Install with pip install ultralytics. Use results[0].show() to view boxes, or pass source=0 for a webcam. Check the licence before commercial use. For browsers, export to ONNX and run with ONNX Runtime Web, as FaceVision does for its models.

How do I make my own object detector?

Collect images of your objects in realistic conditions, label bounding boxes with a tool such as Label Studio or CVAT, split into training, validation and test sets, and fine-tune a pretrained detector. A few hundred labelled images per class often gives useful results.

Steps:

• Define classes clearly and label consistently.
• Include varied lighting, angles, backgrounds and object sizes.
• Fine-tune a small pretrained model first; scale up if needed.
• Evaluate with mean average precision and by looking at mistakes.
• Add hard examples where the model fails, and retrain.
• Export and test speed on your target device.

Data quality matters more than model choice for most custom detectors.

Pranjul Rathour

GenAI Engineer · India

GenAI engineer and AI product builder with 2+ years shipping production-grade AI systems: RAG pipelines, fine-tuned LLMs, hybrid retrieval and multi-modal apps across vision, speech and OCR, architected end to end from ingestion to deployment. Leads engineering for SCULT INDIA's 14-member team, founded the 500+ member TechVerse Enclave community and has mentored 200+ students. Three hackathon first prizes: Changethon 2025 (IIT Roorkee), Product Genesis at Vividhotsava 2025 (CSJMU) and BYTEBATTLE (MeetKats).

Open to GenAI roles, hackathon judging, mentorship sessions and guest talks at colleges. For campus sessions the inviting institution covers the visit end to end: travel (a First AC train ticket for journeys up to five hours, a flight for anything longer), accommodation, meals, and pickup and drop; the honorarium and workshop fees are negotiable on WhatsApp or on the call, and requests are welcome from institutions that can cover the visit this way. Email pranjulrathour41@gmail.com.

PortfolioInvite me to your campusHire meLinkedInGitHubXInstagramThreadsFacebookBlueskyDev.toBloggerHashnodeSCULTEmail