Key takeaways
- Object detection outputs a box, class label and confidence for each object in an image.
- Detectors solve classification for labels and regression for box coordinates.
- Ultralytics YOLO is the fastest route to detection in Python, from pretrained to custom models.
- Segmentation labels pixels, and tracking links detections across frames with stable IDs.
- Edge devices need small, quantised models, and background images reduce false positives.
Object detection finds objects in images or video and marks each with a bounding box, a class label and a confidence score. It powers self-checkout, traffic monitoring, robotics, drones and face pipelines. FaceVision, which I built, starts every recognition with face detection running in the browser through ONNX Runtime Web, so this page covers what object detection means, how it works, Python with YOLO, classification and regression, segmentation, tracking, Raspberry Pi, drones and negative samples.
What does object detection mean in computer vision?
Object detection means finding where objects are in an image and what they are. For each object, a detector outputs a bounding box, a class label such as person or car, and a confidence score.
It sits between image classification, which assigns one label to a whole image, and segmentation, which labels individual pixels. Detection handles multiple objects of different classes in one image, including overlapping ones. Common benchmarks include COCO, with 80 everyday object categories, and quality is measured with mean average precision at various overlap thresholds. Applications include counting people, detecting vehicles and number plates, finding defects in manufacturing, reading shelves in retail, and detecting faces before recognition, as FaceVision does in the browser.
How does object detection work in computer vision?
A neural network backbone extracts features from the image, and a detection head predicts box coordinates and class probabilities at many locations. Non-maximum suppression then removes duplicate boxes, keeping the most confident prediction for each object.
Two-stage detectors such as Faster R-CNN first propose candidate regions, then classify and refine each, giving high accuracy at lower speed. One-stage detectors such as YOLO, introduced by Redmon and colleagues, and SSD predict boxes and classes in a single pass, making them fast enough for real-time video. Transformer-based detectors such as DETR treat detection as set prediction and remove hand-designed steps. Models train on images labelled with boxes, learning by comparing predicted boxes with ground truth using overlap-based losses.
How do you make object detection using Python?
The quickest route is the Ultralytics package: pip install ultralytics, load a pretrained YOLO model, and run it on an image, video or webcam stream. Results include boxes, classes and confidences you can draw with OpenCV.
Also asked as: how to make object detection using python
For custom objects, collect and label images with a tool such as Label Studio, CVAT or Roboflow, export them in YOLO format, describe the dataset in a YAML file, and train starting from pretrained weights. Evaluate with mean average precision on a validation set, then export the model to ONNX or TensorRT for deployment. Alternatives include torchvision's Faster R-CNN and RetinaNet models and Hugging Face's DETR implementations. Check licenses before commercial use, since some popular detection libraries use copyleft terms.
Does object detection include classification?
Yes. Object detection combines localisation, finding where each object is, with classification, deciding what each object is. Every detected box carries a class label and a confidence score for that label.
Unlike image classification, which gives one label per image, detection classifies each object separately, so one image can contain several people, cars and dogs. Some detectors also handle open-vocabulary classification, recognising objects described by text prompts rather than a fixed list, such as Grounding DINO or YOLO-World. Classification quality depends on training data covering each class in varied conditions. If you only need to know whether an image contains something, a classifier is simpler; when you need where and how many, use detection.
Is object detection classification or regression?
Both. Detectors solve a classification problem, predicting the class of each object, and a regression problem, predicting continuous box coordinates such as centre, width and height. Training combines losses for both tasks.
Classification losses such as cross-entropy or focal loss teach the model which class each box contains and whether it contains an object at all. Regression losses, often based on intersection over union, teach accurate box positions. Focal loss, introduced with RetinaNet, handles the imbalance between rare objects and abundant background. Many detectors also predict an objectness score. Segmentation models add a third output, a mask per object. This split helps when debugging: poor labels hurt classification, while inconsistent box annotations hurt regression.
What is object detection and segmentation?
Object detection draws a box around each object, while segmentation labels the exact pixels belonging to objects. Semantic segmentation labels every pixel by class, and instance segmentation separates individual objects with their own masks.
Panoptic segmentation combines both, labelling every pixel with a class and an instance identity. Instance segmentation models such as Mask R-CNN and YOLO segmentation variants output a box and a mask for each object. Meta's Segment Anything Model produces masks from prompts such as points or boxes, often paired with a detector that supplies the boxes. Use detection when location and counts are enough, and segmentation when shape matters, such as measuring areas, medical imaging, background removal or precise robotic grasping.
What is the difference between object detection and tracking?
Detection finds objects in a single image or frame. Tracking follows the same objects across video frames, assigning each a consistent ID so you know that a car in frame ten is the same car in frame one hundred.
Most trackers use tracking by detection: run a detector on each frame, then associate detections with existing tracks using motion predictions, often from a Kalman filter, and appearance features. Popular algorithms include SORT, DeepSORT and ByteTrack, and Ultralytics builds tracking into its YOLO models. Tracking enables counting objects crossing a line, measuring speed, analysing paths and smoothing flickering detections. Challenges include occlusions, crowded scenes and ID switches when similar objects cross. Higher frame rates and stronger appearance features reduce errors.
How do you do object detection using a Raspberry Pi?
Use a lightweight model, such as a YOLO nano variant or MobileNet-SSD, with an optimised runtime like ONNX Runtime, LiteRT or NCNN, and a camera module read through Picamera2. A Raspberry Pi 5 handles small models at modest frame rates.
Also asked as: how to do object detection using raspberry pi
For real-time performance, add an accelerator: Raspberry Pi's AI Kit and AI HAT+ use Hailo chips, and Google's Coral USB accelerator runs quantised models. Reduce input resolution, quantise models to 8-bit, and process every few frames if full speed is not needed. Keep the Pi cool, since thermal throttling slows inference. Export models from your training machine in the format your runtime expects. Typical projects include bird feeders, people counters, parking monitors and home security, all running locally without sending video to the cloud.
How do you do real-time object detection on drones?
Run an efficient detector on onboard edge hardware, such as an NVIDIA Jetson module, optimised with TensorRT, or stream video to a ground station for heavier models. Drone footage needs models trained for small objects seen from above.
Also asked as: how to do real time object detection on drones
Aerial views make objects tiny and change perspective, so fine-tune on aerial datasets such as VisDrone, or on your own labelled footage. Tiled inference, as in the SAHI library, splits large frames into overlapping patches so small objects stay detectable. Balance resolution, frame rate and power, since onboard compute drains batteries. Combine detection with tracking to stabilise results while the drone moves. Follow aviation and privacy regulations, especially when flying over people, and design systems to avoid identifying individuals unless legally permitted.
How do you add negative samples for object detection?
Add background images that contain none of your target objects to the training set with empty label files. They teach the model what not to detect, reducing false positives on similar-looking scenes.
Also asked as: how to add negative samples for object detection
In YOLO format, a negative image simply has an empty or missing annotation file. Ultralytics suggests including a small share of background images, up to around ten percent, though the right amount depends on your data. Choose hard negatives: scenes where the model currently makes mistakes, such as shadows mistaken for people or reflections mistaken for cars. Collect them by running your model on unlabelled footage and reviewing false positives. Retrain, then measure precision and recall again, since too many negatives can reduce recall.
Sources
GenAI engineer and AI product builder with 2+ years shipping production-grade AI systems: RAG pipelines, fine-tuned LLMs, hybrid retrieval and multi-modal apps across vision, speech and OCR, architected end to end from ingestion to deployment. Leads engineering for SCULT INDIA's 14-member team, founded the 500+ member TechVerse Enclave community and has mentored 200+ students. Three hackathon first prizes: Changethon 2025 (IIT Roorkee), Product Genesis at Vividhotsava 2025 (CSJMU) and BYTEBATTLE (MeetKats).
Open to GenAI roles, hackathon judging, mentorship sessions and guest talks at colleges. For campus sessions the inviting institution covers the visit end to end: travel (a First AC train ticket for journeys up to five hours, a flight for anything longer), accommodation, meals, and pickup and drop; the honorarium and workshop fees are negotiable on WhatsApp or on the call, and requests are welcome from institutions that can cover the visit this way. Email pranjulrathour41@gmail.com.
