How YOLO works: training, PyTorch, segmentation, classification, labelling and licences

YOLO, You Only Look Once, is a family of real-time object detection models that predict bounding boxes and classes for a whole image in a single pass of a neural network. Modern versions also handle segmentation, classification, pose and oriented boxes. I run vision models in FaceVision with ONNX Runtime Web, so this page explains how YOLO and its training work, its PyTorch roots, when to use it, segmentation and classification, commercial use, labelling data, and how it compares with Detectron2 and MMDetection.

How does the YOLO algorithm work?

YOLO passes the whole image through a convolutional backbone, combines features at several scales in a neck, and uses detection heads to predict box coordinates and class scores at many positions at once. Overlapping duplicate boxes are then removed.

Also asked as: how does yolo algorithm work

The original YOLO divided the image into a grid, with each cell predicting boxes; later versions added anchor boxes, multi-scale feature pyramids and, more recently, anchor-free heads that predict box centres and distances directly. Post-processing applies a confidence threshold and non-maximum suppression, which keeps the highest-scoring box among heavily overlapping ones; some recent versions train to avoid needing NMS. One forward pass for the whole image is what makes YOLO fast enough for real-time video.

How does YOLO training work?

Training feeds labelled images through the network, compares predicted boxes and classes with ground truth using box regression and classification losses, and updates weights with gradient descent over many epochs.

Also asked as: how does yolo.training work

Ultralytics training is a single command: yolo detect train data=data.yaml model=yolo11n.pt epochs=100 imgsz=640, where data.yaml lists image paths and class names. Starting from pretrained weights, called transfer learning, needs far fewer images; typical custom datasets start with a few hundred images per class. Built-in augmentations such as mosaic, scaling and colour shifts improve robustness. Watch validation mean average precision, not just loss, and keep the best checkpoint. Then export with yolo export to ONNX or TensorRT for deployment.

Does YOLO use PyTorch?

Most modern YOLO versions do. Ultralytics YOLOv5, YOLOv8 and YOLO11 are built on PyTorch, as are YOLOv7, YOLOv9 and YOLOv10. The original YOLOv1 to v3 used Darknet, a framework written in C and CUDA.

PyTorch makes training and customisation accessible in Python, while export to ONNX, TensorRT, OpenVINO, CoreML and TFLite allows deployment without PyTorch. Other implementations exist in TensorFlow, in PaddlePaddle with PP-YOLO, and in JAX, but the PyTorch versions have the largest communities. For browsers, models are exported to ONNX and run with ONNX Runtime Web or converted for TensorFlow.js, which is how I run vision models client-side in FaceVision.

When should you use YOLO?

Use YOLO when you need fast object detection on images or video with a known set of classes, especially in real time or on edge devices: counting, tracking, safety monitoring, inspection, robotics and retail analytics.

Also asked as: when to use yolo

Consider alternatives when you need maximum accuracy on small or crowded objects, where larger transformer detectors may win; when classes are open-ended, where open-vocabulary detectors such as Grounding DINO help; or when you need detailed scene understanding, where vision-language models fit better. YOLO's strengths are speed, mature tooling, easy training and broad export options, which make it the default starting point for most practical detection projects.

Which YOLO model should you use for segmentation?

Use the segmentation variants of recent Ultralytics models, named with a -seg suffix, such as yolo11n-seg for speed or yolo11m-seg and larger for accuracy. They output instance masks alongside boxes and classes.

Also asked as: which yolo model to use for segmentation

Choose the size by hardware: nano and small for edge devices and real-time video, medium for balanced servers, large and extra-large for offline accuracy. For very precise masks, such as medical or measurement tasks, compare against dedicated segmentation models or SAM-based pipelines, which produce finer boundaries. Train on polygon-labelled data in YOLO segmentation format, and evaluate with mask mean average precision rather than box metrics alone. Higher input resolution improves thin structures and small objects.

How does YOLO segmentation work?

YOLO segmentation models add a mask branch to the detector. A small network produces a set of prototype masks for the whole image, and each detected object predicts coefficients that combine those prototypes into its own mask.

This approach, inspired by YOLACT, keeps segmentation fast because masks are built from shared prototypes rather than computed separately per object. The combined mask is cropped to the object's bounding box and thresholded to produce the final instance mask. Training uses polygon labels converted to masks, with a mask loss added to box and class losses. The result is real-time instance segmentation, useful for robotics, agriculture and video effects.

Can you use YOLO for classification?

Yes. Ultralytics provides classification variants, with a -cls suffix, trained on ImageNet, which classify a whole image into one class. Train them on folders of images organised by class name with a single command.

Also asked as: can i use yolo for classification

Use classification when each image contains one main subject, such as identifying plant diseases from leaf photos or sorting product images. Use detection when images contain multiple objects or you need locations. Classification datasets are cheaper to label, since you only sort images into folders. Compare YOLO classification models with alternatives like EfficientNet or ViT if accuracy is critical, and export the best model to ONNX or TFLite for deployment on phones.

Can you use YOLO commercially?

Yes, but check the licence. Ultralytics YOLO models are AGPL-3.0, which requires releasing your source code if you distribute the software or offer it over a network, unless you buy an Ultralytics Enterprise licence.

Some YOLO versions use GPL-3.0, which also has copyleft obligations. Permissively licensed alternatives include YOLOX and RF-DETR, both under Apache 2.0. Pretrained weights and datasets can carry their own terms, such as COCO image licences, so review all three: code, weights and data. For internal tools that are never distributed, obligations may differ, but get legal advice for commercial products rather than relying on assumptions.

How do you label training data for YOLO?

Use an annotation tool such as Label Studio, CVAT or Roboflow to draw boxes, then export in YOLO format: one text file per image, each line holding a class index and the normalised centre x, centre y, width and height.

Also asked as: how to label training data for yolo

Organise images and labels into train and validation folders, and list paths and class names in a data.yaml file. Label consistently: decide rules for partial objects, occlusion and tiny objects, and write them down. Include varied backgrounds, lighting and negative images without objects. Use model-assisted labelling to speed up later batches, correcting predictions instead of drawing every box. Review a random sample for errors, since label quality strongly affects accuracy.

YOLO vs Detectron2 vs MMDetection: which trains faster?

For typical custom datasets, Ultralytics YOLO usually trains fastest and is easiest to set up. Detectron2 and MMDetection are research toolboxes offering many architectures, such as Faster R-CNN, Mask R-CNN and DETR variants, with more configuration and often slower training.

Also asked as: yolo vs detectron2 vs mmdetection: training speed test

Speed depends on model size, image resolution, batch size and hardware, so benchmark on your own data. YOLO's training pipeline is optimised for single commands and strong defaults; Detectron2, from Meta, suits teams extending Mask R-CNN-style models; MMDetection, from OpenMMLab, offers the widest model zoo for research comparisons. Choose YOLO for fast iteration and deployment, and the toolboxes when you need specific architectures or research reproducibility.

Pranjul Rathour

GenAI Engineer · India

GenAI engineer and AI product builder with 2+ years shipping production-grade AI systems: RAG pipelines, fine-tuned LLMs, hybrid retrieval and multi-modal apps across vision, speech and OCR, architected end to end from ingestion to deployment. Leads engineering for SCULT INDIA's 14-member team, founded the 500+ member TechVerse Enclave community and has mentored 200+ students. Three hackathon first prizes: Changethon 2025 (IIT Roorkee), Product Genesis at Vividhotsava 2025 (CSJMU) and BYTEBATTLE (MeetKats).

Open to GenAI roles, hackathon judging, mentorship sessions and guest talks at colleges. For campus sessions the inviting institution covers the visit end to end: travel (a First AC train ticket for journeys up to five hours, a flight for anything longer), accommodation, meals, and pickup and drop; the honorarium and workshop fees are negotiable on WhatsApp or on the call, and requests are welcome from institutions that can cover the visit this way. Email pranjulrathour41@gmail.com.

PortfolioInvite me to your campusHire meLinkedInGitHubXInstagramThreadsFacebookBlueskyDev.toBloggerHashnodeSCULTEmail