How can I better train yolo-detect?

[SnifferCaptain]

I've got a hard task in my private dataset. In validation of every epoch, the box's P , map50 and map50-95 are in a reasonable range(about 0.5 in P, 0.38 in map50, 0.28 in map50-95) However, it somehow act badly in recall(just about 0.3 after 30 epochs). Is there any hyper-settings that can make the net put more focus on recall loss? Or should I code a loss function myself?

epoch	train/box_loss	train/cls_loss	train/dfl_loss	metrics/precision(B)	metrics/recall(B)	metrics/mAP50(B)	metrics/mAP50-95(B)	val/box_loss	val/cls_loss	val/dfl_loss	lr/pg0	lr/pg1	lr/pg2
7	1.3777	1.6216	1.4158	0.50245	0.33914	0.38075	0.28003	1.2912	1.6536	1.4309	0.0004523	0.0004523	0.0004523

[UltralyticsAssistant]

👋 Hello @SnifferCaptain, thank you for reaching out with your training question! It sounds like you're working on an exciting yet challenging project with your dataset 🚀. We recommend checking our Docs where you might find useful insights, especially our Tips for Best Training Results that could help address the recall issue you're experiencing.

If you think a custom loss function might be necessary, our community members on Discord 🎧 enjoy discussions around custom training and might offer novel insights.

Also, note that this is an automated response, and an Ultralytics engineer will assist you shortly. If you wish, please provide additional details about your training setup or a minimum reproducible example to help us support you further.

To ensure you are using the latest features and fixes that might help, make sure your environment is up-to-date. You can upgrade your ultralytics package with the following command:

pip install -U ultralytics

In case you want to ensure you're operating within verified environments, here are suggestions where everything is preinstalled for quick startups:

• Notebooks with free GPU:

For more robust setups:

• Google Cloud, see GCP Quickstart Guide
• Amazon, see AWS Quickstart Guide
• Docker Image, see Docker Quickstart Guide

Rest assured, once the following badge indicates green, it verifies all Ultralytics CI tests are passing:

Your patience is greatly appreciated while awaiting additional support, and we’re excited to see what you'll achieve with your YOLO training! 🎉

[glenn-jocher]

@SnifferCaptain based on your validation metrics, I can suggest a few targeted adjustments to improve recall while maintaining precision. First, try adjusting the confidence threshold during training to a lower value (e.g., conf=0.1) which can help detect more potential objects. You can also modify the hyp.yaml file to increase the obj (objectness) loss weight, which can encourage the model to be more aggressive in detecting objects.

For specific guidance on adjusting these hyperparameters, see the model training tips at https://docs.ultralytics.com/guides/model-training-tips/. If these adjustments don't achieve the desired results, you may want to review your dataset for class imbalance and consider data augmentation techniques.

Let us know if you need help implementing these changes or if you'd like to explore additional optimization strategies.

[SnifferCaptain]

Thank you for your helpful reply!
After so many days of experiment, I finally found that dear Mr. Copilot has helped me to add some additional arguments in my customized Conv block( for some reason I have to change the network's backbone) .
Additionally, I found out that the percision and recall value I show are actually in reasonable range.
Sorry to have inturrupted for my personal reason.
lol

[glenn-jocher]

@SnifferCaptain you're very welcome! No problem at all—experimentation and tweaking are part of the process. Great to hear you've resolved the issue and that your metrics are within a reasonable range. Best of luck with your custom backbone work, and feel free to reach out if you have other questions related to Ultralytics YOLO.

[Skillnoob]

Can you share the labels.jpg that gets generated in the training folder?
Can you also share the command or code you used to train the model?

[SnifferCaptain]

it's the object365 dataset's subset. containing about 100k images with 365 classes.

my training code are as follow:

model=YOLO("./yolov8s.pt",'detect')

results = model.train(data="obj365.yaml",epochs=40,device='0',exist_ok=True,
optimizer='NAdam', optimize = True,imgsz=640,amp=True,close_mosaic=10,
batch=16,lr0 = 0.001, lrf = 0.0001,seed=0,workers=0,val=True,resume=False,name="[test]obj365"
,cos_lr=True,warmup_epochs=1,plots=True,dropout=0.1)

also I've been facing another problom I can't solve myself: The RT-DETR model will go 'nan' during the first epoch and I have no idea why. The condition only exist while training obj365 dataset, and it goes pretty normal while training another dataset with smaller amount of labels and classes.
code are as follow:
model=RTDETR("./rtdetr-l.yaml")

training code are the same#

here are labels.jpg:

[Skillnoob]

Objects365 is a big dataset, bigger than coco. The coco pretrained models have been trained for 600 epochs, you're not gonna be able to get away with 40. Also, why are you changing so many parameters, you should for now leave them at their default values, you're most likely making the results worse.

[SnifferCaptain]

thank you for your reply!
I will re-experiment more.

[Skillnoob]

@SnifferCaptain Change your train command to this:

model.train(data="obj365.yaml", epochs=300, device=0, name="[test]obj365")