problem with int8 quantization of tensorrt for models trained with adam optimizer

[skynn1128]

Search before asking

• I have searched the YOLOv5 issues and found no similar bug report.

YOLOv5 Component

Export

Bug

Hello
When I use the adam optimizer to train a pt model, then convert it to onnx, and then convert it to the tensorrt engine model, there is a problem with the output threshold during testing, but when I use the sgd optimizer to train the model and perform the above steps, the engine model output threshold is normal. What is the reason?
When using sgd, the normal threshold of the int8 engine output is 0.92, but when using adam, the output threshold is 0.14

Environment

No response

Minimal Reproducible Example

No response

Additional

No response

Are you willing to submit a PR?

• Yes I'd like to help by submitting a PR!

[UltralyticsAssistant]

👋 Hello @skynn1128, thank you for your interest in YOLOv5 🚀! Your question about INT8 quantization with TensorRT and the impact of different optimizers is important.

If this is a 🐛 Bug Report, please provide a minimum reproducible example to help us debug it. This would include the specific steps and configurations you used during the conversion process, as well as any logs or errors that occurred.

If you have a ❓ Question about custom training, please give as much detail as possible, including dataset examples and training logs. Additionally, double-check that you are following our best practices for training.

To ensure smooth operation, make sure your environment meets all requirements like Python>=3.8.0 and PyTorch>=1.8. You can easily set up your environment by cloning the YOLOv5 repository and installing the required packages.

YOLOv5 models can be deployed in various environments like cloud services and notebooks. For your exports, ensure all dependencies like CUDA and CUDNN are correctly set up.

If this badge is green, all YOLOv5 Continuous Integration (CI) tests are currently passing. It ensures the reliability of training, validation, inference, export, and benchmarks.

Also, explore our new YOLOv8 models, designed to offer fast and accurate detection, segmentation, and classification to enhance your projects!

Please note, this is an automated response. An Ultralytics engineer will review your issue and provide further assistance soon. Thank you for your patience! 😊

[pderrenger]

@skynn1128 thank you for reporting this issue. It seems like the discrepancy might be related to how the Adam optimizer affects model weights compared to SGD, potentially impacting quantization. I recommend checking if the issue persists with the latest YOLOv5 version and ensuring that the quantization parameters are correctly set for both optimizers. If the problem continues, please provide a minimal reproducible example for further investigation.