(Possibly) Cuda version conflict when training classifcation on custom dataset on Colab

[thiendt2k1]

Search before asking

• I have searched the YOLOv5 issues and discussions and found no similar questions.

Question

I try to train the Yolov5l-cls.pt with custom dataset here on colab, it return a error
../aten/src/ATen/native/cuda/Loss.cu:240: nll_loss_forward_reduce_cuda_kernel_2d: block: [0,0,0], thread: [6,0,0] Assertion t >= 0 && t < n_classesfailed. ../aten/src/ATen/native/cuda/Loss.cu:240: nll_loss_forward_reduce_cuda_kernel_2d: block: [0,0,0], thread: [8,0,0] Assertiont >= 0 && t < n_classesfailed. ../aten/src/ATen/native/cuda/Loss.cu:240: nll_loss_forward_reduce_cuda_kernel_2d: block: [0,0,0], thread: [24,0,0] Assertiont >= 0 && t < n_classesfailed. terminate called after throwing an instance of 'c10::Error' what(): CUDA error: device-side assert triggered CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1. Compile withTORCH_USE_CUDA_DSA` to enable device-side assertions.

Exception raised from c10_cuda_check_implementation at ../c10/cuda/CUDAException.cpp:44 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x7f74e2dcc4d7 in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::string const&) + 0x64 (0x7f74e2d9636b in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10.so)
frame #2: c10::cuda::c10_cuda_check_implementation(int, char const*, char const*, int, bool) + 0x118 (0x7f74e2e68b58 in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10_cuda.so)
frame #3: + 0x1250d5e (0x7f74e4179d5e in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cuda.so)
frame #4: + 0x4d5a16 (0x7f75497e7a16 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_python.so)
frame #5: + 0x3ee77 (0x7f74e2db1e77 in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10.so)
frame #6: c10::TensorImpl::~TensorImpl() + 0x1be (0x7f74e2daa69e in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10.so)
frame #7: c10::TensorImpl::~TensorImpl() + 0x9 (0x7f74e2daa7b9 in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10.so)
frame #8: + 0x75afc8 (0x7f7549a6cfc8 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_python.so)
frame #9: THPVariable_subclass_dealloc(_object*) + 0x305 (0x7f7549a6d355 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_python.so)
frame #10: python3() [0x622134]
frame #11: python3() [0x53e218]
frame #12: python3() [0x58d328]
frame #13: python3() [0x6536c9]
frame #14: python3() [0x53ecfb]

frame #25: python3() [0x5a7951]
frame #27: python3() [0x6bb79b]
frame #28: python3() [0x6bb824]
frame #29: python3() [0x6bbc66]
frame #34: __libc_start_main + 0xf3 (0x7f757ac12083 in /lib/x86_64-linux-gnu/libc.so.6)
`

Is there a cuda version conflict or anything that result in this error, if so, how can i fix it?
Thank you!

Additional

The command i use is:
!python classify/train.py --model yolov5l-cls.pt --data "/content/drive/MyDrive/remake/datasets/classification disease/Classifier-1" --epochs 20 --img 640 --cache --lr 0.0005 --batch-size 80

Info of colab that yolov5 return:
YOLOv5 🚀 v7.0-193-g485da42 Python-3.10.12 torch-2.0.1+cu118 CUDA:0 (NVIDIA A100-SXM4-40GB, 40514MiB)

[github-actions]

👋 Hello @thiendt2k1, thank you for your interest in YOLOv5 🚀! Please visit our ⭐️ Tutorials to get started, where you can find quickstart guides for simple tasks like Custom Data Training all the way to advanced concepts like Hyperparameter Evolution.

If this is a 🐛 Bug Report, please provide a minimum reproducible example to help us debug it.

If this is a custom training ❓ Question, please provide as much information as possible, including dataset image examples and training logs, and verify you are following our Tips for Best Training Results.

Requirements

Python>=3.7.0 with all requirements.txt installed including PyTorch>=1.7. To get started:

git clone https://github.com/ultralytics/yolov5  # clone
cd yolov5
pip install -r requirements.txt  # install

Environments

YOLOv5 may be run in any of the following up-to-date verified environments (with all dependencies including CUDA/CUDNN, Python and PyTorch preinstalled):

• Notebooks with free GPU:
• Google Cloud Deep Learning VM. See GCP Quickstart Guide
• Amazon Deep Learning AMI. See AWS Quickstart Guide
• Docker Image. See Docker Quickstart Guide

Status

If this badge is green, all YOLOv5 GitHub Actions Continuous Integration (CI) tests are currently passing. CI tests verify correct operation of YOLOv5 training, validation, inference, export and benchmarks on macOS, Windows, and Ubuntu every 24 hours and on every commit.

Introducing YOLOv8 🚀

We're excited to announce the launch of our latest state-of-the-art (SOTA) object detection model for 2023 - YOLOv8 🚀!

Designed to be fast, accurate, and easy to use, YOLOv8 is an ideal choice for a wide range of object detection, image segmentation and image classification tasks. With YOLOv8, you'll be able to quickly and accurately detect objects in real-time, streamline your workflows, and achieve new levels of accuracy in your projects.

Check out our YOLOv8 Docs for details and get started with:

pip install ultralytics

[glenn-jocher]

@thiendt2k1 hi there! It seems like you are encountering an error while training YOLOv5 on a custom dataset. The error message suggests a CUDA error with device-side assertions.

This error can sometimes occur due to a CUDA version conflict or compatibility issue. One possible solution is to try compiling with TORCH_USE_CUDA_DSA to enable device-side assertions. This can be done by setting the environment variable TORCH_USE_CUDA_DSA=1 before running the training command.

Here's an example of how you can set the environment variable before running the command:

!export TORCH_USE_CUDA_DSA=1
!python classify/train.py --model yolov5l-cls.pt --data "/content/drive/MyDrive/remake/datasets/classification disease/Classifier-1" --epochs 20 --img 640 --cache --lr 0.0005 --batch-size 80

Please give this a try and let me know if it resolves the issue for you. If you continue to encounter any problems, please provide more details about your CUDA version and any other relevant system information.

Remember that the YOLOv5 repository is a community effort, and the Ultralytics team and the YOLO community play a significant role in maintaining and improving it. If you have any further questions or need additional assistance, feel free to ask.

[thiendt2k1]

Hello sir @glenn-jocher, thank you for your quick reply, i train it on colab with info mentioned
YOLOv5 🚀 v7.0-193-g485da42 Python-3.10.12 torch-2.0.1+cu118 CUDA:0 (NVIDIA A100-SXM4-40GB, 40514MiB)

With nvcc that print:
nvcc: NVIDIA (R) Cuda compiler driver Copyright (c) 2005-2022 NVIDIA Corporation Built on Wed_Sep_21_10:33:58_PDT_2022 Cuda compilation tools, release 11.8, V11.8.89 Build cuda_11.8.r11.8/compiler.31833905_0

If i understand correct, it is CUDA 11.8, gpu is NVIDIA A100 SXM4 with 40GB, python 3.10.12

[github-actions]

👋 Hello there! We wanted to give you a friendly reminder that this issue has not had any recent activity and may be closed soon, but don't worry - you can always reopen it if needed. If you still have any questions or concerns, please feel free to let us know how we can help.

For additional resources and information, please see the links below:

• Docs: https://docs.ultralytics.com
• HUB: https://hub.ultralytics.com
• Community: https://community.ultralytics.com

Feel free to inform us of any other issues you discover or feature requests that come to mind in the future. Pull Requests (PRs) are also always welcomed!

Thank you for your contributions to YOLO 🚀 and Vision AI ⭐

[Frank-Ray]

I have the same question, and I tried to setting the environment variable TORCH_USE_CUDA_DSA=1 before running the training command. The problem still exists.
terminate called after throwing an instance of 'c10::Error'
what(): CUDA error: device-side assert triggered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.

[glenn-jocher]

@Frank-Ray Thank you for confirming the CUDA version and the GPU information. It appears that setting the TORCH_USE_CUDA_DSA environment variable did not resolve the issue for you. Given the CUDA version and GPU details you provided, it seems that there might be another underlying cause contributing to the error.

One potential solution to consider is enabling CUDA_LAUNCH_BLOCKING=1 for debugging, as suggested in the error message. This setting can help provide more accurate stack traces for debugging CUDA errors.

Here's an example of how you can set the CUDA_LAUNCH_BLOCKING environment variable:

!export CUDA_LAUNCH_BLOCKING=1
!python classify/train.py --model yolov5l-cls.pt --data "/content/drive/MyDrive/remake/datasets/classification disease/Classifier-1" --epochs 20 --img 640 --cache --lr 0.0005 --batch-size 80

By enabling CUDA_LAUNCH_BLOCKING, you may obtain more detailed information about the source of the CUDA error, which could help troubleshoot and resolve the issue more effectively.

Additionally, please ensure that the YOLOv5 repository, which is maintained by the YOLO community and the Ultralytics team, is regularly updated to the latest version, as bug fixes and improvements are often included in newer releases.

If the issue persists or if you have any other questions, feel free to reach out for further assistance. Your feedback and insights are invaluable in helping us make YOLOv5 even better.